# 3D Convolutional Neural Networks for Remote Pulse Rate Measurement and Mapping from Facial Video
## 3 Materials and Methods
### 3D CNN for Automatic Pulse Rate Estimation
#### 3.3.3 Pulse Rate Prediction  
The learned model produces a prediction for a volume of 25 \(\times\) 25 \(\times\) 60 pixels. The synthetic data generator used to train the model does not incorporate a stage ensuring that the frames in the 25 \(\times\) 25 patches contain pixels that are naturally arranged and ordered like in real video streams. We therefore chose to break the coherent structure of pixels before predicting the pulse rate by shuffling the pixel position. Note that only the green channel [39] was processed by the model.  
Figure 6: Model architecture. The network integrates a 3D convolution (blue) with its associated 3D pooling (green) layers. The stream converges to two fully dense layers (orange).  
Figure 7: Learning metrics: loss (**a**) and accuracy (**b**) for training and validation.  
Maps of predictions were formed by computing a prediction for each group of pixels in the video stream. The procedure predicts and then shifts the input volume with a constant spatial step of 1 pixel (with overlapping). Typical prediction maps computed from the first 60 frames of subjects #1, #33, and #42 (UBFC-RPPG dataset) are presented in Figure 8. Blue pixels correspond to regions where no distinct PPG variations were identified (e.g., background and hair), while the other colors refer to properly predicted pulse rate values. It is important to emphasize that the network gives a score for each class (all pulse rate values plus a \"No PPG\" class) and that only the class that presents the highest score is saved and presented in these maps.  
We can visually observe that the majority of pulse pixels are located in relevant regions like the cheeks and forehead. These areas contain significant PPG signal-to-noise ratios. The maps are somewhat similar to those presented in [82].  
Figure 8: Maps of predictions and their relative histograms. The model produces a prediction for each group of 25 \(\times\) 25 pixels in the video stream. Blue pixels correspond to regions where no distinct PPG variations were identified, while the other colors refer to properly predicted pulse rate values. Top row: data and results for subject #1 (first 60 frames; histogram main peak: 85 bpm; ground truth: 90 bpm). Middle row: data and results for subject #33 (first 60 frames; histogram main peak: 122.5 bpm; ground truth: 122 bpm). Bottom row: data and results for subject #42 (first 60 frames; histogram main peak: 82.5 bpm; ground truth: 81 bpm).  
The right illustrations on Figure 8 present the histograms computed from the maps of predictions. They exhibit a dominant pulse rate (main peak) of 85, 122.5, and 82.5 bpm for subjects #1, #25, and #42, respectively. The corresponding ground-truth pulse rates for these examples are, respectively, 90, 122, and 81 bpm. The histograms have been normalized so that their total energy is equal to 1. Only the bins that correspond to pulse rates are presented. The final pulse rate was computed by aggregating all of the bins in the histogram of predictions using a weighted average operation:  
\[\widehat{PR}=\frac{\sum_{f=55}^{240}f\times\delta\left(f\right)}{\sum_{f=55}^ {240}\delta\left(f\right)}. \tag{8}\]  
\(\widehat{PR}\) corresponds to the pulse rate value outputted by the method, \(f\) to the frequency (55 to 240 bpm at regular intervals of 2.5 bpm), and \(\delta\left(f\right)\) to the amplitude (number of pixels) of a bin.