FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing
Authors: Jitesh Joshi
Sos Agaian
Youngjun Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: Time-series estimation
remote photo-plethysmography
spatial-temporal attention
non-negative matrix factorization
Primary area: deep_learning_architectures
TL;DR: This work introduces the Factorized Self-Attention Module, which computes multidimensional attention through nonnegative matrix factorization, and integrate it into FactorizePhys, a proposed 3D-CNN model for robust rPPG signal extraction.

Abstract: Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.

FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing
Jitesh Joshi', Sos Agaian", and Youngjun Cho
'Department of Computer Science, University College London, UK *Department of Computer Science, College of Staten Island, City University of New York, USA (jitesh.joshi.2) 20, youngjun.cho @ucl.ac.uk, sos.agaian @csi.cuny.edu
Abstract
Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused this trans- formation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attentis which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiiveness, we de- veloped FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings achieve comprehensive spatial, temporal, and channel atten- tion, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-base rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the- art rPPG methods, each representing different types architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temp features substantiate the ef- fectiveness of FSAM and enhanced cross- generalization in estimating rPPG signals, suggesting its broader potential multidimensional attention mechanism.
Module (FSAM),
The code is accessible at
Introduction
Attention mechanisms in computer vision are inspired by the human ability to identify salient regions in complex scenes. Such mechanisms can be interpreted dynamic weight adjustment process that selects useful features and disregards irrelevant ones in a multidimensional feature space. Recent surveys 22 23 provide comprehensive overview of attention mechanisms and distinctly categorize existing attention mechanisms. Amidst a spectrum of research from convolution block attention (66) to computationally intensive multi- head attention [58), an effective, yet computation and memory efficient, attention mechanism has remained desirable for real-world applications. dimensionality reduction technique, has captured the interest of researchers and has been explored in deep learning research for different objectives [57] 60 17] [8]. This work investigates nonnegative matrix factorization (NMF), matrix decomposition technique, for its potential to efficiently perform multidimenssioal attention and evaluates its effectiveness in the spatial-temporal context of estimating rPPG signal from video frames.
Matrix decomposition
[12 19
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Verkruysse [s9]'s pioneering investigation on extracting photoplehyssmorrahh (PPG) or blood volume pulse (BVP) signals from RGB cameras contactless manner led an exciting research field of imaging- haaed physiological sensing. There exist several potential applications and contexts of noninvasive and contactless measurement techniques, such as stress and mental workload recogni- tion [8] 9 7 driver drowsiness monitoring 77] and social biofeedback interaction [3]. The seminal
works on unsupervised rPPG methods [59 48 I1 either used video frames acquired under stationary conditions or performed skin segmentation (62 region of interest (Rol) tracking preprocessing step. This preprocessing step can be considered as a basic form of attention mechanism that enables the unsupervised models to process only the relevant regions. Some of the supervised rPPG methods, including HR-CNN [54), RhythmNet [45). NAS-HR HR [40], PulseGAN [51]. and Dual-GAN 41 also relied on extracting spatial-tempone features from the tracked Rols as a preprocessing step.
As end-to-end rPPG methods, such as DeepPhys G and MTTS-CAN 36 among several others, take whole facial frames input, they rely on attention mechanisms that enable models to empha- size the relevant spatial-temporal features. Estimating BVP signal from raw facial video frames in end-to-end manner is therefore an interesting downstream task to investigate the attention mechanism in multidimensional feature space. This requires networks to learn to pick the spatial features having the desired temporal signature, while díscarding the variance related to head-motion, illumination, and skin-tones, thus representing one of the challenging other notable end-to-end rPPG methods include PhysNet (83). 3DCNN 44. SAM-IPPNNet (26). RT+PPG B3, and transformer- www.wer-newwwww-beeeeee based methods such as PhysFormer 777]. PhysFormer++ 776. EfficientPhys (37]. JAMSNet (7],, and GLISNet [80]. A recent survey article on visual contactless physiological monitoring in clinical settings (77 highlights susceptibility to disturbance, such as head movement, as one of the key challenges, among others. Some of the recent end-to-end rPPG methods (79 [0] further highlight the need for multidimensional attention, as squeezing features in selective dimensions for deriving attention reduces the feature space to single dimension and is therefore not well suited for the task of signal extraction.
tasks. Few
To address this, our work draws inspiration from the seminal work on NMF [3] which recent work formulated as an approach to design the global information block, referred to as Hamburger [1]. Hamburger [18] implements NMF to derive low-rank embeddings, which serve
global
context block. Despite the low computational complexity of (n)). Hamburger [18] outperformed various attention modules in the semantic segmentation 68 and image generation tasks. In addition, researchers have combined matrix factorization with deep architectures in several ways for different applications such as layer-wise learning of dictionary for classification and clustering (57), adaptive learning of dictionary for image denoising [81]. multi-attention model for recommendation systems [60) and linearly scalable approach to context modeling for medical image segmentation M among several others. Drawing inspiration from these studies, especially those that use matrix factorization to model global context [18] M in vision tasks, we investigate the application of NMF as a multidi- mensional attention block. Although matrix factorization in deep learning has remained a topic of significant interest, has not been investigated in the realm of rPPG, which stands to gain from joint spatial, temporal, and channel attention.
We introduce the Factorized Self-Attention Module (FSAM), which implements NMF to jointly compute spatial-temporal attention and describe an appropriate formulation for the low-rank recovery problem. To investigate the relevance and effectiveness of FSAM in computing multidimensional attention, we build a 3D-CNN architecture FactorizePhys that implements FSAM. We further adapt FSAM for EfficientPhys Kr an end-to-end rPPG architecture that builds on the Temporal Shift Module (TSM) (34). to uniquely learn spatial-temporal features using 2D-CNN layers. Evaluation of FactorizePhys and EfficientPhys 337 with FSAM, against existing SOTA rPPG methods, demon- strates the versatility of FSAM as multidimensional attention along with its effectiveness for the data. In summary, we make the
downstream task of estimating time series from
following contributions.
Factorized Self-Attentinn Module (FSAM): NMF 31 -based novel approach that jointly computes
multidimensional attention within voxel embeddings.
FactorizePhys: an end-to-end 3D-CNN architecture that integrates FSAM for robust estimation of rPPG from spatial-temporal facial video frames.
Thorough assessment of FactorizePhys and FSAM with multiple evaluation metries and the corresponding measure of standard errors to compare cross-dataset generalization performance with SOTA rPPG methods, using four benchmarking rPPG datasets.
2

---

2 Related Work
2.1
Attention Mechanisms in Vision
Varied forms of attention mechanisms have been successful in different visual tasks such image
classification [70
25 66 object detection 1 82
semantic segmentation [78]
16]
18
28
video
understanding 663
SS
33
211, 3D vision 669 24 and multimodal tasks [73 56 among others (00
23
The most widely used attention mechanisms are channel attention [35] 75 spatial attention [66
63
temporal attention 77 74, self-attention or transformer-basee approaches [58] 14. multimodal
attention [56 73]. graph-based approaches [32, as well as different combinations of these types
In addition, researchers have proposed attention mechanisms for video understanding 21 as well as 3D vision [69] [24]. Despite notable advances in different forms of attention mechanisms, some of the existing challenges include the requirement for high computational costs, large training data, the overall efficiency of the model, and a cost-benefit analysis of performance improvement [3]. Additionally, for rPPG research, the impact of attention mechanisms on an ability of models to generalize on unseen datasets is not systematically studied, which we address in this work.
666
16
53
63
15
33
2.2 Attention Mechanisms in rPPG Methods
End-to-end rPPG methods can be categorized into convolution neural networks (CNN) architectures such as PhysNet [83. EfficientPhys-C [37]. 3DCNN 4.. SAM-IPPGNet [26), and RT+PPG (3]. and wwwpeoorreeeeetoo---ooee EfficientPhys-T 37 Among end-to-end rPPG methods, DeepPhys 6 first implemented novel convolutional attention mechanism in an architecture that comprised separate motion and appearance branches, with the latter intended to compute attention for the main motion branch. Inspired by the CBAM attention mechanism [66), originally validated for classification and detection tasks, ST- Attention [46) was proposed to filter salient information from spatial-temporal maps, thus improving remote HR estimation. The similar dual attention mechanism was also found to be effective in the SMP-Net framework [33, which jointly learned the features of RGB and infrared spatial- natiial-emmpral maps to estimate multiple physiological signals.
architectures such as PhysFormer 777). PhysFormer++ (76] and
Recently, EfficientPhys 37 an end-to-end network. presented an efficient single-branch approach with gated attention mechanism. The Triin-Transfommer [39 based version of EfficientPhys [37 insightfully added the TSM 34 module to the Swin transformer [39,, enabling the architecture to perform efficient spatial-temporee modeling and compute attention by combining shifting window partitions spatially and shifting frames temporally. should be noted that the convolution-base version of EfficientPhys (37]. which combined the TSM (34) module and a convolutional attention mechanism 66] showed superior accuracy along with significantly low latency, making it highly suitable for deployment on mobile devices. Recently, there has been an upsurge in transformer-basee based rPPG architectures, some of which include PhysFormer 777.. PhysFormer++ (76), TransPhys [61].
and RADIANT 22).
Unlike other transformer-based architectures that rely on spatial-temporal maps as input, PhysFormer 77 and PhysFormer++ [76] are end-to-end video transformer based architectures, which adaptively aggregate both local and global spatial-temporal features. PhysFormer++ (76] extends PhysFormer 77 by better exploiting temporal contextual and periodic rPPG clues, as it extracts and fuses atten- tional features from slow and fast pathways. In addition, both architectures (77] 76 are trained using label distribution learning and curriculum learning-inspired dynamic constraint in the frequency domain, which helps to alleviate overfitting. Although unlike convolution- EfficientPhys [37]. transformer architectures require significantly higher computational resources.
Most of the light-weight convolutional attention mechanisms require attention to be separately derived in spatial, temporal, and channel dimensions, which is later merged [46] [3] Although 3D-CNN architectures such as PhysNet {33] and iBVPNet 29 have shown promising performances, they have not explored attention mechanisms that can potentially enhance performance in unseen datasets. JAMSNet (79 and GLISNet [0] are recent 3D-CNN architectures that benefit significantly from channel-temporal joint attention (CTJA) and spatial-temporal joint attention (STJA). However, unlike CTJA and STJA (79, 80). we jointly derive attention in temporal, spatial, and channel dimensions, without squeezing any dimension of multidimensional features.
3

---

3 Method
3.1 Primer: Nonnegative Matrix Factorization
Nonnegative matrix factorization is a dimensionality reduction paradigm that decomposes M x N matrix V = [01 02 AN| E IR MxM into nonnegative x basis matrix W = [wl,, w2, WL] E 20
RAx I. and nonnegative L x N coefficient matrix H hN] IR LR depicted in fig. 1 and expressed
Original Matrix
Basis Metric W
Eroor
Approcimated Matrix
Matrik
Coefflolent Matrix
Coefficiant Matrix #
Figure 1: Formulation of Nonnegative Matrix Factorization (NMF)
V = F F
R2 where == [11,, 02, uN] E R M. is reconstructed low-rank matrix and E E TR M is
20
matrix, which is discarded. 1x stands for the set of M x N element-wise nonnegative matrices. Equivalent vector formulation for this approximation can be expressed as:
Bg ~ f3
wHH,;
The objective to represent high- digh-dimensional matrix with fewer basis be achieved only when L chosen such that L < min [M, N). while when L is larger than M, it results over-complete basis. An optimization in W and H to achieve the optimal approximation effectively results in the discovery of inherent correlations between the basis vectors in W and the corresponding coefficients
in H 31 64] The optimization objective is formulated
minw, V - WH 3 Wml 2 Hn 20
(3)
Further, the imposed non-negativity constraints where activation of one or many of the coefficients in H together with the basis vectors in W can reconstruct different interpretable parts of V. For detailed primer on NMF, we refer the reader the seminal work [31] and a survey article (64] that summarizes different NMF models and algorithms.
W and H enable parts-based representations,
The factorization of deeper layer embeddings be elucidated the squeeze of information without reducing the dimensions of the embeddings, unlike the existing attention mechanisms [25]. Therefore, exciting multiplying with the resultant low-rank (information squeezed) embeddings can potentially serve as an attention mechanism. While factorization is formulated for two-dimensional matrix, high-dimensional embeddings can be mapped to two-dimensional matrix. In PyTorch 44 this is achieved with the 'view' operation.
3.2 Factorized Self-Attention Module (FSAM)
For the downstream task of estimating rPPG from video frames, spatial-temporal input data can be expressed as I E Txx xW where T, C, H, and W represents total frames (temporal dimension), channels in a frame (e.g., for RGB frames, C = 3), height and width of pixels in a frame, respectively. is passed through a feature extractor that generates voxel embeddings E e TTXXxaxX with temporal (7), channel (k) and spatial (a, 3) dimensions.
The goal is jointly derive the attention in the multidimensional space of s, without squeezing individual dimensions. For this, we deploy NMF-based matrix factorization to compute low-rank é by reconstructing from the factorized basis matrix W and coefficient matrix H. is essential to factorize E in a way that é approximated through the computed basis and coefficient matrices serves as an effective self-attention. Among several parameters that govern factorization, here we delve into the ones most relevant for the time series estimation task. These include: i) the transformation of the voxel embeddings that maps IR Xxxxx the factorization matrix Ve TR. xN and ii) the rank of the factorization.

---

CH
Ce
+
[c] xx]
Non- Nogative Marik Factonizatiio ( (NMFF)
Aa x xx
022
F
1
62
N
a1
B
Traroformation of Voxel Embedddinje to
Tranaformation of Low- rank Marrrx to Approx Embeddinge
Approximated
Factorization Matrix
Vowel
Embeddings
Embuddding
Instance Norm
Mi
Agprodmmatd Embeddings
Posidual Connection
Escimated BVP
Input Video
Feature Extroctor
Network Head
Signal
Framis
Factorized Self-Attention Module
(r7n6)
Figure 2: Factorized Self-Attenticn Module (FSAM) illustrated for 3D-CNN architecture for rPPG estimation.
For a 2D-CNN architecture with x channels and x spatial features, the transformation (1 (TRaB-+M implemented in the Hamburger module [88] is expressed as:
v E RM - (E E R*xaxp)
M,
N
(4)
where & channels are mapped to M and a x B spatial features are mapped to N, with an underlying assumption that spatial features 3D-CNN architectures, as ε E RTxxxax3 encodes temporal, channel, and spatial features, it is required revisit these mappings. While it can be argued that similar to 2D-CNN architectures, as 3D-CNN architectures have 3D kernels, the spatial-temporal features are inherently correlated. However, it should be noted that the scales of spatial and temporal dimensions are very distinet, owing to which the spatial-temporal patterns to be learned may not be uniformly captured through typical convolutional kernels (c.g. 3 x 3). Adjusting the ppatial-tempora kernel sizes can be heuristic task, and does not guarantee the extraction of desired features, while drastically increasing the model complexity (since for time series estimation, 7 >> a, (). Also, channels are not inherently correlated, and therefore it is crucial to devise multidimensional attention that jointly computes the spatial-tempe temporal and channel attention. To address this, we first consider negative Pearson correlation, a loss function that is commonly deployed to optimize end-to-end tod rPPG methods, expressed as:
inherently correlated due to learnt CNN kernels. However, for
Σ (r. TPPB (g) gPPs
Tpp
V (TtTS )2 )2
V E (0. gopg>²
where, pP99 E I is an estimated rPPG signal and n0 E x corresponds to the ground-trute BVP signal. The optimization of end-to-ent model to estimate vector temporal dimension (rPPg E R RXXT can be leveraged by establishing the correlation of features in spatial and channel dimensions with the features temporal dimension. Factorization a matrix that consists vectors in temporal domain and spatial and channel dimension as the features of the vectors, uniquely offers an opportunity to design the requisite attention. Prior to transforming & V kernels), and ReLU activation to ensure
R M x> it is
pre-processed through convolution layer (with 1 X non-negativity of the embeddings. Following this preprocessing, the temporal features of are mapped the vector dimension (M) in Vst while spatial and channel dimensions are mapped the feature dimension (N) of y This transformation of as depicted in fig 2 can be expressed as:
8 N (6) where, Spre represents preprocessing operation. Factorization of thus formed matrix 4ist with temporal vectors shall result in a low-rank matrix v'st which approximated based on the latent structure that establishes correlation of temporal features with spatial and channel features.
st MxN M (Spro(s RTxxxax8)) T
Vst o(vst)
5

---

where ø represents factorization operation. Vst is transformed back to the embedding space, resulting an approximated voxel embeddings é that selectively retains the spatial and channel features that contribute towards the recovery of salient temporal features in e. The resultant can be expressed as:
é I (Vst E RMXNN
(8)
where PMN--T808 represents matrix transformation operations. We use the one-step gradient optimization based approach [8] factorize Vot, This approach is a linear approximation of the conventional hack-propagation through time algorithm (for time 1 + oc) [65] as proposed with the Hamburger module [18]. Approximated low-rank matrix Vst is transformed back to the embedding space through NM resulting in & that can potentially serve as the requisite attention. is post-processed with a convolution layer (with x 1 kernels), and ReLU activation, followed by element-wise multiplication with s. This multiplication operation serves as an excitation operation, which can be distinctly effective é retains the dimension of while computing the attention. The product is
and added with E that serves as residual connection as depicted in fig. 2 It is to be noted that for each single forward pass through the model, approximation of V requires 4-8 steps, however, FSAM implements NMF within "no_grad" block, that does not require back-propagation through the NMF for model optimization. Representing the network head as w, Spoet as post-procsssing operation and I. as instance normalization, the estimated pPP signal can be expressed as:
pP9 = wis + (A @ Epos((E))
(9)
Next, we look at the rank of the factorization that affects the approximation of V The primary consideration for the rank L < min(M, NN mentioned in $3.1 ensures that Vst is of low rank. Although the choice of L is generally governed by the downstream task, it is often derived empirically. In the context of p estimation, through T that maps temporal features along the M dimension. As we expect only a single signal underlying source of BVP signal across all facial regions, single vector estimation
revisit the formulation of factorization matrix
vo corresponding to rank-1 (i.c., = 1) shall be sufficient to capture the spatial, temporal, and channel features that contribute to the PPG estimation. Experimentation with rank-1 and higher rank factorization (appendix A.3) shows that for the higher ranks, the performance remains at par with that the network without the FSAM, indicating that for rPPG estimation task, rank-1 factorization offers the optimal multidimensional attention, confirming our understanding.
3.3 Deployment of FSAM in 3D-CNN and 2D-CNN Architectures
We deploy FSAM our proposed 3D-CNN model, FactorizePhys and integrate it in an existing 2D-CNN architecture, EfficientPhys 37 assess its versatility.
FactorizePhys Architecture: FactorizePhys, as depicted in fig. B[A] is an end-to-end 3D-CNI architecture for estimating rPPG signal from raw video frames. Skin reflection models 62 discuss the presence of several unrelated stationary and time-varying temporal components, and relatively weaker pulsatile component of interest. eliminate stationary components, FactorizePhys implements a Diff as first layer, inspired by existing rPPG architectures 6 36 37.. The resultant Diff frames are normalized with IN, unlike existing architectures that use BatchNorm. The size
Recimaaion Road
Sphamme
Frumee
S0d:
3000
2400
1d02
200/
3656:
Tetel
101e1
Natraltationn Mustulo
Dow
COn
Dan
Extimared Boad
Dard
Figure 3: (A) Proposed FactorizePhys with FSAM; (B) FSAM Adapted for EfficientPhys 37

---

of the keree and the strides of each convolution layer are depicted in fig. B] For each layer, we use TanH activation followed by IN. Spatial features are gradually aggregated by not padding the features, while we deploy spatial convolution strides only on the 3rd and 6th layers. For temporal features, ame padding retains the input temporal dimension throughout the network, as depicted in fig. A]. Downsizing of temporal features may result in high-amplitdde unrelated time-varying components to outweigh the weaker rPPG related pulsatile component. To provide a clearer overview of the architecture of FactorizePhys, only the spatial and temporal dimensions of the features at multiple layers are shown in fig. 3 A]. while the channel dimension is skipped. FSAM, as elaborated in 3.2 is deployed to jointly compute multidimensional attention, at the layer where the spatial dimension is reduced to 7 x 7, as reported as the optimal spatial dimension in a recent work [].
Adaptation of FSAM for 2D-CNN Architecture: Several SOTA rPPG methods 36 37 leverage the TSM 34 that efficiently models spatial-temporal features using 2D-CNN architectures. The parameter called 'Frame Depth' (tb 3 vb < K channels) controls the number of channels that are shifted along the temporal dimension for modeling temporal features. We investigated the effectiveness of the proposed FSAM with a more recent TSM-based SOTA rPPG architecture, EfficientPhys [i7], which also deploys the Self-Attention Shifted Network (SASN) as the attention module. As vo controls the amount of temporal information that is learned, we use this to formulate the mapping for the factorization matrix. Equation (6) can be adapted for TSM based architectures to appropriately transform the embeddings to factorization matrix as:
(10) Figure 3 [J shows modified EfficientPhys (37 architecture, in which we drop SASN blocks 37 and add single FSAM. Unlike SASN (37), FSAM derives attention without squeezing any individual dimension, which turn can strengthen the correlation between temporal, channel and spatial features. This adaption critically evaluates the proposed FSAM against its counterpart in the SOTA architecture, addressing the recommendations of a recent survey article [23] on visual attention methods.
Viaam E MMM p MN (stam E [R 3 v + X N
4 Experiments
We perform an evaluation with carefully selected end-to-end SOTA rPPG methods that include Phys- Net X 3D-CNN architecture without the attention mechanism, EfficientPhys (37 a 2D-CNN architecture with self-attention, and PhysFormer (77), a transformer- 3D-CNN architecture with multi-head self-attention. Comparison of FactorizePhys and PhysNet [ can indicate the importance of the attention mechanism in 3D-CNN rPPG architectures, while comparison of EfficientPhys with SASN (37] and EfficientPhys
337 with FSAM allows evaluating the effectiveness of the proposed FSAM in 2D-CNN architectures, and thus allows assessing the versatility of FSAM. Similarly, the comparison of Factorize and PhysFormer (77) offers thorough evaluation of the proposed FSAM against multi-head self-attention in 3D-CNN architectures.
Each model was trained on one of the four existing datasets that include iBVP 29 PURE [55].
UBFC-IPPG 2 and SCAMPS 42 and evaluated on the other three. Appendix A. provides our detailed description of these datasets. Our code is based on the rPPG-Toolbox [38] with specific
adaptations described in appendix A.2 We train all models uniformly with 10 epochs [79 on iBVP
29), PURE (55), and UBFC-IPPG 22 datasets, and with one epoch on SCAMPS (42) dataset. For fair evaluation, all model-specific hyperparameters were maintained as provided by the respective SOTA rPPG methods, while the training pipeline related hyperparameters, which include preprocessing steps for images and labels, batch size, number of epochs, learning rate, scheduler, and optimizer were kept consistent for training all the models.
5 Results and Discussion
Ablation Study: First, we train FactorizePhys on UBFC-rPPG 2 dataset and test on PURE (55) and iBVP 29 datasets, to compare different transformations of = E X with temporal, channel and spatial features to factorization matrix Vol E M as tabulated table Superior cross-dataset generalization can be observed when the temporal dimension, 7 is mapped to M, as described in 3.2 We assess contribution of FSAM over base FactorizePhys model and observe consistent performance gains with FSAM, as reported in table 3 in appendix A We then investigate
7

---

Table 1: Ablation Study for Different Mapping of Voxel Embeddings to Factorization Matrix
Mapping of Vouel Embeddiins
Training
Testing
MAE (HI)
RMSE (HR)
MAPR (HR) J
Corr (HR)
SNR (BVP)
MACC(BVP)
Dataset
Dataset
M
N
0.77 0.42
3,29 1.11
0.82
0.99+
13,84
0.02
PURE
TXN H
ax B
0.71 a 0.39
3.05 + 1.03
1.21 0.76
=
13.60+00.1
0.77 = 0.02
UBFC-PPGG
0. B.x.
0.48 0.17
1.39 0.35
0.72 0.28
1,00 0.00
14.16 + 0.83
0.78 0.02
3
2,05 a 0.40
4.65 0.91
2.87 = 0.59
0.88 0.04
5.99 à 0.58
0.55 0.01
iBVP
TXK axp.P.
0.46
+
0.68
0.05
5.83 0.57
0.54 0.01
O xxBxxxx
1,73 0.39
4.38 + 1.06
2.40 = 0.57
0.90 + 0.04
6.61 # 0.58
0.56 = 0.01
residual connection table [] and observe to contribute positively. We also observed that the base FactorizePhys model trained with FSAM retains the performance gains in-spite when FSAM is skipped during the inference. As this eliminates the computational overhead during inference, we report our main results of FactorizePhys trained with FSAM, by running inference without the FSAM. On contrary, in case of TSM (34] based EfficientPhys (37 model trained with FSAM, we observed performance drop when FSAM was skipped during inference, and therefore for EfficientPhys with FSAM, we do not drop FSAM during inference. Evaluation for factorization ranks and optimization steps to solve NMF shows consistent superiority of rank-1 factorization in table 4 in appendix A
FactorizePhys vs. State-of-the-Art: We use heart rate (HR) that include signal-to-noise ratio (SNR) and maximum amplitude of cross-correlation (MACC) [29 [0] for ovaluation. SNR and MACC are direct measures to compare estimated rPPG signals with ground-trute -ruun BVP signals. The HR metrics reported are the mean absolute error (MAE), the square root of the mean square error (RMSE), the mean absolute percentage error (MAPE), and Pearson's 's correlation coefficient (Corr) 67 of the estimated HR. Uncertainty estimates quantifying the variability associated with signal estimation have been shown to be strongly correlated with the absolute error of the estimated HR [52). In addition, for each metries, we report the standard error to estimate the variability of each model. As most SOTA end-to-end models show robust within- dataset performance, we present cross-dataset performance in table 2 while reporting within-dataset performance in table in appendix A
along with BVP metrics
First, we observe that for all the evaluation metrics reported, the proposed FactorizePhys with FSAM outperforms the SOTA methods on PURE (55) and iBV (29] datasets, across all training datasets. This suggests consistent and superior generalization achieved by the proposed method. Cross- dataset evaluation on the UBFC-rPPG [2] dataset further highlights the performance gains of the proposed FactorizePhys model when trained with the iBVP [29] and the SCAMPS [42] datasets, and at-par performance when trained with the PURE dataset. When models are trained with SCAMPS [42 (synthesized dataset), FactorizePhys uniquely outperforms the SOTA methods on all testing datasets further indicating the superior cross-dataset generalization. The performance of EfficientPhys 37 with FSAM exceeds in most cases and remains at par in the rest, compared the EffficientPhys model with SASN (37), suggesting the versatility of FSAM as an attention module. As 3D CNN kernels FactorizePhys can learn spatial-temporal patterns better than the TSM (34) based 2D-CNN model (EfficientPhys (37)). FactorizePhyy with FSAM outperforms EfficientPhys [37] with FSAM across all datasets. Lastly, the proposed method consistently achieves superior SNR and MACC for the estimated rPPG signals, highlighting the enhanced reliability of the extracted signals.
Computation Cost and Latency: We compare computational complexity and latency for all the models in fig. 4 A), and provide further details in table 9 in appendix A The cumulative MAR is computed by averaging cross-dataset performance for respective models across all combinations of training and testing datasets reported in table The proposed FactorizePhys with FSAM not only shows the best performance, it has significantly less number of model parameters and performs par in terms of latency as the 2D-CNN SOTA rPPG method, EfficientPhys (37]. Specifically, dropping FSAM during inference does not result in loss of performance for FactorizePhys, while reducing latency considerably, making it highly suitable for real-time and assource-constriin contrast, when FSAM was dropped after training EfficientPhys (37] with FSAM, it did not retain the performance (results not shown). We interpret that while FSAM effectively influences the 3D convolutional kernels in FactorizePhys to increase the saliency of relevant spatial-temporal features, 2D convolutional kernels cannot benefit adequately due to the limited ability to model spatial-temporal
deployment. In
8

---

Table Cross-datasee Performance Evaluation for rPPG Estimation
Trsining Dataset
Mode
Attention
MAE
RMSE
(HR))
(HR))
BYPIT
(EVP)
Module
Performance Evaluation Dutasct
PhysNei
2.71
a.11
PhysFoomer
TD-MISA
6,58 +: 1.98
16,55 4 3.60
693 1,00
0.56 0.09
9.75 1.96
0 0.00
IBVP
FificieniPPys
SASN
0.56
0.7.
0.01
11,95
EttclentPhys
ESAM
0.34
1.19 0.30
0.4
0,999 0.01
12.64 0.78
Factorizehhys (Ous))
I'A (Ouss)
0,60 0:1
0.87
0,997 0.01
15.19 0.91
0.77 0.02
PhysNet
26.7443.11
46.19
46.3
0.15
22.1
PlysFonmer
TD-MHSA"
2..64
28.13 5.0
30.58
0.51
0,84
SCAMPS
EmicientPhss
SASN
6,21
18,45 4:54
12.16:4.57
0.74 0.09
a.79
0.51 4
FificianPPyys
(Ouns)
15.12+4444
FadieeircPyys
ISAM
5,43
15,80 3.558
a..10
PlysNet
10.38 2:44
21.14 3.90
2001 4,37
11.01 0097 0.9
0.72 0.02
PhysFoomer
TD-MUSA*
8.00
4.677
17.6
4.009
8.3
0.66
UDTC
FfficienPhyys
SASN
7.63
0.50 0.08
8.77
EdicieniPhys
ISAM (Ous)
3,69 1.66
13.27 3.55
5,8 2.63
0.53 0.07
9.65 0.90
0,68 0.02
FadeeircPPyge (Chu)
(Ouns)
0.01
Evaluation UBF C-PPPG
Phyniee
3,09 1.79
10,72
2883
081
7.13 1,53
0,81 0.02
Physioomer
TD MHSA"
9,88 2.95
19.99 5:35
8.42
0.44
2.00
0.70 0.03
FificienPhys
SASN
0.52
0,087
8.71 1.23
EfliieutPyys
PSAM (Ours)
2.87 0.88
1.31
8.54 1.26
0,85 0.61
PactorizePhys (Oum))
ESAM (Orrs)
1
0.23
0.00
0.86 0.01
PhysNet
834
PhysFormer
TD-MHSA*
1,0 0.36
2.40 + 0.69
1.21 0.48
0 0.03
8.42 1.24
0.85 0.01
PURE
EthiieniPhss
SASN
144
3.16
1.68 0.64
0,982 4.00
6.87
0.79 0.02
DfficienPhys
FactoxizePPys (Oun)
FSAM (Ous)
1.04 1.44
2.44 0.69
1.23
0.589 0.13
= 1.30
0,87 0.61
PhysNei
21.24
18.81 4.71
13.55 3.81
0.38 0.17
-0.09 1.02
0.48 0.03
2.29
SCAMPS
SASN
2,18 0.75
4.83 +.43
2.35 0.76
0.96 0.05
4.40 1,03
0.67 0.01
EfficeniPhes
FSAM (Ours)
2.69 0
5.20
3.16
0.95 a 0.06
1.74
0.63 0.02
Performance Evaluation IBVP Dataset
PhysSee
+ 0.33
3.77 e 0.73
2:17 0.42
0.92 0.04
6.8 = 0.60
0.55 0.01
EfficentPhys
SASN
3,80
14,82 3.74
5.15
2.48
0.01
EdicienPPys
PSAM (Ous)
2.10
4,00 0.64
2.4
0.91 0.04
1.54
0.49 0.01
FicterizePPyss (Dun))
2.14
6.7%0055 0.57
Physdet
37.40 3.38
45.62
0.10
.6.1 6.11:0022
PhysFormcr
TD-MHSA*
41.33 1.31
43.80 3.11
$8.56 2.36
0.15 0:10
-9.13
0.14 0.00
SCAMPS
EflicinPPpss
EfficintEhys
ISAM (Ouns)
13,40 1.69
22.10:4:::00
19.93
0.05 0.10
3,46 0.26
0.24
PactorizeEhys (Oum)
FSAM (Ouns)
2,71 3.71&:055
6.22 8: 1.38
3.87 a 0.80
0.81 0.06
1.36 0.47
0.43 0.01
PhysNer
+
PhysEormer
TD MHSA"
7.86
27.1344269
1.71 171100.0
UBFC-PPG
EtticenFPyys
SASN
0.74
7.07 1.81
4.02 1.08
0.74 0.07
4.03 0.55
0.49 0.01
ElliiionPyys
(Ous)
FactorizeEtyss (Ours)
FSA (Ouns)
1.74
4.39
2.42 0.57
0.90
6.99 0.57
0.56
TD- M Temporal Difference Multi-Head Self- Aterrion (7) SASK: Self- Self-Attetion Shifted Network 57 FSAM: Progosed Factorieed Self-Ateemion Medule.
All with dusimal places, exorpe those comelation mensuroo range betweeen

---

Fachobarttv onfenece FS4en)
B
Feature Maps Model Trained without FSAM
Feature Maps of Model Trained with FBAM
Pact STSAnG 1. pPPs du mes
Cumuistie Mean Absollu Enter of Sat (BPM))
Prrcaeethie plosed 3.50 sime'ss B.14M
trickenthe (SADN)
Medel
Sphare
Pheaher
11,908P9/ FS6 2iaama
Labency (ma)
Figure 4: (A) Cumulative cross-dataset performance (MAE) v/s latency' plot. The size of the sphere corresponds to the number of model parameters; (B) Visualization of learned spatial-temporal features from the base 3D-CNN model trained without and with FSAM; System specs: Ubuntu 22.04 OS, NVIDIA GeForce RTX 3070 Laptop GPU, Intel@ CMre 17-10870H CPU @ 2.20GHzz, 16 GB RAM.
features. should also be noted that the higher latency of FactorizePhys compared to EfficientPhys although it has fewer model parameters, be attributed to the difference in floating-point operations (FLOPS) between the 3D-CNN and 2D-CNN architectures.
We compute absolute cosine similarity between the temporal dimension of 4D embeddings (with temporal, spatial, and channel dimensions) and the ground-truth signal to visualize the learned attention for FactorizePhys trained without and with FSAM in fig. B) where each tile represents channel of the embedding layer. A higher cosine similarity score between the temporal dimension of the embeddings and the ground-truth PPG signal, which is observed for FactorizePhys trained with FSAM, indicates higher saliency of temporal features. The spatial spread of high cosine similarity scores in different channels for FactorizePhys trained with FSAM, highlights selectivity of the learned attention, providing clearer evidence that the FactorizePhys model trained with FSAM can effectively pick the spatial features having the strong presence of the rPPG signal (i.e., facial regions with visible skin surface). Figure [ not only suggests the effectiveness of the joint computation of multidimensional attention, but also offers more intuitive visualization of learned spatial-temmp features than existing visualization approaches 779 37
Visualization of Learned Attention:
6 Conclusion
We present FactorizePhys, 3D-CNN model utilizing the Factorized Self- Seff-Attention Module, FSAM, to concurrently extract multidimensional (spatial, temporal, and channel) attention for the downstream task of rPPG estimation from video frames. The assessment performed utilizing various rPPG datasets demonstrates that our proposed method possesses superior generalization capabilities across different datasets, compared to current tftte-ff-hhe-rt methods. Moreover, when adjusted to the 2D-CNN architecture, FSAM achieves performance on par with the established SASN [37 attention, underscoring its adaptability across diverse network architectures.
Broader Impacts and Limitations: The superior performance of FactorizePhys equipped with FSAM to estimate rPPG indicates its potential utility in various healtheare applications that require the estimation of physiological signals through noncontact imaging. Although FSAM has shown efficacy as multidimensionn attention mechanism specifically for the extraction of rPPG signals, more research is needed to determine the efficacy of the proposed method in extracting heart rate variability metrics as well as other physiological signals. Despite the saate-of-the-ar performance of the proposed rPPG method, signal peaks can still be susceptible to challenging real-world scenarios, such as active head movements, occlusions, and dynamic changes in ambient lighting conditions, an issue that is qualitatively illustrated in the waveforms depicted appendix A.IT Moreover, is imperative to conduct additional research to evaluate the effectiveness of FSAM across other spatial- temporal domains, including video understanding. video object tracking, and video segmentation, along with several other downstream tasks that depend multi- muutti-dimensionall input data. In the context of signal estimation tasks, the utilization of NMF variants that integrate temporal or frequency constraints on time series vectors may offer enhanced attention capabilities. These constraints congruent with the characteristics of the ground truth and present avenues for future investigation.
10

---

Acknowledgments and Disclosure of Funding
The author JJ was fully supported by the UCL CS PhD Studentship (GDI Physiological Computing and Artificial Intelligence) which Prof. Cho has secured.
References
[1] Pooya Ashtari, Diana M. Sima, Lieven De Lathauwer, Dominique Ssppey-Marinice Frederik Maes, and Sabine Van Huffel. Factorizer: scalable interpretable approach context modeling for medical image segmentation. Medical Image Analysis, 84:102706, 2023.
[2] Serge Bobbia. Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsupervised skin tissue segmentation for remote photoplethysmogrrphy Pattern Recognition Letters, 124:82-90, 2019.
[3] Deivid Notina-Monsallee Yannick Benezeth, and Johel Miteran. Ruppg: An ultra light 3denn for real-time remote photoplcthyysooraphhy In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2146 2154, June 2022.
[4] Frédéric Bousefsaf, Alain Pruski, and Choubeila Maaoui. 3d convolutional neural networks for remote pulse rate measurement and mapping from facial video. Applied Sciences, 9(20), 2019.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[6] Weixuan Chen and Daniel McDuff. Deepphys: Vidco-based physiological measurement using convo- lutional attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[7] Youngjun Cho. Rethinking eye-blink: Assessing task difficulty through physiological representation of spontaneous blinking. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA, 2021. Association for Computing Machinery.
[8] Youngjun Cho, Nadia Bianchi- Berthouze, and Simon J. Julier. Deepbreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference Affective Computing and Intelligent Interaction (ACII), pages 456-463, 2017.
[9] Youngjun Cho, Simon Julier, and Nadia Bianchi- Berthouze. Instant stress; detection of perceived mental stress through smartphone photoplethysmogrrphy and thermal imaging. JMIR mental health, 6(4)xc10140 2019.
[10] Youngjun Cho, Simon J. Julier, Nicolai Marquardt, and Nadia Bianchi- Berthhizee Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomed. Opt. Express. 8( 8(10))440-4503, Oct 2017.
[11] Gerard de Haan and Vincent Jeanne. Robust pulse rate from chrominance based rppg. IEEE Transactions on Biomedical Engineering, 60(10):2878 227628866,66 2013.
[12] Inderjit S Dhillon and Dharmendra S Modha. Concept decompositions for large sparse text data using clustering. Machine learning, 42:143-175, 2001.
[13] Shuai Ding, Zhen Ke, Zijie Yue, Cheng Song, and Lu Lu. Noncontact multiphysiological signals estimation via visible and infrared facial features fusion. IEFF Transactions on Instrumentation and Measurement, 71:1-13, 2022.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[15] Wenbin Du, Yali Wang, and Qiao. Recurrent spatial- attention network for action recognition in vidcos. IEEE Transactions on Image Processing, 27(3):1347-136) 2018
[16] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

---

[17] Xiao Fu, Kejun Huang, Nicholas D. Sidiropoulos, and Wing-Kin Kin Ma. Nonnegative matrix factorization for signal and data analytics: Identifiability, algorithms, and applications. IEEE Signal Processing Magazine, 36(2):59-88) 80, 2019.
[18] Zhengyang Geng, Meng- Haa Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? International Conference on Learning Representations. 2021.
[19] R.M. Gray and D.L. Neuhoff. Quantization. IEEE Transactions on Information Theory, 44(6):2325-2335 1998.
[20] Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng- Tag Jiang, Tai-Jiang Mu, Song- Zhang, Ralph Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey, Computational visual media, 8(3):331-366 2022.
[21] Xudong Guo, Xun Guo, and Yan Lu. Ssan: Separable self-attention network for video representation learning. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22618-12627, June 2021.
[22] Anup Kumar Gupta, Rupesh Kumar, Lokendra Birla, and Puneet Gupta. Radiant: Better rppg estimation using signal embeddings and transformer. In Proceedings of the IEEEJCVF Winter Conference on Applicaations of Computer Vision (WACV), pages 4976-4986, January 2023.
[23] Mohammed Hassanin, Saeed Anwar, Ibrahim Radwan, Fahad Shahbaz Khan, and Ajmal Mian. Visual attention methods in deep learning: An in-depth survey. Information Fusion, 108:102417, 2024.
[24] Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8417-8427, June 2022.
[25] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[26] Min Hu, Fei Qian, Xiaohua Wang. Lei He, Dong Guo. and Fuji Ren. Robust heart rate estimation with spatial-temporal attention network from facial videos. IEEE Transactions Cognitive and Developmental Systems, 14(2)6639 647, 2022.
[27] Bin Huang, Shen Hu, Zimeng Liu, Chun-Liang Lin, Junfeng Su, Changehen Zhao, Li Wang, and Wenjin Wang. Challenges and prospects of visual contactless physiological monitoring clinical study. NPJ Digital Medicine, 6( 6(1):331, 2023.
[28] Jitesh Joshi, Nadia Berthouze, and Youngjun Cho. aelf-aaverlariall multi-scale contrastive learning for semantic segmentation of thermal facial images. 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 2 21-24, 2022. BMVA Press, 2022.
[29] Jitesh Joshi and Youngjun Cho, iBVP Dataset: RGB-Thermal rPPG Dataset with High Resolution Signal Quality Labels. Electronics, 13(7), 2024.
[30] Jitesh Joshi, Katherine Wang, and Youngjun Cho. PhysioKit: An Open-Source, Low-Cost Physiological Computing Toolkit for Singlc-and Multi-User Studies. Sensors, 23(19), 2023.
[31] Daniel Lee and Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. nature, 401 (01(6755)77887771 1999.
[32] John Boaz Lee, Ryan Rossi, Sungchul Kim. Nesreen K. Ahmed, and Eunyee Koh. Attention models in graphs: A survey. ACM Trans. Knowl. Discow. Data, 13(6), nov 2019.
[33] Dong Li, Ting Yao, Ling- Yu Duan, Tao Mei, and Yong Rui. Unified spatio-temporal attention networks for action recognition in videos. IEEE Transactions on Multimedia, 21(2):416 428, 2019.
[34] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[35] Tonglai Liu. Ronghai Luo, Longgin Xu, Dachun Feng, Liang Cao. Shuangyin Liu, and Jianjun Guo. Spatial channel attention for deep convolutional neural networks. Mathematics, 10(10):1750, 2022.
[36] Xin Liu, Josh Fromm. Shwetak Patel, and Daniel McDuff. Multi task temporal shift attention networks for on-ievice contactless vitals measurement. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19400-194 19411. Curran Associates, Inc., 2020.
12

---

[37] Xin Liu, Brian Hill, Ziheng Jiang, Shwetak Patel, and Daniel McDuff. Efficienthhy:: Enabling simple, fast and accurate camera-based cardiac measurement. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pages 5008-5017, January 2023.
[38] Xin Liu, Girish Narayanswamy, Akshay Paruchuri, Xiaoyu Zhang, Jiankai Tang, Yuzhe Zhang. Roni Sengupta, Shwetak Patel, Yuntao Wang, and Daniel McDuff. rppg-toolbox: Deep remote ppg toolbox. In Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 68485-68510 10. Curran Associates, Inc., 2023.
[39] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages
October 2021.
[40] Hao Lu and Hu Han. Nas-hr: Neural architecture search for heart rate cstimation from face videos. Virtual Reality & Intelligent Hardware, 3 (1):33-42, 2021. Emotion recognition for human-computer interaction.
[41] Hao Lu, Hu Han, and S. Kevin Zhou. Dual-gan: Joint byp and noise modeling for remote physiological measurement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition June 2021.
(CVPR), pages 12404-12411
[42] Daniel McDuff, Miah Wander, Xin I Liu, Brian Hill, Javier Hernandez, Jonathan Lester, and Tadas Bal- trusaitis. Scamps: Synthetics for camera measurement of physiological signals. Advances in Neural Information Processing Systems, 35:3744-3757 2022.
2022.
[43] Clara Moge, Katherine Wang, and Youngjun Cho. Shared user interfaces of physiological data: Systematic review of social biofeedback systems and contexts in hei. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-16, 2022.
[44] Girish Narayanswamy, Yujia Liu, Yuzhe Yang, Chengqian Ma, Xin Liu, Daniel McDuff, and Shwetak Patel. Bigsmall: Efficient multi-task learning for disparate spatial and temporal physiological measurements. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7914-7924, 2024.
[45] Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. Rhythmmet: End-to-end heart estimation from face via spatial-temporal representation. IEEE Transactions on Image Processing, 29:2409-2423 2020.
[46] Xuesong Niu, Xingyuan Zhao, Hu Han, Abhijit Das, Antitza Danteheva, Shiguang Shan, and Xilin Chen. Robust remote heart rate estimation from face utilizing spatial-temporal attention. In 2019 14th IEFE International Conference on Automatic Face Gesture Recognition (FG 20/9), pages 1-8, 2019.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pere-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[48] Ming- Zher Poh, Daniel McDuff, and Rosalind W. Picard. Non-contact, automated cardiac pulse mcasurements using vidco imaging and blind source separation. Opt. Express, 18(10) 10774, May 2010.
[49] Delong Qi. Weijun Tan, Qi Yao. and Jingfeng Liu. YoloSface: Why reinventing face detector. In European Conference on Computer Vision pages 228-244. Springer, 2022.
[50] Leslie Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369-386. SPIF, 2019.
[51] Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. Pulsegan: Learning to generate realistic pulse waveforms in remote photoplethysmographyy IEEE Journal of Biomedical and Health Informatics, 253 1384, 2021.
[52] Reneheng Song, Han Wang, Haojic Xia, Juan Cheng, Chang Li, and Xun Chen. Uncertainty quantification for deep lcarning-basee based remote photoplethysmography. IEEE Transactions on Instrumentation and Measurement, 2023.
[53] Sijie Song, Cuiling La, Junliang Xing, Wenjun Zeng, and Jiaying Liu. An end-to-end spatio-temporal attention model for human action recognition from skeleton data. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.
[54] Radim Spetlik, Vojtech Franc, and Jirí Matas. Visual heart rate estimation with convolutional neural network. In Proceedings the british machine vision conference, Newcastle, UK, pages 3-6, 2018.
13

---

[55] Ronny Stricker, Steffen Müller, and Horst- Gross. Non-contact video-based pulse rate measurement mobile service robot. In The 23rd IEEE International Symposium Robot and Human Interactive Communication, pages 1056-1062. IEEE, 2014.
[56] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-hert: Pre-training of generic visual- linguusii representations. In International Conference on Learning Representations, 2020.
[57] Snigdha Tariyal, Angshul Majumdar, Richa Singh, and Mayank Vatsa. Deep dictionary learning. IEEE Access, 1010099--119,, 2016.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[59] Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. Remote plethysmographic imaging using ambient light. Opt. Expresx, 16(26): 1434-2144455 Dec 2008.
[60] Jing Wang and Lei Liu. A multi-attention deep neural network model base on embedding and matrix factorization for recommendation. International Journal of Cognitive Computing in Engineering, 1:70-77, 2020.
[61] Rui- Rui-Xuan Wang, Hong-Mei Sun, Kong-Rong Hao, Ang Pan, and Rui-Sheng Jia. Transphys: Transformer- based unsupervised contrastive learning for remote heart rate measurement. Biomedical Signal Processing and Control, 86:105058, 2023.
[62] Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan. Algorithmic principles of remote ppg. IEEE Transactions on Biomedical Engineering, 64(7): 1479-119,,, 2017.
[63] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018. 8.
[64] Yu-Xiong Wang and Yu-Jin Zhang. Nonnegative matrix factorization: A comprehensive review. IEEE Transactions on Knowledge and Data Engineering, 25(6):1336 13566,11333120 2013.
[65] P.J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE. 78100):1550-166 1990.
[66] Sanghyun Woo, Jongchan Park, Joon- Joon-Young Lee, and In So Kweon. Cham: Convolutional block attention module. Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[67] Hanguang Xiao, Tianqi Liu, Yisha Sun, Yulin Li, Shiyi Zhao, and Alberto Avolio. Remote photoplethys- mography for heart rate measurement: A review. Biomedical Signal Processing and Control, 88: 105608, 2024.
[68] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages -10900. Curran Associates, Inc., 2021.
[69] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang. Yiming Zhang. Kai Xu, and Jun Wang. Mlevnet: Multi- level context votenet for 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition (CVPR), June 2020.
[70] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pages 048870057 Lille, France. 07-09 Jul 2015. PMLRR
[71] Ming Xu, Guang Zeng. Yongjun Song, Yue Cao. Zeyi Liu, and Xiao He. Ivrr-ppg: An illumination variation robust remote-ppg ppg algorithm for monitoring heart rate of drivers. IEEE Transactions on Instrumentation and Measurement, 72:1-10, 2023.
[72] Shuangjie Xu. Yu Cheng, Kang Gu, Yang Yang, Shiyu Chang, and Pan Zhou. Jointly attentive spatial- temporal pooling networks for video-based person re- re-ieetifiiation. In Proceedings of the IEEE interna- tional conference on computer vision, pages 4733-4742, 2017.
14

---

[73] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Atingan: Fine-grained text image generation with attentional generative adversarial networks. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition (CVPR). June 2018.
[74] Chenggang Yan, Yunbin Tu, Xingzheng Wang. Yongbing Zhang, Xinhong Hao, Yongdong Zhang, and Qionghai Dai. Stat: Spatial-temporal attention mechanism video captioning. IEEE Transactions on Multimedia, 22(1))229-22) 241, 2020.
[75] Zongxin Yang. Linchao Zhu, Yu Wu, and Yang. Gated channel transformation for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). June
2020.
[76] Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Yawen Cui, Jiehua Zhang. Philip Torr, and Guoying Zhao. Physformer++: Facial vidco-basee basod physiological measurement with slowfast temporal difference transformer. International Journal of Computer Vision, 131(6)1307-1338) February 2023.
[77] Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip H.S. Torr, and Guoying Zhao. Physformer: Facial video-based physiological measurement with temporal difference transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4186-4196, June 2022.
[78] Yuhui Yuan, Lang Huang, Jianyuan Guo. Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object context for semantic segmentation. International Journal of Computer Vision, 13753)22355- 2021.
[79] Changchen Zhao, Hongsheng Wang, Huiling Chen, Weiwei Shi, and Yuanjing Feng. Jamsnet: remote pulse extraction network based on joint attention and multi-scale fusion. IEEE Transactions on Circuits and Syxtems for Video Technology, 3366):7783-7997, 2023.
[80] Changehen Zhao, Menghao Zhou, Zheng Zhao, Bin Huang, and Bing Rao. Learning spatio-temporal pulse representation with glohal-local interaction and supervision for remote prediction of heart rate. IEEE Journal of Biomedical and Health Informatics, 28(2):609-220 2024.
[81] Hongyi Zheng, Hongwei Yong, and Lei Zhang. Deep convolutional dictionary learning for image denoising. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 630-641, June 2021.
[82] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable (detr): Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2021.
[83] Yu Zitong, Li Xiaobai, and Guoying Zhao. Remote photoplethysmograph signal measurement from facial videos using seatio-temporal networks. In 30th British Machine Vision Conference (BMVC), 9th-12th September 2019, Cardiff. UK, September 2019.
15

---

A Appendix Supplemental Material
A.1 Datasets
All datasets provide video recordings with a resolution of 640 x 480, and frame rate of 30 FPS. Below provide data-specific details.
The iBVP dataset consists of 124 synchronized RGB and thermal infrared videos from 31 subjects, acquired under controlled conditions. Each video is 3 minutes in duration, and the ground truth BVP signals were acquired from the ear using PhysioKit [3.. Data were acquired under different conditions that include controlled breathing, math tasks, and head movements. BVP signals are marked with the signal quality, enabling the use of the video frames only where the quality of ground-truth BVP signal is high. In this work, use only RGB frames to train the models.
iBVP 29]:
This data set comprises video recordings from subjects, with the ground-trui BVP and SpO2 signals acquired from the subject's finger. For each participant, six recordings are acquired under varied motion conditions, offering a range of data reflecting different physical states.
PURE [55]:
This data set contains video recordings of 43 subjects acquired under indoor conditions with combination of natural sunlight and artificial illumination.
UBFC-rPPG [2:
This dataset comprises 2800 videos of synthetic avatars that were generated through high-fidelity quasi-photorealistic renderings. Although the videos introduce various conditions such as head motions, facial expressions, and changes ambient illumination, they are often used as a training set rather than a validation test set.
SCAMPS 42
A.2 Implementation Overview
The preprocessing steps for video frames include face detection using the YOLO5Face [49] face detector at an interval of 30 frames and using the detected facial bounding box to crop 30 subsequent frames, prior to performing the next face detection. The cropped facial frames are resized to a resolution of 72 x 72, which has been shown to be sufficient to estimate the rPPG. Additionally, to ensure uniform input data for all models, we add Diff layer to the PhysNet [3] and PhysFormer 77 architectures, implemented by EfficientPhys (37 and the proposed FactorizePhys models, and train all the models from scratch using uniformly preprocessed video frames.
The number of frames in video chunk is maintained as 161, which after the Diff layer becomes 160, making the spatial-temporal input data size 160 x 72 72. Ground-truth BVP signals are also uniformly standardized for training all models. This is different from some of the recent work 37 that applics Diff in addition to standardization. We empirically found that all models perform significantly better when trained with the standardized BVP signals, although when the Diff is applied to the video frames.
All models were trained with 10 epochs on, following a recent work [79]. a higher number of epochs, e.g. 30 epochs as used in rPPG-Toolbox [8] resulted in poor generalization for all models. However, we used only one epoch for all models to train on the SCAMPS [42] dataset, since this dataset is a synthesized dataset with generated BVP signals that are easier for models to learn, unlike real-world datasets. Training beyond one epoch resulted in poorer cross-dataset performance for all the models. The batch size of 4 was used consistently throughout the training and the maximum learning rate was set to 1 x 10 with 1 cycle learning rate scheduler [50] for all CNN models.
In addition, CNN models were optimized using negative Pearson correlation as loss function. The learning rate for PhysFormer [77] was set to x 10-4 and it was optimized using dynamic loss composed of several hyperparameters, a negative Pearson loss, a frequency cross-entropy loss and a label distribution loss as used by the authors and implemented in the rPPG- Toolbox (38). Before computing HR for performance evaluation. both ground truth and estimated BVP signals were filtered using bandpass filter (low cutoff = 0.60 Hz, high cutoff 3.30 Hz) to accommodate HR ranges of 36 to 198 BPM. HR was then computed using the FFT-peaks-bar based approach as implemented in iPPG-Toolbox (38).
16

---

A.3 Ablation Studies for FactorizePhys
We conduct ablation studies evaluate optimal architectural choices and hyperparameters for the pro- posed FactorizePhys and FSAM. In table 3 we compare base FactorizePhys without FSAM and with FSAM and observe consistent performance gains with FSAM. Evaluation with and without residual connection indicates performance gains when residual connection around FSAM is implemented.
Table Ablation study to assess residual connection FSAM Module, and compare the models trained with FSAM, for their inferences without FSAM
Training
Testing
Training
MAB-IRRI
(10)
(IR)
(IR)
(NVP)
Paloset
Dalsed
Bise
Base
137 1.02
7.9
2.5 2.07
0.8 0.04
1374 088
0.77
Base PSAM
0.71 0.39
305 4 1.02
1.20 = 0.76
0.9 = 0.0
088
0.77 000
0.99
081
Bise + FSAM Res
Bise: FSAM Ros
0.48 s 0.17
0.72
1 IUI
14.16 e RAA
0.7% 6.12
UBFC- JPPG
Besa
0.48 = 0.87
0.39
0.72 0.28
1 RMI
14.16 0.8
0.78 0.02
1,99
Bose FSAM
Baso PSAM
1.90
0,99
2,66 0,50
0.11 0004
5,82
0.5 000
:BVP
Rivos
1.88 = 0:33
3,89 0.78
2.59 - 0.49
0.91 RJ94
5.50 0.57
0.54 0.01
=
4.58 1.06
0.90 0.04
4.50
2.42
0.90 0.04
0.56
Retention performance gains despite FSAM being skipped during inference, for FactorizePhys trained with FSAM offers insight into the mechanics of how FSAM functions. This can be interpreted as follows: Optimization of a network having FSAM implemented as an attention mechanism influences the network to increase the saliency of the most relevant features, so that factorized approximation of embeddings retains these features, while discarding the less important features. Due to the increased saliency of relevant features and the presence of residual connection, FSAM can be skipped during inference, significantly reducing computational overhead.
Table 4: Performance Evaluation of Models on PURE Dataset [55] Trained with UBFC-rPPG Dataset 2 using Different Ranks and Optimization Steps for Factorization
Optimization Steps
Rank
MAE (IR)
RMSE (II))
MAPE (HR))
Corr (HI)
SNR ( dB, BVP)
MACC (BVP)
for Matrix Factorization
Mean
SE
Mean
SE
Mean
Mean
SE
Meun
SE
Meun
SR
Base
1.37
1.02
7.97
2.84
2.55
2.07
0.94
0.04
13.74
0.81
0.77
0.02
0.48
0.17
0.35
0.72
0.28
1.00
0.01
14.16
0.83
0.78
0.02
2
1.40
1.02
7.98
2.84
2.59
2.07
0.94
0.04
13.71
0.81
0.77
0.02
2.25
3.09
4.36
0.05
0.82
0.02
8
1.44
1.02
7.98
2.84
2.64
2.07
0.94
0.04
13.70
0.83
0.77
0.02
2.20
1.30
10.22
3.09
4.26
0.91
0.05
13:55
0.82
0.77
0.02
0.80
0.39
3.11
1.03
1.33
0.77
0.99
0.02
13.60
0.81
0.77
0.02
1.31
0.84
2.30
2.45
0.96
0.04
6
4
1,53
0.90
7.10
2.32
2.91
1.86
0.96
0.04
13.54
0.82
0.77
0.02
2.22
3.09
4.29
0.05
16
1.43
1.02
7.98
2.84
2.65
2.07
0.94
0.04
13.62
0.81
0.77
0.02
0.73
0.39
1.24
0.99
0.02
13,67
0.81
0.77
0.02
2
1.44
1.02
7.98
2.84
2.64
2.07
0.94
0.04
13.35
0.82
0.77
0.02
0.78
1.30
0.99
0.80
0.02
8
0.73
0.39
3.06
1.02
1.24
0.77
0.99
0.02
13.50
0.82
0.77
0.02
0.73
0.39
3.06
1.02
1.34
0.99
0.02
13.55
0.83
0.77
0.02
17

---

In table 4 we present results to compare the performance obtained for different ranks L, as well as the optimization steps used solve factorization. For all experiments, FactorizePhys is trained with the UBFC-rPPG dataset [2] and the performance is presented for the PURE dataset [55] We can observe that the best performance was achieved for rank 7. for the different steps used solve the factorization. For higher ranks, performance remains on par with that of the network without the FSAM, indicating that for the rPPG estimation task, the rank-1 factorization offers the optimal spatial-temporal attention. These results align with the expected single source of the underlying BVP signals in different facial regions.
A.4 Statistical Significance of the Main Results
We performed repeated experiments with 10 different random seed values between 1 and 1000 to compare the proposed FactorizePhys trained with FSAM with the best performing SOTA rPPG method. For the cross-dataset generalization results reported in table [ EfficientPhyys with SASN 37 was found to perform the best among the existing SOTA methods.
Table 5: Performance Evaluation of Models on PURE Dataset, Trained with UBFC-IPPG Dataset, using Different Random Seed Values
Model
Random Seed Value
(HR)
RMSE (HR)
MAPE (HR).
(HR) 1
SNR RVP)
MACC
SE
Mean
Mean
3,75
1.62
12.97
3.53
5.60
2.50
0.84
0.07
8.60
38
1.74
14.09
3.59
7.18
0.81
0.08
8.69
1.79
3.65
7.50
2.97
8.69
1.79
3.65
128
3.65
7.68
Phlicien:Pyss
138
3.65
SASN Mediule
3.65
308
3.65
319
3.65
7.55
8.96
900
3.65
7.48
Average
1.77
14.31
3.63
7.27
0.81
1100
10
7.64
2.71
2.52
0.04
38
4.31
1.86
3.79
12.52
55
1.30
10.22
3.09
4.22
0.91
11.05
13.21
100
0.48
0.17
1.39
0.35
0.72
0.28
1.00
14.16
0.78
0.02
128
0.39
3.08
1.03
0100
13.23
Proposed PratocizePPyys
138
0.52
0.19
1.56
0.40
0.72
0.27
13.08
with ESAM Attencinn Module
212
2.15
1.22
9.63
2.88
4.19
2.50
0.92
0.05
13.58
0.77
0.02
308
1.50
0.98
7,70
2.71
2.79
1.99
0.95
0.04
13.39
0.82
0.77
0.02
319
1.38
0.84
6.61
2.30
2,60
1.73
0.96
0.04
13.54
0.81
0.77
0.02
900
3.34
1.70
13.46
3.69
5.21
2.78
0.83
0.00
12.76
0.83
0.76
0.02
Average
1.80
0.96
7.62
2.30
3.14
1.81
0.93
1104
13:33
01.2
0.76
0.002
Pairol Test
0.0001
0,0014
0.0011
0.0004
0.000
0.0000
For each random seed value, we trained the proposed FactorizePhys with FSAM and EfficientPhys with SASN (37] on the UBFC-rPPG [2] dataset and evaluated them on the PURE dataset (55). Paired T tests for each reported evaluation metries suggest that the performance gains achieved with the
proposed method are statistically significant compared against the best performing SOTA rPPG method, highlighting its effectiveness and thereby highlighting contributions of this work in the research field of end-to-end rPPG estimation from video frames.
A.5 Within Dataset Performance
In this work, we primarily focus on comparing rPPG methods for their cross-dataset generalization, which offers more critical evaluation and reliable estimates of how models perform on unseen or
18

---

cit-of-dittibuttion data. Within -ataset performance signifies an representation ability of model fit the data, derived from the same distribution, serving an essential criteria. Therefore, for
completeness, in table 6 we report within-dataset evaluation on iBVP (29 55 and UBFC-rPPG
datasets, where we observe at-par performance of FactorizePhys as compared with the SOTA rPPG methods.
Table 6: Within Dataset Performance Evaluation
Attention
Model
MAE (HI)
RMSE (HR)
MAPE (HR))
Corr (HR)
SNR (BB, BYP)
MACC (BVP)
Module
Performance Evaluation on iBV Dataset, Subject-wise Split: Training (0.0 0.7)), Test (0.7 1.0)
PhysNet
0.29
2,10 0.51
1.64 0.42
0.03
10.63 1.05
0.68 0.82
PhysFormer
TD-MHSA"
1.96 a 0.63
4.22 a: 1.47
2.49 0.72
0.91 = 0.07
10.72 1.04
0.66 2 0.03
EfficientPhys
SASN
2.74 + 0.96
6.28 + 2,14
3.56 + 1.13
0.81 = 0.10
7.01 1.03
0.58 + 0.03
EflicienPPhy
FSAM (Ours)
2.34
0.46
0.99
0.02
PactorizePbys (Ours)
FSAM (Ours)
1.13 1 0.36
2.42 # 0.77
1.52 = 0.50
0.97 = 0.04
9.73 1.05
0.65 = 0.02
Performance Evaluation on PURE Dataset, Subject-wise Split: Training (0.0 0.7), Test (0.7 1.0)
PhysNet
1,28
PhysFommer
TD-MHSA"
D.68 = 0.26
1.31 + 0.46
1.08 = 0.43
1.00 = 0.02
19.05 + 1.07
0.87 = 0.01
EflicienPPly
SASN
0.49 0,26
1,21 x 0.46
0.73 0.42
1,00 0.02
15,25 +: 1,20
0.80 0.02
EfliciemPPhss
FSAM (Ours)
1.28
0.92 0.44
15.42 1.25
0.80 0.02
FactorizePbys (Ours)
FSAM (Ours)
0.49 =: 0.26
1.21 + 0.46
0.73 = 0.42
1.00 = 0.02
19.63 4: 1.40
0.86 = 0.01
Performance Evaluation on L/BRC-PPG Daraser, Subject-wise Splir: Training (0.0 Q.7)). Test (0.7-
PhysNet
3/08
0.90
PhysFommer
TD-MHSA"
1.76 = 0.79
3.36 + 1,30
1.60 = 0.74
0.96 = 0.08
6.10 1,86
0.90 0.01
EfficienPhyy
SASN
2.30 1,40
5.54 + 2,53
2.28 1.44
0.90 = 0.13
6.75 1.76
0.87 0.01
EfficienPPhys
FSAM (Ours)
5.88
2.79
0.87 0.01
PactorizePhys (Ours)
FSAM (Ours)
2.84 1.42
5.87 + 2,52
2.73 + 1.46
0.88 0.14
6.33 2.000
0.91 = 0.01
TD-MHSA* Temporal Difference Multi-Head Self-Atcatiin bin
Scli-Antction Shifled Network
FSAM: Proposed Factorized Module
A.6 Scalability Assessment of FSAM
We further investigate FSAM for its scalability to higher spatial-temporal resolution. For this, we perform within-dataset evaluation on the UBFC-IPPG dataset (2), which is pre-processed with the regular input dimension of 160 72 72 as well as with a higher spatial and temporal dimension of 240 x 128 x 128. Repeatable experiments are conducted with 10 different random seeds between and 1000 to compare the performance of FactorizePhys with FSAM for each apatial-temporte input dimension.
Comparable performance, observed in 7 for both spatial- -tmmoral input dimensions, suggests that FSAM can easily deployed for different spatial-temporal scales. should also be noted that the higher spatial dimension of video frames (i.e., 128 x 128) does not produce improved performance, indicating that the spatial dimension of 72 72 sufficient to extract rPPG signals with end-to-end methods.
19

---

Table 7: Scalability Assessment of FSAM for Higher Spatial and Temporal Dimensions
Input
Random
MAE (HR)
RMSE (HR) 1
MAPE (HR)L
Corr (HR)
SNR dB, BVP)
MACC (BVP)
Dimension
Seed Value
Mean
Mean
SE
Mean
SE
Mean
Mean
Mean
10
5.87
2.52
2.73
1.45
0.88
0.14
6.49
2.03
0.01
38
2.84
1.43
5.87
2.52
2,73
1.45
0.88
0.14
6.68
2.00
0.91
0.01
55
2.97
1.41
5.89
2.52
2.84
1.44
0.88
0.14
6.52
1.98
0.91
0.01
2.73
2.01
0.01
128
2.97
1.41
5.89
2.52
2.84
1.44
0.88
0.14
6.42
1,99
0.91
0.01
160x72x72
138
2.84
1.43
5.87
2.52
2.73
1:45
0.88
0.14
6.48
1.96
0.91
0.01
212
2.91
1.42
5.88
2.52
2.79
1:45
0.88
0.14
6.40
1.99
0.91
0.01
308
2.84
1,43
5.87
2.52
2.73
1.45
0.88
0.14
6.51
1.98
0.91
0.01
319
2.91
1.42
5.88
2.52
2.79
1.45
0.88
0.14
6.44
2.03
0.91
0.01
900
2.91
1.42
5.88
2.52
2.79
1.45
0.88
0.14
6.55
2.01
0.91
0.01
2.77
0.01
10
3.04
1.92
7.56
3.64
3.22
2,18
0.83
0.17
6.68
1.93
0.90
0.01
38
2.91
1.93
7.54
3.64
3.10
2.19
0.84
0.16
6.86
1.93
0.90
0.01
55
2.97
1.92
7.54
3.64
3.16
2.18
0.84
0.16
6.63
1.11
0.91
0.01
100
2.91
1.93
7.54
3.64
3.10
2.19
0.84
0.16
6.87
1.91
0.90
0.01
128
3.11
1.91
7.56
3.64
3.28
2.17
0.84
0.17
6.71
1.87
0.90
0.01
120x128x128
138
2.91
1.93
7.54
3.64
3.10
2.19
0.84
0.16
6.63
1.95
0.91
0.01
0.01
308
2.91
1.93
7.54
3.64
3,10
2.19
0.84
0.16
6.71
1.92
0.90
0.01
319
3.04
1.92
7.56
3.64
3.22
2.18
0.83
0.17
6.83
1.93
0.91
0.01
3.64
3.22
0.01
Mcon
2.99
7.55
3.64
3.17
2.18
0.84
0.17
6.74
1.92
0.01
A.7
Multimodal rPPG Extraction
As iBVP dataset offers synchronized RGB and thermal infrared video frames, we conducted brief experiment using FactorizePhys with FSAM investigate whether combining both modalities can result in performance gains for the estimation of rPPG. For this, we also individually trained FactorizePhys on RGB and thermal frames keeping the identical data split of 70%-30%. Results
Table 8: Performance Evaluation on iBVP Dataset, Subject- Split: Train (70%), Test (30%)
Modality
MAE (II))
RMSE (II) 1
MAPE (UR)
Corr (HI) 1
SNR (dB, BVP) T
MACC (BVP)
Input Frames
Mean
SE
Mean
SE
Mean
SE
Mean
SE
Mean
SE
Mean
SE
T
6.40
0.97
8.58
2.11
8.66
1.26
0.84
0.09
-3.27
0.40
0.20
0.01
RGB
1.13
0.36
0.77
0.50
0.97
0.04
9.75
0.65
0.02
RGBT
1.10
0.36
2.42
0.77
1.49
0.50
0.97
0.04
9.65
1.04
0.64
0.02
in table 8 suggest weaker presence of rPPG signal in thermal infrared frames, leading poorer performance when FactorizePhys is trained only on thermal frames, while not showing significant performance gains when jointly trained with RGB and thermal frames.
A.8 Visual Overview of Cross-Datasel Generalization and Latency
Figure 5 offers a quick visual summary of the cross-dataset generalization performance on different evaluation metrics, their respective standard error, and latency for the proposed and existing SOTA
20

---

methods. The performance reported on Y-axis of each plot is cumulative cross cross-dataset performance
Factor cottrsa FSAM)
PactoriosPtyy (nference FBAM)
4-d0is 1. LG BPPK 9.8
Fastorio (SSAM)
FactortosPhye (TSAM) 1.76 0.52 SPH 188 mo
Camuuattvve Cross-Daaett
MAE MfE Extimntet (BP(H) (BPM)
Curmulativv Crose- Datsaet of Estimated
Emiclentiiee (FSSA])
EriientPtyys (SASN) 5.44 1 BPPM) 1,5ms
Efficersityye (SAGI)) 40 2.81 BPM//1 1-3
Model
Model
MAE SE/ Latency
RMBE BEI Lubency
Sphere Size MAME Standard Enree
Phyehhen
Sphore Size AMSI Standard Erro
17.40 2.14 BIPR/ mu4
PlysForncr
11.50 BPM 22.9
18.00 BPM//88.ms
Latency (me)
Latency (ms)
Factorbredhye (nneernce wis (SSAM)
2.70 0 BPW mg
PuctorizoPtyyo Inforonco FBAM)
(PGAM)
C.D4/
Pactortosltyya PASAPO 0.43
MAPE of Estimated HR (BPM)
Cumuletive Cross-Deraeet
EffisensPPts (PBAi)) 6.00 a DPPa s.d
Cumulative Cross- Cross- Dafaest Extimated
EfficlenFhyee (SAGN)
(FSAN)
8.75 2:00 BiHH 1.3
00.00
Effilenn
Model
0.71
0,74
0.08 11.3
Model
4APE SE/ Lonenay
Corr SEI Latency
Sphere Sizee MAPE Standard Error
0.22
Sphere Size Car Standard
Pryaahis
Phyohett
10.35 2.62 6PH 114
0.20
0.00 0.03
PhyoFormer 16 de 2 9m.
0.95
0,08 1.......10/.255.
Latency (ms)
Latency (ms)
10.5
Pact Iideeencs wio FSAM]
6:75
EaazoriosPhpe (iteeencs wits FSAM))
PactoriaufPyyu
01/22
0.70
0.01
m1.3
Factortuulage (FSAA))
0.70 0..01 /..01/.8mmm
Cumulative Cross- etaaset SNR (dB) (dB) of Estimeted BVP
Cumulathe Crosse Detadnt
MACC of Extimatedd
0.00
Efficiersityyy (SASN)
ElicientPPtp (PSAM)
6.68 0.83
Model Latency
Model MACC SE/ Latency
Sphare Size Standard Error
Sphers MACO Ener
4.40 8.83 dB/ 1,4mms
Physhet
0.57 0.027 1.4
Phyyl fonmer 3.40 1.16 /28.8ms
0.25
Phyoframmer 0,50 - /26
Latency (ms)
Latency (me)
Figure 5: Cross-dataset performance comparison between SOTA and the proposed method reported with cumulative evaluation metrics, their standard error (SE) and latency
for respective models, averaged over different training and testing datasets. The proposed method outperforms existing state-of-the-art methods in all evaluation metries by a significant margin, while achieving at-par latency.
A.9
Computational Cost and Latency
Table compares model parameters, latency on GPU and CPU, and model size of the proposed FactorizePhys with FSAM with that of the existing SOTA rPPG methods. Considering the identical inference time performance of the base FactorizePhys. when trained using the proposed FSAM. the proposed method uses an order of magnitude fewer parameters and achieves a par latency on both CPU and GPU systems. Relatively higher latency compared to the EfficientPhys (37 model, despite the fewer model parameters, is due to the difference in the number of floating point operations (FLOPS). FactorizePhys, being a 3D-CNN architecture, requires more FLOPS compute 3D features at each layer compared the fewer FLOPS for EfficientPhys [37] which implements the 2D-CNI
21

---

architecture. It should be noted that the FLOPS are also dependent on the input dimension. which kept consistent for all the models. For resource critical deployment, FLOPS can be significantly reduced by decreasing the spatial dimension of input from 72 x 72 to 8 x 8 as found optimal for RTyPPG or to as used in the small branch the Bigsmall model for rPPG estimation.
Table 9: Comparison of FactorizePhys based on Model Parameters, Latency and Model Size
Model
Inference Time
Inference Time
Model
Model
Parameters
on CPU (ms)f
GPU (ms)
Size (MB)
PhysFormer
7380871
450.47
26.86
29.80
PhysNet
768583
272.89
1.36
3.10
EfficientPhys with SASN
2163081
371.08
1.31
8.70
EfficientPhys with FSAM (ours)
140655
82.19
0.57
PactorizePhys Base (ours)
51840
96.80
1.94
0.22
FactorizePhys with FSAM (ours)
52168
95.75
8.97
0.22
+CPU Spees: IntelØ COreTM 17-10870H CPU 2.20GHz
GB RAM.
4GPU Specs: NVIDIA GeForce RTX 3070 Laptop GPU (CUDA cores 5120).
A.10
Visualization of Learned Attention
In fig. we present additional samples of learned spatial-temporal features. For FactorizePhys trained with FSAM, we can observe superior cosine similarity and more relevant spatial distribution specifically under challenging scenarios with occlusions such as arising from hairs, eye-glasses and beard.
A.11 Qualitative Comparison with Estimated rPPG Signals
Qualitative comparison the estimated rPPG signals between the proposed method and the best performing SOTA method (i.e., EfficientPhys [37 is presented for different test datasets iBVP (2)
(fig. 7
PURE [55 (fig. 8). and UBFC-rPPG 2 (fig. 9).
A.12 Safeguards
We intend to release our rPPG estimation code only for academic purposes, with Responsible Al license (RAIL) Research areas that will benefit directly from this work include human- computr interaction and contactless health tracking vital signs monitoring. Although the methods presented in this work may potentially benefit certain clinical scenarios, thorough validation studies, with appropriate ethics approval, are required critically assess performance in such settings.
In addition, in some recent work, rPPG methods have been indicated effective in detecting deep- fake videos. In this context, we would like to caution such use, considering the main results presented for the models trained using the SCAMPS [42] dataset, consisting of synthesized avatars. We argue that the rPPG signal can be embedded in the synthesized (or deep-fake) videos, with similar approach as used for generating the SCAMPS [42] dataset. In such scenarios, in spite of high accuracy in estimating rPPG signals, such methods can be fooled by the synthesized videos that embed BVP signals. Therefore, we highlight that necessary to use the rPPG signal estimation methods in this context with great caution.
22

---

Without FSAM
With FSAM
Without FSAM
With FSAM
Cosine Similarity Score
Figure 6: Visualization of Learned Opatial-Temporal Features
23

---

aetfinss ficans
28_5
55
62_12
86_5
14c 13
Citene1 liber CRDTTLON CoSSnLen Sitenn Letrie? lebrass Fitrisy
19d 25
20a_
27c 23
28d_4
Figure 7: Comparison of Estimated rPPG Signals on iBVP Dataset for Models Trained with PURE, SCAMPS and UBFC-rPPG Datasets
24

---

105.3
202
304_6
404_3
153 3
603_7
7
805
0
1001 2
Figure 8: Comparison of Estimated rPPG Signals PURE Dataset for Models Trained with iBVP. SCAMPS and UBFC-rPPG Datasets
25

---

133003 1100
23.3
REdrET litfe
litcise
Figure 9: Comparison of Estimated rPPG Signals on UBFC-IPPG Dataset for Models Trained with iB VP, PURE and SCAMPS Datasets
26

---

B NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make thorough evaluation of the claims and results support the claims, which are appropriately highlighted in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion section, we have dedicated a paragraph to mention the broader impact and discuss the limitations and future directions that can be investigated.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [NA]
Justification: This work does not propose a new theory, as it focuses on adapting estab- lished foundational algorithm for low-rank recovery using nonnegative matrix factorization into deep learning architecture to jointly compute spatial-temporal attention. We present empirical results of thoroughly conducted experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In sections appendix A. and appendix we provide a detailed description necessary to reproduce the main experimental results. In addition, we make our code available attps//gghhoooomm//yyyoggggollll
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
28

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Although this paper does not present new data, we provide access code accompanied by comprehensive usage instructions. Our code extends the TPoGoox [38], a widely embraced tool by the rPPG research community, and includes exhaustive guidelines to allow replication of our main results. Furthermore, we urge researchers who encounter any difficulties in reproducing the results to report them through issue tracker on our repository
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-sourg benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/pips.cccpubli//guiii..
ssiniioisooyycc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Through $ and appendix A.2 we specify data-splits as well hy- perparameters. We provide results from the ablation study in $ and appendix A..3 for key hyperparameters that in this work. In addition, the code released at includes config files used to train and
test all models.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
29

---

Answer: [Yes]
Justification: We conduct a rigorous assessment of the proposed models by means of an fair comparison with SOTA rPPG methods, and report standard error for all evaluation metrics. This examination is further supplemented with statistical significance analysis of the principal results, detailed in table 5 appendix Our evaluation encompasses the most effective SOTA rPPG method and includes a paired T-test result derived from 10 experiments that use distinct random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/://oourrpp...c///////aaas
Answer: [Yes]
Justification: Our work is in accordance with the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
30

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Although briefty highlight the potential positive impacts in 6 to prevent any negative societal impact, we have safeguarded our code with Responsible Al License specifying the appropriate restrictions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have mentioned safeguard measures in appendix A.12
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
31

---

Answer: [Yes]
Justification:
We cite all the code, data and models used in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have prepared ReadMe for the code repository, as well as we have added code comments where applicable to provide the needed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does involve direct collection of data from human subjects, uses existing datasets, which have been cited.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
32

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Although this work did not require working with human subjects, have institutional ethics covered for this research, which is approved by the Ethics Committee of the University College London Interaction Center (ID Number: UCLIC/1920000//taf//CC Approval date: 20 May 2020).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "CA9HR9dywV", "reviewer": "Reviewer_a4X8", "summary": "The paper proposed the Factorized Self-Attention Module (FSAM), which jointly computes multi-dimensional attention across spatial, temporal, and channel dimensions using non-negative matrix factorization (NMF). The FSAM is integrated into a new end-to-end 3D-CNN architecture called FactorizedPhys, designed to estimate blood volume pulse signals from video frames. Extensive experiments on multiple datasets demonstrate FSAM's effectiveness.", "questions": "- Are the methods sensitive to the hyperparameters? Lack of sensitivity analysis.\n\n-The authors could consider visualizing more about the outputs of the models.\n\n-Lack of scalability analysis of the methods, especially when the NMF is adopted. For large datasets, it is valuable to know whether the method is scalable compared to other methods.", "limitations": "The limitations are mentioned in the paper but could be highlighted more clearly in the discussion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The overall writing of the paper is clear. The proposed method leverages the strengths of matrix factorization to capture global spatial-temporal context effectively. Specifically:\n\n- Unlike most current methods that compute attention disjointly across different dimensions, the proposed method jointly computes multi-dimensional attention using non-negative matrix factorization to gain the modeling advantage.\n\n-Ablation studies are conducted to assess the different mappings of voxel embeddings in the factorization matrix.\n\n- Model Complexity is analyzed experimentally.", "weaknesses": "-Lack of visualization to show how the results of the proposed method are superior (say, an image of the output to show the learned attention).\n\n-Lack of variance of the results in the experiments table.\n\n-It is still unclear why and in what situation the proposed method is better than the SOTA methods. Say, pick some cases from the model output that show that FactorizePhys is better than the others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for acknowledging the strengths of our contributions, providing constructive feedback and suggestions. We will revise manuscript to reflect responses to your comments that we describe here.\n\nFor your comments on visualization, variance and scalability, we request you to refer our global response. We address rest of the comments here:\n\n* Describing the cases when the proposed method is better than the SOTA methods:\nWe will revise the discussion section to describe the key findings as follows: \n\nFor all reported evaluation metrics, the proposed method outperforms SOTA methods for its cross-dataset evaluation (revised Table-1 in PDF) on PURE and iBVP datasets, for all respective training datasets. This suggest consistent and superior generalization achieved by the proposed method. Cross-dataset evaluation on UBFC-rPPG dataset further highlights the performance gains of the proposed FactorizePhys model when it is trained using iBVP and SCAMPS datasets, and at-par performance with SOTA models when trained using PURE dataset. FactorizePhys uniquely outperforms the SOTA methods on all testing datasets when SCAMPS dataset is used for training, further stressing the superior generalization achieved from synthesized dataset with the proposed method. Lastly, the proposed method consistently offers superior SNR and MACC for the estimated rPPG signals, which highlights enhanced reliability. We further report intra-dataset evaluation (Table-7 in PDF), where performance of FactorizePhys is at-par with the SOTA methods. \n\n* Are the methods sensitive to the hyperparameters? Lack of sensitivity analysis\n\nWhile we follow fair training-testing strategy as described below and report statistical significance of results in the supplementary data, we specifically evaluated sensitivity for factorization rank – which is the most important hyper-parameter for the proposed FSAM module. The results shown in the appendix Table 4 of the submitted manuscript indicate low sensitivity for a set of low-ranks, while the performance drops with higher rank, which is an expected behaviour. High-rank factorization results in low matrix approximation error which does not remove redundancy in the embeddings, and thereby not serving as an effective attention mechanism. Factorization rank is therefore an important hyperparameter that is required to be adjusted based on the architecture, placement of the module within the architecture and the downstream task. For rPPG signal extraction downstream task, we expect only one underlying signal source and therefore rank-1 factorization of temporal vectors offers optimal performance.\n\nFair training-testing strategy: All model-specific hyper-parameters were maintained as provided by the respective SOTA methods, while the training pipeline related hyper-parameters were kept consistent for training all the models. Training pipe-line related hyper-parameters that we maintained consistent include - pre-processing steps for images and labels, batch-size, number of epochs, learning rate, scheduler and optimizer. However, we noticed that the number of epochs that we initially kept as 30, resulted in extremely low training-loss for all SOTA methods, affecting their cross-dataset generalization. The proposed method did not show such an overfitting problem and was not found senstive to the number of training epochs after convergence. This offered advantage to the proposed method. For fair evaluation against SOTA methods, we revised the training, validation and testing strategy, in which instead of using the validation set based best epoch from 30 epochs training, we re-trained all models for 10 epochs, similar to a recent work [1], and used the last epoch for cross-dataset as well as within-dataset evaluation. Table 6 in PDF highlights that the updated strategy offers fair evaluation against the SOTA methods. Revised Table 1 and new appendix Table 7 provide detailed results for cross-dataset and regular (within-dataset) evaluation respectively, with statistical variance mentioned for all the evaluation metrics.\n\n\n* The authors could consider visualizing more about the outputs of the models.\n\nWe appreciate this feedback and we will add a figure in the revised manuscript to show the learned attention map as well as output of the models. Here, in Figure 5 of the PDF, we have added a sample plot that compares the output rPPG signals of the proposed FactorizePhys (in orange) and EfficientPhys (best performing SOTA model, in blue) with the ground-truth BVP signal (GT, in black). We have provided large sample set of such comparisons in the supplementary data shared with the ACs. To fit more plots in the supplementary data, we used higher JPEG compression for the figures, due to which images may appear noisy, for which we apologize. \n\n\n* The limitations are mentioned in the paper but could be highlighted more clearly in the discussion section.\n\nWe appreciate reviewer for this comment. Following are few limitations which we would like to clearly highlight in the discussion section:\n\n(a) While the proposed FSAM module has shown to be effective spatial-temporal attention for rPPG signal extraction task, its efficacy as an attention mechanism for other spatial-temporal tasks such as video understating, video object tracking, and video segmentation, needs to be further investigated.\n\n(b) For signal extraction tasks, different forms of constrained NMF were not investigated. Specifically the variants of NMF which put temporal or frequency constraints on the time-series vectors may offer more effective attention, since such constraints can be chosen based on the characteristics of the ground-truth signal to be estimated.\n\n\n\n\n1. C. Zhao, H. Wang, H. Chen, W. Shi and Y. Feng, \"JAMSNet: A Remote Pulse Extraction Network Based on Joint Attention and Multi-Scale Fusion,\" 2023, doi: 10.1109/TCSVT.2022.3227348.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9RdNkS1kS1", "reviewer": "Reviewer_c6jF", "summary": "The paper presents a novel attention block FSAM devised for handling spatio-temporal data. It is benchmarked against a range of SOTA architectures on a suitable selection of different datasets, and found to perform strongly.", "questions": "Is FSAM anticipated to have significantly wider impact on spatio-temporal applications, beyond rPPG?  \n\nHow do the different candidate methods compare in their scalability to higher temporal and spatial resolutions?", "limitations": "There is a discussion on potential societal impact in the Appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well presented and clearly structured\n\nThe is a good degree of novelty in the proposed architecture, and the application is of high interest.\n\nThe experiments are conducted over a well selected range of real world datasets.\n\nA good level of detail is provided on the methodology, and the performance of the proposed approach appears strong.", "weaknesses": "While the paper's experimental results are promising, they do not appear to be accompanied by any uncertainty estimates. Including these is crucial to allow the reader to draw meaningful conclusions from the results.  I would refer the authors to question 6 in the checklist at the end of the paper. The answer given to the question 'does the paper report [...] statistical significance of the experiments' is given as \"N/A\", while the guidelines advise that this answer indicates that the paper does not include any experiments.\n\nAlso, while it is acceptable to focus on the intra-dataset performance within the main text, it is crucial to at least include the regular results in the Appendix.\n\nA couple of minor typos:\nOn line 196:\n\"device a\" -> \"devise a\"\nOn 236:\n\"It's\" -> \"Its\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for acknowledging the novelty of our contributions and significance of the reported results. We highly value your suggestions based upon which we will further revise manuscript.\n\nFor your comments on statistical significance, uncertainty estimates and scalability, we request you to refer our global response. We address rest of the comments here:\n\n\n* Regular results in the Appendix:\n\nWe would like to clarify that our main results are for cross-dataset evaluation, which offers better insights into the real-world performance for unseen data distribution. If we understand your suggestion correctly, it relates to reporting within-dataset performance. Based on this interpretation, we present intra-dataset/ within-dataset results in Table-7 in the PDF attached to the global response.\n\n\n* A couple of minor typos: On line 196: \"device a\" -> \"devise a\" On 236: \"It's\" -> \"Its\"\n\nThank you for spotting the typos. We will address these and we will thoroughly proof-read the manuscript for its revision.\n\n\n* Is FSAM anticipated to have significantly wider impact on spatio-temporal applications, beyond rPPG?\n\nAmong the existing works, NMF as implemented in the Hamburger module [1] demonstrated its effectiveness in capturing global context, specifically for the semantic segmentation and the conditional image generation tasks. While Hamburger module [1] outperformed transformer based network, authors did not implement their method as an attention mechanism, and the embeddings were limited to spatial and channel dimensions. \n\nFSAM builds upon it and investigates NMF based factorization of embeddings as multi-dimensional attention with additional temporal dimension. \n\nIn principle, the factorization of embeddings results in a low-rank approximation of embeddings. Optimization of a network having factorization module implemented as an attention mechanism, influences the network to increase saliency of the most relevant features, such that factorized approximation of embeddings retain these features, while discarding the less salient features. \n\nAs factorization can handle multiple dimensions simultaneously with appropriate transformation or mapping, it underlines its unique potential as muti-dimensional attention mechanism, unlike existing candidates. \n\nEnd-to-end estimation of rPPG signal from facial video frames represents one of the challenging spatial-temporal tasks as it requires network to learn to pick the spatial features having the desired temporal signature, while discarding the variance related to head-motion, illumination and skin-tones. \n\nWhile this work is limited to the evaluation of FSAM for a spatial-temporal task of estimating rPPG signal, based on the consistent performance gains across the datasets, and versatility of the module in 3D-CNN and 2D-CNN models, we envisage FSAM to have wider impact on spatio-temporal applications such as video segmentation, video object tracking, and action recognition, which we consider as future extension of this work.  \n\n\n1. Z. Geng, M.-H. Guo, H. Chen, X. Li, K. Wei, and Z. Lin, ‘Is Attention Better Than Matrix Decomposition?’, in International Conference on Learning Representations, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atQdXStcOC", "reviewer": "Reviewer_ECso", "summary": "The paper introduces FactorizePhys that utilizes Non-negative Matrix Factorization (NMF) to decompose voxel embeddings. By integrating FSAM into both 3D-CNN and 2D-CNN architectures, FactorizePhys can estimate blood volume pulse signals from video frames. Through evaluations and comparisons with state-of-the-art rPPG methods, the effectiveness of FSAM and FactorizePhys are demonstrated.", "questions": "Please check the weakness part.", "limitations": "The authors mentioned the limitations of the work and proposed future directions such as time series estimation.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Introducing the Factorized Self-Attention Module (FSAM) for computing multi-dimensional attention from voxel embeddings. \n3. Evaluation of FSAM and FactorizedPhys against state-of-the-art rPPG methods\n4. Integration of FSAM into existing 2D-CNN-based and 3D-CNN-based rPPG architectures to demonstrate its versatility.", "weaknesses": "1. The motivation to use the non-negative matrix factorization is unclear. Why can the factorized matrix be used as the attention? Authors should give more insights about the non-negative matrix factorization and show the attention maps to demonstrate the effectiveness.\n2. There are other matrix factorization methods such as SVD and QR. Why is the non-negative matrix factorization chosen?\n3. The factorization needs to solve an optimization problem which should do gradient descent steps. In line 216, the Authors only use a one-step gradient for matrix factorization. Is the one-step gradient enough to achieve satisfactory factorization?\n4. The method part is not clearly illustrated. e.g., equation 5 is confusing, and the symbols in the equation are not well explained. The symbol mentioned in the main text is not well shown in the main figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for valuable comments and insightful questions. We will revise manuscript to reflect our responses.\n\nFor your comments on visualization, we request you to refer our global response. We address rest of the comments here:\n\n\n* W1: The motivation to use the non-negative matrix factorization (NMF) as the attention:\n\nFactorization or matrix decomposition results in a low-rank approximation, and among such methods, NMF is highly researched area [1-4]. From prior works, Hamburger module [7] that implemented NMF, demonstrated its effectiveness in capturing global context, specifically for semantic segmentation and conditional image generation tasks. Hamburger module [7] outperformed transformer based network, though it was not implemented as attention mechanism, and the embeddings were limited to spatial and channel dimensions. Further, there have been efforts in reducing the redundancy in feature embeddings [6]. \n\nThese prior works inspire us to investigate if low-rank approximation of embeddings (intuitively - a type of squeeze operation [5],  that squeezes information without reducing dimension of the embeddings), when multiplied with the original embeddings (similar to excitation), forms an effective attention. The proposed FSAM module builds upon Hamburger module [7]  and extends NMF based factorization for multi-dimensional embeddings having spatial, temporal and channel dimensions. Optimization of a network having factorization module implemented as an attention mechanism, can influence the network to increase saliency of the relevant features, such that factorized approximation of embeddings retain these features, while discarding the less salient features. As factorization can handle multiple dimensions simultaneously with appropriate transformation or mapping, it underlines its unique potential as muti-dimensional attention mechanism, unlike existing candidates that reduces one or more dimensions of embeddings to compute attention.\n\n* W2: NMF v/s SVD, QR\n\nThe rationale for choosing NMF compared to other decomposition techniques is as follows: i) The only constraint posed by NMF on the matrix, its vectors and the features is non-negativity, while SVD, QR and Vector Quantization (VQ) assume statistical independence or orthogonality between vectors of the approximated matrix. For deep-layer embeddings, constraints of orthogonality or statistical independence may not be relevant, and therefore such decomposition methods are not well-suited. ii) Owing to non-negativity or purely additive constraints, NMF effectively learns parts-based representation, which further enhances interpretability of the learned features [3].\n\n* W3: Gradient steps\n\nWe fully agree with the reviewer that factorization needs to solve optimization problem. However, we would like to clarify in the manuscript that one-step gradient relates to a gradient step for each iteration that solves factorization with multiplicative update in one gradient step per iteration, as proposed in the Hamburger module [7].  Empirically we find 4 to 6 iterations to be sufficient to obtain the desired level of approximation. It is to be noted that our objective is not to achieve perfect fit with sub-zero approximation error. On the contrary, factorization serves as a better attention mechanism when it optimally approximates only salient features and discards less relevant features. \n\n* W4: Methods section, main figure and equation-5\n\nWe will revise the methods section to sufficiently clarify the use of the symbols in the equations, along with clearer depiction of the symbols in the main figure. In regards to the equation 5, it mentions the negative Pearson correlation as an objective function for the downstream task of rPPG signal estimation task. We notice the typo in the description, and further understand the use of “i” and “T” in equation 5 can be clarified for better readability. To clarify here, “T” refers to total number of samples  of the estimated and ground-truth signal, which is also equal to the total number of input video frames. \n\n\n\n1.\tY. -X. Wang and Y. -J. Zhang, \"Nonnegative Matrix Factorization: A Comprehensive Review,\" doi: 10.1109/TKDE.2012.51\n2.\tLee, D., & Seung, H. S. (2000). Algorithms for non-negative matrix factorization. Advances in neural information processing systems.\n3.\tLee, Daniel D., and H. Sebastian Seung. (1999) \"Learning the parts of objects by non-negative matrix factorization.\" nature 401.6755\n4.\tGan, Jiangzhang, et al. \"Non-negative matrix factorization: a survey.\" The Computer Journal 64.7 (2021): 1080-1092.\n5.\tHu, Jie, Li Shen, and Gang Sun. \"Squeeze-and-excitation networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition, 2018\n6.\tHan, Zongyan, Zhenyong Fu, and Jian Yang. \"Learning the redundancy-free features for generalized zero-shot object recognition.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n7.\tZ. Geng, M.-H. Guo, H. Chen, X. Li, K. Wei, and Z. Lin, ‘Is Attention Better Than Matrix Decomposition?’, in International Conference on Learning Representations, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to sincerely thank all reviewers for their valuable feedback that helps stregthening our contributions. We would like to respond to common comments here, while responding individually for the rest.

* a4X8, ECso: 
Visualization of the attention maps to demonstrate the effectiveness the proposed FSAM module:

We fully agree that visualization of learned attention can provide insight into the efficacy of the proposed method. In figure 4 of PDF, we have added the same for the network trained without and with the proposed FSAM module. Each tile represents a channel of 4D embedding (spatial, temporal and channel), and the figure shows all the channels of an embedding layer. For each channel, we compute cosine similarity between the temporal dimension of a spatial-temporal voxel and the ground-truth signal. The resultant color-coded cosine-similarity score-map offers more intuitive visualization of the learned spatial-temporal attention, as compared to that used in the existing works [1] that disjointly present spatial and temporal attention. Given this comment, we will introduce a new figure in the results section of revised manuscript, along with a brief description on our approach to generate visualization of the learned attention map. Further, our supplementary data with more samples of learned embeddings, model outputs of estimated signals, detailed results and code is shared with the ACs.

* a4X8, c6jF:
Reporting of statistical significance of the experiments, variance and uncertainty estimates:

To report statistical significance, we picked the best performing SOTA method (EfficientPhys [2]) from the main cross-dataset evaluation. For the SOTA and the proposed method, we performed 10 rounds of training and testing with random seed values. Paired T tests show that the proposed method outperforms SOTA with statistical significance obtained for all reported evaluation metrics. Results are added to an excel-sheet that is a part of the supplementary data, which we have recently shared with the ACs. 

For rPPG signal estimation task, total uncertainty measure has been shown to highly correlate with the absolute error obtained from the heart-rate computed using the estimated rPPG signal and the ground-truth heart-rate [3]. While authors used CHROM [5] rPPG method to show this correlation, they clarified that their method is agnostic to the rPPG method [3] and their findings can be generalized to time-series estimation task [3, 4]. Based on this understanding, we report mean absolute error as well as mean values of other relevant rPPG specific evaluation metrics in the submitted manuscript. We have further revised Table-1 (in attached PDF) to provide the corresponding variance measures to draw meaningful conclusions from the experiments. We understand that the reported evaluation metrics along with the variance measures thoroughly compare the uncertainty estimates of the proposed method with that of the SOTA methods.

* a4X8, c6jF:
Scalability Analysis:

i) Performance of FSAM for higher temporal and spatial resolution: 
In new appendix Table-8 (in PDF), we compare the performance of the proposed module by varying spatial resolution and temporal dimension. With higher spatial resolution and more temporal frames, we observe small performance gain. We further conducted repeated tests (10 paired tests with different seed values) that showed non-significant performance difference between the models trained with 160x72x72x3 (low-res) and 240x128x128x3 (high-res) dimensional input video frames. Results of the repeated tests for scalability are also added to the supplementary data.

ii) Overview of computational complexity when FSAM is adapted for higher temporal and spatial resolution: 
The number of trainable parameters of the FSAM module are dependent only on the parameters in pre and post convolution layers, while the factorization operation is implemented as no-grad, thereby not adding trainable parameters as well as not directly affecting the gradients flow for optimization of the main network.
For low-res and high-res inputs, trainable parameters of FSAM are just 328 within our implemented 3D-CNN architecture having 56200 parameters. For low-res and high-res, dimension of embeddings approximated by FSAM are 160x392 and 240x5000 respectively, which are both approximated in 4 iterations, executed only during forward pass. For very high-resolution embeddings, one of the high-resolution dimensions can be appropriately splitted to execute batched optimization for factorization. Thus FSAM adds negligible overhead making it highly suitable for scalability. 
Our additional ablation study (in supplementary data) highlight that the models trained using the FSAM retain the same performance, when deployed without the FSAM module for evaluation. This eliminates inference time latency and computational overhead of the module. The results reported in the revised Table-1 (in PDF) are obtained from the proposed FactorizePhys model deployed without FSAM module. This further improves the scalability of the proposed method.


1. C. Zhao, H. Wang, H. Chen, W. Shi and Y. Feng, "JAMSNet: A Remote Pulse Extraction Network Based on Joint Attention and Multi-Scale Fusion," June 2023, doi: 10.1109/TCSVT.2022.3227348
2. Liu, Xin, et al. "Efficientphys: Enabling simple, fast and accurate camera-based cardiac measurement." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2023
3. R. Song, H. Wang, H. Xia, J. Cheng, C. Li and X. Chen, "Uncertainty Quantification for Deep Learning-Based Remote Photoplethysmography," 2023, doi: 10.1109/TIM.2023.3317379.
4. W. Qian, D. Zhang, Y. Zhao, K. Zheng and J. J. Q. Yu, "Uncertainty Quantification for Traffic Forecasting: A Unified Approach," 2023, doi: 10.1109/ICDE55515.2023.00081.
5. De Haan, Gerard, and Vincent Jeanne. "Robust pulse rate from chrominance-based rPPG." IEEE transactions on biomedical engineering 60.10 (2013): 2878-2886.

Meta-review: The paper is well-written and clearly presents a novel method that leverages the strengths of Non-negative Matrix Factorization (NMF) to effectively capture global spatial-temporal context. While NMF is powerful for vision tasks, the authors apply it within a new end-to-end 3D-CNN architecture aimed at estimating blood volume pulse signals from video frames.

The reviewers unanimously agree that the empirical results on remote Photoplethysmography (rPPG) are promising.

However, to further strengthen the paper, the authors are encouraged to clarify the motivation behind using NMF in this context. Incorporating additional details such as the visualization of attention maps and providing statistical significance, as suggested by the reviewers, would enhance the paper's robustness and the clarity of its contributions.

I recommend accepting this paper, provided the authors address these points in their final submission.
Decision: Accept (poster)