FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing

Authors: jitesh  joshi
sos s agaian
youngjun  cho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:52:54

Abstract: Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.

Main paper: 
 attention mechanisms in vision: 
 Varied forms of attention mechanisms have been successful in different visual tasks such as image classification [70, 25, 66] , object detection [5, 82] , semantic segmentation [78, 16, 18, 28] , video understanding [63, 15, 33, 21] , 3D vision [69, 24] , and multimodal tasks [73, 56] among others [20, 23] . The most widely used attention mechanisms are channel attention [35, 75] , spatial attention [66, 63] , temporal attention [72, 74] , self-attention or transformer-based approaches [58, 14] , multimodal attention [56, 73] , graph-based approaches [32] , as well as different combinations of these types [66, 16, 53] . In addition, researchers have proposed attention mechanisms for video understanding [63, 15, 33, 21] as well as 3D vision [69, 24] . Despite notable advances in different forms of attention mechanisms, some of the existing challenges include the requirement for high computational costs, large training data, the overall efficiency of the model, and a cost-benefit analysis of performance improvement [23] . Additionally, for rPPG research, the impact of attention mechanisms on an ability of models to generalize on unseen datasets is not systematically studied, which we address in this work.End-to-end rPPG methods can be categorized into convolution neural networks (CNN) architectures such as PhysNet [83] , EfficientPhys-C [37] , 3DCNN [4] , SAM-rPPGNet [26] , and RTrPPG [3] , and transformer-network-based architectures such as PhysFormer [77] , PhysFormer++ [76] , and EfficientPhys-T [37] . Among end-to-end rPPG methods, DeepPhys [6] first implemented a novel convolutional attention mechanism in an architecture that comprised separate motion and appearance branches, with the latter intended to compute attention for the main motion branch. Inspired by the CBAM attention mechanism [66] , originally validated for classification and detection tasks, ST-Attention [46] was proposed to filter salient information from spatial-temporal maps, thus improving remote HR estimation. The similar dual attention mechanism was also found to be effective in the SMP-Net framework [13] , which jointly learned the features of RGB and infrared spatial-temporal maps to estimate multiple physiological signals.Recently, EfficientPhys [37] , an end-to-end network, presented an efficient single-branch approach with a gated attention mechanism. The Swin-Transformer [39] based version of EfficientPhys [37] insightfully added the TSM [34] module to the Swin transformer [39] , enabling the architecture to perform efficient spatial-temporal modeling and compute attention by combining shifting window partitions spatially and shifting frames temporally. It should be noted that the convolution-based version of EfficientPhys [37] , which combined the TSM [34] module and a convolutional attention mechanism [6] showed superior accuracy along with significantly low latency, making it highly suitable for deployment on mobile devices. Recently, there has been an upsurge in transformer-based rPPG architectures, some of which include PhysFormer [77] , PhysFormer++ [76] , TransPhys [61] , and RADIANT [22] .Unlike other transformer-based architectures that rely on spatial-temporal maps as input, PhysFormer [77] and PhysFormer++ [76] are end-to-end video transformer-based architectures, which adaptively aggregate both local and global spatial-temporal features. PhysFormer++ [76] extends PhysFormer [77] by better exploiting temporal contextual and periodic rPPG clues, as it extracts and fuses attentional features from slow and fast pathways. In addition, both architectures [77, 76] are trained using label distribution learning and a curriculum learning-inspired dynamic constraint in the frequency domain, which helps to alleviate overfitting. Although unlike convolution-based EfficientPhys [37] , transformer architectures require significantly higher computational resources.Most of the light-weight convolutional attention mechanisms require attention to be separately derived in spatial, temporal, and channel dimensions, which is later merged [46, 13] . Although 3D-CNN architectures such as PhysNet [83] and iBVPNet [29] have shown promising performances, they have not explored attention mechanisms that can potentially enhance performance in unseen datasets. JAMSNet [79] and GLISNet [80] are recent 3D-CNN architectures that benefit significantly from channel-temporal joint attention (CTJA) and spatial-temporal joint attention (STJA). However, unlike CTJA and STJA [79, 80] , we jointly derive attention in temporal, spatial, and channel dimensions, without squeezing any dimension of multidimensional features. EQUATION)whereV = [ v1 , v2 , ..., vN ] ∈ R M ×N ≥0 is reconstructed low-rank matrix and E ∈ R M ×N ≥0 is an error matrix, which is discarded. R M ×N ≥0stands for the set of M × N element-wise nonnegative matrices. Equivalent vector formulation for this approximation can be expressed as:EQUATIONThe objective to represent high-dimensional matrix with fewer basis can be achieved only when L is chosen such that L ≪ min(M, N ), while when L is larger than M , it results in over-complete basis. An optimization in W and H to achieve the optimal approximation effectively results in the discovery of inherent correlations between the basis vectors in W and the corresponding coefficients in H [31, 64] . The optimization objective is formulated as: min W,H ∥V -W H∥ 2 F ∋ W ml ≥ 0, H ln ≥ 0 (3) Further, the imposed non-negativity constraints on W and H enable parts-based representations, where activation of one or many of the coefficients in H together with the basis vectors in W can reconstruct different interpretable parts of V . For a detailed primer on NMF, we refer the reader to the seminal work [31] and a survey article [64] that summarizes different NMF models and algorithms.The factorization of deeper layer embeddings can be elucidated as the squeeze of information without reducing the dimensions of the embeddings, unlike the existing attention mechanisms [25] . Therefore, exciting or multiplying with the resultant low-rank (information squeezed) embeddings can potentially serve as an attention mechanism. While factorization is formulated for two-dimensional matrix, high-dimensional embeddings can be mapped to two-dimensional matrix. In PyTorch [47] , this is achieved with the 'view' operation.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In the conclusion section, we have dedicated a paragraph to mention the broader impact and discuss the limitations and future directions that can be investigated.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
factorized self-attention module (fsam): 
 For the downstream task of estimating rPPG from video frames, spatial-temporal input data can be expressed as I ∈ R T ×C×H×W , where T, C, H, and W represents total frames (temporal dimension), channels in a frame (e.g., for RGB frames, C = 3), height and width of pixels in a frame, respectively. I is passed through a feature extractor that generates voxel embeddings ε ∈ R τ ×κ×α×β , with temporal (τ ), channel (κ) and spatial (α, β) dimensions.The goal is to jointly derive the attention in the multidimensional space of ε, without squeezing individual dimensions. For this, we deploy NMF-based matrix factorization to compute low-rank ε by reconstructing it from the factorized basis matrix W and a coefficient matrix H. It is essential to factorize ε in a way that ε approximated through the computed basis and coefficient matrices serves as an effective self-attention. Among several parameters that govern factorization, here we delve into the ones most relevant for the time series estimation task. These include: i) the transformation of the voxel embeddings that maps ε ∈ R τ ×κ×α×β to the factorization matrix V st ∈ R M ×N and ii) the rank of the factorization. For a 2D-CNN architecture with κ channels and α×β spatial features, the transformation (Γ καβ →M N ) implemented in the Hamburger module [18] is expressed as:EQUATION) where κ channels are mapped to M and α × β spatial features are mapped to N , with an underlying assumption that spatial features are inherently correlated due to learnt CNN kernels. However, for 3D-CNN architectures, as ε ∈ R τ ×κ×α×β encodes temporal, channel, and spatial features, it is required to revisit these mappings. While it can be argued that similar to 2D-CNN architectures, as 3D-CNN architectures have 3D kernels, the spatial-temporal features are inherently correlated. However, it should be noted that the scales of spatial and temporal dimensions are very distinct, owing to which the spatial-temporal patterns to be learned may not be uniformly captured through typical convolutional kernels (e.g. 3 × 3 × 3). Adjusting the spatial-temporal kernel sizes can be heuristic task, and does not guarantee the extraction of desired features, while drastically increasing the model complexity (since for time series estimation, τ >> α, β). Also, κ channels are not inherently correlated, and therefore it is crucial to devise a multidimensional attention that jointly computes the spatial-temporal and channel attention. To address this, we first consider negative Pearson correlation, a loss function that is commonly deployed to optimize end-to-end rPPG methods, expressed as:η p = 1 - T i (r ppg i -r ppg )(g ppg i -g ppg ) T i (r ppg i -r ppg ) 2 T i (g ppg i -g ppg ) 2 (5)where, r ppg ∈ R 1×T is an estimated rPPG signal and g ppg ∈ R 1×T corresponds to the ground-truth BVP signal. The optimization of end-to-end model to estimate a vector in temporal dimension (r ppg ∈ R 1×T ) can be leveraged by establishing the correlation of features in spatial and channel dimensions with the features in temporal dimension. Factorization of a matrix that consists of vectors in temporal domain and spatial and channel dimension as the features of the vectors, uniquely offers an opportunity to design the requisite attention. Prior to transforming ε to V st ∈ R M ×N , it is pre-processed through a convolution layer (with 1 × 1 × 1 kernels), and a ReLU activation to ensure non-negativity of the embeddings. Following this preprocessing, the temporal features of ε are mapped to the vector dimension (M ) in V st , while spatial and channel dimensions are mapped to the feature dimension (N ) of V st . This transformation of ε as depicted in fig. 2 , can be expressed as:EQUATIONwhere, ξ pre represents preprocessing operation. Factorization of thus formed matrix V st with temporal vectors shall result in a low-rank matrix V st which is approximated based on the latent structure that establishes correlation of temporal features with spatial and channel features.EQUATIONwhere ϕ represents factorization operation. V st is transformed back to the embedding space, resulting in an approximated voxel embeddings ε that selectively retains the spatial and channel features that contribute towards the recovery of salient temporal features in ε. The resultant ε can be expressed as:EQUATIONwhere Γ M N →τ καβ represents matrix transformation operations. We use the one-step gradient optimization based approach [18] to factorize V st . This approach is a linear approximation of the conventional back-propagation through time algorithm (for time t → ∞) [65] , as proposed with the Hamburger module [18] . Approximated low-rank matrix V st is transformed back to the embedding space through Γ M N →τ καβ , resulting in ε that can potentially serve as the requisite attention. ε is post-processed with a convolution layer (with 1 × 1 × 1 kernels), and a ReLU activation, followed by element-wise multiplication with ε. This multiplication operation serves as an excitation operation, which can be distinctly effective as ε retains the dimension of ε while computing the attention. The product is instance-normalized, and added with ε that serves as residual connection as depicted in fig. 2 . It is to be noted that for each single forward pass through the model, approximation of V st requires 4-8 steps, however, FSAM implements NMF within "no_grad" block, that does not require back-propagation through the NMF for model optimization. Representing the network head as ω, ξ post as post-processing operation and IN as instance normalization, the estimated r ppg signal can be expressed as:EQUATIONNext, we look at the rank of the factorization that affects the approximation of V st . The primary consideration for the rank L ≪ min(M, N ) as mentioned in §3.1 ensures that V st is of low rank. Although the choice of L is generally governed by the downstream task, it is often derived empirically. In the context of r P P G estimation, we revisit the formulation of factorization matrix through Γ τ καβ →M N that maps temporal features along the M dimension. As we expect only a single signal underlying source of BVP signal across all facial regions, single vector estimation vst 0 corresponding to rank-1 (i.e., L = 1) shall be sufficient to capture the spatial, temporal, and channel features that contribute to the r P P G estimation. Experimentation with rank-1 and higher rank factorization (appendix A.3) shows that for the higher ranks, the performance remains at par with that of the network without the FSAM, indicating that for rPPG estimation task, rank-1 factorization offers the optimal multidimensional attention, confirming our understanding.We deploy FSAM in our proposed 3D-CNN model, FactorizePhys and integrate it in an existing 2D-CNN architecture, EfficientPhys [37] to assess its versatility.FactorizePhys Architecture: FactorizePhys, as depicted in fig. 3[A] , is an end-to-end 3D-CNN architecture for estimating rPPG signal from raw video frames. Skin reflection models [62, 6] discuss the presence of several unrelated stationary and time-varying temporal components, and a relatively weaker pulsatile component of interest. To eliminate stationary components, FactorizePhys implements a Diff as first layer, inspired by existing rPPG architectures [6, 36, 37] . The resultant Diff frames are normalized with IN , unlike existing architectures that use BatchNorm. The size Figure 3 : (A) Proposed FactorizePhys with FSAM; (B) FSAM Adapted for EfficientPhys [37] of the kernel and the strides of each convolution layer are depicted in fig. 3[A] . For each layer, we use TanH activation followed by IN . Spatial features are gradually aggregated by not padding the features, while we deploy spatial convolution strides only on the 3 rd and 6 th layers. For temporal features, same padding retains the input temporal dimension throughout the network, as depicted in fig. 3[A] . Downsizing of temporal features may result in high-amplitude unrelated time-varying components to outweigh the weaker rPPG related pulsatile component. To provide a clearer overview of the architecture of FactorizePhys, only the spatial and temporal dimensions of the features at multiple layers are shown in fig. 3[A] , while the channel dimension is skipped. FSAM, as elaborated in §3.2, is deployed to jointly compute multidimensional attention, at the layer where the spatial dimension is reduced to 7 × 7, as reported as the optimal spatial dimension in a recent work [3] .Adaptation of FSAM for 2D-CNN Architecture: Several SOTA rPPG methods [36, 37] leverage the TSM [34] that efficiently models spatial-temporal features using 2D-CNN architectures. The parameter called 'Frame Depth' (ψ ∋ ψ ≪ κ channels) controls the number of channels that are shifted along the temporal dimension for modeling temporal features. We investigated the effectiveness of the proposed FSAM with a more recent TSM-based SOTA rPPG architecture, EfficientPhys [37] , which also deploys the Self-Attention Shifted Network (SASN) as the attention module. As ψ controls the amount of temporal information that is learned, we use this to formulate the mapping for the factorization matrix. Equation ( 6) can be adapted for TSM based architectures to appropriately transform the embeddings to factorization matrix as:EQUATION) Figure 3 [B] shows modified EfficientPhys [37] architecture, in which we drop SASN blocks [37] and add a single FSAM. Unlike SASN [37] , FSAM derives attention without squeezing any individual dimension, which in turn can strengthen the correlation between temporal, channel and spatial features. This adaption critically evaluates the proposed FSAM against its counterpart in the SOTA architecture, addressing the recommendations of a recent survey article [23] on visual attention methods.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We conduct a rigorous assessment of the proposed models by means of an fair comparison with SOTA rPPG methods, and report standard error for all evaluation metrics. This examination is further supplemented with a statistical significance analysis of the principal results, as detailed in • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Although we briefly highlight the potential positive impacts in §6, to prevent any negative societal impact, we have safeguarded our code with Responsible AI License, specifying the appropriate restrictions. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 We perform an evaluation with carefully selected end-to-end SOTA rPPG methods that include Phys-Net [83] , a 3D-CNN architecture without the attention mechanism, EfficientPhys [37] , a 2D-CNN architecture with self-attention, and PhysFormer [77] , a transformer-based 3D-CNN architecture with multi-head self-attention. Comparison of FactorizePhys and PhysNet [83] can indicate the importance of the attention mechanism in 3D-CNN rPPG architectures, while comparison of EfficientPhys with SASN [37] and EfficientPhys [37] with FSAM allows evaluating the effectiveness of the proposed FSAM in 2D-CNN architectures, and thus allows assessing the versatility of FSAM. Similarly, the comparison of FactorizePhys and PhysFormer [77] offers a thorough evaluation of the proposed FSAM against multi-head self-attention in 3D-CNN architectures.Each model was trained on one of the four existing datasets that include iBVP [29] , PURE [55] , UBFC-rPPG [2] and SCAMPS [42] and evaluated on the other three. Appendix A.1 provides our detailed description of these datasets. Our code is based on the rPPG-Toolbox [38] , with specific adaptations described in appendix A.2. We train all models uniformly with 10 epochs [79] on iBVP [29] , PURE [55] , and UBFC-rPPG [2] datasets, and with one epoch on SCAMPS [42] dataset. For fair evaluation, all model-specific hyperparameters were maintained as provided by the respective SOTA rPPG methods, while the training pipeline related hyperparameters, which include preprocessing steps for images and labels, batch size, number of epochs, learning rate, scheduler, and optimizer were kept consistent for training all the models. 
results and discussion: 
 Ablation Study: First, we train FactorizePhys on UBFC-rPPG [2] dataset and test on PURE [55] and iBVP [29] datasets, to compare different transformations of ε ∈ R τ ×κ×α×β with temporal, channel and spatial features to factorization matrix V st ∈ R M ×N as tabulated in table 1. Superior cross-dataset generalization can be observed when the temporal dimension, τ is mapped to M , as described in §3.2. We assess contribution of FSAM over base FactorizePhys model and observe consistent performance gains with FSAM, as reported in table 3 in appendix A. We then investigate 3 , and observe it to contribute positively. We also observed that the base FactorizePhys model trained with FSAM retains the performance gains in-spite when FSAM is skipped during the inference. As this eliminates the computational overhead during inference, we report our main results of FactorizePhys trained with FSAM, by running inference without the FSAM. On contrary, in case of TSM [34] based EfficientPhys [37] model trained with FSAM, we observed performance drop when FSAM was skipped during inference, and therefore for EfficientPhys with FSAM, we do not drop FSAM during inference. Evaluation for factorization ranks and optimization steps to solve NMF shows consistent superiority of rank-1 factorization in table 4 in appendix A.FactorizePhys vs. State-of-the-Art: We use heart rate (HR) [67] along with BVP metrics that include signal-to-noise ratio (SNR) and maximum amplitude of cross-correlation (MACC) [29, 10] for evaluation. SNR and MACC are direct measures to compare estimated rPPG signals with ground-truth BVP signals. The HR metrics reported are the mean absolute error (MAE), the square root of the mean square error (RMSE), the mean absolute percentage error (MAPE), and Pearson's correlation coefficient (Corr) [67] of the estimated HR. Uncertainty estimates quantifying the variability associated with signal estimation have been shown to be strongly correlated with the absolute error of the estimated HR [52] . In addition, for each metrics, we report the standard error to estimate the variability of each model. As most SOTA end-to-end models show robust withindataset performance, we present cross-dataset performance in table 2, while reporting within-dataset performance in table 6 in appendix A.First, we observe that for all the evaluation metrics reported, the proposed FactorizePhys with FSAM outperforms the SOTA methods on PURE [55] and iBVP [29] datasets, across all training datasets. This suggests a consistent and superior generalization achieved by the proposed method. Crossdataset evaluation on the UBFC-rPPG [2] dataset further highlights the performance gains of the proposed FactorizePhys model when trained with the iBVP [29] and the SCAMPS [42] datasets, and at-par performance when trained with the PURE dataset. When models are trained with SCAMPS [42] (synthesized dataset), FactorizePhys uniquely outperforms the SOTA methods on all testing datasets further indicating the superior cross-dataset generalization. The performance of EfficientPhys [37] with FSAM exceeds in most cases and remains at par in the rest, compared to the EfficientPhys model with SASN [37] , suggesting the versatility of FSAM as an attention module. As 3D CNN kernels in FactorizePhys can learn spatial-temporal patterns better than the TSM [34] based 2D-CNN model (EfficientPhys [37] ), FactorizePhys with FSAM outperforms EfficientPhys [37] with FSAM across all datasets. Lastly, the proposed method consistently achieves superior SNR and MACC for the estimated rPPG signals, highlighting the enhanced reliability of the extracted signals.Computation Cost and Latency: We compare computational complexity and latency for all the models in fig. 4 [A], and provide further details in table 9 in appendix A. The cumulative MAE is computed by averaging cross-dataset performance for respective models across all combinations of training and testing datasets reported in table 2 . The proposed FactorizePhys with FSAM not only shows the best performance, it has significantly less number of model parameters and performs at par in terms of latency as the 2D-CNN SOTA rPPG method, EfficientPhys [37] . Specifically, dropping FSAM during inference does not result in loss of performance for FactorizePhys, while reducing latency considerably, making it highly suitable for real-time and resource-constrained deployment. In contrast, when FSAM was dropped after training EfficientPhys [37] with FSAM, it did not retain the performance (results not shown). We interpret that while FSAM effectively influences the 3D convolutional kernels in FactorizePhys to increase the saliency of relevant spatial-temporal features, 2D convolutional kernels cannot benefit adequately due to the limited ability to model spatial-temporal TD-MHSA*: Temporal Difference Multi-Head Self-Attention [77] ; SASN: Self-Attention Shifted Network [37] ; FSAM: Proposed Factorized Self-Attention Module. † All metrics are shown with two decimal places, except those correlation measures that range between 0.99 and 1.0 features. It should also be noted that the higher latency of FactorizePhys compared to EfficientPhys [37] , although it has fewer model parameters, can be attributed to the difference in floating-point operations (FLOPS) between the 3D-CNN and 2D-CNN architectures.Visualization of Learned Attention: We compute absolute cosine similarity between the temporal dimension of 4D embeddings (with temporal, spatial, and channel dimensions) and the ground-truth signal to visualize the learned attention for FactorizePhys trained without and with FSAM in fig. 4[B] ,where each tile represents a channel of the embedding layer. A higher cosine similarity score between the temporal dimension of the embeddings and the ground-truth PPG signal, which is observed for FactorizePhys trained with FSAM, indicates a higher saliency of temporal features. The spatial spread of high cosine similarity scores in different channels for FactorizePhys trained with FSAM, highlights selectivity of the learned attention, providing clearer evidence that the FactorizePhys model trained with FSAM can effectively pick the spatial features having the strong presence of the rPPG signal (i.e., facial regions with visible skin surface). Figure 4 [B] not only suggests the effectiveness of the joint computation of multidimensional attention, but also offers more intuitive visualization of learned spatial-temporal features than existing visualization approaches [79, 37] . 
conclusion: 
 We present FactorizePhys, a 3D-CNN model utilizing the Factorized Self-Attention Module, FSAM, to concurrently extract multidimensional (spatial, temporal, and channel) attention for the downstream task of rPPG estimation from video frames. The assessment performed utilizing various rPPG datasets demonstrates that our proposed method possesses superior generalization capabilities across different datasets, compared to current state-of-the-art methods. Moreover, when adjusted to the 2D-CNN architecture, FSAM achieves performance on par with the established SASN [37] attention, underscoring its adaptability across diverse network architectures.Broader Impacts and Limitations: The superior performance of FactorizePhys equipped with FSAM to estimate rPPG indicates its potential utility in various healthcare applications that require the estimation of physiological signals through noncontact imaging. Although FSAM has shown efficacy as a multidimensional attention mechanism specifically for the extraction of rPPG signals, more research is needed to determine the efficacy of the proposed method in extracting heart rate variability metrics as well as other physiological signals. Despite the state-of-the-art performance of the proposed rPPG method, signal peaks can still be susceptible to challenging real-world scenarios, such as active head movements, occlusions, and dynamic changes in ambient lighting conditions, an issue that is qualitatively illustrated in the waveforms depicted in appendix A.11. Moreover, it is imperative to conduct additional research to evaluate the effectiveness of FSAM across other spatialtemporal domains, including video understanding, video object tracking, and video segmentation, along with several other downstream tasks that depend on multi-dimensional input data. In the context of signal estimation tasks, the utilization of NMF variants that integrate temporal or frequency constraints on time series vectors may offer enhanced attention capabilities. These constraints are congruent with the characteristics of the ground truth and present avenues for future investigation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We have mentioned safeguard measures in appendix A.12. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Verkruysse [59] 's pioneering investigation on extracting photoplethysmography (PPG) or blood volume pulse (BVP) signals from RGB cameras in a contactless manner led to an exciting research field of imaging-based physiological sensing. There exist several potential applications and contexts of noninvasive and contactless measurement techniques, such as stress and mental workload recognition [8, 9, 7] , driver drowsiness monitoring [71] and social biofeedback interaction [43] . The seminal works on unsupervised rPPG methods [59, 48, 11] either used video frames acquired under stationary conditions or performed skin segmentation [62] or region of interest (RoI) tracking as a preprocessing step. This preprocessing step can be considered as a basic form of attention mechanism that enables the unsupervised models to process only the relevant regions. Some of the supervised rPPG methods, including HR-CNN [54] , RhythmNet [45] , NAS-HR [40] , PulseGAN [51] , and Dual-GAN [41] also relied on extracting spatial-temporal features from the tracked RoIs as a preprocessing step.As end-to-end rPPG methods, such as DeepPhys [6] , and MTTS-CAN [36] among several others, take whole facial frames as input, they rely on attention mechanisms that enable models to emphasize the relevant spatial-temporal features. Estimating BVP signal from raw facial video frames in an end-to-end manner is therefore an interesting downstream task to investigate the attention mechanism in multidimensional feature space. This requires networks to learn to pick the spatial features having the desired temporal signature, while discarding the variance related to head-motion, illumination, and skin-tones, thus representing one of the challenging spatial-temporal tasks. Few other notable end-to-end rPPG methods include PhysNet [83] , 3DCNN [4] , SAM-rPPGNet [26] , RTrPPG [3] , and transformer-network-based methods such as PhysFormer [77] , PhysFormer++ [76] , EfficientPhys [37] , JAMSNet [79] , and GLISNet [80] . A recent survey article on visual contactless physiological monitoring in clinical settings [27] highlights susceptibility to disturbance, such as head movement, as one of the key challenges, among others. Some of the recent end-to-end rPPG methods [79, 80] further highlight the need for multidimensional attention, as squeezing features in selective dimensions for deriving attention reduces the feature space to a single dimension and is therefore not well suited for the task of signal extraction.To address this, our work draws inspiration from the seminal work on NMF [31] which a recent work formulated as an approach to design the global information block, referred to as Hamburger [18] . Hamburger [18] implements NMF to derive low-rank embeddings, which serve as a global context block. Despite the low computational complexity of O(n), Hamburger [18] outperformed various attention modules in the semantic segmentation [68] and image generation tasks. In addition, researchers have combined matrix factorization with deep architectures in several ways for different applications such as layer-wise learning of dictionary for classification and clustering [57] , adaptive learning of dictionary for image denoising [81] , multi-attention model for recommendation systems [60] , and linearly scalable approach to context modeling for medical image segmentation [1] among several others. Drawing inspiration from these studies, especially those that use matrix factorization to model global context [18, 1] in vision tasks, we investigate the application of NMF as a multidimensional attention block. Although matrix factorization in deep learning has remained a topic of significant interest, it has not been investigated in the realm of rPPG, which stands to gain from joint spatial, temporal, and channel attention.We introduce the Factorized Self-Attention Module (FSAM), which implements NMF to jointly compute spatial-temporal attention and describe an appropriate formulation for the low-rank recovery problem. To investigate the relevance and effectiveness of FSAM in computing multidimensional attention, we build a 3D-CNN architecture FactorizePhys that implements FSAM. We further adapt FSAM for EfficientPhys [37] , an end-to-end rPPG architecture that builds on the Temporal Shift Module (TSM) [34] , to uniquely learn spatial-temporal features using 2D-CNN layers. Evaluation of FactorizePhys and EfficientPhys [37] with FSAM, against existing SOTA rPPG methods, demonstrates the versatility of FSAM as multidimensional attention along with its effectiveness for the downstream task of estimating time series from spatial-temporal data. In summary, we make the following contributions.• Factorized Self-Attention Module (FSAM): NMF [31] -based novel approach that jointly computes multidimensional attention within voxel embeddings. • FactorizePhys: an end-to-end 3D-CNN architecture that integrates FSAM for robust estimation of rPPG from spatial-temporal facial video frames. • Thorough assessment of FactorizePhys and FSAM with multiple evaluation metrics and the corresponding measure of standard errors to compare cross-dataset generalization performance with SOTA rPPG methods, using four benchmarking rPPG datasets.2 Related Work 
 
 Appendix: All datasets provide video recordings with a resolution of 640 × 480, and frame rate of 30 FPS. Below we provide data-specific details.iBVP [29] : The iBVP dataset consists of 124 synchronized RGB and thermal infrared videos from 31 subjects, acquired under controlled conditions. Each video is 3 minutes in duration, and the ground truth BVP signals were acquired from the ear using PhysioKit [30] . Data were acquired under 4 different conditions that include controlled breathing, math tasks, and head movements. BVP signals are marked with the signal quality, enabling the use of the video frames only where the quality of ground-truth BVP signal is high. In this work, we use only RGB frames to train the models.PURE [55] : This data set comprises video recordings from 10 subjects, with the ground-truth BVP and SpO2 signals acquired from the subject's finger. For each participant, six recordings are acquired under varied motion conditions, offering a range of data reflecting different physical states.This data set contains video recordings of 43 subjects acquired under indoor conditions with a combination of natural sunlight and artificial illumination.SCAMPS [42] : This dataset comprises 2800 videos of synthetic avatars that were generated through high-fidelity, quasi-photorealistic renderings. Although the videos introduce various conditions such as head motions, facial expressions, and changes in ambient illumination, they are often used as a training set rather than a validation or test set.The preprocessing steps for video frames include face detection using the YOLO5Face [49] face detector at an interval of 30 frames and using the detected facial bounding box to crop 30 subsequent frames, prior to performing the next face detection. The cropped facial frames are resized to a resolution of 72 × 72, which has been shown to be sufficient to estimate the rPPG. Additionally, to ensure uniform input data for all models, we add Diff layer to the PhysNet [83] and PhysFormer [77] architectures, as implemented by EfficientPhys [37] and the proposed FactorizePhys models, and train all the models from scratch using uniformly preprocessed video frames.The number of frames in a video chunk is maintained as 161, which after the Diff layer becomes 160, making the spatial-temporal input data size 160 × 72 × 72. Ground-truth BVP signals are also uniformly standardized for training all models. This is different from some of the recent work [37] that applies Diff in addition to standardization. We empirically found that all models perform significantly better when trained with the standardized BVP signals, although when the Diff is applied to the video frames.All models were trained with 10 epochs on, following a recent work [79] , as a higher number of epochs, e.g. 30 epochs as used in rPPG-Toolbox [38] resulted in poor generalization for all models. However, we used only one epoch for all models to train on the SCAMPS [42] dataset, since this dataset is a synthesized dataset with generated BVP signals that are easier for models to learn, unlike real-world datasets. Training beyond one epoch resulted in poorer cross-dataset performance for all the models. The batch size of 4 was used consistently throughout the training and the maximum learning rate was set to 1 × 10 -3 with 1 cycle learning rate scheduler [50] for all CNN models.In addition, CNN models were optimized using negative Pearson correlation as a loss function. The learning rate for PhysFormer [77] was set to 1 × 10 -4 and it was optimized using a dynamic loss composed of several hyperparameters, a negative Pearson loss, a frequency cross-entropy loss and a label distribution loss as used by the authors and implemented in the rPPG-Toolbox [38] . Before computing HR for performance evaluation, both ground truth and estimated BVP signals were filtered using a bandpass filter (low cutoff = 0.60 Hz, high cutoff = 3.30 Hz) to accommodate HR ranges of 36 to 198 BPM. HR was then computed using the FFT-peaks-based approach as implemented in rPPG-Toolbox [38] .We conduct ablation studies to evaluate optimal architectural choices and hyperparameters for the proposed FactorizePhys and FSAM. In table 3 , we compare base FactorizePhys without FSAM and with FSAM and observe consistent performance gains with FSAM. Evaluation with and without residual connection indicates performance gains when residual connection around FSAM is implemented. Retention of performance gains despite FSAM being skipped during inference, for FactorizePhys trained with FSAM offers insight into the mechanics of how FSAM functions. This can be interpreted as follows: Optimization of a network having FSAM implemented as an attention mechanism influences the network to increase the saliency of the most relevant features, so that a factorized approximation of embeddings retains these features, while discarding the less important features. Due to the increased saliency of relevant features and the presence of residual connection, FSAM can be skipped during inference, significantly reducing computational overhead. In table 4 , we present results to compare the performance obtained for different ranks L, as well as the optimization steps used to solve factorization. For all experiments, FactorizePhys is trained with the UBFC-rPPG dataset [2] and the performance is presented for the PURE dataset [55] . We can observe that the best performance was achieved for rank L = 1 for the different steps used to solve the factorization. For higher ranks, performance remains on par with that of the network without the FSAM, indicating that for the rPPG estimation task, the rank-1 factorization offers the optimal spatial-temporal attention. These results align with the expected single source of the underlying BVP signals in different facial regions.We performed repeated experiments with 10 different random seed values between 1 and 1000 to compare the proposed FactorizePhys trained with FSAM with the best performing SOTA rPPG method. For the cross-dataset generalization results reported in table 2, EfficientPhys with SASN [37] was found to perform the best among the existing SOTA methods. For each random seed value, we trained the proposed FactorizePhys with FSAM and EfficientPhys with SASN [37] on the UBFC-rPPG [2] dataset and evaluated them on the PURE dataset [55] . Paired T tests for each reported evaluation metrics suggest that the performance gains achieved with the proposed method are statistically significant compared against the best performing SOTA rPPG method, highlighting its effectiveness and thereby highlighting contributions of this work in the research field of end-to-end rPPG estimation from video frames.In this work, we primarily focus on comparing rPPG methods for their cross-dataset generalization, which offers more critical evaluation and reliable estimates of how models perform on unseen or out-of-distribution data. Within-dataset performance signifies an representation ability of model to fit the data, derived from the same distribution, serving as an essential criteria. Therefore, for completeness, in table 6, we report within-dataset evaluation on iBVP [29] , [55] , and UBFC-rPPG [2] datasets, where we observe at-par performance of FactorizePhys as compared with the SOTA rPPG methods. TD-MHSA*: Temporal Difference Multi-Head Self-Attention [77] ; SASN: Self-Attention Shifted Network [37] ; FSAM: Proposed Factorized Self-Attention ModuleWe further investigate FSAM for its scalability to higher spatial-temporal resolution. For this, we perform within-dataset evaluation on the UBFC-rPPG dataset [2] , which is pre-processed with the regular input dimension of 160 × 72 × 72 as well as with a higher spatial and temporal dimension of 240 × 128 × 128. Repeatable experiments are conducted with 10 different random seeds between 1 and 1000 to compare the performance of FactorizePhys with FSAM for each spatial-temporal input dimension.Comparable performance, as observed in table 7, for both spatial-temporal input dimensions, suggests that FSAM can be easily deployed for different spatial-temporal scales. It should also be noted that the higher spatial dimension of video frames (i.e., 128 × 128) does not produce improved performance, indicating that the spatial dimension of 72 × 72 is sufficient to extract rPPG signals with end-to-end methods. As iBVP dataset offers synchronized RGB and thermal infrared video frames, we conducted a brief experiment using FactorizePhys with FSAM to investigate whether combining both modalities can result in performance gains for the estimation of rPPG. For this, we also individually trained FactorizePhys on RGB and thermal frames keeping the identical data split of 70%-30%. Results A.9 Computational Cost and Latency Table 9 compares model parameters, latency on GPU and CPU, and model size of the proposed FactorizePhys with FSAM with that of the existing SOTA rPPG methods. Considering the identical inference time performance of the base FactorizePhys, when trained using the proposed FSAM, the proposed method uses an order of magnitude fewer parameters and achieves a par latency on both CPU and GPU systems. Relatively higher latency compared to the EfficientPhys [37] model, despite the fewer model parameters, is due to the difference in the number of floating point operations (FLOPS). FactorizePhys, being a 3D-CNN architecture, requires more FLOPS to compute 3D features at each layer compared to the fewer FLOPS for EfficientPhys [37] which implements the 2D-CNN Justification: We make thorough evaluation of the claims and results support the claims, which are appropriately highlighted in the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper is well-written and clearly presents a novel method that leverages the strengths of Non-negative Matrix Factorization (NMF) to effectively capture global spatial-temporal context. While NMF is powerful for vision tasks, the authors apply it within a new end-to-end 3D-CNN architecture aimed at estimating blood volume pulse signals from video frames.

The reviewers unanimously agree that the empirical results on remote Photoplethysmography (rPPG) are promising.

However, to further strengthen the paper, the authors are encouraged to clarify the motivation behind using NMF in this context. Incorporating additional details such as the visualization of attention maps and providing statistical significance, as suggested by the reviewers, would enhance the paper's robustness and the clarity of its contributions.

I recommend accepting this paper, provided the authors address these points in their final submission.

Reviews: review: paper_summary: paper_summary: The paper proposed the Factorized Self-Attention Module (FSAM), which jointly computes multi-dimensional attention across spatial, temporal, and channel dimensions using non-negative matrix factorization (NMF). The FSAM is integrated into a new end-to-end 3D-CNN architecture called FactorizedPhys, designed to estimate blood volume pulse signals from video frames. Extensive experiments on multiple datasets demonstrate FSAM's effectiveness.
strength_weakness: strength_weakness: -Lack of visualization to show how the results of the proposed method are superior (say, an image of the output to show the learned attention).

-Lack of variance of the results in the experiments table.

-It is still unclear why and in what situation the proposed method is better than the SOTA methods. Say, pick some cases from the model output that show that FactorizePhys is better than the others.
questions: questions: - Are the methods sensitive to the hyperparameters? Lack of sensitivity analysis.

-The authors could consider visualizing more about the outputs of the models.

-Lack of scalability analysis of the methods, especially when the NMF is adopted. For large datasets, it is valuable to know whether the method is scalable compared to other methods.
limitations: limitations: The limitations are mentioned in the paper but could be highlighted more clearly in the discussion section.
paper_summary: paper_summary: The paper presents a novel attention block FSAM devised for handling spatio-temporal data. It is benchmarked against a range of SOTA architectures on a suitable selection of different datasets, and found to perform strongly.
strength_weakness: strength_weakness: While the paper's experimental results are promising, they do not appear to be accompanied by any uncertainty estimates. Including these is crucial to allow the reader to draw meaningful conclusions from the results.  I would refer the authors to question 6 in the checklist at the end of the paper. The answer given to the question 'does the paper report [...] statistical significance of the experiments' is given as "N/A", while the guidelines advise that this answer indicates that the paper does not include any experiments.

Also, while it is acceptable to focus on the intra-dataset performance within the main text, it is crucial to at least include the regular results in the Appendix.

A couple of minor typos:
On line 196:
"device a" -> "devise a"
On 236:
"It's" -> "Its"
questions: questions: Is FSAM anticipated to have significantly wider impact on spatio-temporal applications, beyond rPPG?  

How do the different candidate methods compare in their scalability to higher temporal and spatial resolutions?
limitations: limitations: There is a discussion on potential societal impact in the Appendix.
paper_summary: paper_summary: The paper introduces FactorizePhys that utilizes Non-negative Matrix Factorization (NMF) to decompose voxel embeddings. By integrating FSAM into both 3D-CNN and 2D-CNN architectures, FactorizePhys can estimate blood volume pulse signals from video frames. Through evaluations and comparisons with state-of-the-art rPPG methods, the effectiveness of FSAM and FactorizePhys are demonstrated.
strength_weakness: strength_weakness: 1. The motivation to use the non-negative matrix factorization is unclear. Why can the factorized matrix be used as the attention? Authors should give more insights about the non-negative matrix factorization and show the attention maps to demonstrate the effectiveness.
2. There are other matrix factorization methods such as SVD and QR. Why is the non-negative matrix factorization chosen?
3. The factorization needs to solve an optimization problem which should do gradient descent steps. In line 216, the Authors only use a one-step gradient for matrix factorization. Is the one-step gradient enough to achieve satisfactory factorization?
4. The method part is not clearly illustrated. e.g., equation 5 is confusing, and the symbols in the equation are not well explained. The symbol mentioned in the main text is not well shown in the main figure.
questions: questions: Please check the weakness part.
limitations: limitations: The authors mentioned the limitations of the work and proposed future directions such as time series estimation.

Comments: title: Source code from authors
comment: In continuation of the previous comment, below we share revised main results (after changing the low-cutoff for band-pass filtering) for all the models trained using UBFC-rPPG dataset and evaluated on PURE dataset.


| Model                | Attention Module | MAE  |  (HR) ↓ | RMSE  | (HR) ↓ | MAPE  | (HR)↓ | Corr | (HR) ↑ | SNR   | (dB, BVP) ↑ | MACC | (BVP) ↑ |
| -------------------- | ---------------- | ---- | ------- | ----- | ------ | ----- | ----- | ---- | ------ | ----- | ----------- | ---- | ------- |
|                      |                  | Mean | STD     | Mean  | STD    | Mean  | STD   | Mean | STD    | Mean  | STD         | Mean | STD     |
| PhysNet              | \-               | 7.70 | 2.16    | 18.30 | 113.63 | 13.15 | 3.87  | 0.66 | 0.10   | 11.48 | 1.12        | 0.74 | 0.02    |
| PhysFormer           | TD-MHSA\*        | 7.47 | 2.18    | 18.33 | 131.60 | 11.72 | 3.47  | 0.65 | 0.10   | 9.22  | 1.12        | 0.68 | 0.02    |
| EfficientPhys        | SASN             | 3.39 | 1.58    | 12.59 | 103.66 | 3.65  | 1.35  | 0.84 | 0.07   | 10.27 | 1.17        | 0.69 | 0.02    |
| EfficientPhys        | FSAM (Ours)      | 2.10 | 1.19    | 9.37  | 76.23  | 2.60  | 1.18  | 0.92 | 0.05   | 11.25 | 1.07        | 0.71 | 0.02    |
| FactorizePhys (Ours) | FSAM (Ours)      | 0.54 | 0.21    | 1.70  | 1.56   | 0.77  | 0.31  | 1.00 | 0.01   | 15.18 | 0.96        | 0.80 | 0.02    |

\* TD-MHSA*: Temporal Difference Multi-Head Self-Attention \cite{yu2022PhysFormer};  SASN: Self-Attention Shifted Network \cite{liu2023EfficientPhys}; FSAM: Proposed Factorized Self-Attention Module


We further notice the large RMSE STD errors for SOTA methods attributed to the low SNR of the estimated rPPG signal, affecting waveform morphology, and thereby the computed HR for the edge cases. As the SNR of rPPG signals estimated by FactorizePhys is higher, it significantly outperforrms on edge cases and unseen datasets. The superior cross-dataset generalization of our proposed FSAM module is further evidenced with these revised results.
We thank you for your comment and time to help improve our manuscript.

While preparing for our previous response, we couldn’t spot out your main concern “which is how the NMF is related to the rPPG task”, which we are addressing below. 

To begin with, we fully agree that NMF is useful for general tasks or to put it in other words, it is agnostic to the downstream tasks. Here, we would like to highlight that rPPG estimation task is concerned with a series of spatial-temporal tasks including video object tracking, video segmentation and video action recognition from the perspective of learning spatial-temporal features. Therefore, rPPG estimation task benefits from the attention mechanism like any other spatial-temporal downstream task.

As described in our response to Reviewer a4X8, the proposed approach helps jointly compute multi-dimensional attention using NMF to gain the modeling advantage, which is unlike most current methods that compute attention disjointly across different dimensions, which our reviewer highlighted as our strength. 

Further insights about the relevance of the proposed method for rPPG estimation task can be drawn from the visualization of learned attention maps, that we provided in Figure-4 of the PDF submitted along with global response. In these learned attention maps, higher cosine-similarity score can be observed for the model trained with the proposed FSAM module. Higher cosine-similarity score between the temporal dimension of the embeddings and the ground-truth PPG signal indicates higher saliency of temporal features. The spatial spread of high cosine-similarity scores highlights that the learned attention is selective to the regions of the face that have exposed skin surface (where rPPG signal can be found). This provides clearer evidence that the model trained with the FSAM module can appropriately pick the spatial features that are the sources of the desired temporal signal. Thus, the presented comparison of the visualization offers greater insights into the effectiveness of the joint computation of multi-dimensional attention for rPPG estimation task.

On top of the joint computation of multi-dimensional attention using the proposed FSAM module, transformation of embeddings to factorization matrix as formalized in Equation-6 of the submitted manuscript plays a key role in obtaining significant performance gains as observed for our main, cross-dataset evaluation, which highlight superior generalization ability in the given task. Specifically, as per the Equation-6, temporal dimension of the embeddings is mapped to the vectors of factorization matrix, and the rest of the dimensions form features of the factorization matrix. This mapping enables explainable selection of rank of factorization in rPPG extraction case. Across the entire facial region, we expect only a single rPPG source signal, and similarly we expect this to be represented within the embeddings. Performing factorization of embeddings, with an optimally chosen rank (=1) is therefore highly suited for the given rPPG extraction task. Through an overall optimization of the network in presence of rank-1 approximation of embeddings, model learns to increase the saliency of the most relevant spatial-temporal features, which explains high effectiveness of the proposed method in rPPG estimation.

We hope that the above explanation addresses the concern on “how the NMF is related to the rPPG task.”
We sincerely thank you for appreciating our efforts on preparing visualization of learned attention maps. We also apologize for not making it sufficiently clear about why the proposed FSAM module achieves significant performance gains. Here we would like to provide further insights. 

Firstly, aligned with your suggestion, through our visualization presented in Figure-4 of the PDF submitted with the global response, we compare embeddings of the identical base 3D CNN model – trained without and with FSAM module (which performs joint computation of multi-dimensional attention) . 

In these visualized learned attention maps, higher cosine-similarity score can be observed for the model trained with the proposed FSAM module. Higher cosine-similarity score between the temporal dimension of the embeddings and the ground-truth PPG signal indicates higher saliency of temporal features. Further look at the spatial spread of high cosine-similarity scores highlights that the learned attention is selective to the regions of the face with the exposed skin surface (where rPPG signal can be found). This provides a clearer evidence that the model trained with the FSAM module can appropriately pick the spatial features that are the sources of the desired temporal signal. Thus, the presented comparison of the visualization offers greater insights into the effectiveness of the joint computation of multi-dimensional attention.

To compare the performance against existing attention modules, we picked the best performing SOTA (EfficientPhys that implements SASN attention) based on our main results and adapted our proposed FSAM for 2D-CNN based EfficientPhys architecture. We then replaced SASN module with the proposed FSAM module. Our main (cross-dataset) results, in Table-1 of PDF and within-dataset results (Table-7) of the PDF in global response, provide direct comparison of the performance of FSAM and SASN in EfficientPhys architecture. While EfficientPhys with FSAM module performs at-par with the EfficientPhys having SASN attention module, it can be inferred that 2D-CNN based architecture is not able to leverage the full potential of the joint spatial-temporal attention, which is achieved in the proposed 3D-CNN based architecture.

We understand that in addition to the joint computation of multi-dimensional attention using the proposed FSAM module, transformation of embeddings to factorization matrix as formalized in Equation-6 of the submitted manuscript plays a key role in obtaining significant performance gains as observed for our main, cross-dataset evaluation, which highlight superior generalization ability. Specifically, as per the Equation-6, temporal dimension of the embeddings is mapped to the vectors of factorization matrix, and the rest of the dimensions form features of the factorization matrix. This mapping enables explainable selection of rank of factorization in rPPG extraction case. Across the entire facial region, we expect only a single rPPG source signal, and similarly we expect this to be represented within the embeddings. Performing factorization of embeddings, with an optimally choosen rank (=1) is therefore highly suited for the given rPPG extraction task. Through an overall optimization of the network in presence of rank-1 approximation of embeddings, model learns to increase the saliency of the most relevant features, which explains high effectiveness of the proposed method.

We will revise the results and discussion section to offer these deeper insights.

Hope above response address your main concern.
Thank you for finding our responses satisfactory and for increasing the score. Below we respond to your further queries:

* Very large RMSE STD: 

Thanks for spotting out this error.

We investigated this thoroughly and discovered an edge case scenario of one participant in the PURE dataset, with ground-truth HR=46. The data of only this participant was driving the RMSE STD very high. We first inspected the estimated rPPG signals for all the which were found well aligned with the ground-truth BVP signal. 

The root cause was low-cut-off freq (0.75 Hz), of the band-pass filter as implemented in rPPG-Toolbox [1], upon which our code is built. This filter is applied both to the ground-truth BVP signal and estimated rPPG signals before computing HR. This low-cut value impacts the FFT-peak based computation of HR for HR=46, as the main peak is suppressed and results are driven by the harmonics, leading to the observed large errors. We changed this to 0.5 Hz to accommodate low HR cases and re-evaluated ours as well as SOTA methods on PURE dataset, first for within-dataset case. We thoroughly verified that this change in low-cut-off frequency did not alter any outcome for HR higher than 46, which is the case with other datasets. 

Below, we present the revised within-dataset results for PURE dataset, where we observe expected RMSE STD range for all models.

| Model                | Attention Module | MAE (HR) ↓ |  | RMSE (HR) ↓ |  | MAPE (HR)↓ |  | Corr (HR) ↑ |  | SNR ( dB, BVP) ↑ |  | MACC (BVP) ↑ |  |
| -------------------- | ------------------- | ---------- | ---------- | ----------- | ----------- | ---------- | ---------- | ----------- | ----------- | ---------------- | ---------------- | ------------ | ------------ |
|                      |                     | Mean       | STD        | Mean        | STD         | Mean       | STD        | Mean        | STD         | Mean             | STD              | Mean         | STD          |
| PhysNet              | \-                  | 0.88       | 0.50       | 2.29        | 4.17        | 1.25       | 0.63       | 0.99        | 0.03        | 22.25            | 1.99             | 0.92         | 0.01         |
| PhysFormer           | TD-MHSA\*           | 0.98       | 0.49       | 2.31        | 4.17        | 1.41       | 0.62       | 0.99        | 0.03        | 20.85            | 1.87             | 0.89         | 0.01         |
| EfficientPhys        | SASN                | 0.68       | 0.48       | 2.13        | 4.16        | 0.86       | 0.53       | 0.99        | 0.03        | 17.72            | 1.81             | 0.84         | 0.02         |
| EfficientPhys        | FSAM (Ours)         | 0.88       | 0.50       | 2.29        | 4.17        | 1.25       | 0.63       | 0.99        | 0.03        | 17.42            | 1.87             | 0.83         | 0.02         |
| FactorizePhys (Ours) | FSAM (Ours)         | 0.78       | 0.50       | 2.25        | 4.18        | 1.06       | 0.62       | 0.99        | 0.03        | 21.13            | 2.05             | 0.89         | 0.01         |

To ensure the validity of our main results on PURE, we compared ours and the best-performing SOTA, trained using UBFC-rPPG. 

| Model                | Attention Module | MAE (HR) ↓ |  | RMSE (HR) ↓ |  | MAPE (HR)↓ |  | Corr (HR) ↑ |  | SNR ( dB, BVP) ↑ |  | MACC (BVP) ↑ |  |
| -------------------- | ---------------- | ---------- | ---------- | ----------- | ----------- | ---------- | ---------- | ----------- | ----------- | ---------------- | ---------------- | ------------ | ------------ |
|                      |                  | Mean       | STD        | Mean        | STD         | Mean       | STD        | Mean        | STD         | Mean             | STD              | Mean         | STD          |
| EfficientPhys        | SASN             | 3.39       | 1.58       | 12.59       | 103.66      | 3.65       | 1.35       | 0.84        | 0.07        | 10.27            | 1.17             | 0.69         | 0.02         |
| FactorizePhys (Ours) | FSAM (Ours)      | 0.54       | 0.21       | 1.70        | 1.56        | 0.77       | 0.31       | 1.00        | 0.01        | 15.18            | 0.96             | 0.80         | 0.02         |

The large RMSE STD error of the SOTA method here is due to the low SNR of the estimated rPPG signal, affecting waveform morphology, and thereby the computed HR for the edge cases. As the SNR of rPPG signals estimated by FactorizePhys is higher, its results are further improved. This further underlines the superior cross-dataset generalization that our proposed model with FSAM module is able to achieve.

Given these observations, we will further revise all cross-dataset results for evaluation performed on PURE dataset.

* Bold for 2.78, vs 2.83:  
Apologies for this typo. We will present the revised results with correctly “bolded” entries in the revised manuscript.

[1] Liu, Xin, et al. "rppg-toolbox: Deep remote ppg toolbox." NeurIPS 2024
Please find the code from authors 

https://drive.google.com/file/d/1agSB6Ez4ywA7E-rH3Bn2OOdp4WbVaU3h/view?usp=drive_link
Thank you for responding thoughtfully to my concerns. Overall my main concerns have been addressed so I shall update my score accordingly. 

> If we understand your suggestion correctly, it relates to reporting within-dataset performance.

Yes that is correct, thank you for including these.

Regarding the new Table 7:

- Some of the STD values seem very large, for example the RMSE STD is around ten times larger than the mean. Is this a typo or just a feature of the dataset?

- For the MAE values in "Performance evaluation on PURE", it seems as though Physnet should be bolded since it has 2.78, vs 2.83 which is currently in bold.
The authors answer most of my questions. I especially appreciate the effort of putting in more visualization.  But the concern "It is still unclear why and in what situation the proposed method is better than the SOTA methods" is still not answered well. I understand the experimental results are better than the SOTA but still not clear how the proposed method could gain significant improvement. If it is due to the use of joint computation of multi-dimensional attention, the case study should compare the difference(either visualization or some deep explanation of results) of different approaches that use attention as well.
Thanks for the authors's response. My main concern still exists, which is how the NMF is related to the rPPG task. I think NMF is useful for general tasks, but the relation to rPPG is not that strong. Therefore, I will keep the score.