Title: FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time-series estimation

remote photo-plethysmography

spatial-temporal attention

non-negative matrix factorization
Abstract: Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.


Peer reviews:
comments: SUMMARY:
The paper introduces FactorizePhys that utilizes Non-negative Matrix Factorization (NMF) to decompose voxel embeddings. By integrating FSAM into both 3D-CNN and 2D-CNN architectures, FactorizePhys can estimate blood volume pulse signals from video frames. Through evaluations and comparisons with state-of-the-art rPPG methods, the effectiveness of FSAM and FactorizePhys are demonstrated.

STRENGTHS:
1. Introducing the Factorized Self-Attention Module (FSAM) for computing multi-dimensional attention from voxel embeddings. 
3. Evaluation of FSAM and FactorizedPhys against state-of-the-art rPPG methods
4. Integration of FSAM into existing 2D-CNN-based and 3D-CNN-based rPPG architectures to demonstrate its versatility.

WEAKNESSES:
1. The motivation to use the non-negative matrix factorization is unclear. Why can the factorized matrix be used as the attention? Authors should give more insights about the non-negative matrix factorization and show the attention maps to demonstrate the effectiveness.
2. There are other matrix factorization methods such as SVD and QR. Why is the non-negative matrix factorization chosen?
3. The factorization needs to solve an optimization problem which should do gradient descent steps. In line 216, the Authors only use a one-step gradient for matrix factorization. Is the one-step gradient enough to achieve satisfactory factorization?
4. The method part is not clearly illustrated. e.g., equation 5 is confusing, and the symbols in the equation are not well explained. The symbol mentioned in the main text is not well shown in the main figure.

QUESTIONS:
Please check the weakness part.

LIMITATIONS:
The authors mentioned the limitations of the work and proposed future directions such as time series estimation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

comments: SUMMARY:
The paper presents a novel attention block FSAM devised for handling spatio-temporal data. It is benchmarked against a range of SOTA architectures on a suitable selection of different datasets, and found to perform strongly.

STRENGTHS:
The paper is well presented and clearly structured

The is a good degree of novelty in the proposed architecture, and the application is of high interest.

The experiments are conducted over a well selected range of real world datasets.

A good level of detail is provided on the methodology, and the performance of the proposed approach appears strong.

WEAKNESSES:
While the paper's experimental results are promising, they do not appear to be accompanied by any uncertainty estimates. Including these is crucial to allow the reader to draw meaningful conclusions from the results.  I would refer the authors to question 6 in the checklist at the end of the paper. The answer given to the question 'does the paper report [...] statistical significance of the experiments' is given as "N/A", while the guidelines advise that this answer indicates that the paper does not include any experiments.

Also, while it is acceptable to focus on the intra-dataset performance within the main text, it is crucial to at least include the regular results in the Appendix.

A couple of minor typos:
On line 196:
"device a" -> "devise a"
On 236:
"It's" -> "Its"

QUESTIONS:
Is FSAM anticipated to have significantly wider impact on spatio-temporal applications, beyond rPPG?  

How do the different candidate methods compare in their scalability to higher temporal and spatial resolutions?

LIMITATIONS:
There is a discussion on potential societal impact in the Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed the Factorized Self-Attention Module (FSAM), which jointly computes multi-dimensional attention across spatial, temporal, and channel dimensions using non-negative matrix factorization (NMF). The FSAM is integrated into a new end-to-end 3D-CNN architecture called FactorizedPhys, designed to estimate blood volume pulse signals from video frames. Extensive experiments on multiple datasets demonstrate FSAM's effectiveness.\n\nSTRENGTHS:\nThe overall writing of the paper is clear. The proposed method leverages the strengths of matrix factorization to capture global spatial-temporal context effectively. Specifically:\n\n- Unlike most current methods that compute attention disjointly across different dimensions, the proposed method jointly computes multi-dimensional attention using non-negative matrix factorization to gain the modeling advantage.\n\n-Ablation studies are conducted to assess the different mappings of voxel embeddings in the factorization matrix.\n\n- Model Complexity is analyzed experimentally.\n\nWEAKNESSES:\n-Lack of visualization to show how the results of the proposed method are superior (say, an image of the output to show the learned attention).\n\n-Lack of variance of the results in the experiments table.\n\n-It is still unclear why and in what situation the proposed method is better than the SOTA methods. Say, pick some cases from the model output that show that FactorizePhys is better than the others.\n\nQUESTIONS:\n- Are the methods sensitive to the hyperparameters? Lack of sensitivity analysis.\n\n-The authors could consider visualizing more about the outputs of the models.\n\n-Lack of scalability analysis of the methods, especially when the NMF is adopted. For large datasets, it is valuable to know whether the method is scalable compared to other methods.\n\nLIMITATIONS:\nThe limitations are mentioned in the paper but could be highlighted more clearly in the discussion section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}