Cluster-Phys: Facial Clues Clustering Towards Efficient Remote Physiological Measurement

Authors: wei  qian
kun  li
dan  guo
bin  hu
meng  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 04:26:16

Abstract: Remote photoplethysmography (rPPG) measurement aims to estimate physiological signals by analyzing subtle skin color changes induced by heartbeats in facial videos. Existing methods primarily rely on the fundamental video frame features or vanilla facial ROI (region of interest) features. Recognizing the varying light absorption and reactions of different facial regions over time, we adopt a new perspective to conduct a more fine-grained exploration of the key clues present in different facial regions within each frame and across temporal frames. Concretely, we propose a novel clustering-driven remote physiological measurement framework called Cluster-Phys, which employs a facial ROI prototypical clustering module to adaptively cluster the representative facial ROI features as facial prototypes and then update facial prototypes with highly semantic correlated base ROI features. In this way, our approach can mine facial clues from a more compact and informative prototype level rather than the conventional video/ROI level. Furthermore, we also propose a spatial-temporal prototype interaction module to learn facial prototype correlation from both spatial (across prototypes) and temporal (within prototype) perspectives. Extensive experiments are conducted on both intra-dataset and cross-dataset tests. The results show that our Cluster-Phys achieves significant performance improvement with less computation consumption. The source code will be available at https://github.com/VUT-HFUT/ClusterPhys.

Main paper: 
 introduction: 
 Due to the periodic nature of the human heartbeat [20, 21, 54] , blood volume undergoes corresponding changes, manifesting in variations in the skin's light absorption rate [21, 47] . Although the skin color change is imperceptible to the naked eye, this subtle change in skin color can be recorded by an ordinary camera. Building on this premise, remote photoplethysmography (rPPG) technology [20, 21, 35, 36, 58] has been developed for physiological signal measurements like estimating heart rate (HR), heart rate variability (HRV), and respiration frequency (RF). With the convenience and non-intrusion nature, vision-based rPPG-based physiological measurement has become a research hotspot and has been widely applied in driver monitoring [10] , atrial fibrillation screening [25] , and face anti-spoofing [13, 56] .Early studies primarily analyze the subtle skin color changes with traditional signal processing algorithms, such as blind source separation [14, 33] and color space transformation [3, 49] . However, they heavily rely on prior knowledge or assumptions such as skin reflection model [3, 49] or linear combination assumption [33, 34] , only applicable in well-controlled laboratory environments. Then, data-driven methods [26, 27, 43, 58, 60] dominate this domain. 2D-CNN [23, 32, 41] and 3D-CNN methods [42, 44, 55, 58] usually estimate the rPPG signal from the original video frame and utilize attention mechanism [9, 23] to highlight high-quality facial regions. To Diameter 31% better 0.68x smaller [23] [60][59][30] [43] Figure 1 : Performance and complexity evaluation for crossdataset testing on MMSE-HR dataset . The proposed Cluster-Phys achieves 31% performance gain on RMSE than Phys-Former++ [59] while being 0.68× smaller than CVD [30] . The diameter of the circle indicates the model size.reduce the high computation costs of 3D-CNN, the spatial-temporal map is proposed, and the researchers [26, 27, 30, 31, 38] focus on the design of various architecture to embedding the spatial-temporal maps for better alignment with rPPG signals. To increase the robustness of rPPG estimation, some methods [26, 30] attempt to disentangle the physiological information with non-physiological features from the spatial-temporal map to obtain distilled physiological features. Recently, Transformer [46] is introduced for this field in terms of capturing the spatiotemporal contexts of rPPG [24, 59, 60] .In fact, because different physiological tissue structures, capillary density, and blood flow exist in different facial regions, their light absorption is not constant. This brings huge challenges to modeling the rPPG signals.Considering the subtle changes in the face are imperceptible by the human eye, we magnified the video frame by the magnification methods [48, 51] , thereby better revealing these subtle changes. As shown in Fig. 2 (a), we can see that different facial regions absorb light differently and exhibit different reactions over time. For example, in the first frame, the forehead exhibits a high reaction (i.e., bright) while the right cheek exhibits a high reaction in the 3rd frame. The video-based rPPG estimation aims to capture these subtle changes from all frames. Intuitively, this motivates us to first model the light absorption of different facial regions in each frame and then capture the periodically changing rPPG signals by modeling across frames. To this end, as shown in Fig. 2 (b), we attempt to build compact and representative facial prototypes for each frame to represent the predominant light absorption from different facial regions. Then, the facial prototypes from all frames are used for rPPG estimation. Such a facial prototype-based method enables us to achieve superior performance while using much fewer parameters. As shown in Fig. 1 , our method surpasses the previous SOTA method PhysFormer++ [59] by 31% in terms of the RMSE metric and the GFLOPs number of our method is ×68% smaller than CVD [30] .Specifically, we propose a novel rPPG-based physiological signal measurement framework named Cluster-Phys, which aims to capture the latent periodic physiological signals from all frames through progressive facial ROI prototypical clustering (FPC). As shown in Fig. 3 , in the facial ROI prototypical clustering module ( § 3.2), we aim to iteratively build compact and representative facial prototypes using a "learn-and-cluster" manner. In the learn phase, we first learn a soft assignment matrix through semantic similarity distance measurement between initialized prototypes and facial ROI features. In the cluster phase, we use the soft matrix to adaptively cluster representative facial ROI prototypes. Considering that facial physiological signals change over time, we devise a prototype updating strategy based on the semantic similarity between prototypes and ROI features to update the prototypes. The prototype updating strategy updates facial ROI prototypes of all frames, making it better capture global physiological signal changes for rPPG estimation. The established prototypes are processed in our spatial-temporal prototype interaction module ( § 3.3) where we focus on mining spatial and temporal correlations among prototypes to further enrich them. Considering that rPPG estimation necessitates modeling spatial changes in facial regions, and the temporal smoothness and periodic pattern consistency in rPPG signals, we separately use spatial and temporal interactions to perceive the spatial correlations among prototypes within each frame and periodic pattern consistencies. We recurrently perform the above prototype clustering and interaction processes to build the compact facial representation. Finally, we use the rPPG regression head ( § 3.4) to predict the rPPG signal based on the compact and representative prototypes learned above.In summary, our contributions can be summarized as follows:• To mine informative facial ROI clues for remote physiological measurement, we propose a Facial ROI Prototypical Clustering method to progressively learn and utilize representative ROI prototypes. To the best of our knowledge, it is the first attempt at prototypical clustering in this field. • To capture the spatiotemporal contexts in ROI prototypes, we propose a spatial-temporal prototype interaction module that performs interaction on spatial and temporal separately.• Extensive experiments show that our method achieves stateof-the-art performance on the four benchmark datasets referring to both intra-dataset and cross-dataset testing. 
related work: 
 Deep learning-based rPPG Measurement. In the early years, CNN-based methods [8, 41] with native backbone [37] are proposed to capture time-frequency representation for rPPG prediction. To capture the dynamic changes in temporal contexts, attention-based CNN method [2, 23, 44] and CNN-RNN based methods [15, 57] are proposed capture spatiotemporal features to eliminate the negative effect of head movements. To enhance the robustness of the rPPG predictor against unseen noises, Generative Adversarial Networks (GANs)-based methods [26, 38] are proposed to improve the distinguishability of the rPPG predictor. Recently, with the great success of transformer [17, 18, 52, 53, 63] , Transformer-based methods [24, 60] have been proposed to aggregate long-range spatiotemporal features for rPPG estimation. Different from the above approaches, the proposed Cluster-Phys employs a facial ROI prototypical clustering method to build representative facial ROI prototypes, and uses a spatiotemporal interaction module to mine facial cues in the prototypes for rPPG estimation. Prototypical Clustering for Video Feature Aggregation. Representative and compact feature representation is the pursuit of video understanding [7, 11, 12, 19, 62] , such as group activity recognition [19] , video retrieval [11] . Among these works, prototypical clustering aims to capture representative embeddings for groups of semantically similar instances. For example, Li et al. [19] designed a clustered attention transformer that captures both intraand inter-group relations, thereby constructing better group informative features for group action recognition. In this paper, we focus on rPPG measurement from facial videos. Existing studies in this domain primarily concentrate on modeling subtle and sensitive physiological signals [2, 16, 58, 60] , and mainly design models based on the original video feature sequences. We hypothesize that prototypical clustering can facilitate the extraction of crucial clues related to rPPG, and endeavor to verify this hypothesis in this field. 
methodology 3.1 overview: 
 Let an RGB facial video as X 𝑣 ∈ R 𝑇 ×𝐻 ×𝑊 ×3 , where 𝑇 , 𝐻 , and 𝑊 denote frame number, height, width of the video, respectively. The quasi-periodic pulse signal originates from subtle light reflections of blood vessels under the skin. The remote physiological measurement task aims to predict one-dimensional rPPG signal 𝑠 𝑝𝑟𝑒 ∈ R 𝑇 that can reflect the quasi-periodic heartbeat from facial videos. Considering the motivation outlined in the introduction, i.e., facial ROI is composed of different facial regions and each ROI has different physiological signals, we build the prototypes for the facial ROIs of each frame to learn representative embedding of semantically similar physiological information. To capture the spatial correlations between prototypes with each frame and the periodic variation in the temporal dimension, we design the spatiotemporal prototype interaction to capture its information separately. As illustrated in Fig. 3 , we first transform the video X 𝑣 into an ROI-based MSTmap X 𝑚𝑎𝑝 ∈ R 𝑇 ×𝑁 ×6 , where 𝑁 denotes the detected facial ROI number.Then, we embed the MSTmap X 𝑚𝑎𝑝 to high-dimensional feature X ∈ R 𝑇 ×𝑁 ×𝐷 . Subsequently, we take the X as the input and design a facial ROI prototypical clustering strategy to build prototypes. Then, facial ROI prototypes are fed into the spatial-temporal prototype interaction module to learn the global spatial and temporal correlation of rPPG clues. Finally, we use an rPPG regression head to predict the one-dimensional rPPG signal 𝑠 𝑝𝑟𝑒 ∈ R 𝑇 .In this field, the gap between facial ROI-level features and latent rPPG clues has not been completely elaborately explored. Intuitively, physiological information implicit in facial ROIs is closely related to the rPPG signal. In this work, we focus on the modeling of facial ROI information and discovering crucial semantics from subtle differences in faces. The rPPG estimation is based on the periodic changes in optical absorption of a local tissue with changes in blood volume, corresponding to the heartbeats. Therefore, the facial ROI will exhibit periodic variation within the time cycles.In addition, the facial ROI is composed of different facial regions and each region has different physiological signals. To capture this periodic variation for accurate rPPG estimation, we build the prototypes for the facial ROIs of each frame to learn representative embedding of semantically similar instances.Facial ROI Prototype Initialization. Our facial ROI prototype initialization includes two steps, i.e., facial ROI feature preparation and prototype initialization.Facial ROI feature Preparation. Compared with the background, the face contains wealthy rPPG-related physiological information. In addition, since blood flow in blood vessels under the skin varies in different facial areas, different facial areas need to be analyzed separately. In our work, we detect facial landmarks of the video and divide the whole facial region into multiple facial ROI as MSTmap [27, 29, 30] . Specifically, we detect 6 meta-ROI (i.e., forehead, left upper cheek, left lower cheek, right upper cheek, right lower cheek, and chin) of the face, and generate 𝑁 = (2 6 -1) = 63 informative ROI combination blocks. Then, the average pixel values of each facial ROI are concatenated along the temporal dimension to build the MSTmap X 𝑚𝑎𝑝 ∈ R 𝑇 ×𝑁 ×6 , where 6 represents {R,G,B,Y,U,V} color channels. Finally, we embed the MSTmap 𝑀 to high-dimensional feature X ∈ R 𝑇 ×𝑁 ×𝐷 using a fully connected layer as the ROI feature representation.Prototype Initialization. In view of the previous discussion of motivation, here we implement facial clue clustering. Given the input ROI feature embeddings X ∈ R 𝑇 ×𝑁 ×𝐷 , we adopt the density peaks clustering algorithm to initialize facial ROI prototypes as C ∈ R 𝑇 ×𝐾 ×𝐷 , where 𝐾 = 𝑁 × 𝜌 is the number of facial ROI prototypes, and 𝜌 is the cluster sparse ratio hyper-parameter. Specifically, for the input feature embeddings {𝑥 𝑖 } 𝑁 𝑖=1 ∈ R 𝑁 ×𝐷 of each frame, we first calculate its local density by:EQUATIONwhere 𝜎 (•, •) denotes the Euclidean distance, KNN (𝑥 𝑖 ) denotes the 𝑘-nearest neighbors of 𝑥 𝑖 . Subsequently, we compute the relative distance indicator between 𝜑 𝑖 and 𝜑 𝑗 as follows:EQUATIONIntuitively, 𝜑 𝑖 represents the local density around 𝑥 𝑖 , and 𝛿 𝑖 represents 𝑥 𝑖 's distance to other high-density features. In order to select the appropriate initial prototype (or cluster center), we use the product method to simultaneously consider the local density 𝜑 𝑖 of features 𝑥 𝑖 and its degree 𝛿 𝑖 of isolation from other dense regions. We define the 𝑥 𝑖 's score as 𝜑 𝑖 × 𝛿 𝑖 , and select representative top-𝐾 of them as the learnable facial ROI prototypes C (0) ∈ R 𝑇 ×𝐾 ×𝐷 .Clustering. After getting the learnable prototypes C (0) in prototype initialization, we perform prototype clustering to aggregate rPPG-related information from facial ROI features and build a prototype center. Considering that we aim to mine rPPGrelated information from the input feature X ∈ R 𝑇 ×𝑁 ×𝐷 , we use the Expectation-Maximization (E-M) approach cluster prototype in 𝑀 iterations. As shown in Fig. 3 , in 𝐸 phase, we first construct a soft assignment matrix A (𝑚) to model the correlation between the initial prototypes C (0) ∈ R 𝑇 ×𝐾 ×𝐷 and the input feature X ∈ R 𝑇 ×𝑁 ×𝐷 : 𝐸-phase:EQUATIONwhere 𝑚 ∈ {1, • • • , 𝑀 }, Q 𝐶 ∈ R 𝑇 ×𝐾 ×𝐷 denotes the query vector projected from the facial ROI prototype C, and K 𝑋 ∈ R 𝑇 ×𝑁 ×𝐷 correspond to the key vectors projected from the input features X.The soft assignment matrix globally represents the correlation between prototype and feature, unlike DPC-KNN which only updates clustering using hard-matching with the features of prototype.Based on the soft assignment matrix A (𝑚) from the E phase, we use it to cluster facial ROI prototypes in M phase:EQUATIONwhere V 𝑋 ∈ R 𝑇 ×𝑁×𝐷 indicates the value vectors projected from the input features X. In each iteration 𝑚, this prototype clustering strategy iteratively learns the soft assignment matrix A (𝑚) in 𝐸 phase and use it to cluster the facial ROI prototypes C in 𝑀 phase. This dynamic "learn-and-cluster" strategy can continuously learn the soft assignment matrix between prototypes and the facial ROI features to build better prototypes.Based on the aforementioned prototype clustering strategy, we obtained the facial ROI prototypes by considering the soft assignment correlation between input features and initial learnable prototypes. However, considering that facial physiological signals change over time [34, 39] , there will be significant differences between facial ROI prototypes at different times. Therefore, we need to make the prototype more representative of facial ROI prototypes at all times so that it can better capture global physiological signal changes for rPPG estimation. To this end, we propose the prototype updating strategy based on the semantic similarity between prototypes and input features. Specifically, for each time 𝑡, we use the cosine similarity to calculate the semantic similarity S ∈ R 𝑁 ×𝐾 between the facial ROI prototypes C 𝑡 ∈ R 𝐾 ×𝐷 and input features X 𝑡 ∈ R 𝑁 ×𝐷 :EQUATIONwhere S 𝑖,𝑗 denotes the similarity between 𝑖-th features 𝑥 𝑖 in X 𝑡 and 𝑗-th prototype 𝑐 𝑗 in C 𝑡 . To expand the influence of semantic closer input features to the prototypes while also avoiding potential flaws from irrelevant features, we apply the nearest matching strategy to divide the input features into different prototype clusters and update the prototypes based on the intra-cluster associations. We define 𝑆𝑒𝑡 (𝑐 𝑗 ) as the intra-cluster features, which denote the set of input features assigned to the prototype 𝑐 𝑗 . Next, we use a similaritybased weighting fusing method to update the prototype from the clustering level. For intra-cluster features, features close to the prototype should have a greater impact on clustering and require greater weight. Therefore, we calculate the weights of prototype and intra-cluster features respectively according to the semantic similarity S:EQUATION)For each 𝑐 𝑗 in C 𝑡 , we use weights 𝑊 𝑖 and 𝑊 𝑗 to update its feature:EQUATIONFinally, for each frame 𝑡, we can get the updated facial ROI prototype C ′ 𝑡 = {𝑐 ′ 𝑖 } 𝐾 𝑖=1 ∈ R 𝐾 ×𝐷 . Therefore, the video-level prototype representation is C ′ ∈ R 𝑇 ×𝐾 ×𝐷 . To gradually mine informative rPPG clues to update the prototypes, we perform 𝐿 steps for facial ROI prototypical hierarchically. The spatial-temporal interaction module is followed by each facial ROI prototypical clustering block.After the facial ROI prototypical clustering mentioned above, we obtained the compact prototypes C ′ that enhanced the representation of the facial ROI. Considering that rPPG estimation necessitates spatial-temporal modeling changes in facial signals, it is required to learn the spatial correlation within these facial ROI prototypes. Furthermore, to maintain temporal smoothness and periodic pattern consistency in video-based rPPG estimation, temporal interactions of facial ROI prototypes are also required. We employ a Prototype Interactor to capture both spatial and temporal contextual information. 1). For spatial-wise interaction, we first divide C ′ into C ′ 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 = {c ′(𝑡 ) ∈ R 𝐾 ×𝐷 |𝑡 = 1, . . . ,𝑇 }, then feed it into the ProtoInter(.) operator to model the spatial correlation between 𝐾 prototypes at each frame. 2). Similarly, for temporal-wise interaction, we convert C ′ into C ′ 𝑡𝑒𝑚𝑝𝑜𝑟𝑎𝑙 = {c ′(𝑘 ) ∈ R 𝑇 ×𝐷 |𝑘 = 1, . . . , 𝐾 }, then feed it into the ProtoInter(.) operator to model temporal correlation between 𝑇 frames of each prototype. Suppose the facial ROI prototype features C ′ ∈ R 𝑇 ×𝐾 ×𝐷 . The ProtoInter can be formulated as follows. The input C ′ (C ′ 𝑠𝑝𝑎𝑡𝑖𝑎𝑙 and C ′ 𝑡𝑒𝑚𝑝𝑜𝑟𝑎𝑙 ) of Eq. 8 is determined by spatial-wise or temporal-wise operation.EQUATIONwhere MSA, LN, and FFN are multi-head attention, layer norm and feed-forward layer in [46] .From the entire Facial ROI Prototype modeling, we obtain the final facial ROI prototypes C ′′ ∈ R 𝐾 (𝐿) ×𝑇 ×𝐷 . Next, we use an rPPG regression head consisting of a spatial average pooling layer and a linear projection layer to predict 1D rPPG signal 𝑠 𝑝𝑟𝑒 ∈ R 𝑇 based on the prototype features C ′′ . Following the practice [26, 30, 60] , the standard Negative Pearson Correlation loss is used to minimize error between 𝑠 𝑝𝑟𝑒 and ground truth rPPG signals 𝑠 𝑔𝑡 ∈ R 𝑇 :EQUATIONwhere 𝐶𝑜𝑣 (𝑥, 𝑦) denotes the covariance of variables 𝑥 and 𝑦. Implementation Details. For HR estimation, following previous work [24, 60] , we apply a 1st-order Butterworth filter to convert the rPPG signal into an HR value with a cutoff frequency range of [0.75Hz, 2.5Hz], corresponding to [45, 150] beats per minute. Subsequently, we perform the PSD [50] to estimate HR values for video Finally, we get a video-level HR by averaging HR values from all segments. 
intra-dataset testing: 
 HR Estimation on UBFC-rPPG. We first evaluate the proposed method on the UBFC-rPPG dataset, which has simple scenarios and high-quality data. As illustrated in Table 1 , the proposed method outperforms both traditional and deep learning-based methods.Compared with state-of-the-art Dual-GAN [26] , our method significantly reduces MAE and RMSE by 43% and 25%, respectively. Furthermore, compared to the recent supervised version of Contrast-Phys+ [43] based on contrastive loss, we have achieved a leading advantage in most metrics. These noteworthy advancements underscore the effectiveness of our clustering strategy in capturing informative rPPG clues, leading to a notable improvement in HR estimation accuracy. HR Estimation on PURE. Unlike the UBFC-rPPG dataset, the extensive head movements in PURE significantly disrupt rPPG acquisition, posing significant challenges. Nevertheless, our model achieves substantial improvements, such as a 53% lower RMSE compared to Dual-GAN [26] and a 62% reduction compared to Li et al. [22] . These results indicate that our clustering strategy effectively consolidates crucial facial signals even in complex environments, successfully minimizing the impact of external disturbances. HR Estimation on VIPL-HR. The large-scale VIPL-HR dataset is known for its extremely complex external noise challenges, such as head movements, talking, and dark and bright scenarios. Following the protocol in [27, 29, 60] , we conduct the subject-exclusive 5-fold cross-validation. As shown in Table 1 , we can see that our model surpasses previous state-of-the-art NEST (MAE of 4.76 bpm and RMSE of 7.51 bpm) by a large margin, indicating the effectiveness of Cluster-Phys in mitigating the impact of various noises. While noise-induced variation may be more pronounced than rPPG signals, our aggregation strategy allows the model to focus on more high-quality facial cues and filter the noisy information.HRV Estimation on UBFC-rPPG. In addition to HR estimation, we also conduct the heart rate variability (HRV) and respiration frequency (RF) estimation. HRV and RF estimations require accurately measured high-quality rPPG signals. As shown in Table 2 , we can see that the proposed approach outperforms the existing state-of-the-art traditional methods by a large margin.Cross-dataset testing is essential for assessing the model's generalization capabilities in unseen scenarios. Following previous works [24, 26, 60] , we conduct three cross-dataset tests . The experimental results are reported in Table 3 . PURE→UBFC-rPPG. Since the PURE dataset is much more complex than the UBFC-rPPG dataset, this adaption process could be relatively easy. Compared with state-of-the-art Dual-GAN [26] , our method achieves further breakthroughs in all metrics. Notably, the RMSE of our method is lower than 1 bpm, indicating that the predicted HRs are closely aligned with the ground truth HRs. UBFC-rPPG→PURE. From Table 3 , we observe that all deep learning-based methods performed poorly, with RMSE exceeding 10 bpm. When compared to the complex to simple test (PURE→UBFC-rPPG), the significant performance degradation in this mode may be attributed to the introduction of head movement scenarios in the PURE dataset. In such situations, the methods struggle to extract rPPG signals from chaotic facial features, resulting in inaccurate HR estimations. In contrast, our approach attains an RMSE lower than 10 bpm and a Pearson 𝑟 close to 1, significantly outperforming previous methods. These results show that our clustering strategy prefers to learn the intrinsic rPPG clues of face regions.VIPL-HR→MMSE-HR. In this cross-dataset test, VIPL-HR contains different motion patterns and light changes, while MMSE-HR contains diverse facial expressions. As shown in Table 3 , our method achieves the lowest MAE (1.74), lowest RMSE (3.51), and highest 𝑟 (0.96), respectively. Compared to the EfficientPhys-T1 [24] and PhysFormer family [59, 60] based on spatial-temporal Transformer, our Cluster-Phys has significant advantages across all metrics. We attributed it to the proposed clustering strategy that can extract representative rPPG-aware prototypes, which shows robustness in this challenging cross-dataset test.Impact of Main Components. As shown in 3.87. This result reflects that facial prototype clustering can capture crucial rPPG-related clues for rPPG estimation. When incorporating the prototype cluster, the model will perform better as it learns better features from the cluster. From rows 3-5, we can conclude that both the spatial prototype interaction and temporal prototype interaction can help the model learn the spatial and temporal correlations for boosting rPPG estimation. Furthermore, we employ Bland-Altman plots to analyze the data consistency between our predicted and ground-truth HR. As shown in Fig. 6 , we can see that our facial ROI prototypical clustering strategy contributes to better data consistency, i.e., smaller standard deviation. Since the evaluated VIPL-HR dataset has extreme noise, such as head movements and light flickering, therefore the model without FPC leads to a significant standard deviation from the ground truth. Impact of Clustering Strategy. We evaluate different clustering strategies, i.e., "Random", "K-means", "DPC", and the proposed "FPC". As shown in (1) Visualization of Feature Clustering. As depicted in Fig. 4 , we visualize the prototypes in our facial ROI prototypical clustering model. Given a video, in the 1st stage, 63 meta-ROIs are clustered into 32 prototypes, we can see that (a1) captures the forehead and chin, while (a2) captures the left cheek, the right cheek, and the chin. In the 2nd stage, (b) captures the most representative ROIs, i.e., the left cheek and the right cheek. Similarly, in the 3rd and 4th stages, (c) and (d) capture the left and right cheek. The facial prototypical clustering strategy adaptively aggregates the most representative ROIs to construct compact prototypes for rPPG estimation.(2) Robust HR Estimation Cases. We visualize four complex scenarios in Fig. 5 to verify the robustness of the model. From examples (a)-(c), we can obviously observe that not only on rPPG periodicity but also on the continuity and smoothness of the rPPG curve, our Cluster-Phys performs much better than PhysFormer [60] . It indicates the good robustness and accuracy of our method. In addition, we also give an extremely complex rPPG measurement scenario with both head movement and occlusion noises in Fig. 5 (d ). Despite the challenges we encountered with our approach, it still outperforms PhysFormer.As shown in Table 6 , we give detailed statistics on the complexity of the model of our method and open-source methods. Compared to the CNN-based method EfficientPhys-C, our method significantly outperforms it in RMSE with lower GFLOPs and fast inference speed. Similarly, compared to the Transformer-based method Phys-Former, our method consumes only 5.28% GFLOPs and achieves better performance. In summary, these results validate the lower complexity and superior performance of our method. 
conclusion: 
 In this paper, we propose Cluster-Phys, a novel facial prototypical aggregation architecture for remote physiological measurement. We propose a Facial ROI Prototypical Clustering (FPC) module to progressively mine and aggregate facial ROI clues to build prototypes. To capture the spatial correlations among prototypes within each frame and ensure the temporal periodic pattern consistency in rPPG signals, we devise a spatial-temporal prototype interaction module that separately perceives these correlations and consistencies. Extensive experiments conducted on both intra-dataset and cross-dataset tests show that our Cluster-Phys achieves new state-of-the-art performance. 
 
 Appendix:  


Decision: All reviewers agree that the proposed method is interesting and innovative. All reviewers comment that the facial ROI Prototypical Clustering method to progressively learn and utilize representative ROI prototypes is novel and contribute significantly to the increased performance of the presented approach. The reviewers are satisfied with the presented experimental study. The rebuttal addressed a large majority of additionally raised questions. The authors are advised to include the relevant explanation in the final version of the paper. After reading the rebuttal, two of the reviewers upgraded the final score. Given a general appreciation of the work by the reviewers, I believe that the paper will be of interest to the audience attending ACM MM and would recommend a presentation of the work as a poster.

Reviews: review: paper_summary: paper_summary: This presents a framework for remote physiological measurement using video-based analysis of facial regions. This work proposes to use prototypical clustering, focusing on facial regions of interest (ROIs) to capture the subtle changes in the face. Experiments are done on both intra-dataset and cross-dataset to show its effectiveness.
strength_weakness: strength_weakness: 1. The use of prototypical clustering for rPPG signals is a novel approach that effectively captures critical physiological signals.
2. The method outperforms existing models in terms of both accuracy and computational efficiency.
limitations: limitations: 1. The initialization process for prototypes in this study involves using an MSTmap, which is an aggregation of average pixel values from each facial region of interest (ROI) over time. While this method simplifies the input data by reducing its dimensionality, it also raises concerns about the potential loss of critical spatial information. Averaging pixel values can smooth out fine details that might be crucial for detecting subtle physiological changes.

2. The “learn-and-cluster” strategy mentioned in the paper appears to draw inspiration from the deep clustering technique described by Caron et al. (2018). How about directly apply deepcluster on this task? This could be a baseline.
 [1]Caron, Mathilde, et al. "Deep clustering for unsupervised learning of visual features." Proceedings of the European conference on computer vision (ECCV). 2018.

3. The paper could benefit from a more detailed discussion on scenarios where Cluster-Phys underperforms or fails, providing insights into potential limitations or areas for improvement.  For example, how it performs when landmark is incorrectlty detected thus ROI is wrongly determined.
paper_summary: paper_summary: This paper proposes a novel clustering-driven remote physiological measurement framework (Cluster-Phys), which employs a facial ROI prototypical clustering module to adaptively cluster the representative facial ROI features as facial prototypes and then update facial prototypes with highly semantic correlated base ROI features. Experimental results on four public datasets illustrate that the proposed method is superior to these existing methods.
strength_weakness: strength_weakness: Compared with most existing methods, this work proposes a new rPPG measurement framework, Cluster-Phys, which is a facial ROI Prototypical Clustering method to progressively learn and utilize representative ROI prototypes. The authors also mentioned that it was the first attempt at prototypical clustering in this field. This idea sounds novel. And extensive experiments have been conducted to validate the performance of the proposed method.
limitations: limitations: 1. Although the framework of the proposed method (Cluster-Phys) is drawn clearly and a lot of experiments are implemented to illustrate the performance of Cluster-Phys, the descriptions for Cluster-Phys are not clear. The writing should be earnestly modified. Additionally, in the first page, Fig.1 is given but the introduction for Fig.1 is missed in the main context.   
2. In Section 3.3, how to achieve the prototype interaction? Give more details for this. 
3. In Section 3.2, is the feature of each input image updated with the network learning? How to set the number of prototype clustering? 
3. In Fig.6, why are the sizes of inputted figures consistent? The size of the middle figure is bigger than others.
paper_summary: paper_summary: The paper proposes a novel rPPG-based physiological signal measurement framework named Cluster-Phys, which aims to capture latent periodic physiological signals across all frames through progressive facial ROI prototype clustering. The effectiveness of the proposed method is validated through experiments in Intra-dataset Testing and Cross-dataset Testing.
strength_weakness: strength_weakness: 1. The experiments are thorough and show excellent results, achieving better performance with less computational cost.
2. Code release is promised.
3. The proposed method based on prototypical clustering can inspire other researchers in the community.
limitations: limitations: 1. The selection of facial ROIs is crucial. It is currently unclear how 63 ROI combination blocks are derived from the 6 meta-ROIs. Additionally, why are these 6 meta-ROI regions selected as the criteria? How would other ROI criteria, such as facial action unit ROIs, perform? These ROI selections might provide a more detailed facial region analysis.
2. The description is unclear. In lines L268 and L322 of the manuscript, it is not clear what D represents in T × N × D.
3. The manuscript lacks a deeper discussion of the method, such as its 'limitations'.