Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for extracting the blood volume pulse signal via photoplethysmography from a video.  This use case of computer vision has several positive applications, that could help make scalable physiological sensing possible.  The authors do a thorough job summarizing the existing literature, including many of the most recent papers.  While the summary could be made a little more readable, I think it is a well synthesized introduction to the paper on whole.  The proposed method uses existing but computationally efficient neural layers to extract a PPG signal from the video.  The results overall appear reasonable and the method performs well compared to the baselines in the comparisons that are given.

strength_and_weaknesses: The authors correctly highlight deficiencies with the existing public datasets which are invariably small and limited in diversity. They present a new dataset that they collected which is arguably larger than the existing data. However, their dataset is also limited in diversity with regard to skin types and age, based on the description. I think their claims that their dataset is particularly large in comparison to existing datasets is not really true - it is comparable in order of magnitude. I think that those claims should be tempered a bit. It was not very clear to me whether the dataset will be made available as a benchmark set?  Please clarify - making the contribution of the dataset a central claim in the paper without releasing it is frustrating to readers as it doesn’t help to advance the communities’ efforts, beyond the insights gained from a single paper.

The authors state that for some existing methods the "input batch size is 128 frames, which means it has a latency of over 1s. To get a real-time BVP signal, the latency should be as small as possible." However, it is not that clear to me in what application a delay of 1 or even a couple of seconds would be a problem.  In heart rate estimation a period of at least 5 seconds would be need at least any way.  Can the authors clarify why a latency of 1 second is significant here?  I can see the memory or compute constraints with a longer temporal window being more significant which may better support the authors arguments here. 

The approach to solving the problem does appear to be novel.  It is interesting that the authors collapse the input to RGB vectors thereby losing spatial information, which is often useful for segmentation.  I would question how sensitive their method is the any face detection or segmentation step they perform. I would appreciate a comment on this, again with little visibility on the dataset they collected it is hard for a reader to ascertain this.  To add to this point, the paper is short on experimental details and while the supplementary material contains model implementations, it does not actually include end-to-end training code which is disappointing as their are often details in steps prior to, or following, a model that impact performance (e.g., segmentation steps, filtering steps etc.). I would appreciate the authors comments on any additional details that could be provided to help ensure reproducibility.

Section 5:  It is still a bit unclear to me after reviewing the paper why the proposed method cannot be trained on other datasets.  Yes, some datasets do not have very highly synchronized PPG and video, but some do.  In my experience, for example, training models with PURE or SCAMPS is possible.  Can the authors be more precise about why training with those existing datasets isn’t possible? I would appreciate a clearer indication or examples to illustrate this.  
And in Table 4 - why wasn’t EfficientPhys, PulseGAN or DualGAN trained on the collected dataset?  Is that because those models were not implemented and the results taken from prior work?

The above point applied to training, for testing it would seem that the authors could use any dataset.  Why did the authors only choose to test on UBFC and not any of the other benchmark datasets?  This is a significant weakness of the work as almost all other papers manage to test on at least 2-3 public datasets.  Could the authors produce results on these other datasets? 

In Section 5 the authors write: Unlike the time spent per frame in Liu et al. (2020), we calculated the time spent to process a whole batch, which is more reflective of the latency caused by the computation. However, while this is true about latency, it doesn’t reflect the computational cost which I think is more the point (see the comment above about the difference between latency and computational cost). I don’t think latency is really a bottleneck in this application, but computation is. 

clarity,_quality,_novelty_and_reproducibility: I have some concerns and the reproducibility and as a related topic, to the comprehensiveness of the results, as described  in the previous section.
summary_of_the_review: Overall, I believe this application is very exciting.  I think that the proposed method has some novelty and focusing on fast/computationally light algorithms is good.  The paper has the makings of a good contribution that I’d like to support.  However, I feel like the paper as it is currently falls short in empirical support for their method and theoretical support. If the authors can address all these points above - I would be willing to increase my rating.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a simple method, followed by a dataset for remote PPG/BVP monitoring. The proposed method is a simple neural model with reasonable results, and the dataset has 58 participants.
strength_and_weaknesses: The main strength of the paper is the dataset component of the work. However, the paper, unfortunately, has a number of weaknesses that needs to be addressed. These weaknesses are as follows:

1- The writing, presentation, and editing are not very polished. 

2- The paper is very hard to follow. I find it hard to put my finger on what exactly the contributions of the paper are.

3- The proposed method is very simple and trivial, and does not provide any new "representation learning" insights.

4- The paper cites several well-known datasets in the area, including MAHNOB-HCI, VIPL-HR, and others, some of which have more participants and are more diverse. It is therefore not clear why a new dataset was needed.

5- The dataset is not very diverse in terms of participants, and many details are missing, e.g. M/F ratio, age range/SD, etc

6- There is no confirmation that ethics approval has been secure, which is really important.

7- The dataset is not made public (and doesn't seem to be planned to go public). So I am not sure if it can be considered a contribution to the area. 
clarity,_quality,_novelty_and_reproducibility: The work requires polishing and editing, and perhaps a re-write in a few areas. The contributions are not clear. The methodology is trivial, and many details are missing.
summary_of_the_review: Please see my review above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a new dataset for blood volume pulse (BVP) estimation which addresses some shortcomings of existing datasets. A method to estimate BVP from RGB frames is also introduced. Results on the collected dataset and other publicly available datasets are presented and the proposed approach achieves competitive performance without requiring complex pre-processing.


strength_and_weaknesses: Strengths

A new dataset is presented which will be a nice contribution (if it becomes available).

Weaknesses

A major weakness in the paper is that it’s not easy to follow the discussion. There are no references to tables or figures, so the reader is left searching for the corresponding tables/figures.

The introduction reads more like a review of existing methodologies, it’s hard for the reader to understand what the main contributions are since they are described in different paragraphs. It would be good to have a paragraph which lists the main contributions. For example the end of the last paragraph could be extended to summarise all contributions.  

An ablation study is missing in the paper. This would reveal the impact of each component.
For example, the authors use an image of 8 by 8 in order to alleviate privacy concerns. However, this is a very low resolution and it would be good to test the performance of the proposed approach as a function of the input resolution. Another example is studying the impact of the number of convolutional and spectral layers and the impact of the input sequence length.

One of the main contributions of this paper is the introduction of a new dataset, however only a brief description of the dataset is given and it is not mentioned if the dataset will become publicly available. The latter Is quite important as if the dataset is not released it diminishes the paper’s contribution. It would also be useful to add more details in the dataset description, e.g., age/gender distribution, did all subjects do all tasks, total number of videos and duration of the entire dataset.

clarity,_quality,_novelty_and_reproducibility: As mentioned above it is not easy to follow the results discussion. The paper also needs proofreading. There are several typos and sentences which need to be rewritten. A non-exhaustive list from the introduction only is the following (but all sections need proof-reading):

(ICA), In Poh et al. (2010b), -> (ICA). In Poh
new color space and separates -> …which separates
separate present -> maybe separately present (?)
facial image segmentation can limit its deployment on mobile devices because it is currently difficult to achieve on mobile devices effectively. -> this sentence needs to be rewritten
movements. the large size -> movements. The large

The authors provide several details about their method but still the results are not fully reproducible since the training and test sets are generated randomly.

summary_of_the_review: There are several weaknesses as explained above which need to be addressed before the paper can be considered for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed a new method for extracting the blood volume pulse signal via photoplethysmography from a video, which is simple, fast and accurate. While the proposed technique may be practically useful, reviewers raised major weakness concerns about the incremental contributions, limited novelty, weak experimental results and lack of clarity in the presentation. While the authors tried to give responses to address the review questions, the overall quality of this work is still below the acceptance bar.
Decision: Reject