Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a synthetic dataset for visual physiological signals estimation. Physiological signals supported include respiration and heart rate markers. Authors demonstrate viable physiological markers estimation on top of their visual synthesis flow.

strengths: - Supporting scalable digital health is an urgent societal need. And although this early work is not perfect, it is a very worthwhile endeavour
- High-fidelity modelling and data synthesis with open-source Blender  
- Good volume of data
weaknesses: - while initial distribution of supported physiologies in Fig. 6 is very good, a wider range of normal sinus rhythm (e.g., below 50 and above 150) could make the dataset more representative---as will the inclusion of more exotic morphologies (e.g, arrhythmia)
- limited evaluation (cf. Table 2)

correctness: The physiological modelling assumptions and the synthesis flow are reasonable.
clarity: Paper is well written.
relation to prior work: Work is well positioned.
additional feedback: Extending this initial effort with QRS-complex-/morphology-accurate modelling could make possible future work beyond vision-based estimation, e.g., by using 3D Blender models (pre-rendering) for say ray-tracing wireless responses.
For more information on morphologies to support in the future, see ECG-related literature, and commercial offerings for typically supported physiological ranges in the medical domain:
[1] A. T. Reisner, G. D. Clifford, and R. G. Mark. The physiological basis of the electrocardiogram. Artech House Publishers, 2006.
[2] Rigel Medical. Rigel 333 ECG Patient Simulator. http://www.rigelmedical.com/products/simulators/ ecg- patient- simulator.
[3] Laerdal Medical Limited. SimMan Patient Simulator.
http://www.laerdal.com/gb/doc/86/SimMan.
review_title: Synthetic visual physiological signals dataset

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a synthetic face-centric video dataset, with physiological- and biologically-plausible properties.
To achieve this, the authors start with facial scans of real subjects, and render synthetic videos with a computer graphics pipeline by including physiological signals (e.g., heart rate) as well as human movements (e.g., breathing movements, face movement...).
While interesting, the dataset would benefit from better insights in the rendering process and distribution information about its properties.
strengths: (+) The paper proposes an interesting synthetic dataset with videos of faces, where physiological signals are present. It is quite unique as it combines rendering techniques from computer graphics with physiological signals to make the human faces more realistic.

(+) The paper evaluates the pertinence of such dataset on existing models and show its utility on existing evaluation datasets for the task of heart rate prediction.
weaknesses: (-) The dataset lacks evaluation on its physiological plausibleness
* While the paper claims to be physiological plausible by including physiological signals such as the heart rate or breathing movements, there is no evaluation to assess this physiological plausibleness.
* What would be the next steps to make the data more plausible? e.g., would ballistic forces from the aorta be useful?

(-) The dataset lacks details on subject and scene parameters distribution
* While the paper claims to stem from "a range of skin types/tones, genders and ages", it doesn't provide any information on these demographics statistics regarding the original facial scans or the rendered videos in the proposed dataset. Such information matters to understand any potential fairness bias of the dataset or audit potential model issues (e.g., heart rate is usually more challenging to perform on darker skins).
* It would be interesting to also know the distribution of the background scenes used, as well as the illumination intensity. Such information would inform about the diversity of the scene and could help understanding potential model issues (e.g., not working well in night time).

(-) The dataset lacks explanation regarding the rendering process
* It would be appreciated if some parameters choice would be better justified though (e.g., heart rate in Sec 3, or subsurface scattering coefficients in Sec 4).
* On a side note, would the rendering engine be open-source for the community to generate their own dataset? (e.g. it could be useful to understand the causal effect of some parameters by changing them one by one rather than having random parameters for every video).

(-) The limitations and terms of use could be expanded regarding physiological information
* For example, it is possible to infer a person identity from the QRS complex. Such potential usage should be discussed.

(-) No comments on subject informed content
* Given that the dataset stems from facial scan of humans, it is important to know whether they provided their informed consent for the dataset creation.
correctness: Yes.
clarity: Yes. The paper is well written and easy to read.
relation to prior work: Yes. It is appreciated that the paper has trained several related models on the proposed dataset.
additional feedback: n.a.
review_title: The paper proposes an interesting synthetic dataset of face videos with physiological signals, but would benefit from a better documentation.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces a synthetic multi-modal dataset that can useful for training models to measure physiological signals using face-captured videos. The authors have used their domain expertise to embed subtle changes in the skin tones of an avatar as breathing and other physiological signs vary. This way one can obtain a perfectly synchronized multi-modal dataset to train the models for measuring physiological signals. The dataset of 2800 videos (1.68 million frames) covers a range of environments, body motions, illumination conditions, and physiological states. The authors have also ensured the dataset is diverse in terms of skin tone, gender and age. Since this is a synthetic dataset, it is also accompanied with segmentation masks for hair, bread, skin, etc. They show that a model trained on the synthetic dataset has decent generalizability on the real-world dataset. 
strengths: 1. Valuable contribution: A multi-modal dataset which is perfectly synchronize to measure physiological signals (using only videos) is challenging to capture. Therefore, paper's contribution of a large synthetic dataset in this field is valuable for the community. In addition to training models to measure physiological signals, this dataset can be used to train/pre-train segmentation models, multi-modal self-supervised models, conditional GANs, etc.
2. Non-trivial synthetic dataset: Creation of this dataset is non-trivial. To create this dataset, it requires diverse know-how of the interactions between lighting conditions, environmental conditions, skin tones and physiological signals. Additionally, the computational resources required for creating this dataset is also not easily available (24 machines each with an NVIDIA M40 GPU running for 720 hours each)
3. Diverse dataset: The dataset of 2800 videos (1.68 million frames) covers a range of environments, body motions, illumination conditions, and physiological states. 
4. Overall paper quality:  The paper is well written. It anticipates questions of the reviewers and addresses them in the manuscript.

weaknesses: 1. Although the entire pipeline for generating the synthetic dataset is explained in the paper, it would not be enough to reproduce it. Would it be possible for the authors to share a detailed code/ step-by-step guide for generating the dataset? This can/may include steps that require external tools/library like Blender.org.
2. To alleviate any ethical concerns with respect to gender, race or age biases, it would be nice if the authors can include their distributions in the appendix (separate distribution for gender, race and age should work)
3. Since there are some real world datasets, it would interesting to see if a model pre-trained on this synthetic dataset and then finetuned on the real world data is better than a model just trained on the real-world data. This would further emphasize the utility of the dataset if models pretrained on synthetic dataset out-performs other pre-trained/ trained from scratch models. 
correctness: Yes. 
clarity: Yes.
relation to prior work: Yes.
additional feedback: None
review_title: Diverse synthetic multi-modal dataset to measure physiological signals

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The works presents SCAMPS, a synthetic dataset for estimating physiological signals (PPG, pulse, breath) from human avatars. The authors provide 2800 high quality videos with a diverse range of simulated human faces and environments. The authors introduce facial action units and multiple types of head movements (side-to-side, up and down), which are influenced by breathing.


strengths: The proposed a novel synthetic dataset is diverse in terms of populations, environments and confounding factors. The dataset is large, having 2800 RGB videos, totalling 1.68M frames. 

The authors took care into providing simulated semi-realistic physiological patterns in terms of PPG, pulse rate and breathing, which are synchronized with the videos. Benchmark results showcasing direct transfer to real-world datasets imply a promising direction in the field.
weaknesses: The authors simulate heart-rate variability by applying random perturbations to the beat timings. However, there is also a relationship between breathing and heart rate, in the form of respiratory sinus arrhythmia, in which the heart rate synchronizes with the breathing. Moreover, the heart rate increases slightly on the inhale, and slows down on the exhale, which I’m assuming affects photoplethysmography readings in some subtle ways. 

It was not clear to me if the heart rate and breathing patterns are linked in the simulated videos. If not, why? 

The videos have a heavy focus on side-to-side rotations of the head. Why isn't there more variability with regards to the camera angle (not only horizontal but also vertical angles)?
correctness: The dataset is constructed in a sound way. Benchmarks do not have explicit accompanying code, but the authors indicate an external tool to reproduce the results.
clarity: The paper is clearly written.
relation to prior work: The authors clearly discussed in the Introduction the differentiating aspects between their proposed synthetic dataset and existing real-world datasets.
additional feedback: Overall, it is a good contribution, being the first synthetic dataset designed for physiological signs estimation.
review_title: A novel synthetic dataset for physiological signals estimation from RGB videos

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors proposed a synthetic dataset that could lower the costs of collecting physiological data using camera measurement methods.

This dataset is especially relevant as, in real-world scenarios, the synchronization of multimodal data sources is a challenge.

Moreover, the authors own the infrastructure for the synthesis of this dataset. Future updates are a possibility.
strengths: The proposed dataset claims that no personal information is shared.

Moreover, the dataset includes occlusion masks that can help experimental segmentation designs.

It is worth noting that the authors illustrate generalizability on a real-world public dataset in their experiments.
weaknesses: Despite the clear contributions of this dataset towards new methods in camera measurement of physiological signals, the Authors should discuss some issues further.

One of such discussion points should be simulating scar tissue and its potential impact on heart rate measurement.

Moreover, makeup or other skin augmentation such as tattoos or piercings could be included and mentioned explicitly.
correctness: The claims presented in this paper are not exaggerated. Authors build upon some existing work and provide valuable material.

Unfortunately, the code that the Authors used for experiments was not published.
clarity: Although most of the article is correct, the authors could address minor clarity issues. Unfortunately, the authors provided a version of the article that does not contain line numbers, so it makes pointing them out a little challenging nevertheless, please refer to the issues below:
- The "Introduction" section contains a minor stylistic issue, where the word "physiological" is used twice in the same sentence.
- Table 1 contains datasets that were previously available. When showcasing the available variables, two are similar enough that they might be discussed in more detail. This is especially relevant in the case of the MAHNOB dataset containing "Breath - Breathing waveform" and "BR - Breathing rate". It seems possible to calculate breathing rate from the breathing waveform. This could be discussed further.
- Section 4 "Video Synthesis", paragraph "Facial actions" seems to contain a typo: "lid tightened" instead of "lip tightened". Moreover, the same paragraph mentions that the system used for action coding is widely used. There seems to be a lack of citation towards this system or its full explicit definition. Pointing readers toward this system documentation might be a good step toward extending the clarity of this work.

Within the article, there seem to be a few instances where past tense could be used, but it is a matter of stylistic preference rather than the merits of the introduced dataset. On the other hand, I would recommend reviewing the whole article once again and verifying if the authors think the writing could be improved. In some instances, "will be publicly available" should be "are publicly available".
relation to prior work: Authors claim their dataset is a first attempt at generating synthetic data for camera measurement of physiological signals - verifying that information is impossible. A straightforward comparison is made to other camera measurement physiological datasets.
additional feedback: Questions towards this work would be as follows:
- Can you provide the code that was used for baseline experiments.
- Will the infrastructure for generating this dataset be published as open-source?
- Do you plan on including skin augmentation and makeup in the future version of this dataset.
review_title: Very good approach at generating synthetic physiological data.

Meta-review: metareview: The reviewers are positive regarding the high level of the contribution of the work for the NeurIPS 2022 Track Datasets and Benchmarks. The authors properly addressed all reviewers comments and concerns during the rebuttal period.
confidence: 5: The area chair is absolutely certain
Decision: Accept