Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a dataset for benchmarking the performance of ECG and PPG waveform imputation methods by leveraging publicly available ICU waveform data and using realistic missingness patterns to create simulated gaps to be imputed. While I believe that some of the baseline model choices don’t make sense, overall, the paper is well written and timely. They share the waveform data along with missingness masks, along with train/validation/test splits so that others can easily compare imputation methods to baseline model performance. Several baseline models from the literature were implemented and tested, along with a novel transformer model to serve as an additional baseline. 
strengths: Strengths of the dataset include the scale and the realistic masking procedure used to generate missingness that tries to closely mirror patterns seen in mHealth systems. I also appreciate that a downstream prediction task is also included in the dataset, since often users care about how imputation quality will affect application performance. Many of the potential concerns I had with the dataset selection (e.g., using ICU patients, mHealth-specific missingness patterns, differences in ECG sensor quality from hospital to mHealth) were preemptively by the authors’ explanation in Appendix A1. 
weaknesses: I’m not sure that mean and linear imputation necessarily make sense as baselines. If a peak is defined as a local maximum (i.e., $(x[i] > x[i-1])$ && $(x[i] > x[i+1])$ at index $i$ in signal $x$), then by definition there will be no peaks with mean or linear interpolation and the peak classification will fail. This may explain the NaN values and zero sensitivity of the mean and linear imputation in Table 2. Why not use a simple FFT-based baseline since these are quasi-periodic signals?

The choice to not utilize the MIMIC matched waveform database so that covariates can be used either in the modeling process or during evaluation is also a weakness. Many parameters such as resting heart rate, max heart rate, etc. vary as a function of age, disease status, medication usage, etc. and as a benchmarking dataset it would be nice to be able to incorporate that data.

correctness: What do the authors mean by “pooled all channels together” (line 131)? Are multiple types of ECG leads (e.g., I, II, III, aVR, V, etc.) included in the ECG dataset? Based on the IDs in the MIMIC_III_ECG_filenames.txt file, it appears like this may be the case. If so, I think that’s a potential issue. As the authors mention in line 177, mHealth applications usually use lead I (e.g., Apple watch, AliveCor use lead I). Some clarification would be helpful. 
clarity: The paper is nicely written. 
relation to prior work: Yes, the authors make it clear how their work differs from prior work. 
additional feedback: Questions

- Why did the authors not include confidence intervals in Table 2? In general, without confidence intervals it’s difficult to tell if proposed methods improve over baselines or state-of-the-art if only point estimates of performance metrics are given.

Comments
- In the Appendix (lines 28-29) the authors write “Because these paradigms are rhythm-based, the morphological differences are not relevant, only the timing of the signals, which is exactly what our PPG Heart Beat Detection task tests.” I disagree with this point - I think that if the goal is to measure quality of imputation, the reconstruction of morphology is important. If waveform morphology was irrelevant, then imputation algorithms should focus on estimating the average heart rate or beat-to-beat intervals (i.e., R-R interval), and likely downstream tasks such as atrial fibrillation classification wouldn’t benefit from using the raw waveform compared to derived features. I would advocate that the waveform imputation community consider adding morphology features such as T-wave identification in the ECG waveforms or left ventricle ejection time (LVET) in the PPG waveforms as target quantities of interest. 

Suggestions
- One suggestion is for the authors add the number of patients in the datasets described in Sections 3.2 and 3.3.
- Another suggestion would be to break the training dataset into chunks in case users want to download a subset of the data without downloading such a huge file. 
- An extension/addition the authors could consider is adding additional waveform data for a subset of users/patients such that models could be fine-tuned or personalized during an initial data gathering stage before being fully deployed (e.g., see Deng et al. 2021). 


Deng, Y., Lu, L., Aponte, L. et al. Deep transfer learning and data augmentation improve glucose levels prediction in type 2 diabetes patients. npj Digit. Med. 4, 109 (2021). https://doi.org/10.1038/s41746-021-00480-x
review_title: Proposed dataset for benchmarking waveform imputation methods fills an important gap in the waveform imputation community. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: An existing architecture was slightly transformed and transferred to another domain (i.e. wearables) to address a gap in physiologic signal imputation.  The approach presented shows superior performance compared to existing work.
strengths: The authors demonstrate that the current state-of-the-art algorithms trained on ECG and PPG waveform perform poorly on signals obtained from wearables.  
Table 1 outlines a clear comparison to existing work. The explanation why multi-channel is out of scope is solid. 

weaknesses: The approach was only tested on one dataset for each domain. Validation on other datasets would strengthen the paper.
correctness: The authors' claim to have created the largest publicly-available ECG and PPG waveform dataset is inaccurate.  The project is more of a transformation of existing datasets rather than new data.
The use of KNN or other non-deep learning techniques is not a disadvantage per se as the authors seem to suggest. Specific algorithmic limitations that cannot be solved with KNN or similar methods should be elaborated further. A deep learning approach is not necessarily better in all KPIs.
clarity: The paper is clearly written but not quite appropriate for the dataset and benchmark track.
relation to prior work: The submission failed to discuss how the approach presented compares to previous work when it comes to explainability of predictions.
additional feedback: The submission is better suited for the main conference.
review_title: PulseImpute: A Novel Benchmark Task for Physiological Signal Imputation 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper raises the issue of missing data in pulsative signals collected from wearable devices and introduces an imputation benchmark (PulseImpute). Specifically, the authors extracted missingness patterns from real-world mHealth settings and applied them to two existing pulsative signal datasets. They reproduced several existing imputation methods and proposed a new imputation method and used them as baselines. Applying these baseline imputation methods to their processed pulsative signal datasets with missing values, the authors proposed benchmarks for the downstream tasks.

strengths: - This paper defines an interesting research question. 
- Code to replicate the baseline results.
weaknesses: - I can see why the authors chose to apply the missingness patterns in the mHealth settings to the existing pulsative signal datasets, rather than collecting data directly from mHealth wearables (due to a possible lack of ground truth). Admittedly, the authors also compared several differences between data collected from mHealth settings and clinical settings. However, the lack of a direct comparison of the differences in missing patterns in the data collected from the two settings, and what exactly the missing patterns are in the data collected from mHealth wearables, makes the choice of curated datasets to address the question raised in this paper less convincing.
- What makes pulsative signals more interesting/compelling than other signals from a missingness perspective was less motivated. 
- The paper is a bit hard to follow and there are some inconsistencies and inaccessible references. 

correctness: I have a few concerns about the correctness of the methods used in this paper.
- Although the authors discussed their motivation for using clinical pulsative waveform datasets to mimic mHealth pulsative waveforms in A1, I’m still somewhat concerned as there is no direct comparison in missing patterns between pulsative signals collected from wearables and clinical settings to support their choice is an appropriate approach.
- As the paper applies missingness patterns from real-world mHealth settings to the existing pulsative signal datasets, it is crucial to have a description/review of missingness patterns the authors used to help reviewers/readers to justify this application is reliable. However, I referred to references [7] and [29] and supplemental materials but did not find any description of the missingness patterns.
- I wonder if the data for the third downstream task (predict diagnosis) is balanced? If not, they may want to add balanced accuracy and kappa as metrics to evaluate performance.

clarity: This paper is a bit hard to follow. Specifically,
- it reads to me that PulseImpute is a combination of a set of datasets for imputation tasks, nine baseline imputation models (including eight replicated existing models + one newly proposed model), and downstream task performance benchmarks. So in the last paragraph of section 1, it might be better to state that the main contribution is PulseImpute, and then to detail the three sub-contributions. In addition, I’m a bit confused as to whether the nine baseline models mentioned in line 51 contain the one proposed by the authors. Referring to Table 2, it does seem to contain the newly proposed imputation method. Then, I suggest reversing the second and third bullets in lines 51 and 52 and making it clear that the nine baseline models include the eight existing imputation methods and your new proposed method.
- there are some inconsistencies in this paper. For example, in sections 3.1, 3.2, and 3.3, the authors said that F1 score is used to measure classification performance. However, in Table 2, they reported F1, precision, and sensitivity, and in Figure 3, they reported AUC. As another example, the authors used nine models as baselines, however, in figure 1, it only contains visualization of imputation results from six baselines without explaining why the other three baselines were left out (although the authors do show the visualization of imputation results from all baselines in the appendix).
- some of the statements in the paper should be more precise. For example, in lines 38-39, the authors claim that “Table 1 describes six criteria that the PulseImpute challenge provides which are lacking in prior work.”. In fact, all six criteria were partially presented in prior work, but no work covered these six criteria simultaneously.
- some parts might need to be reorganized. For example, In section 3 (PulseImpute Challenge Description), lines 101-111 read to me like a review of two datasets the authors used for further curation, should it be part of related work? In addition, lines 112-119 look like brief summaries of each section of the paper. It seems better to put them in the introduction.

relation to prior work: This paper identified a gap in the existing literature and made efforts to fill it. 

additional feedback: Following are several minor issues to be addressed.
- In section 2 (related work), instead of reviewing prior work and highlighting its limitations simultaneously in each subsection, the authors may consider systematically reviewing prior work in each subsection, and then summarizing the issues with existing work in the last paragraph of this section.
- In section 2 (related work) line 61, EHR needs to be specified.
In section 2 (related work) line 75, LOCF was mentioned, but I’m confused about which work uses LOCF for imputation. Also, LOCF needs to be specified (i.e., Last Observation Carried Forward).
- In section 3.1 line 128, ABP needs to be specified.
Some references are hard to access. For example, [24] in line 377 and [33] in line 397, I’m not sure which paper/work the authors are referencing.
- In section 3.1 lines 145-146, “the majority of” and “some of” should be quantified in this sentence “the majority of missingness gaps are 3-9 seconds long but some gaps can last more than a minute.”. Specifically, whether the majority is 90% or 70%?
- Some statements in this paper need references. For example, in section 3.1 lines 134-135, “We sampled 5 min waveforms, as this is a standard duration for tasks like HRV that are based on beat detection”. As another example, in section 4 lines 208-210, this sentence needs a reference “Our Our primary focus is on transformer-based approaches because self-attention is an attractive method for modeling the long-range structure of pulsative signals as a means to leverage quasi-periodicity in imputing large blocks of missing samples.”.
review_title: Interesting topic, but needs better presentation

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The usage of wearable sensors for medical purposes promises better monitoring with high frequency information. However, the usage of such sensors worn in day-to-day life often leads to gaps in the sensory information. Imputation techniques attempt to fill such gaps, but are lacking for pulsative signals like ECGs and PPGs. This submission supplies datasets, methods to mimic realistic ‘missingness’ in the data, as well as challenge tasks to evaluate pulsative signal imputation. Further, the authors propose a benchmark transformer model, where the signal tokenizer is realized via dilated convolution, and empirically demonstrate significant outperformance compared to previous work on their benchmark tasks.
strengths: The proposed datasets and challenges appear highly relevant to facilitate broad usage of wearable sensors for medical purposes. The definition of evidenced-based patterns of missingness fills a gap in previously published work to that end. 
That definition is made accessible by the proposed challenge, which is based on existing (and peer-reviewed) datasets, incorporating the patterns missingness. The experiment design is clear and thoughtful. The downstream taks are both difficult and relevant tasks for pulsative signals to test imputation methods. The data provided makes it easy for researchers to work on the topic. The authors propose a model architecture which demonstrates significant outperformance on the benchmark tasks.
weaknesses: In my view, there are no major weaknesses. Two minor issues:

First, the presentation of the BDC architecture may be a bit too confident. In ll. 211 following the authors state their requirements for their benchmark model and state that ‘no existing transformer models address all three issues sufficiently’. I’d argue that these are rather common issues. As an example: images / pixels require local context, require some measure against permutation equivariance and have to deal with scaling of long sequences. [Vision transformers](https://arxiv.org/abs/2010.11929) (ViT) have found ways to deal with that, so has the [Perceiver](https://arxiv.org/abs/2103.03206) architecture, directly on the data. [CvT](https://openaccess.thecvf.com/content/ICCV2021/papers/Wu_CvT_Introducing_Convolutions_to_Vision_Transformers_ICCV_2021_paper.pdf) adds convolution data encoders to ViT, similar to the author’s approach. That does not take anything away from their model architecture, which they show works fine. I’d find it fair though to put it into context of other work. The authos further state that positional encodings don’t have a good inductive bias in their setting (line 239). I’m curious, is that empirical or are there other reasons to think that? On other domains (again, images as example), position encodings work surprisingly well to contextualize. 
 
Second, equation 1) seems somewhat over-complicated. Self-attention is commonplace and a softmax would have simplified it greatly.  
correctness: Dataset, benchmark tasks as well as paper seem of very good quality. The motivation is made clear, design decisions for the dataset selection, creation of missingness or benchmark architecture are explained and justified. The benchmark results seem sound and convincing.  
If there is space, I would appreciate more insights or discussion of the results on extended missingness (Figure 3, right). Based on the plots, BDC appears much richer in information than, e.g., BRITS, which nonetheless seems to work well on the Diagnostic label group. Are those spurious correlations, or features carried over by BRITS from the true signal? Vice-versa, do the authors have insights into what mode of information is lost by BDC that could explain the particular drop in predictive performance?

clarity: The paper is very well written. The flow from motivation over the background to the challenges and benchmark make it a rewarding read from which I could gain something.  
The only minor issue:  I had a few questions on the exact ‘realistic patterns of missingness’ (l 39) until the authors explained where they come from in Section 3. Maybe that can be caught a little earlier? 

relation to prior work: The proposed dataset and task appear novel and original. Previous work on health data imputation either worked on non-pulsative signals, was not publicly available, did not use realistic missingness patterns or could not be used as benchmarks. 
additional feedback: None
review_title: Review – A solid challenge and benchmark for a relevant problem

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, the authors proposed a new benchmark task for physiological sensor signal imputation. Specifically, the authors focus on ECG and PPG signals and use public datasets for evaluation. They first simulated realistic missing patterns for ECG and PPG signals by ablating samples and built a dataset with realistic missing data. They implemented eight existing traditional or modern time-series imputation techniques. Moreover, they also proposed and developed a new bottleneck dilated convolution (BDC) self-attention architecture that fits the characteristic ECG/PPG data: these data usually have a long-range structure as pulsative signals. The authors then evaluated the performance of these algorithms on 1. the raw signal reconstruction task, and 2. three downstream tasks: a) heartbeat detection in ECG, b) heartbeat detection in PPG, and c) cardiac pathophysiology classification in ECG. Their results indicate that the new BDC technique is significantly better than all baselines.
strengths: - The topic of signal sensor imputation is an important, realistic, and very practical problem in mHealth daily physiological data collection.
- Both the raw construction error and the three downstream tasks are valid experiment designs.
- The design of the new architecture does leverage the pattern of ECG/PPG properties and the advantage over the baseline methods are encouraging.
weaknesses: - Physiological signal types may go beyond ECG/PPG.
While ECG/PPG is arguably one of the most commonly collected physiological signals in mHealth applications. There are other common sensors that are not covered in this paper, such as IMU (accelerometer, gyroscope, and magnetometer), GSR, EEG, etc. I am not arguing that the authors must evaluate their techniques on these signals. But their characteristics could be very different from ECG/PPG signals. For example, they may not have a clear pulsative pattern. I am curious to know the authors' consideration of the generalizability of the technique and the potential necessity to tone down the paper's framing (or clarify the scope of the paper).

- Baseline method selection.
The authors compare the new technique against eight baseline techniques, which is great. However, why not compare against the two existing papers that specifically focus on imputing mHealth pulsative signals (i.e., [20, 40] - the citation number in the paper)? Please justify.

- Extremely low recall for baseline technique.
Related to the previous point, in Table 2, the performance of the baseline techniques all have very low recall (thus low F1 score). The authors provided some reasons in the text, which is great. But such a low performance raises a concern that whether these baselines are too easy to beat. Comparing against the SOTA technique could provide more valid results.

[20] Arman Iranfar, Adriana Arza, and David Atienza. Relearn: A robust machine learning framework in presence of missing data for multimodal stress detection from physiological signals. arXiv preprint arXiv:2104.14278, 2021.

[40] Hillol Sarker, Matthew Tyburski, Md Mahbubur Rahman, Karen Hovsepian, Moushumi Sharmin, David H Epstein, Kenzie L Preston, C Debra Furr-Holden, Adam Milam, Inbal Nahum-Shani, et al. Finding significant stress episodes in a discontinuous time series of rapidly varying mobile sensor data. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 4489–4501, 2016.
correctness: My concern of the pulsative signals only being a part of the wide range of physiological sensing signals has been mentioned above. Other than that, I do not have concern on the correctness of the paper. Evaluation methods and the experiment design are appropriate and valid.
clarity: The paper is written well and easy to follow. The figures clearly convey the right message.
relation to prior work: The paper covers a good amount of related work. Table 1 is very helpful to highlight the distinction of this work. My only concern is mentioned above in terms of selecting the right baseline techniques to compare.
additional feedback: Please see my comments above. I look forward to seeing the authors' response!
review_title: A valid contribution to the sensory data imputation benchmark, together with a new imputation method

Meta-review: metareview: This paper develops a new benchmark for missing data imputation in pulsative signals like ECG using realistic missingness models. I expect such a dataset to drive important developments in this understudied area; indeed, the authors show that standard SOTA methods fail. The reviewers' enthusiasm makes this paper a clear accept.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept