Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Overview: The authors present an approach to learn blood oxygen levels using respiratory signal data. This might be relevant for various contexts such as ARDS or COVID-19. Unlike using pulse oximetry, it doesnt rely on wearing a sensor and less biased on darker skinned individuals.

Clarity and Quality: The problem description and aim of the paper are very clear. Unfortunately, I think the abstract of the paper and the introduction are quite misleading however as the authors entirely focus on COVID-19, while only a tiny fraction of the results are actually demonstrated on COVID. The paper thus makes bigger claims for COVID than what is demonstrated in the experiments which is misleading.

Originality and Significance: Certainly the problem of learning blood oxygen levels is a significant one in high-risk patients in ICU for instance. However, the originality of the modelling approach is not that unique. Specifically the authors use auxiliary learning to improve on standard ML approaches. The model proposed is a multiheaded model that incorporates auxiliary oxygen-related variables into the prediction process.

Pros: 1. The paper addresses a relevant biomedical problem
2.  The authors demonstrate the performance of their approach on three separate datasets

Cons:
1. Unfortunately, the authors never explicitly clarify what is meant by a "breathing signal". Is this the respiratory rate or a pulse or something else? I find this term neither technically sound from the medical perspective nor informative to a non-expert. This is also crucial to understanding how good the model is. For instance, if the task is to predict blood oxygen from "breathing signals", how noisy are these measurements? How are these recorded? Are these recorded via contactless sensing? 

2. In the experiment for COVID, the model is only applied to a case of a single COVID-19 patient. The reader needs more information of what is the prior condition of the patient, is he in ICU? If so, how do you account for treatment policies or protocol changing for the duration of stay; this could affect overall outcomes (and oxygen saturation)? Moreover, the paper oversells this demonstration on a single COVID-19 patient almost as its key contribution. Yet this is only a single patient.
review_title: An ok paper with a potentially misleading abstract

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
##########################################################################

Summary:

 
The paper proposes neural network models to predict oxygen saturation from breathing signals. The architecture implements "feature switches" that partition the data in a multi-head manner with the ultimate goal of predicting auxiliary variables which will help the prediction. The results are very encouraging especially given the immediate application to COVID-19 non-invasive monitoring. 

##########################################################################

Reasons for score: 

 
Overall, I vote for marginal accept. I like the idea of predicting one modality from another. My major concern is about the experimental section of the paper and some additional ablations (see cons below). Hopefully, the authors can address my concern in the rebuttal. 

 
##########################################################################Pros: 

 
1. The paper leverages one of the most important biomarkers for COVID-19 monitoring. If it's possible to predict SpO2 robustly from other signals, it's a big advance.
 
2. The proposed architecture seems to be superior to vanilla models, as reported in the experiments section. 

 
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework.

 
##########################################################################

Cons: 

 
1. My major concern is the use of PSG and RF signals. Since both of them require prohibitively-expensive equipment (a minimum of 22 wire attachments, and a big antenna I suppose, respectively), I am curious why the paper is not using easier to obtain sensors. For example, some of the medical datasets already used here include heart rate measured through PPG signals or ECG (now common in smartwatches). Could the same framework be applied to heart rate->SpO2?

 
2. The motivation behind the feature switches seems a bit inflated. The paper ends up using the gradient cosine similarities to inform the final outputs. Why do we need to inspect the gradients in contrast to -say- the validation loss? Couldn't this be achieved with traditional feature importance methods? For example, permutation feature importance could inform which feature set is insignificant or you could even apply SHAP to the vanilla model with all features.
 
3. In terms of baselines, the paper starts from a single-headed neural network. However, it would be good to explore the "lower bound" of the dataset with a naive model. What would be the error of linear regression or a random forest regressor trained on features extracted by the signals? I am aware that here the output is multi-step, but you could apply the models iteratively on the predicted values. Also, in general, I would like to see more information about the parameter size of the models, batch size etc. In the appendix, we read that "the breathing signals we used in all the experiments last several hours per night", which means that the output is also a signal that spans hours? How do you average the error for that kind of multi-step forecasting? Does the error increase as we forecast further in the future?

 
##########################################################################

Questions during the rebuttal period: 

 
Please acknowledge the potential use of cheaper and more accessible sensors.

The structure of the sections could be improved as well. Section 3 is not the right place to mention (some) results.

Figure 4 --> is this an observation made for the Multi-head model only? How is the distribution of the single-headed model?

learning to throw the data to the right function --> informal, please rephrase


 
#########################################################################

Many typos (please proofread the entire paper again): 

-These result is quite interesting 

-there are medical evidence 

-..

review_title: Encouraging result in a very topical domain

final_score: rating: 3: Clear rejection
review_content: Summary: A model that appears to very effectively predict SpO2 from respiration signals. I find the model and experiments well-designed, but have serious concerns about the motivation and prediction task itself.

**Update**: After reading the rebuttal and revised paper, I am keeping my score the same. There is a lot in this paper I really like -- an important prediction target and race analysis among them -- but still have concerns about clinical utility. In standard photoplethysmography, the causal graph is PaO2-->light absorbance, which leads to a clear story of what a fingertip SpO2 sensor is doing (modeling an imperfect but mostly unidirectional relationship). The causal graph in this paper is less clear but probably has some kind of cycle like ventilation-->SpO2-->ventilation, which is  a much more challenging story. I think there are two good routes this paper can take: (1) focus on how to clinically justify a model with such a potentially complicated causal graph; is the model learning the vent->O2 relationship, O2->vent, or some combination of the two? How can we be confident we know that? To which scenarios will and won't a model that has learned such relationships generalize? (The rebuttal has analysis that is a good start, but I think not enough to fully answer these questions) Or (2) acknowledge that the scope will necessary be limited when causal structure is so complicated and poorly understood, and actually limit the tasks the model is capable performing to those on which we can be confident it will do well. I think either (1) or (2) would involve major changes to the structure and goals of the paper, and probably require new experiments. 

Objective: Predict blood oxygen sequence information from respiration sequence information, leverage auxiliary variables and demonstrate feasibility of RF-based contactless oxygen prediction. I view the first as the primary goal, bolstered by the auxiliary variables, which enables the RF-based prediction.

Strengths:
Overall, I think the model design and evaluation is very good! I like the paper a lot overall and think the authors did a good job addressing most methodological and experimental concerns.
* Impressive, accurate predictions. 
* Experiments are well-designed. In particular, analysis w.r.t. race is well-motivated and well conducted. 
* RF experiment demonstrates both RF feasibility and ability of the model to generalize across respiration measurement devices/methodologies.

Weaknesses:
Major weaknesses (only 1):
My most important concern is the choice of problem and prediction task. There is a well-discussed medical distinction between ventilation and respiration -- simply moving air vs actually adding oxygen to the blood stream. There is a good reason why respiratory rate and pulse oximetry are measured separately: resp. rate measures ventilation while oximetry measures respiration. Many conditions, i.e. pulmonary edema or embolism, affect the lungs' ability to exchange oxygen without affecting the ability to move air, and many conditions, i.e. traumatic or neurological, affect the ability to move air without affecting the ability to exchange oxygen in the lungs. Because of this, I see SpO2 as a variable that contains *independent* information not in the ventilation signal, and believe it is dangerous to display a patient SpO2 signal that is entirely *dependent* on ventilation - containing no respiration signal. Such a signal could look plausible enough but fail in the most important clinical cases.

I understand there is low error on the tasks shown, but how does the model work in cases where we'd expect respiratory problems but no ventilation problems -- or vice versa? Beyond sleep studies or ambulatory settings, is it likely to work in cases of pulmonary embolism or ARDS in an acutely ill COVID-19 patient? Some of the problems discussed in the paper (e.g., sleep-time drops in oxygen saturation) seem like they could plausibly be predicted from ventilation signal, but for these cases, why not predict a binary target instead of the full SpO2 signal? The only reason for predicting the SpO2 signal would seem to be if you think it's a true inference of the physiology that will hold true even in cases you didn't examine in the paper -- and I think there are a lot of prior physiological reasons to believe that's not the case. My prior is that there are at least 2 causal paths where you can predict SpO2 from ventilation signal: (1) hypoventilation-->hypoxemia and (2) hypoxemia-->hyperventilation. I'm concerned that the model may be able to pick up SpO2 signals that follow these patterns (and are accurate in the datasets used) without learning a general, "true" relationship between breathing and blood oxygen. This is important because the paper is primarily clinical: the value is the ability to predict a new clinical outcome. Thus, I think the paper should only be accepted if it gives a good idea of in what cases the model would be clinically useful at predicting SpO2, and I don't see much of this analysis in the paper. If these issues were discussed at all (and ideally in detail), I'd be more open to seeing clinical value in the work. 

Minor points:
* Methodologically, it seems the auxiliary variable strategy works well but I'm not convinced it's the only way or the best way to solve the problem -- both multitask and multi-headed seem good at capturing the shape of the signal, with multitask often off by a constant. This is what we'd expect from an MSE model trained on a large dataset -- the overall signal shape will be pulled towards the mean. Gating into separate models for groups with different baselines would reduce this problem (because there will be less variance within each group). This is not a huge issue (and I know L1 loss is used here rather than MSE), but there are other ways to handle such problems -- for example using shape and time distortion losses like DILATE.
* I did not find the COVID analysis particularly compelling because there's only a single patient and mostly qualitative analysis -- it's hard to draw any clear conclusions form the example about the method's value overall in such cases. I think I'd prefer a more rigorous evaluation on a non-COVID task to something that's a bit speculative but COVID related.
* Some writing could be smoother and more terse in the Method section, i.e. " In this paragraph, we answer
the critical question..."

review_title: Nice model and experiments, but concerned about the prediction task.

Meta-review: Two referees indicate reject, one supports (weak) accept. My impression is that major points of criticism raised by the reviewers -- mostly about limited novelty and somewhat inconclusive experimental results -- could not be addressed in a clearly convincing way during the rebuttal phase. I will therefore recommend rejection.

Decision: Reject