Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors of the paper have introduced a multilingual cardiac signal captioning framework. The authors have proposed a neural framework that generates captions for the cardiac signals in multiple languages simultaneously. The authors add an auxiliary task of identifying the language of some of the tokens to improve the performance of the decoder. The proposed framework achieves on par performance with state of the art pre-training methods. 
main_review: The strengths of the paper: 
1. The paper is well presented and the problem introduced in the paper is interesting as well as important. 
2. The authors have tried multiple pre-training methods to train their decoder and use a relevant auxiliary task to supplement the performance of the proposed model.
3. The reports generated by the proposed framework seems to be quite coherent and similar to the gold standard captions.

The weaknesses of the paper: 
1. The authors have mentioned that though there has been work around EEG captioning (Biswal et al 2019, 2020), the previous papers did not explore the multilingual captioning of the cardiac signals. But in the case of the proposed framework the gold-label captions for languages other than English are trained on captions generated via Google Translate API. So the ceiling of the decoder model is as good as the Translate API itself. If that is the case, wouldn't it be better to just generate the English caption and then translate them via the API. The cited paper (Conneau et al) uses such a strategy to generate for different languages to augment the dataset and generate new translations which the model might not have seen. 
2. I do not understand the use case of generating all languages at once. The more suitable framework would be where the report is generated as per the requirement in which the source language is provided as an input to the decoder (<source_lg such as en, fr etc.><START>). 
3. If MLM performs as good or better, in most cases, for generating the captions. What would justify the use-case of RTLP? It is an interesting approach but I do not see an added advantage of the proposed model.
4. The application is highly interesting but almost all parts of the proposed framework/models have been explored in a similar setting except for the multilingual aspect, which I am having a hard time understanding the use case of. In the proposed problem formulation, I would try to learn a really good cardiac signal captioning 'en' model and then either use the API itself or train a translator model. This would reduce the model parameters and make the whole framework more efficient.
summary_of_the_review: The paper proposed a solution to an interesting problem but as mentioned in the weakness section above I could not comprehend clearly the use-case of the multilingual decoder. But I would be highly interested in discussing with the authors and other reviewers during the rebuttal period if I might be missing the aspect of using multilingual data for the framework.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The goal of this paper is to develop an approach for generating multilingual ECG reports. The authors propose a new multilingual pretraining method in which tokens are randomly replaced with those from a different language, and the model must learn to identify the language of all tokens. The method – RTLP – performs similarly to MLM, ELECTRA, and MARGE according to BLEU-1, METEOR, and ROUGE-L metrics on generation of ECG reports, and the authors qualitatively assess that the generated reports are clinically accurate. The paper compares monolingual and multilingual versions of the models and find that RTLP benefits from multilingual training.
main_review: Strengths: This paper is very well written. Multilingual text generation is an interesting task, and more multilingual work is needed in the ML for health space. 

Weaknesses: I have several concerns about the clinical utility of this task as well as the evaluation approach.  
-	First of all, I think clarification is needed to describe the utility of the task setup. Why is the task framed as generation of the ECG report rather than framing the task as multi-label classification or slot-filling, especially given the known faithfulness issues with text generation? There are some existing approaches for automatic ECG interpretation. How does this work fit into the existing approaches? A portion of the ECG reports from the PTB-XL dataset are actually automatically generated (See Data Acquisition under https://physionet.org/content/ptb-xl/1.0.1/). Do you filter out those notes during evaluation? How does your method compare to those automatically generated reports?
-	A major claim in the paper is that RTLP generates more clinically accurate reports than MLM, yet the only analysis in the paper related to this is a qualitative analysis of a single report. A more systematic analysis of the quality of generation would be useful to support the claim made in the appendix. Can you ask clinicians to evaluate the utility of the generated reports or evaluate clinical utility by using the generated reports to predict conditions identifiable from the ECG? I think that it’s fine that the RTLP method performs comparable to existing methods, but I am not sure from the current paper what the utility of using RTLP is. 
-	More generally, I think that this paper is trying to do two things at once – present new methods for multilingual pretraining while also developing a method of ECG captioning. If the emphasis is on the former, then I would expect to see evaluation against other multilingual pretraining setups such as the Unicoder (Huang 2019a). If the core contribution is the latter, then clinical utility of the method as well as comparison to baselines for ECG captioning (or similar methods) is especially important. 
-	I’m a bit confused as to why the diversity of the generated reports is emphasized during evaluation. While I agree that the generated reports should be faithful to the associated ECG, diversity may not actually be necessary metric to aim for in a medical context. For instance, if many of the reports are normal, you would want similar reports for each normal ECG (i.e. low diversity). 
-	My understanding is that reports are generated in other languages using Google Translate. While this makes sense to generate multilingual reports for training, it seems a bit strange to then evaluate your model performance on these silver-standard noisy reports. Do you have a held out set of gold standard reports in different languages for evaluation (other than German)? 

Other Comments:
-	Why do you only consider ECG segments with one label assigned to them? I would expect that the associated reports would be significantly easier than including all reports. 
-	You might consider changing the terminology from “cardiac arrythmia” categories to something broader since hypertrophy (one of the categories) is not technically a cardiac arrythmia (although it can be detected via ECG & it does predispose you to them)
-	I think it’d be helpful to include an example of some of the tokens that are sampled during pretraining using your semantically similar strategy for selecting target tokens. How well does this work in languages that have very different syntactic structures compared to the source language?
-	Do you pretrain the cardiac signal representation learning model on the entire dataset or just the training set? If the entire set, how well does this generalize to setting where you don’t have the associated labels?
-	What kind of tokenization is used in the model? Which Spacy tokenizer?
-	It’d be helpful to reference the appendix when describing the setup in section 3/5 so that the reader knows that more detailed architecture information is there.
-	I’d be interested to know if other multilingual pretraining setups also struggle with Greek. 
-	It’d be helpful to show the original ECG report with punctuation + make the ECG larger so that they are easier to read
-	Why do you think RTLP benefits from fine-tuning on multiple languages, but MARGE does not?


summary_of_the_review: Overall, the combined unclear clinical utility, lack of support for a major claim in the paper (that RTLP generates more clinically accurate reports), and concerns about the evaluation strategies lead me to recommend this paper for rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to build a multilingual cardiac signal captioning system to generate ECG reports, which describe the clinical findings in the input of electrocardiogram (ECG) signals. In particular, the proposed system can generate desirable and fluent reports in multiple languages, i.e., German, Greek, English, Spanish, French, Italian, and Portuguese. The experiments on a public dataset verify the effectiveness of the proposed approach, which performs on par with state-of-the-art language pre-training methods.
main_review: Strengths:
1. This paper is clearly written. The paper is easy to follow and understand.
2. The targeted problems, i.e., cardiac signal captioning and multilingual captioning, are novel and important in both artificial intelligence and clinical medicine.
3. The proposed multilingual cardiac signal captioning system is well-motivated, novel, and interesting.
4. The experiments and analysis are extensive and solid.

Weaknesses:
1. The presentation can be further improved. In the Abstract and Introduction, can you give more explanations about "Generating these reports, however, can be time-consuming and error-prone, while also exhibiting a high degree of intra- and inter-physician variability"? For example, why generating these reports can be error-prone? What problems will be brought by the high degree of intra- and inter-physician variability (can you give some examples in the Introduction)? 

2. Although the targeted problems are important and novel, after reading this paper, I am still confused about how your proposed method is related to this cardiac signal captioning. I think the proposed approach can be used for conventional image captioning as well. In other words, have you solved the challenges and problems that are unique to the multilingual cardiac signal captioning?

3. The experiment should be improved. Firstly, the evaluation metrics used in this paper, e.g.,  BLEU and ROUGE, are all general metrics for text generation tasks. So, it is unclear why your proposed approach can bring improvements. In what aspects can the proposed method improve the performance of the multilingual cardiac signal captioning? Secondly, in Table 1, why is your proposed method lower than the baselines in some settings? Can you give more explanations? Thirdly, the Google Translate model is not specifically designed for biomedical texts, so you can give more analysis of the Google Translate model. 

4. The paper is written in an optimistic tone that leads the reader to assume the proposed approach is rather good. However, I am more interested in knowing if the approach brings errors? And what type of errors does it bring? And why?

5. The related work is insufficient. It is suggested to add more discussions about the report generation for other types of medical signals, e.g., chest X-rays, which have been widely explored in existing papers [1][2][3][4][5][6][7].

Missing References:

[1] On the Automatic Generation of Medical Imaging Reports. In ACL, 2018

[2] Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports. In ACL, 2019.

[3] When Radiology Report Generation Meets Knowledge Graph. In AAAI, 2020.

[4] Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation. In CVPR, 2021.
summary_of_the_review: The paper is well-written and the motivation sounds reasonable. The targeted problems, i.e., cardiac signal captioning and multilingual captioning, are novel and important. The presented approach is novel and interesting. Thus, I tend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new report generation system for cardiac signals. It applied replaced token language prediction (RTLP) settings to improve the report generation performance. Experiments show that the proposed RTLP framework can achieve comparable performance with SOTA models. Extensive analyses were conducted to examine the diversity and the performance in multilanguage settings. 
main_review: This paper proposed a novel multilingual pretraining language model settings. The idea is very interesting and the experiment setting is solid.   This paper conducted many result analyses which are helpful to understand the model's performance. 

1. Using BLUE and ROUGE as the metrics is not enough in the evaluation of clinical report generation. The precision of content is critical in clinical reports. High BLUE/ROUGE does not represent the report is well written. In my opinion,  the performance of clinical text generation should be validated based on its content, although it is very hard and requires extensive manual work. 

2. The gold-standard reports for non-English languages came from Google translation. This setting may lead to evaluation bias. It would be better to conduct a sensitivity analysis by using other translation tools.   

3. It is not clear which network structure was used to represent the cardiac signals? Convolutional NN? Transformor? Other time series models?

4. It is not clear how to build the categorical distribution when selecting the semantic-similar target language tokens (page 4).

5. The "blessing of multilinguality"  comes from the low performance of RTLP in a single language. Based on this low baseline, the improvement from multilanguage has limited value. (RTLP + multilanguage can not beat MARGE).
summary_of_the_review: The proposed model is novel and interesting. While the evaluation metrics in clinical text generation are not persuasive. And some of the important details were missing in the current draft.
review_title: null

Meta-review: The paper addresses the problem of generating captions for ECG signals where they extend the task in the literature from monolingual to multilingual captions. The model proposed in this work is a variant of mask language model where they augment the target by switching the words from one language to another. In addition to predicting the actual words, they also predict the language associated of the words.

Pros
+ The problem is motivated by real world application and need. 
+ The presentation is clear and the authors compare the performance of the model with appropriate recent models.

Cons
- The model has higher complexity in both training and parameters, yet achieves only comparable performance to simpler models.
- Empirical evaluation of the multi-lingual output is inadequate since the ​ground-truth is derived from Google Translate.
- There are also a number of other specific concerns raised by the reviewers (e.g., VuBG, HBxE).

Reviewers raised questions about several shortcomings. In response the authors updated the paper and were very engaged in the discussion with the reviewers. However, at the end of the day, the paper in its current form has serious shortcomings and left the reviewers unconvinced. I suggest the authors take advantage of the feedback from the reviewers and address them fully in their next iteration.
Decision: Reject