Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
Authors: Dani Kiyasseh
Tingting Zhu
David A. Clifton
Venue: ICLR 2022 Submitted (2022)
Keywords: multilingual representation learning
cardiac signal captioning

Abstract: Cardiac signals convey a significant amount of information about the health status of a patient. Upon recording these signals, cardiologists are expected to manually generate an accompanying report to share with physicians and patients. Generating these reports, however, can be time-consuming and error-prone, while also exhibiting a high degree of intra- and inter-physician variability. To address this, we design a neural, multilingual, cardiac signal captioning framework. In the process, we propose a discriminative multilingual representation learning method, RTLP, which randomly replaces tokens with those from a different language and tasks a network with identifying the language of all tokens. We show that RTLP performs on par with state-of-the-art pre-training methods such as MLM and MARGE, while generating more clinically accurate reports than MLM. We also show that, with RTLP, multilingual fine-tuning can be preferable to its monolingual counterpart, a phenomenon we refer to as the \textit{blessing of multilinguality}.

Under review as a conference paper at ICLR 2022
LET YOUR HEART SPEAK IN ITS MOTHER TONGUE: MULTILINGUAL CAPTIONING OF CARDIAC SIGNALS
Anonymous authors Paper under double-blind review
ABSTRACT
Cardiac signals convey a significant amount of information about the health status of patient. Upon recording these signals, cardiologists are expected to manually generate accompanying report to share with physicians and patients. Gener- ating these reports, however, can be time-consuming and error-prone, while also exhibiting high degree of intra- and inter-physician variability. To address this, we design a neural, multilingual, cardiac signal captioning framework. In the pro- cess, propose discriminative multilingual representation learning method, RTLP, which randomly replaces tokens with those from a different language and tasks network with identifying the language of all tokens. We show that RTLP- generated reports are of high quality and clinical utility, and are on par with reports generated by networks pre-trained with state-of-the-an methods such as MLM and MARGE. We also show that generated reports exhibit higher quality and clinical utility when RTLP is fine-tuned in multilingual setting than in monolingual setting, a phenomenon we refer to as the blessing of multilinguality.
INTRODUCTION
Cardiac signals convey a significant amount of information about the status of patient. Upon recording these signals, an expert cardiologist often required to interpret the findings, manually generate an accompanying textual report (Richley & Walters 2020), and share it with fellow physi- cians (and patients). Such report is crucial within medicine, impacting communication between physicians, facilitating clinical decision-making, and holding care teams accountable for their ac- tions (Waegemann et al. 2002). Manually generating these reports, however, can be time-consuning and error-prone, while also exhibiting high degree of intra and inter-physician variability. Com- bined, these can detract from a physician's ' time with a patient, thus decreasing patient satisfaction, and hinder communication between physicians (Hibbard et al., 2001 Keselman & Smith 2012). thus potentially compromising patient outcomes Brailer et al.] 1997).
One way to address this challenge is by collapsing clinical report into multiple pertinent medical conditions and designing a system that identifies such conditions. However, abstracting away the details of report tinence of conditions can be disease-specific and non-trivial, Based on these observations, we look address the following question: how do we automatically generate clinically accurate reports that reliably summarize cardiac signals? To address this question, the fields of visual-language representation learning and captioning hold promise. In this setting, rich representations of visual inputs and language are learned to automatically generate captions. Representations of the latter can be learned in a discriminative monolingual setting, as with ELECTRA (Clark al., 2020), or in a generative multilingual setting, with MARGE (Lewis et al., 2020). However, discriminative multilingual framework has yet to explored. Moreover, previous work has focused exclusively on the captioning of medical images (Hasan al. 2018 Zeng et al. 2020). It has not explored the captioning of cardiac time-series signals.
obseure the full assessment of patient's health status and defining the per-
In this paper, we address the outlined question by designing a neural, multilingual, cardiac signal captioning system while exploiting a large-scale electrocardiogram (ECG) database. Our contribu- tions are the following: (1) We propose discriminative multilingual representation learning method, replaced token language prediction (RTLP), in which we randomly replace tokens with those from other languages, and task network with predicting the language of all tokens. (2) We show that

---

Under review as a conference paper at ICLR 2022
RTLP allows networks to generate reports, in multiple languages, that clinically accurate as those generated by masked language modelling (MLM), To the best of our knowledge, we are the first to propose multilingual cardiac signal captioning system. (3) We conduct extensive studies to show that generated reports exhibit higher quality and clinical utility when RTLP is fine-tuned in multilingual setting than in monolingual setting, phenomenon refer to the blessing of multilinguality
RELATED WORK
Language representation learning Representation learning is integral to modern natural lan- guage processing. Generative tasks such as MLM show promise (Devlin et al. 2019: Liu et al., 2019b), even in the multilingual setting (Conneau & Lample| 2019; Conneau et al. 2020b Liu et al. 2020). MARGE (Lewis et al. (220)) retrieves documents, in potentially different languages, and attempts to reconstruct a target document. Others propose to jointly learn textual and visual rep- resentations et 2019 2019 Lu et al. 2019 Zhang al.) 2020b). Similar to our work is ELECTRA (Clark et al. 2020), where tokens are replaced with those from a generative model, and network is tasked with discriminating between the original and replaced tokens. This approach remains com- putationally expensive where both an MLM and discriminative network are learned. It also does not optimize multilingual objective. To the best of our knowledge, we are the first propose a discriminative multilingual representation learning method in the context of cardiac signals.
Multilingual representation learning Pre-training and fine-tuning networks on multiple lan- guages confers benefits to NI.P tasks (Conneau et al.] 2020b Pratap et al.] 2020) Conneau et al,] 2020a Artetxe et al., 2020). For example, Conncau et al. (2020b) and Artetxe et al. (2020) show that multilingual pre- training outperforms its monolingual counterpart in solving downstream NLP tasks. These findings have been partially driven by monolingual datasets that machine-transe translated to multiple languages, as with XNLI (Conneau et al.) (018). We similarly translate ground-truth ECG reports into multiple languages in order to guide the generation of multilingual reports. Al- though Huang et al. (2019) propose a pre-training setup similar to ours, we explore more languages and define different pre-training task (RTLP) which we exploit for cardiac signal captioning.
Captioning in healthcare Biomedical image captioning has traditionally focused on chest X-rays (Kisilev et al. 2016)Ding et al. [2017] Hasan et al. 2018) et al. Zeng et al. (2000 Liu et al.,
2021]_ Liu et (2019a) condition their captioning system on the medical topic to be discussed and
Wang et al. (2018) propose multi-level attention model that attends to both the image and the text.
Previous work which captioned elcctroencephaalogran (EEG) signals (Biswal et al., 2019: (020) did not explore multilingual representation learning method and does not extend to the generation of reports in multiple languages. To the best of our knowledge, we are the first to propose the multilingual captioning of cardiac signals.
3 BACKGROUND
3.1 CARDIAC SIGNAL CAPTIONING
We begin by assuming access to a dataset, D = (æ;, ). comprising N cardiac signals, æ E RD, and their associated captions, cap = {w,}==] which consist of words, w,. The goal of cardiac signal captioning is generate a caption (report) that reliably summarizes the physiological state of a patient as manifested in a cardiac signal. To extract features from the signal, an encoder, fo RP {v, RM17 parameterized by 0, maps D-dimensional instance, to set of T representations, (v, E RRM Jt=1 each of which is M -i-imensional (see Fig. left).
To convert the captions into format ingestible by a network, we first convert each word, Ws, in caption, cap = [w,15_ to a token, ug, to form a sequence of tokens {u,}5_1- Such tokenization can involve lower-casing and stemming words, in addition to removing punctuation. After deriving tokens from all captions in a training set, we form a fixed vocabulary, V = { of C unique tokens where [V| = C. We then define an embedding matrix (lookup table), E E RCXM u - e RM, which maps each token, u, an M-dimensional token embedding, e. These embeddings
2

---

Under review as a conference paper at ICLR 2022
Key
Network
Head
Represent
Embed.
Token
Replaced Token Language Prediction
normal
sinus
rhythm
<end>
English
English
English
Spanish
12-lead
Poy
Pyu
Pg
Pe
Newe
Bugger
o
Encoder fo
Shared Decoder Bs
Shared Decoder
Cugare
<start-
mormal
rrythhm
cstarts
noomal
senus
cilmo
L (languages)
sourre
tokens
target
tolun
Figure 1: (Left) Multilingual cardiac signal captioning pipeline. We feed 12-lead ECG into an encoder, fe. to extract representations, v. These are fed, alongside embeddings, c, of tokens from particular language, decoder, Ia. to generate token representations, h. We feed h into language-specific head, Pwy, to generate a caption in a specific language. (Right) Replaced token language prediction framework. We randomly replace source tokens with those from target language and task the network with classifying the language of tokens. In doing so, we encourage the network to capture relationships between representations of tokens from different languages.
typically aundomly-nitialiin and learned in an end-to-end manner via gradient descent. As such, every caption can now be represented as a sequence of token embeddings, {e.]5=1-
(h, E RM pa- rameterized by Φ, attends to all encoder representations, {wr) while mapping a token em- bedding, Cs, at step, to an M of-dimensional token representation, h,. Each token representa- tion. h. in the sequence of representations, {h.)==1- is then fed into linear classification head, Pw h E RM - Ys E RCC, to output probability distribution, Ys. over the C tokens in the vocabulary. This forms a sequence of outputs, )= At each step, 8, in the sequence, the goal to maximize the likelihood of generating the token of the next step, Us++1 Therefore, by identifying the most probable output token at each step, we can form sentence of words (i.e., a caption).
To extract features from language, decoder, Do fur {e,)==1
METHODS
4.1 MULTILINGUAL CAPTIONING OF CARDIAC SIGNALS
At the surface, a monolingual captioning framework would appear to be sufficient for physicians communicating single language. However, the motivation for multilingual captioning frame- work, in which reports are simultancously generated in multiple languages, is threefold. First, multilingual framework would obviate the cumbersome process having train distinct model for different languages (Conneau et al., 2020a). Second, recent work has demonstrated the benefits of incorporating additional languages into the learning process (Artetxe al., 2020). Lastly, mul- tilingual framework exhibits greater flexibility than its monolingual counterpart, it can always be collapsed, during inference, to generate captions from a single language.
To enable multilingual captioning, we first assume access to I. language-specife datasets, {Dif where D, {æ;, cap!) comprises N cardiac signals, and captions in specific language, I E L {en, es................. }. where en and es represent English and Spanish, respectively. Note that the cardiac signals are shared across the datasets. We follow the same encoder-decoder approach mentioned in the previous section with one exception. We replace the single classification head with L language-specift heads account for the distinct vocabularies of the L languages. In doing so, we exploit recent observations that demonstrated the utility of having a network with parameters that are both language-specific and shared across languages (Zhang et al.) 2020a).
3

---

Under review as a conference paper at ICLR 2022
Formally, for each language, 4, we have linear classification head, Pwy h1 E RM + y RC, with wi E [Wen- Wes, .. reflecting language-specific parameters. Each head maps the token representation, hi at each step, s, in the sequence reflects the size of a language-specific token vocabulary. When doing this for each step in the sequence, we arrive at set of probability distributions, {v[]s )..m As with traditional language models, at each step in the sequence, 8 [1, S]. we maximize the likelihood of observing the next token in the sequence, u from particular language, I E L. Therefore, for mini-batch B captions single language, we would optimize the categorical cross-entropy loss at each step, To extend this to L languages, we load L mini-batches and optimize the following multi-task categorical cross-entropy loss.
probability distribution, y over C tokens
where C' E ([Ven), [Val,"
L
1 LBS
fmultilingual
log (V m1,)
(1)
IEL 11 8=1
4.2 REPLACED TOKEN LANGUAGE PREDICTION
To facilitate achieving the downstream task of multilingual cardiac signal captioning, we design discriminative multilingual pre-training task that learns the decoder parameters, á and the token embeddings, e. At high-level, this task involves randomly selecting tokens in sequence, replac- ing them with semantically-simin classifying the language of all tokens (see Fig. right).
tokens from a different language, and tasking a network with
The intuition behind our framework is that a network exposed to semanticall-ssiiilar tokens from distinct languages which share the same context (neighbouring tokens) learn that such tokens are indeed similar to one another. We hypothesize that encouraging this behaviour can lead to the learning of token representations that beneficial for the downstream task of mudtilingual captioning since these tokens (from different languages) are likely to arise in the same generated report, albeit in different language. As such, the generated multilingual reports might be more likely to contain plausible tokens, and thus be clinically accurate. From hereon forward, we refer to this method replaced token language prediction (RTLP) and describe its mechanics next.
Source token selection Given tokens, {uf-re )5=. in a sequence of length. S, from source lan- guage, Lare E L, we first sample K distinct steps, {sx K where Sy E [] from a uniform distribution, U. We then replace the corresponding source tokens, {u ) Jkm with those from target language, {usz" K
We outline how to select the target language and token next.
1k=1
Target language and token selection For each source token, uhre, we sample target language, let U(L), uniformly at random from the set of remaining languages L' L 1sre where [ 1. Given legt, we now sample a target token, usz (VV..y uniformly at random from the language-specific vocabulary of tokens, Vi Such random sampling, however, can lead to the selection of a target token that is semantically different from the source token. As such, the network may discriminate between source and target tokens by using a detrimental shortcut that is based on semantics instead of language.
Lyt
To avoid this behaviour, we instead adopt strategy where the target token is likely to be seman- tically similar to (c.g., a noisy translation of) the source token. Formally, we quantify the cosine similarity, simy, between the source token embedding, e'qre E M and the embedding, ej E B of each token, " similarities, we form categorical distribution, 9. with elements, gj. from which we sample Aeg As the token embeddings become more meaningful during training, the sampled target token is more likely to be semantically similar to the source token.
Legt Vl.g in the vocabulary of the target language. By taking the softmax of these
cxp (sim;)
elare
Lage
lage %%
q
where
qj
Σ
Viegel m
exp (sim.)
simy
le
le
Objective function Equipped with a sequence of tokens in source and target languages, we predict the language of such tokens. To do so, define classification head, pe h. y R._.
4

---

Under review as a conference paper at ICLR 2022
parameterized by 2, that maps each token representation, h, probability distribution, y, over L languages. Formally, given a mini-batch of B captions of length, S, in the source language, I, we optimize the categorical cross-entropy loss. This is achieved for the tokens in the source (r 0) and target (r = 1) languages. To extend this to source languages, we load I. mini-batches and optimize the following multi-task categorical cross-entropy loss where II is the indicator function.
L
B
9
1 1, log Pu = lerelu ) 1,=1 logpolus. = -egclu (eae) (3)
CRTLP = LBS TET. ==1
5 EXPERIMENTAL DESIGN
Datasets We evaluate our framework on the PTB-XL dataset (Wagner et al., 2020 the only publicly-available dataset which comprises ECG signals with corresponding clinical report. Each ECG signal (from a total of 18,885 patients) is also associated with cardiac abnormality labels which we group into 5 classes (Strodthoff al. 2020). We provide further details in Appendix A.
Representation learning of cardiac signals
Supervised pre- tre-training remains effective way to learn rich, generalizable representations. As such, we pre-train the encoder, Jo. to map 12-lead ECG signals in the training set to cardiac abnormalities. We decide this task because ECG reports likely to reflect such abnormalities. Therefore, grounding the captioning process in representations of cardiac signals that can discriminate between cardiac abnormalities can be beneficial.
Representation learning of clinical reports To tokenize the ECG reports, exploit L language- specific tokenizers from SpaCy (see Appendix C.5 for details). For simplicity. we lower-case the text and remove any punctuation. By keeping track of unique tokens, we form L. distinct vocabularies. Each vocabulary also includes language-speciff tokens to indicate the start and end of the report, and the [PAD] and [OOV] tokens to refer to padded entries and tokens observed during inference that are not seen during training, respectively. We also introduce the [MASK] token where appropriate. The details of network architectures can be found in Appendix C.I
Multilingual cardiac signal captioning After pre-training the encoder, fa. and the decoder, Já- independently of one another, we exploit the learned parameters, {0, Ó). and token embeddings to solve the task of cardiac signal captioning (see Fig. left). After experimenting with several variants of our framework, we chose to freeze the encoder parameters and extract multiple representations per cardiac signal. Multilingual captioning also requires multilingual ground -trth ECG reports. Since such paired reports do not exist, we translate the original set of reports in English (en) to six lan- guages ( (German (de), Greek (el). Spanish (es), French (fr), Italian (it), Portuguese (pt)) using the Google Translate API, strategy similar to that adopted by Conneau et al. (2018). We open- source these reports] and provide further details in Appendix B Although translated reports can be imperfect ground-truth -tuth reports, we hypothesize (and indeed show) that the net effect of multilingual- ity on the quality and clinical utility of generated reports is advantageous.
Baseline Methods As the first to propose multilingual captioning of cardiac signals, we cannot trivially compare to previous methods. However, compare our pre-training method, RTLP, to the following state-of-the-ant pre-training methods: 1) MLM (Devlin et al. 2019), masked language modelling where the decoder is tasked with identifying masked tokens, 2) ELECTRA (Clark al.) 2020). where the decoder is tasked with identifying whether tokens have been replaced with those from an MLM model, and 3) MARGE (Lewis al.) 2020), multilingual generative language rep- resentation learning approach where source documents in various languages are retrieved to generate similar target document (see Appendix C33 for details how we adapted these methods).
Evaluation Metrics
We quantify the quality of the generated reports by comparing the degree of overlap of tokens such reports to those in the ground report. Specifically, we use the BLEU-- (Papineni et al. 2002). METHOR (Banerjee & Lavie 2005), and ROUGE. 2004) scores (see Appendix C.4 for more details).
"Code and data: https:tp/////. htapp:///iinyrre www.///////aacaseeee waw.eeaiiiaaaaaaaaiaaiaaaaaaa
5

---

Under review as a conference paper at ICLR 2022
Hyperparameters We conduct experiments using PyTorch Paszke al. 2019). We pre-train the encoder and decoder with patience value of 10 and 25 epochs, respectively, on the validation loss. When transferring the parameters the task of cardiac signal captioning. we use those asso- ciated with the lowest validation loss. When fine-tuning, we checkpoint the parameters associated with the highest validation BLEU score. We implement a greedy decoding mechanism by taking the argmax of the probability distribution over tokens at each step. We leave for future work other decoding mechanisms such as beam search (Graves 2012).
EXPERIMENTAL RESULTS
6.1 QUANTITATIVE EVALUATION OF GENER ATED REPORTS
We begin by quantitatively evaluating the ability of the pre-training methods to generate high quality multilingual reports. In Table II present the BLEU-1, METEOR, and ROUGE-L scores reports generated in seven different languages.
scores of
We find that RTLP performs par with state-of-the-art generative pre-training methods. For example, on average, RTLP achieves BLEU-1 = 28.5 whereas MLM and MARGE achieve BLEU-1 29.4 and 28.9, respectively. This finding holds across languages and evaluation met- rics. Furthermore, we find that RTLP outperforms the state-of-the discriminative pre-training method, ELECTRA. For example, average, RTLP and ELECTRA achieve ROUGE-L, 33.4 and 0.5, respectively. We note that we were unable to achieve satisfactory performance with ELEC- TRA despite extensive experimentation (please see code for reproducibility). One hypothesis for this stems from the difficulty of optimizing its objective function which comprises a generative and discriminative term. We also find that, regardless of the pre-training method implemented, perfor- mance varies significantly across languages. For example, RTLP achieves BLEU-1 19.8 and 33.5 on Greek (el) and English (en) reports, respectively. We hypothesize that this due to a high level of dissimilarity between the Greek vocabulary and that of the remaining languages. This, in turn, reduces the amount knowledge transferred from the other languages the Greek language. Overall, our findings suggest that RTLP can be effective discriminative multilingual pre-training method. Qualitative evidence to support this claim is provided in the next section.
Table I: Multilingual cardiac signal captioning performance of pre-training methods. Results are shown on the test set across five seeds. The standard deviation shown in brackets. For clarity, we have highlighted our method in gray. We find that RTLP performs on par with state-of-the-art language pre-training methods, MLM and MARGE.
Language Pre- Pre-training Method
German
Greek
English
Spanish (es)
French (fr)
Italian (i0)
Portuguese (pt)
Average
BLEU-]
al. 2019
25.9 (0.6)
(0.3)
31.3 (0.5)
(0.6)
30.3
(0.7)
(4.6)
ELECTRA (Crrk et 0120
0.1 (0.1)
0.2
0.2 (0.2)
0.31
0.6 (0.1)
0.5 (0.1)
0.5 (0.1)
0.3 (0.2)
MARGE
(1.0)
19.5 (0.9)
(0.5)
(2.5)
29.7 (0.6)
29.4 (0.5)
(1.0)
28.9 (4.8)
RTLP
(1.1
19.8 (0.6)
30.0 (0.7)
33.1 (0.9)
28.3 (0.8)
30.0 (0.1)
(1.0)
28.5 (4.5)
METEOR
MLM Devlin al.
2019
(1.0
23.6 (0.2)
(1.1)
38.6 (0.6)
33.5 (0.7)
33.9 (0.7)
38.8 (0.7)
(5.0(
ELECTRA (Clakk 1010
0.3 (0.5)
0.2
0.2 (0.1)
0.5 (0.4)
1.1 (0.3)
(0.))
0.5 (0.2)
0.5 (0.4)
MARGE
Lowis ot
2120
(1.5
22.2 (1.0)
36.5 (0.8)
37.1 (0.6)
33.1 (1.0)
32.9 (0.9)
37.8 (L.1)
33.6(5.1)
36.5 (0.7)
(2.6
36.0
38.5 (0.8)
(2.440
33.7 (0.9)
(7.6
(3.9
ROUGE-L
MLM Devlin et al. 2019
(0.8)
11.4 (1.9)
28.5 (0.2)
39.3 (1.3)
34.5 (0.8)
36.9 (0.5)
39.1 (0.6)
(3.3
ELECTRA
(.0)
MARGE
Low an 2020)
33.2 (0.7)
11.1 (2.3)
38.1 (0.5)
39.2 (0.6)
34.4 (0.8)
36. (0.6)
39.0 (0.5)
(9.4)
(4.0
36.3
33.1 (0:9)
(6.0
(0.9)
6.2 QUALITATIVE EVALUATION OF GENERATED REPORTS
We also manually inspect the multilingual clinical reports generated by RTLP. Ideally, such reports should accurately reflect clinical information. In Fig2, we present a 12-lead ECG segment alongside the multilingual ground-truth reports and those generated by RTLP and MLM.

---

Under review as a conference paper at ICLR 2022
There are three main takeaways from Fig First, RTLP allows networks to generate reports that accurately capture the aberrant morphology (shape) of the ECG signal. For example. in Spanish (es), the ground-truti -ruth and RTLP-gennrain queo de rama izquierda" " (left bundle branch block). Second, we find that RTL -TT.--generated reports manage capture critical aspects of the ECG signal that their MLM counterparts struggle with. For example, in English (en). both the ground-truth and RTLP-generatee reports both explicitly mention "left hypertrophy possible", whereas this phrase is noticeably absent from the MLM- --g-errerttee re- port. Such an absence is problematic as it might result in a physician overlooking this aspect, and thus failing to act accordingly. Lastly, we find that MLM-ggnerrate reports can include additional erroneous phrases which are neither present in the ground-truth report nor in the RTL.-genenrn report. For example, in Portuguese (pt), the --n-ggeratee report mentions "atrial flutter", which is noticeably (and correctly) absent from the remaining reports. Similar erroneous inclusions can also be found in the German (de) and French (fr) reports. This is also problematic since physicians can be misled by such statements, potentially resulting in unnecessary medical treatments. These findings indicate that RTLP allows networks to generate reliable and clinically accurate reports.
reports both explicitly mention the cardiac abnormality "blo-
German ide)
Spanish teal
Frends (Ie)
ueberdiclter licketyp
pa p-antetperrdiiee valcado cama teguleeda lipertoita loquiends powiibe 6,46 Informe confimado
blogauo
shristooondial
gaodhe lox
intawchenkotliiek linlddt purtiophhi indilih 4.66
beanche gauchs sloe de brundie gaudha hnperteopiic powible 6,45 suppott men cxetimae
inbertiigtt beriche
kiusdlsthhnn piiiistooannilll linkoty
dumn p-diismncaddile contemmdo
loquiendo del informe
type bloe breecie gruche hopertnophle confiree
eythma sinuual
basobbii 4.4 unbs etugger bencht
de
possibl riportt
sututllinmem' flatieaa udberdrchee tinkutyp eiatipeer
extnsiaol ventriculer cimno sinuaal lipe iaquicndo Hoquas iqqueeds
fihrillaation audecllate
type puuche beanche gouche ropport
VI
Gruandltuuaa nport
repoct
rxpart
English feal
type bock hypertophyy possible uncolffimed report
<iros rhythm -isiemenntise oweetumee keft left
bundle branch block lelt hogpetingh possibled t6 unsortaaeed repert
left bundle tranch blocs 46 unnordrreedd repect
atos rhythm oveeturned
Greek (el)
Itllian
Portugaene (pt)
geltua p- sintenecardiiee dvwntgomt agentagee remou ageruorr Boorn csuraa chore qui podkin opar sigj
dnussle sintoroo coddaate rowesciajo upo stnicro biooco di braus a bloco di branaa aisidto aisidto ipettofii simaats postiille 4.60 nelecto ran onfermato
cmioal p-entenocardise invertico tipo euguoedo Noqueioo de camo eiqueeco hiper aoda esquerdi possivel 4 adaticio nan confirmado
intogngeegie milkedi 6,40 pip immiiffanuuii quily cov copppaa
Sonnaaann Buii ensergooot auarrs virus agurenery Nenum smmrna cnoerne pertinn enpornerr) vntopagoela it milksort E.36 poy chijighnsuuurr ateppogh
sitno imusele sinictro cardiale rovescatoo tipo sinitzo Bloew di brunca uniero blocen di branss strat::
rime sirusd revirado tipo esqueeoo blogueoo de camo caquazdo hiperiodia eiqurria pa siiie & solaiinto náo contemaado
sperools dinietra poocibie 1.66 nevetto non confermato
vou peaquazerng
ditn sirusale albegpmmnno caperolto
diristro
Abrtaato atrial Elatter t euquredo Kieuuat de namo esqueedo hiperrotia esguedda possivel 4,60 ne nesooito stt confimmdoo
cesso auperoop beout unnumrdoenn dejua pinkic apurting) intapngegpa in milkeei 6.40 tad comptelansgiie envnpead
booco beanca siriago Hocoe di branda sintitoo
ppertneia smrat postistle 6.46 relerto nan condermaato
Figure 2: 12-lead ECG, multilingual ground truth reports, and those generated by RTLP and
ML.M. We show some phrases in bold which exhibit high level of agreement in both the ground- truth report and that generated by RTLP, in blue which are captured by RTLP and not captured by MLM (false negatives), and in red which MLM erroncously includes (false positives). Overall, we show that RTLP can generate reports that accurately capture the pathology of the cardiac signal.
6.3 QUANTIFICATION OF CLINICAL UTILITY OF GENERATED REPORTS
So far, we have qualitatively shown that RTLP is capable of generating clinically accurate and plau- sible reports. It could be argued, however, that such a qualitative evaluation presents a limited view of the clinical utility of these reports. More broadly, our evaluation setup can be questioned since are comparing generated reports to potentially noisy multilingual "ground-trutt reports, which were, for the most part, generated via translation. In this section, we aim to allay this concern and demonstrate the clinical utility of our generated reports in systematic manner, explained next.
To demonstrate that the translated ground-truth reports exhibit clinically useful information, we examine their ability to identify characteristics (e.g.. cardiac abnormalities) of the corresponding cardiac signal. The intuition is that a report which is highly predictive of such characteristics, which inform clinical decision-making, is likely to be of clinical utility. For example, this report could be
7

---

Under review as a conference paper at ICLR 2022
used for decision support. As such, we set out to learn model that maps translated ground-truth reports to a single cardiac abnormality label (5 classes) associated with the ECG. We represent these words (BoW) which Random Forest model is trained (see Appendix D further details). In F we illustrate the performance of such models when trained and evaluated on mutually-exclusive ground-truth reports in different languages (Target). We find that ground- truth reports are indeed highly predictive of cardiac abnormalities. This is evident by the strong performance of these models (e.g., AUC 0.90). Such a finding suggests that the translated ground-trutt reports are of clinical value.
reports bag-of-words
To demonstrate the clinical utility of the generated reports, we follow a procedure similar to the one just described with one exception. Instead of training model on ground-truth reports, we do so on held-out reports that are generated by a framework (e.g., RTLP). Importantly, we evaluate all models on the same set of ground-truth reports. The intuition here is that high quality generated reports should allow for the learning of predictive model that can generalize to the ground-truth reports. In Fig. we illustrate the performance of such models when trained on MLM- and RTLP- generated reports. There are two main findings here. First, we see that the generated reports exhibit lower clinical utility than ground-truth reports. This is evident by the & AUC when comparing the Target setting to the MLM setting. For example, in German (de), AUC ~ 0.92 + 0.82. Second, show that the generated reports predictive of cardiac abnormalities. This is evident by AUC of models trained on such reports. For example, in Italian (it), RTT. P--geerrated reports lead to AUC 0.84. Such a finding supports the claim that RTLP geen--geeertted reports are of clinical value.
Target
MLM
RTLP
1.0
0.9
AUG
0.8
0.7
pt
Figure 3: Performance of models trained on clinical reports, either ground-truth or generated, to predict cardiac abnormalities. In all experiments, Random Forest model is trained either on ground-truth reports (Target) those generated by MIL and RTLP Models are evaluated on mutually-exclusive set of ground-truth reports across five random seeds. Translated ground-truth reports and those generated, for example, by RTLP are predictive of cardiac abnormalities. Such finding demonstrates the clinical utility of reports, and by extension, the system that generates them.
6.4 INVESTIGATION OF THE CURSE OF MULTILINGUALIY
Multilingual neural systems can experience the curse of multilinguality (Conneau et al.. 20206). Concisely, this attributes the potentially poorer performance of multilingual models relative their monolingual counterparts to interference between the various languages. Intuitively, tasking a net- work with generating reports in multiple languages, analogous to multi-task learning (Caruana 1993). can be too demanding and thus hinder its ability to generate sensible reports. We explore this curse through the lens of the quality and clinical utility of generated reports. To do so, we first pre-train our networks, as per usual, and fine-tune them in the monolingual setting. We then com- pare the quality (BLFU--1 and clinical utility (AUC) of the generated reports the multilingual setting to those in the monolingual setting. Such comparison is presented in Fig.
In Fig. (left column), we find that MARGE does not experience the curse of multilinguality when evaluated along the dimension of either performance or clinical utility. This can be seen by the similar performance achieved by MARGE irrespective of whether it is fine-tuned in the monolin- gual multilingual setting. For example, when generating Spanish (es) reports, MARGE achieves BLEU-1 32 and AUC x 0.80 in both settings. Such a finding suggests that incorporating mul- tiple languages into the fine-tuning process has little to effect the quality and clinical utility
8

---

Under review as a conference paper at ICLR 2022
MARGE
RTLP
Manalingaal
Muttlingual
Monalingual
Multlingual
30
BLEU-1
20
10
09
pt
0.9
0.7
AUG
0.7
0.8
0.8
0.5
pt 0.5
07 85
pt.
Figure 4: Effect of multilingualltyy on the (top row) quality and (bottom row) clinical utility of the generated reports. The multilingual setting involves simultaneously generating reports in all seven languages, L fde, cl, en, es, fr, it, pt}. Results are shown across five random seeds. Through the lens of report quality and clinical utility, MARGE does not consistently experience curse of multilinguality. In contrast, RTLP benefits significantly from incorporating multiple languages into the fine-tuning process, phenomenon we refer to as the blessing of multilingualiy.
of the generated reports. This is in contrast to findings with RTLP (Fig. right column), which reflect a significant benefit from the inclusion of multiple languages. This can seen by the higher quality BLEU-1) 1) and clinical utility (1 AUC) of reports generated by RTLP in the multilingual setting than in the monolingual setting. For example, for French (fr) reports and both settings, the network achieves BLFU-1 R 30 and 2, respectively, and AUC 0.80 and 0.52, respectively. We denote this positive effect of multilinguality on the quality and clinical utility of reports as the blessing of multilinguality. We hypothesize that such benefits stem from the transfer and sharing of knowledge across languages. We also note the poorer performance of RTLP in the monolingual set- ting relative the multilingual setting. For English (en) reports, RTLP achieves BLEU-1 = 31, respectively. We hypothesize that this is due to the relative importance placed by the model on particular languages during multilingual RTLP pre-training. For example, such pre-training may implicitly weight languages differently. As such, the learned language-specific token representations may differ in their expressiveness. In light this, networks fine-tuned the monolingual setting may perform poorly.
and
DISCUSSION
In this paper, we proposed a neural multilingual cardiac signal captioning framework. In the process, designed a discriminative multilingual pre-training method, RTLP, which randomly replaced kens in caption with those from a different language and tasked the network with identifying the language of all tokens. We showed that -eneergee reports exhibit high quality and clinical util- ity, and are on par with those generated by state-of offte--f-the-artt pre- pre-training methods such as MLM. We also showed that generated reports exhibit higher quality and clinical utility when RTLP fine-tuned in multilingual setting than in monolingual setting, phenomenon we refer to as the blessing of multilinguality. Based on these findings, we recommend exploiting RTLP for multilingual cardiac signal captioning. Several exciting paths remain. These include the design of captioning frame- work that incorporates multiple modalities (e.g.. medical videos, electronic health records). Doing may result in higher quality reports and increase the likelihood of system adoption by medical professionals. order to further validate our framework, we aim exploit it to generate reports of higher level of complexity.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mikel Artetxe, Gorka Labaka, and Eneko Agire. Translation artifacts in cross-lingual transfer learning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7674-7684, 2020.
Satanjeev Banerjee and Alon Lavie Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Siddharth Biswal, Cao Xiao, Brandon Westover, and Jimeng Sun. Eegtotext: learning to write medical reports from eeg recordings. In Machine Learning for Healthcare Conference, pp. 513- 531. PMLR, 2019.
Siddharth Biswal, Cao Xiao, Lucas M Glass, Brandon Westover, and Jimeng Sun. Clara: Clinical report auto-completion. In Proceedings of The Web Conference 2020, pp. 541-550, 2020.
David Brailer, Eugene Kroch, and Mark V Pauly, The impact of computer-assisted test interpre- tation physician decision making: the case of clectrocardiograms. Medical Decision Making, 17(1):80-86, 1997.
Richard A Caruana. Multitask connectionist learning. In Proceedings of the 1993 Connectionist Models Summer School. Citeseer, 1993.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators, arXiv preprint urXiv:2003.10533 2020.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, pp. 7059-7069, 2019.
Alexis Conneau, Ruty Rinotr, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp- 2475-2485 2018
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint httt:22006139779 2020a.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudbary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsuper- vised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, 2020b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186. 2019.
Alex Graves. Sequence transduction with recurrent neural networks. arXiv:1211.3111,222 2012.
arXiv preprint
Sadid A Hasan, Yuan Ling, Joey Liu, Rithesh Sreenivasan, Shreya Anand, Tilak Raj Arora, Vivek Datla, Kathy Lee, Ashequl Qadir, Christine Swisher, et al. Attention-base medical caption gen- eration with image modality classification and clinical concept mapping. International Confer- ence of the Cross-Languet Evaluation Forum for European Languages, pp. 2018.
Springer,
Judith H Hibbard, Ellen Peters, Paul Slovic, Melissa L Finucane, and Martin Tusler. Making health care quality reports easier to use. The Joint Commission journal on quality improvement, 27(11): 591-604, 2001.
10

---

Under review as a conference paper at ICLR 2022
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjum Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross- cross-lingual tasks. arXiv preprint arXi::1909.0099: 2019.
Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. arXiv preprini arXiv: 1711111911...,. 2017.
Baoyu Jing, Zeya Wang, and Eric Xing. Show, describe and conclude: On exploiting the structure information of chest x-ray reports. arXiv preprint arXiv:2004. 12274, 2020.
Alla Keselman and Catherine Arnott Smith. A classification of errors in lay comprehension of medical documents. Journal of biomedical informatics, 45(6):1 2012.
Pavel Kisilev, Eli Sason, Ella Barkan, and Sharbell Hashoul. Medical image captioning: Learn- ing to describe medical image findings using multi-task-loss cnn. Deep Learning for Precision Medicine, Riva del Garda, Italy, 2016.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettle- moyer. Pre-training via paraphrasing. Advances in Neural Information Processing Systems, 33, 2020.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. Exploring and distilling posterior and prior knowledge for radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13753-13762, 2021.
Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Pe- ter Szolovits, and Marzych Ghassemi. Clinically accurate chest x-ray report generation. arXiv preprint arXiv: 190.0093, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint MrXiv:1907, 1692, 2019b.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans- actions of the Association for Computational Linguistics, 8:726-742, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task tank-agnostic visiolinguis- tic representations for vision- and language tasks. In Advances in Neural Information Processing Systems, pp. 13-23, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. arXiv preprint arXiv: 1912.0/73, 2019.
Vineel Pratap, Anuroop Sriram, Paden Tomasello, Awni Hannun, Vitaliy Liptchinsky, Gabriel Syn- naeve, and Ronan Collobert. Massively multilingual asr: 50 languages, model, billion param- eters. Proc. Interspeech 2020, pp. 4751-4755, 2020.
David Richley and Harriet Walters. Clinical guidelines by consensus recommendations for ECG reporting standards and guidance.. 2020. URL https::/ssss..ogg wtwp://ooerr.u ott.////ss////sst...rrr///ooooo.
uploads/2022 2ttp//200000//S421...2 1-....-eee. ECG Report inporriing.oo Suwdeliieeeeseeee June 20000000000 20200pdff
Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojeiech Samek. Deep learning for ecg analysis: Benchmarks and insights from ptb-xl. IEEE Journal of Biomedical and Health Infor- matics, 25(5):1519-1525 2020.
11

---

Under review as a conference paper at ICLR 2022
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 7464-7473, 2019.
C Peter Waegemann, Claudia Tessier, Andrew Barbash, Barry H Blumenfeld, Jonathan Borden, Robert M Brinson Jr, Todd Cooper, Peter L Elkin, J Michael Fitzmaurice, Susan Helbig, et al. Healtheare documentation: report on information capture and report generation. Newton, MA: Medical Records Institute, 2002.
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Wojciech Samek, and Tobias Scha-
effter. PTB-XL. large publicly available eleetrocardiography dataset, 2020. URL https:
//physionet org/ wontent/ttstttte xttpoxxx11..111...
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9049-9058, 2018.
Xianhua Zeng, Li Wen, Banggui Liu, and Xiaojun Qi. Deep learning for ultrasound image caption generation based on object detection. Neurocomputing, 392: 132-141,, 2020.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1628-1633 16391, 2020a.
Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Con- trastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020b.
12

---

Under review as a conference paper at ICLR 2022
A DATASETS
A. DATA PREPROCESSING
The ECG frames consisted of 2500 samples and consecutive frames had no overlap with one another. Data splits were always performed at the patient-level.
PTB-XL (Wagner al., 2020). Each ECG recording was originally 10 seconds with a sampling rate of 500Hz. We extract 5-seoond non-overlapping segments of each recording generating frames of length 2500 samples. We follow the diagnostic class labelling setup suggested by (Strodthoff al. 2020) which resulted in five classes: Conduction Disturbance (CD), Hypertrophy (HYP), Myocar- dial Infarction (MI), Normal (NORM), and Ischemic ST-T Changes (STTC). Furthermore, we only consider ECG segments with one label assigned to them. The ECG frames were standardized to follow a standard Gaussian distribution.
A.2 DATA SAMPLES
In this section, we outline the number of instances used during training.
Table 2: Number of instances (number of patients) used during training. These represent sample sizes for all 12 leads.
Dataset
Train
Validation
Test
PTB-XL.
22,670 (11,335) 3,284 (1,642) 3,304 (1,152)
A.3 VOCABULARY TOKENS
In this section, we outline the number of language-specific tokens available each language's vo- cabulary for the two datasets.
Table 3: Number of language-specific tokens each dataset
Dataset
de
el
fr
it
pt
PTB-XI.
2206
2662
1606
1950
1974
1866
2010
13

---

Under review as a conference paper at ICLR 2022
B TRANSLATION DETAILS
In this section, we outline the steps taken to translate the ECG reports originally found the PTB- XL dataset. We remind readers that although these ECG reports mixture of English and Ger- man, they are predominantly in the latter. As result, we treat German as the source language from which we translate the reports to other languages. More specifically, we follow these steps.
1. We leverage the Google Translate APp to first detect the source language of each ECG report. Although the majority of the reports are in German, some are in English, and this language detection step ensures that the ultimate translation of higher quality.
2. We continue to leverage the Google Translate API to translate ECG reports from the iden- tified source language to the target language of interest.
3. Due to imperfections the Google Translate API, certain ECG reports may not be trans- lated in full translated all. To minimize the incidence of such cases, we repeat Step 2 several times and stop once we reach the following criterion: we deploy the language detection module of the Google Translate API on the translated reports confirm that over 90% of them are indeed in the translated language. Although this implies that the final translated reports may have some noise, we found that this did not prevent our algorithm from learning appropriately.
14

---

Under review as a conference paper at ICLR 2022
C IMPLEMENTATION DETAILS
C.I NETWORK ARCHITECTURES
In this section, we outline the neural network architectures used for our encoder and decoder. More specifically, we use the architecture shown in Table for the encoder and that shown in Table 5 for the decoder.
Table 4: Encoder architecture used for experiments conducted on the PTB-XL dataset. P. Cin- and Cou represent the kernel size, number of input channels, and number of output channels, respec- tively. A stride of 3 was used for all convolutional layers. M represents the dimension of the final representation. We only use layer 5 when performing supervised pre-training. When captioning, layer 4 outputs 7 temporal features.
Layer Number
Layer Components
Kernel Dimension
Conv ID BatchNorm
7 12 32 (P Cin X Cou)
ReLU
(axPool(2)
Dropout(0.1)
Conv 1D
7 32 64
BatchNorm
2
ReLU
MaxPuol(2)
Dropout(0.1)
Conv ID
7 x 64 x 128
BatchNorm
3
ReLU
MaxPool(2)
Dropout(0.1)
Linear
128 M
4
ReLU
5
Lincar
M x C (classes)
Table 5: Decoder architecture used for experiments conducted on the PTB-XL dataset. = 300 represents the dimension of the representations from the encoder and the representations of the decoder tokens. H = 4 represents the number of heads used in each of the self and cross-attention modules. Clang represents the number of tokens in a specific language.
Layer Number
Layer Components
Kernel Dimension
Transformer Decoder Layer
E. H
2
Transformer Decoder Layer
E, H
Transformer Decoder Layer
E, H
Transformer Decoder Layer
E. H
Linear
E Chung
15

---

Under review as a conference paper at ICLR 2022
Table 6: Batchsize and learning rates used for training. The Adam optimizer was used for all experiments.
Stage
Batchsize
Learning Rate
Encoder
Supervised Pre-training
128
10
Decoder
MLM Pre-training
128
10
ELECTRR Pre-training
128
10
RTLP Pre-training
128
10
MARGE Pre-training
64
10a
Combined
Fine-tuning
128
10
C.2 ENCODER PRE-TRAINING
In this section, outline the task used to pre-train the encoder of the captioning system in pervised manner. Specifically, we learn an encoder, fo u E RxD + y E RC parameterized by 0, that maps P = 12 D-ddmenssional ECG signals, u, (where P represents the number of leads) to a C-dimensional output representing the probability assigned to each of the cardiac arrhythmia classes. When leveraging the PTB-XL dataset, C 5. For a mini-batch of size, B, and where e; represents the ground-truth class for particular instance, 2ú, we learn this behaviour by optimizing the following categorical cross-entropy loss.
1 B
Lee
Σ
Po = 0.)
We checkpoint, and eventually exploit, the parameters, 0, that coincide with the lowest loss observed on the validation set. This ensures that we use parameters that do not exhibit overfitting.
16

---

Under review as a conference paper at ICLR 2022
C.3 BASELINE IMPLEMENTATIONS
Masked Language Modelling. Masked language modelling (MLM) can be thought of as analogous to a denoising autoencoder. Inputs are perturbed and the network is tasked with generating the original, unperturbed version of the input. In the context of natural language processing, fraction == of the tokens in a sentence are chosen to be masked. Of these chosen tokens, 80% are replaced with the token [MASK], 10% are replaced with random token from the vocabulary, and the final 10% are not replaced at all. The motivation behind this task lies in the ability of the network leverage the context of masked tokens to correctly predict them. This, in turn, allows for the learning of rich representations. In our context, and to allow for a fair comparison to the multilingual pre-traim pre-training methods, we follow the original implementation introduced by Devlin et al. (2019) for each of the language mini-batches. More specifically. at each iteration, we load N mini-batches corresponding to N languages and perform MLM on each of these batches.
ELECTRA. ELECTRA, opposed to MLM introduced above, is discriminative language rep- resentation learning method. ELECTRA builds upon the implementation of MLM in the following ways. First, instead of masking tokens and tasking the network with generating the original token, ELECTRA performs a binary classification of whether a token was replaced or not. The motiva- tion for doing so lies in the alleged unnecessary complexity associated with generative language representation learning methods. Moreover, instead of replacing tokens with the [MASK] token, ELECTRA proposes to do so by exploiting the predicted outputs of MLM. This increases the likelihood that replaced tokens are in-distribution. As a result, ELECTRA simultaneously trains an MLM network and binary classifier. In our context, follow the original implementation introduced by Clark et al. (2020) for each of the mini-batches.
MARGE. MARGE is generative multilingual language representation learning method that ex- ploits source documents in various languages to generate text from a similar yet distinct target doc- ument. For example, M source documents with M S tokens are encoded and leveraged by decoder to generate the ' tokens in the target document. In doing so, the network is able to capture relationships between languages and thus learn representations useful for downstream multilingual tasks. In the original implementation (Lewis et al., 2020), similar documents need to be retrieved from database. In our context, however, our ECG reports are available N different languages and thus the target document is formed by report in one language and the source documents are formed by reports in the remaining N - languages. Since our ECG reports were translated from single original language, used reports in this language target documents. For PTB-XL, this amounts to using German.
C.4 EVALUATION METRICS
As we are mainly interested in the cardiac captioning task, we we leverage three automatic metrics com- monly used to evaluate image-captioning (BLEU score (Papineni et al. 2002). METEOR (Banerjee) & Lavie 2005) and ROUGE-L (Lin, 2004). At high-level, these metrics quantify the degree of overlap of n-grams between ground-truth sentence and a generated sentence. An n-gram can be thought of as a combination of tokens (words) that neighbour one another. For example, a 1- gram simply consists of all individual tokens in a sentence whereas 2-gram consists of all pairs of adjacent tokens in the sentence.
BLEU Score. The BLEU score first requires calculating the precision of n-grams for particular value of n. This precision is defined as the number of overlapping n-grams between the generated and ground-truth sentence, Overlapn. divided by the total number of n-grams in the generated sentence Total,- Such a calculation is repeated for multiple values of n E [1,..., .....,N N] before being averaged and weighted according to a brevity penalty, BP, which penalizes generated sentences which are shorter than the ground-truth sentence.
Overlap
BLEU - N = BP NII Precissoo))
Precision, Total
(5)
METEOR Score. The METEOR score designed, for the most part, to explicitly account for recall in n-gram overlap calculations, an operation not included in the BLEU score. Specifically, it aligns the ground-truth and generated sentences with one another, calculates the unigram (1-
17

---

Under review as a conference paper at ICLR 2022
gram) precision and recall, and derives an F-score with greater emphasis recall than precision.
Similar to BLEU, it weights the F-score with a sentence brevity penalty, BP.
10 Precision Recall
METEOR = 9 Precision + Recall
(6)
ROUGE Score. Although there exist multiple ROUGE scores (e.g.. ROUGE-N, ROUGE-L ROUGE-W. W, and ROUGE-S), we opt for the ROUGE-L score because it obviates the need to set value of n for the n-grams. Formally, ROUGE-L calculates the F-score (geometric mean of pre- cision and recall) of the longest common sub-sequence (LCS) of tokens between the ground-truth sentence with tokens and the generated sentence with tokens.
2 Precision Recall Precision + Recall
ROUGE I.
Precision LCS and Recall = LCS
C.5 CLINICAL REPORT TOKENIZERS
Throughout the manuscript, we exploit SpaCy as the main natural language processing package. Specifically, we use language-specifer pipelines that are publicly-availabb such as the following for the Italian language: htttss//spacy:: io models/it litee__too_ee______ttt______lee_ porr__ewss___ mcl Pipelines similar this allow us tokenize the reports in specific languages. For all languages, except for English, we use the pipelines with the suffix "core news md". For English reports, we use the pipeline with the suffix "core web md".
18

---

Under review as a conference paper at ICLR 2022
D QUANTIFICATION OF CLINICAL UTILITY OF REPORTS
In this section, we outline the implementation details for conducting the experiments that attempt to demonstrate the clinical utility of reports. In order to allay concerns about the quality of the trans- lated ground-truth reports, and to demonstrate the clinical utility of generated reports, we propose to train model that maps such reports to single-label cardiac abnormalities reflected the correspond- ing cardiac signals. The intuition is that reports which are highly predictive of such abnormalities (which are used for clinical recision-making are likely to of clinical value.
From a mechanistic perspective, we consider the translated ground-truth reports found exclusively in held-out set (3284 reports). We split this subset of reports into a training, validation, and test set using 50 : 20 30 split. This amounts to 1642, 656, and 986 reports, respectively. Given the relatively small size of the corpus, and the simplicity of the reports, and to avoid overfitting, we opted for a straightforward bag-of-words (BoW) unigram feature representation (maximum features 50). We train a Random Forest model on such representations in the training set to perform a multi- class classification of cardiac abnormalities. These abnormalities Conduction Disturbance (CD), Hypertrophy (HYP), Myocardial Infarction (MJ), Normal (NORM), and Ischemic ST-T Changes (STTC). We then evaluate the model on the mutually-exclusive set of reports in the test
In order to demonstrate the clinical utility of generated reports, we follow the same procedure as the one described above, with one exception. Instead of training on ground-truth reports, now train on generated reports. In all cases, we continue to evaluate on the same set of ground-truth reports in order allow for fair comparison between the models in both settings. The intuition of this setup is, once again, that reports of higher clinical utility should allow for a model that is more predictive of cardiac abnormalities.
19


Reviews:
[{"review_id": "ekRSXGXbYwm", "reviewer": "Reviewer_KZQE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper: \n1. The paper is well presented and the problem introduced in the paper is interesting as well as important. \n2. The authors have tried multiple pre-training methods to train their decoder and use a relevant auxiliary task to supplement the performance of the proposed model.\n3. The reports generated by the proposed framework seems to be quite coherent and similar to the gold standard captions.\n\nThe weaknesses of the paper: \n1. The authors have mentioned that though there has been work around EEG captioning (Biswal et al 2019, 2020), the previous papers did not explore the multilingual captioning of the cardiac signals. But in the case of the proposed framework the gold-label captions for languages other than English are trained on captions generated via Google Translate API. So the ceiling of the decoder model is as good as the Translate API itself. If that is the case, wouldn't it be better to just generate the English caption and then translate them via the API. The cited paper (Conneau et al) uses such a strategy to generate for different languages to augment the dataset and generate new translations which the model might not have seen. \n2. I do not understand the use case of generating all languages at once. The more suitable framework would be where the report is generated as per the requirement in which the source language is provided as an input to the decoder (<source_lg such as en, fr etc.><START>). \n3. If MLM performs as good or better, in most cases, for generating the captions. What would justify the use-case of RTLP? It is an interesting approach but I do not see an added advantage of the proposed model.\n4. The application is highly interesting but almost all parts of the proposed framework/models have been explored in a similar setting except for the multilingual aspect, which I am having a hard time understanding the use case of. In the proposed problem formulation, I would try to learn a really good cardiac signal captioning 'en' model and then either use the API itself or train a translator model. This would reduce the model parameters and make the whole framework more efficient.", "summary_of_the_paper": "The authors of the paper have introduced a multilingual cardiac signal captioning framework. The authors have proposed a neural framework that generates captions for the cardiac signals in multiple languages simultaneously. The authors add an auxiliary task of identifying the language of some of the tokens to improve the performance of the decoder. The proposed framework achieves on par performance with state of the art pre-training methods. ", "summary_of_the_review": "The paper proposed a solution to an interesting problem but as mentioned in the weakness section above I could not comprehend clearly the use-case of the multilingual decoder. But I would be highly interested in discussing with the authors and other reviewers during the rebuttal period if I might be missing the aspect of using multilingual data for the framework. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R5SqYYmumRF", "reviewer": "Reviewer_HBxE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is clearly written. The paper is easy to follow and understand.\n2. The targeted problems, i.e., cardiac signal captioning and multilingual captioning, are novel and important in both artificial intelligence and clinical medicine.\n3. The proposed multilingual cardiac signal captioning system is well-motivated, novel, and interesting.\n4. The experiments and analysis are extensive and solid.\n\nWeaknesses:\n1. The presentation can be further improved. In the Abstract and Introduction, can you give more explanations about \"Generating these reports, however, can be time-consuming and error-prone, while also exhibiting a high degree of intra- and inter-physician variability\"? For example, why generating these reports can be error-prone? What problems will be brought by the high degree of intra- and inter-physician variability (can you give some examples in the Introduction)? \n\n2. Although the targeted problems are important and novel, after reading this paper, I am still confused about how your proposed method is related to this cardiac signal captioning. I think the proposed approach can be used for conventional image captioning as well. In other words, have you solved the challenges and problems that are unique to the multilingual cardiac signal captioning?\n\n3. The experiment should be improved. Firstly, the evaluation metrics used in this paper, e.g.,  BLEU and ROUGE, are all general metrics for text generation tasks. So, it is unclear why your proposed approach can bring improvements. In what aspects can the proposed method improve the performance of the multilingual cardiac signal captioning? Secondly, in Table 1, why is your proposed method lower than the baselines in some settings? Can you give more explanations? Thirdly, the Google Translate model is not specifically designed for biomedical texts, so you can give more analysis of the Google Translate model. \n\n4. The paper is written in an optimistic tone that leads the reader to assume the proposed approach is rather good. However, I am more interested in knowing if the approach brings errors? And what type of errors does it bring? And why?\n\n5. The related work is insufficient. It is suggested to add more discussions about the report generation for other types of medical signals, e.g., chest X-rays, which have been widely explored in existing papers [1][2][3][4][5][6][7].\n\nMissing References:\n\n[1] On the Automatic Generation of Medical Imaging Reports. In ACL, 2018\n\n[2] Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports. In ACL, 2019.\n\n[3] When Radiology Report Generation Meets Knowledge Graph. In AAAI, 2020.\n\n[4] Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation. In CVPR, 2021.", "summary_of_the_paper": "This paper aims to build a multilingual cardiac signal captioning system to generate ECG reports, which describe the clinical findings in the input of electrocardiogram (ECG) signals. In particular, the proposed system can generate desirable and fluent reports in multiple languages, i.e., German, Greek, English, Spanish, French, Italian, and Portuguese. The experiments on a public dataset verify the effectiveness of the proposed approach, which performs on par with state-of-the-art language pre-training methods.", "summary_of_the_review": "The paper is well-written and the motivation sounds reasonable. The targeted problems, i.e., cardiac signal captioning and multilingual captioning, are novel and important. The presented approach is novel and interesting. Thus, I tend to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "No8x_RPCDoD", "reviewer": "Reviewer_VuBG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper is very well written. Multilingual text generation is an interesting task, and more multilingual work is needed in the ML for health space. \n\nWeaknesses: I have several concerns about the clinical utility of this task as well as the evaluation approach.  \n-\tFirst of all, I think clarification is needed to describe the utility of the task setup. Why is the task framed as generation of the ECG report rather than framing the task as multi-label classification or slot-filling, especially given the known faithfulness issues with text generation? There are some existing approaches for automatic ECG interpretation. How does this work fit into the existing approaches? A portion of the ECG reports from the PTB-XL dataset are actually automatically generated (See Data Acquisition under https://physionet.org/content/ptb-xl/1.0.1/). Do you filter out those notes during evaluation? How does your method compare to those automatically generated reports?\n-\tA major claim in the paper is that RTLP generates more clinically accurate reports than MLM, yet the only analysis in the paper related to this is a qualitative analysis of a single report. A more systematic analysis of the quality of generation would be useful to support the claim made in the appendix. Can you ask clinicians to evaluate the utility of the generated reports or evaluate clinical utility by using the generated reports to predict conditions identifiable from the ECG? I think that it’s fine that the RTLP method performs comparable to existing methods, but I am not sure from the current paper what the utility of using RTLP is. \n-\tMore generally, I think that this paper is trying to do two things at once – present new methods for multilingual pretraining while also developing a method of ECG captioning. If the emphasis is on the former, then I would expect to see evaluation against other multilingual pretraining setups such as the Unicoder (Huang 2019a). If the core contribution is the latter, then clinical utility of the method as well as comparison to baselines for ECG captioning (or similar methods) is especially important. \n-\tI’m a bit confused as to why the diversity of the generated reports is emphasized during evaluation. While I agree that the generated reports should be faithful to the associated ECG, diversity may not actually be necessary metric to aim for in a medical context. For instance, if many of the reports are normal, you would want similar reports for each normal ECG (i.e. low diversity). \n-\tMy understanding is that reports are generated in other languages using Google Translate. While this makes sense to generate multilingual reports for training, it seems a bit strange to then evaluate your model performance on these silver-standard noisy reports. Do you have a held out set of gold standard reports in different languages for evaluation (other than German)? \n\nOther Comments:\n-\tWhy do you only consider ECG segments with one label assigned to them? I would expect that the associated reports would be significantly easier than including all reports. \n-\tYou might consider changing the terminology from “cardiac arrythmia” categories to something broader since hypertrophy (one of the categories) is not technically a cardiac arrythmia (although it can be detected via ECG & it does predispose you to them)\n-\tI think it’d be helpful to include an example of some of the tokens that are sampled during pretraining using your semantically similar strategy for selecting target tokens. How well does this work in languages that have very different syntactic structures compared to the source language?\n-\tDo you pretrain the cardiac signal representation learning model on the entire dataset or just the training set? If the entire set, how well does this generalize to setting where you don’t have the associated labels?\n-\tWhat kind of tokenization is used in the model? Which Spacy tokenizer?\n-\tIt’d be helpful to reference the appendix when describing the setup in section 3/5 so that the reader knows that more detailed architecture information is there.\n-\tI’d be interested to know if other multilingual pretraining setups also struggle with Greek. \n-\tIt’d be helpful to show the original ECG report with punctuation + make the ECG larger so that they are easier to read\n-\tWhy do you think RTLP benefits from fine-tuning on multiple languages, but MARGE does not?\n\n", "summary_of_the_paper": "The goal of this paper is to develop an approach for generating multilingual ECG reports. The authors propose a new multilingual pretraining method in which tokens are randomly replaced with those from a different language, and the model must learn to identify the language of all tokens. The method – RTLP – performs similarly to MLM, ELECTRA, and MARGE according to BLEU-1, METEOR, and ROUGE-L metrics on generation of ECG reports, and the authors qualitatively assess that the generated reports are clinically accurate. The paper compares monolingual and multilingual versions of the models and find that RTLP benefits from multilingual training.", "summary_of_the_review": "Overall, the combined unclear clinical utility, lack of support for a major claim in the paper (that RTLP generates more clinically accurate reports), and concerns about the evaluation strategies lead me to recommend this paper for rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MPEaJFq8orr", "reviewer": "Reviewer_VEwE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed a novel multilingual pretraining language model settings. The idea is very interesting and the experiment setting is solid.   This paper conducted many result analyses which are helpful to understand the model's performance. \n\n1. Using BLUE and ROUGE as the metrics is not enough in the evaluation of clinical report generation. The precision of content is critical in clinical reports. High BLUE/ROUGE does not represent the report is well written. In my opinion,  the performance of clinical text generation should be validated based on its content, although it is very hard and requires extensive manual work. \n\n2. The gold-standard reports for non-English languages came from Google translation. This setting may lead to evaluation bias. It would be better to conduct a sensitivity analysis by using other translation tools.   \n\n3. It is not clear which network structure was used to represent the cardiac signals? Convolutional NN? Transformor? Other time series models?\n\n4. It is not clear how to build the categorical distribution when selecting the semantic-similar target language tokens (page 4).\n\n5. The \"blessing of multilinguality\"  comes from the low performance of RTLP in a single language. Based on this low baseline, the improvement from multilanguage has limited value. (RTLP + multilanguage can not beat MARGE).", "summary_of_the_paper": "This paper proposed a new report generation system for cardiac signals. It applied replaced token language prediction (RTLP) settings to improve the report generation performance. Experiments show that the proposed RTLP framework can achieve comparable performance with SOTA models. Extensive analyses were conducted to examine the diversity and the performance in multilanguage settings. ", "summary_of_the_review": "The proposed model is novel and interesting. While the evaluation metrics in clinical text generation are not persuasive. And some of the important details were missing in the current draft. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of generating captions for ECG signals where they extend the task in the literature from monolingual to multilingual captions. The model proposed in this work is a variant of mask language model where they augment the target by switching the words from one language to another. In addition to predicting the actual words, they also predict the language associated of the words.

Pros
+ The problem is motivated by real world application and need. 
+ The presentation is clear and the authors compare the performance of the model with appropriate recent models.

Cons
- The model has higher complexity in both training and parameters, yet achieves only comparable performance to simpler models.
- Empirical evaluation of the multi-lingual output is inadequate since the ​ground-truth is derived from Google Translate.
- There are also a number of other specific concerns raised by the reviewers (e.g., VuBG, HBxE).

Reviewers raised questions about several shortcomings. In response the authors updated the paper and were very engaged in the discussion with the reviewers. However, at the end of the day, the paper in its current form has serious shortcomings and left the reviewers unconvinced. I suggest the authors take advantage of the feedback from the reviewers and address them fully in their next iteration.
Decision: Reject