Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals

1 INTRODUCTION . Cardiac signals convey a significant amount of information about the status of a patient . Upon recording these signals , an expert cardiologist is often required to interpret the findings , manually generate an accompanying textual report ( Richley & Walters , 2020 ) , and share it with fellow physicians ( and patients ) . Such a report is crucial within medicine , impacting communication between physicians , facilitating clinical decision-making , and holding care teams accountable for their actions ( Waegemann et al. , 2002 ) . Manually generating these reports , however , can be time-consuming and error-prone , while also exhibiting a high degree of intra and inter-physician variability . Combined , these can detract from a physician ’ s time with a patient , thus decreasing patient satisfaction , and hinder communication between physicians ( Hibbard et al. , 2001 ; Keselman & Smith , 2012 ) , thus potentially compromising patient outcomes ( Brailer et al. , 1997 ) . One way to address this challenge is by collapsing a clinical report into multiple pertinent medical conditions and designing a system that identifies such conditions . However , abstracting away the details of a report can obscure the full assessment of a patient ’ s health status and defining the pertinence of conditions can be disease-specific and non-trivial . Based on these observations , we look to address the following question : how do we automatically generate clinically accurate reports that reliably summarize cardiac signals ? To address this question , the fields of visual-language representation learning and captioning hold promise . In this setting , rich representations of visual inputs and language are learned to automatically generate captions . Representations of the latter can be learned in a discriminative monolingual setting , as with ELECTRA ( Clark et al. , 2020 ) , or in a generative multilingual setting , as with MARGE ( Lewis et al. , 2020 ) . However , a discriminative multilingual framework has yet to be explored . Moreover , previous work has focused exclusively on the captioning of medical images ( Hasan et al. , 2018 ; Zeng et al. , 2020 ) . It has not explored the captioning of cardiac time-series signals . In this paper , we address the outlined question by designing a neural , multilingual , cardiac signal captioning system while exploiting a large-scale electrocardiogram ( ECG ) database . Our contributions are the following : ( 1 ) We propose a discriminative multilingual representation learning method , replaced token language prediction ( RTLP ) , in which we randomly replace tokens with those from other languages , and task a network with predicting the language of all tokens . ( 2 ) We show that RTLP allows networks to generate reports , in multiple languages , that are as clinically accurate as those generated by masked language modelling ( MLM ) . To the best of our knowledge , we are the first to propose a multilingual cardiac signal captioning system . ( 3 ) We conduct extensive studies to show that generated reports exhibit higher quality and clinical utility when RTLP is fine-tuned in a multilingual setting than in a monolingual setting , a phenomenon we refer to as the blessing of multilinguality . 2 RELATED WORK . Language representation learning Representation learning is integral to modern natural language processing . Generative tasks such as MLM show promise ( Devlin et al. , 2019 ; Liu et al. , 2019b ) , even in the multilingual setting ( Conneau & Lample , 2019 ; Conneau et al. , 2020b ; Liu et al. , 2020 ) . MARGE ( Lewis et al. , 2020 ) retrieves documents , in potentially different languages , and attempts to reconstruct a target document . Others propose to jointly learn textual and visual representations ( Sun et al. , 2019 ; Lu et al. , 2019 ; Zhang et al. , 2020b ) . Similar to our work is ELECTRA ( Clark et al. , 2020 ) , where tokens are replaced with those from a generative model , and a network is tasked with discriminating between the original and replaced tokens . This approach remains computationally expensive where both an MLM and discriminative network are learned . It also does not optimize a multilingual objective . To the best of our knowledge , we are the first to propose a discriminative multilingual representation learning method in the context of cardiac signals . Multilingual representation learning Pre-training and fine-tuning networks on multiple languages confers benefits to NLP tasks ( Conneau et al. , 2020b ; Pratap et al. , 2020 ; Conneau et al. , 2020a ; Artetxe et al. , 2020 ) . For example , Conneau et al . ( 2020b ) and Artetxe et al . ( 2020 ) show that multilingual pre-training outperforms its monolingual counterpart in solving downstream NLP tasks . These findings have been partially driven by monolingual datasets that are machine-translated to multiple languages , as with XNLI ( Conneau et al. , 2018 ) . We similarly translate ground-truth ECG reports into multiple languages in order to guide the generation of multilingual reports . Although Huang et al . ( 2019 ) propose a pre-training setup similar to ours , we explore more languages and define a different pre-training task ( RTLP ) which we exploit for cardiac signal captioning . Captioning in healthcare Biomedical image captioning has traditionally focused on chest X-rays ( Kisilev et al. , 2016 ; Jing et al. , 2017 ; Hasan et al. , 2018 ; Jing et al. , 2020 ; Zeng et al. , 2020 ; Liu et al. , 2021 ) . Liu et al . ( 2019a ) condition their captioning system on the medical topic to be discussed and Wang et al . ( 2018 ) propose a multi-level attention model that attends to both the image and the text . Previous work which captioned electroencephalogram ( EEG ) signals ( Biswal et al. , 2019 ; 2020 ) did not explore a multilingual representation learning method and does not extend to the generation of reports in multiple languages . To the best of our knowledge , we are the first to propose the multilingual captioning of cardiac signals . 3 BACKGROUND . 3.1 CARDIAC SIGNAL CAPTIONING . We begin by assuming access to a dataset , D = { xi , capi } Ni=1 comprising N cardiac signals , x ∈ RD , and their associated captions , cap = { ws } Ss=1 , which consist of S words , ws . The goal of cardiac signal captioning is to generate a caption ( report ) that reliably summarizes the physiological state of a patient as manifested in a cardiac signal . To extract features from the signal , an encoder , fθ : x ∈ RD → { vt ∈ RM } Tt=1 , parameterized by θ , maps a D-dimensional instance , x , to a set of T representations , { vt ∈ RM } Tt=1 , each of which is M -dimensional ( see Fig . 1 left ) . To convert the captions into a format ingestible by a network , we first convert each word , ws , in a caption , cap = { ws } Ss=1 , to a token , us , to form a sequence of tokens { us } Ss=1 . Such tokenization can involve lower-casing and stemming words , in addition to removing punctuation . After deriving tokens from all captions in a training set , we form a fixed vocabulary , V = { ui } Ci=1 , of the C unique tokens where |V | = C. We then define an embedding matrix ( lookup table ) , E ∈ RC×M : u→ e ∈ RM , which maps each token , u , to an M -dimensional token embedding , e. These embeddings are typically randomly-initialized and learned in an end-to-end manner via gradient descent . As such , every caption can now be represented as a sequence of token embeddings , { es } Ss=1 . To extract features from language , a decoder , gφ : { vt } Tt=1 , { es } Ss=1 → { hs ∈ RM } Ss=1 , parameterized by φ , attends to all encoder representations , { vt } Tt=1 , while mapping a token embedding , es , at step , s , to an M -dimensional token representation , hs . Each token representation , hs , in the sequence of representations , { hs } Ss=1 , is then fed into a linear classification head , pω : hs ∈ RM → ys ∈ RC , to output a probability distribution , ys , over the C tokens in the vocabulary . This forms a sequence of outputs , { ys } Ss=1 . At each step , s , in the sequence , the goal is to maximize the likelihood of generating the token of the next step , us+1 . Therefore , by identifying the most probable output token at each step , we can form a sentence of words ( i.e. , a caption ) . 4 METHODS . 4.1 MULTILINGUAL CAPTIONING OF CARDIAC SIGNALS . At the surface , a monolingual captioning framework would appear to be sufficient for physicians communicating in a single language . However , the motivation for a multilingual captioning framework , in which reports are simultaneously generated in multiple languages , is threefold . First , a multilingual framework would obviate the cumbersome process of having to train a distinct model for different languages ( Conneau et al. , 2020a ) . Second , recent work has demonstrated the benefits of incorporating additional languages into the learning process ( Artetxe et al. , 2020 ) . Lastly , a multilingual framework exhibits greater flexibility than its monolingual counterpart , as it can always be collapsed , during inference , to generate captions from a single language . To enable multilingual captioning , we first assume access to L language-specific datasets , { Dl } Ll where Dl = { xi , capli } Ni=1 comprises N cardiac signals , x , and captions in a specific language , l ∈ L = { en , es , · · · } , where en and es represent English and Spanish , respectively . Note that the cardiac signals are shared across the datasets . We follow the same encoder-decoder approach mentioned in the previous section with one exception . We replace the single classification head with L language-specific heads to account for the distinct vocabularies of the L languages . In doing so , we exploit recent observations that demonstrated the utility of having a network with parameters that are both language-specific and shared across languages ( Zhang et al. , 2020a ) . Formally , for each language , l , we have a linear classification head , pωl : h l s ∈ RM → yls ∈ RC , with ωl ∈ { ωen , ωes , · · · } reflecting language-specific parameters . Each head maps the token representation , hls , at each step , s , in the sequence to a probability distribution , y l s , over C tokens where C ∈ { |Ven| , |Ves| , · · · } reflects the size of a language-specific token vocabulary . When doing this for each step in the sequence , we arrive at a set of probability distributions , { yls } Ss=1 . As with traditional language models , at each step in the sequence , s ∈ [ 1 , S ] , we maximize the likelihood of observing the next token in the sequence , uls+1 , from a particular language , l ∈ L. Therefore , for a mini-batch of B captions in a single language , we would optimize the categorical cross-entropy loss at each step , s. To extend this to L languages , we load L mini-batches and optimize the following multi-task categorical cross-entropy loss . Lmultilingual = − 1 LBS L∑ l∈L B∑ i=1 S∑ s=1 log pωl ( y l i , s = u l i , s+1 ) ( 1 )

Summary: This paper aims to build a multilingual cardiac signal captioning system to generate ECG reports, which describe the clinical findings in the input of electrocardiogram (ECG) signals. In particular, the proposed system can generate desirable and fluent reports in multiple languages, i.e., German, Greek, English, Spanish, French, Italian, and Portuguese. The experiments on a public dataset verify the effectiveness of the proposed approach, which performs on par with state-of-the-art language pre-training methods.