dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2021). A major issue holding back progress is that most current stateof-the-art MMT models  (Yin et al. , 2020; Elliott and Kádár, 2017; Wu et al. , 2021; Li et al. , 2022)  are trained solely on the ∼30k examples of the Multi30k dataset  (Elliott et al. , 2016) , comprising image captions and their translations. This causes two issues: (i) the models do not exploit the large amount of text-only data available and therefore perform poorly in comparison to state-of-the-art text-only MT systems, and (ii) we show that very few examples require images to be correctly translated, which means that the datasets are ill-adapted to evaluating the use of the image modality. In this article, we aim to overcome these problems by proposing (i) a new MMT approach that is able to exploit (text-only) monolingual and parallel data as well as (multimodal) captioning data, and that reaches a good balance between maintaining high MT quality and effectively exploiting images, and (ii) a test set, CoMMuTE, containing contrastive evaluation pairs, where images provide the necessary context to disambiguate between multiple meanings of the same source sentence.
Related Work Multimodal MT data. The reference dataset to train and evaluate MMT models is Multi30k  (Elliott et al. , 2016). However, recent work has shown that most MMT systems trained and evaluated on it do not effectively exploit the image information;  Elliott (2018)  showed that replacing the ground truth image with a random one does not lead to the drop in performance that would be expected, while  Wu et al. (2021)  argued that the observed gain in performance was due to a regularisation effect. It is also notoriously difficult to beat text-only baselines on this benchmark  (Barrault et al. , 2018). This may be due to (i) some subsets of Multi30k having been translated independently from the images  (Elliott et al. , 2016)  and (ii) most of the time, the source text being sufficient in theory to produce a perfect translation (i.
e. the image is not necessary; see Section 5. 2 for our own analysis). Based on this, alternative test sets and evaluation methods have been proposed. Caglayan et al. (2019)  proposed to probe the use of images in MMT models, while  Li et al. (2021)  proposed another training corpus and evaluation benchmark to evaluate MMT systems, but their work is only based on gender ambiguity and requires specific training data to train MMT models. Lala and Specia (2018) released a lexically ambiguous MMT evaluation dataset to evaluate models ability to disambiguate source sentences, but we found that text context is generally sufficient to translate the evaluation dataset correctly. Contrastive MT datasets. Another means of evaluating (and the one we adopt here) is to target specific phenomena through the use of contrastive test sets.
They involve evaluating models based on their ability to rank pairs of translations, where one is correct and the other incorrect. They have been used for the evaluation of different linguistic phenomena, including grammaticality  (Sennrich, 2017) , multi-sense word disambiguation (Rios  Gonzales et al. , 2017; Raganato et al. , 2019) , pronoun translation  (Müller et al. , 2018; Bawden et al. , 2018; Voita et al. , 2019)  and lexical coherence/consistency  (Bawden et al. , 2018; Voita et al. , 2019). Bawden et al. (2018)  introduced the idea of conditioning which of the translations is correct depending on linguistic context, and we adopt the same strategy here with our CoMMuTE dataset, composed of lexically ambiguous sentences whose translations are determined by the visual context.
We concatenate local (MDETR) features and global (CLIP) features to the text inputs. We choose to use global features too, since the source sentence can describe more general aspects of the image than mere objects (such as scenes). We jointly train the non-frozen parts of our model on two distinct objectives: multimodal MT (MMT) and visuallyconditioned masked language modelling (VMLM), as described in Section 3. 1. We also introduce a guided self-attention to exploit image information in a straightforward manner (see Section 3. 2) in the encoder (while the decoder uses regular self-and cross-attentions and can only attend to embeddings related to text positions). We call our approach Visually Guided and Adapted Machine Translation (VGAMT).
Combining training objectives As shown in Figure  2 , we jointly train VGAMT on two objectives: visual masked language modelling (VMLM) and multimodal MT (MMT). VMLM (resp. MMT) consists in predicting masked tokens (resp. translating the sentence) conditioned on the image. 5 The use of the VMLM objective in addition to MMT ensures that the model does not learn to ignore the visual inputs when translating (since Multi30k is mainly composed of very standard and unambiguous parallel sentences). We make sure to mask a high percentage (25%) of the text inputs so that the model is forced to attend to the image when producing translations.
Contrastive Multilingual Multimodal Translation Evaluation (CoMMuTE) To overcome the flaws of existing benchmarks (see Section 5. 2), we introduce CoMMuTE, a Contrastive Multilingual Multimodal Translation Evaluation dataset 6. It is composed of 155 lexically ambiguous sentences in English, each associated with two translations corresponding to two of the possible meanings of each sentence and two images that determine which of the translations is correct. It covers English→French, English→German and English→Czech. An example is given in Figure  4. 2018 ), and we created the remaining ones. 7 We collected two images for each sentence under Creative Commons license (either Google Images or our own photos), so that the image illustrates without ambiguity one of the two meanings of the sentence.
We do not restrict the image-text relation to be strictly descriptive (as for image captions) in order to have a more general evaluation dataset. Each sentence was translated into two possible translations (each corresponding to one of the images) by a native speaker of the target language. Appendix A provides some basic statistics. The idea of CoMMuTE is to use MMT models to rank each of the two translations based on image information. The perplexity of a sentence for a given model is defined as: P P L q (y) = N i=1 q(y i ) -1 N , where q is the probability distribution output by the model, N is the sequence length and y 1 ,. , y N is the sequence of tokens. Now, let y 1 ,. , y N 1 be the sequence of tokens of the correct translation and y ′ 1 ,. , y ′ N 2 the sequence of tokens of the incorrect translation, a model makes a correct prediction if P P L q (y) ≤ P P L q (y ′ ).
Text-only data All our experiments are based on the strong MT model mBART 8  (Liu et al. , 2020) , which we finetune on parallel text (see Table  1 ). We use Open- 1 2 We'll have to get rid of that mole. Il va falloir enlever ce grain de beauté. Subtitles2018 9   (Lison et al. , 2018) , Wikipedia  (Wołk and Marasek, 2014) , Ted Talks  (Reimers and Gurevych, 2020)  and the Books datasets (Tiedemann, 2012). We preprocess the data using Moses scripts  (Koehn et al. , 2007). 10.
, 2014)  for which the textual context is often sufficient for disambiguation. These benchmarks remain useful to make sure MMT systems do not perform worse than text-only MT models on examples where images are not necessary to translate correctly. However, we consider them insufficient to assess how well MMT systems exploit images to improve translation. Monolingual multimodal data. For the VMLM objective, we train our model on the Conceptual Captions (CC) dataset  (Sharma et al. , 2018)  composed of 3. 3M 12  images aligned with English text.
Baselines We consider several text-only and multimodal baselines. All baselines except the MT models finetuned from mBART were trained from scratch with the original codebases and features released by the papers' authors. Models trained on the (multimodal) MT objective only where trained on Multi30k, while models jointly trained on the (multimodal) MT and (V)MLM objectives were trained on Multi30k and Conceptual Captions. Text-only. We trained a text-only Seq2seq Transformer  (Vaswani et al. , 2017)  from scratch and a text-only Seq2Seq Transformer initialised from TLM weights  (Conneau and Lample, 2019). We refer to these models as Vanilla MT and TLM + MT respectively. We also trained several MT models initialised from pretrained mBART  (Liu et al. , 2020 ) and which we fine-tuned on parallel data  (Lison et al.
Results and Analysis Quatre cyclistes font une course sur un parcours avec une foule en arrière-plan. Quatre motards font une course sur un parcours avec une foule en arrière-plan. Tables  3  and 4  show BLEU, COMET and accuracy scores for all models compared on several En→{Fr,De,Cs} test sets including CoMMuTE. An initial observation is that the text-only model is a strong baseline on the three standard benchmarks (Test2016, Test2017 and MSCOCO). As mentioned in Section 5. 2, most of these evaluation datasets do not need visual context to be correctly translated. Our model VGAMT is on average on par with its counterpart text-only mBART+MT w/ adapters baseline for all Multi30k En→Fr test sets, while being on average just below this baseline on En→{De,Cs} Multi30k benchmarks. It outperforms other MMT models with a large margin due to both the effective use of textual knowledge from the frozen MT model but also guided self-attention.
Note that the scores reported for the baselines are lower than the ones reported in the original papers of the models for several reasons. First, we computed the scores on fully detokenised data to have a uniform evaluation between all models. We also report the average score from three different runs using different seeds and not the best score obtained over a single run. More importantly, our VGAMT obtains strong improvements over both text-only baselines and state-of-the-art MMT systems on CoMMuTE; our model can use visual context to disambiguate sentences. This can be seen in Figure  5  (one of the.
Limitations In this work, we focused on En→{Fr,De,Cs} multimodal MT. At the time of writing, our method can only be applied for En→X MMT. It is indeed necessary to have access to a modulated object detector in the source language to extract the features and the image-text relationship exploited by our model. This type of modulated object detector is only available in English for the moment. We leave the extension of our method to non-English source languages to future work. Moreover, our method requires large amount of captioning data to perform well. It is therefore computationally expensive. French, German and Czech. Given the region-text couples extracted by MDETR (Figure  9a ), only the token fans can attend to the MDETR region features. The normalised attention scores of the embedding of the token fans on these regions are low in comparison to the scores on the text part and on the CLIP embedding.
Acknowledgements This work was granted access to the  HPC resources of IDRIS  under the allocation  2022-AD011013908  and  2022-AD011012254  made by  GENCI. It was also partly funded by the last four authors' chairs in the PRAIRIE institute funded by the  French national agency ANR  as part of the  "Investissements d'avenir" programme  under the reference  ANR-19-P3IA-0001.
g. functional dedifferentiation or hemispheric asymmetry reduction, as well as cognitive outcomes. Furthermore, we compare the results from this simulated mechanistic framework against empirical imaging and cognitive data of a large population-based sample of older adults. To this end, we developed personalized brain network models (BNMs), which integrate the individual SCs from the diffusion-weighted imaging (DWI) data of the participants of 1000BRAINS dataset (n = 649, age-range = [55-85] years, n females = 317)  (Caspers et al. , 2014)  to simulate functional data. Fig. 1  provides an overview of the general workflow. Each individual BNM is an ensemble of nodes, representing the different brain regions, constrained by the corresponding SC, which serves as the weighted adjacency matrix for the network.
Eligibility for MRI measurements included any history of neurosurgery, cardiac pacemakers, coronary artery stents, surgical implants or prostheses in head or trunk, tattoos or permanent make-up on the head. Participants gave written informed consent prior to inclusion in 1000BRAINS and MR imaging was only conducted as there were no dental implants found which could cause artifacts in the brain images and as participants did not experience claustrophobia. The study protocol of 1000BRAINS was approved by the Ethics Committee of the University of Essen, Germany. In the current work we focused on particularly older adults (above 55 years, n = 970), from which a total of 871 had functional scans available. 16 participants had to be excluded due to insufficient data quality. Of these 855 subjects, 718 also had diffusion data available, from which 649 were of good quality (see more details under Imaging).
Structural image processing Using the CAT12 toolbox implemented in SPM12  (Ashburner, 2009) , we created tissue probability maps (TPM) for gray matter (GM), white matter (WM) as well as corticospinal fluid (CSF) from the participant's T1 data. These brain masks were used to optimally extract the brain from the T1 data by superimposing and thresholding them at 0. 5 (small enclosed holes were filled). The T1 brain was bias field corrected, rigidly aligned to MNI152 template space and resampled to 1. 25 mm isotropic voxel size. We corrected the diffusion MRI data for eddy current and motion artifacts including interpolation of slices with signal dropouts  (Andersson et al. , 2016). Subsequently, all diffusion data were visually controlled for ghosting, remaining signal dropouts or very noisy data and volumes or datasets that were considered as suboptimal were excluded from further analysis (n = -69).
For dMRI -T1 alignment, the first b0 images from each dMRI data with b1000 and b2700 were extracted and rigidly aligned to T1 dataset using mutual information as cost function  (Wells et al. , 1996). Based on the estimated transforms, the dMRI data were transferred to the individual T1 space, separately for both b-values. Implicitly, the realignment resamples the data to 1. 25 mm and rotates the b-vectors according to the corresponding transformations. As there are no field maps or b0 volumes with opposite EPI readout directions available in the current study, we computed Anisotropic Power Maps (APM) from the b2700 dMRI data in 1. 25 mm space to account for susceptibility artifacts and optimize image registration  (Dell'Acqua et al. , 2014). APM contrasts are very similar to those of the T1 image and therefore provide an optimized base for image registration.
Hence, APMs were used to compute the non-linear transformation from diffusion to anatomical space, thereby taking EPI induced distortions into account. These non-linear transformations were then used to transform the TPMs to diffusion space. Finally, the two datasets with b1000 and b2700 were merged into one single file and corrected for different echo times. This correction was computed by a voxelwise multiplication of the b2700 data with the ratio of the non-diffusion-weighted data respectively for the two datasets. Lastly, local modeling and probabilistic streamline tractography were performed using the MRtrix software package version 0. 3. 15  (Tournier et al. , 2012). We computed the constrained spherical deconvolution (CSD) local model using multi-tissue CSD of multi-shell data  (Jeurissen et al.
The virtual aging brain pipeline For the investigation of the causal link between white matter degeneration and functional changes and its association with the heterogeneity of cognitive decline, we performed a two-fold analysis. In the first part, we addressed the questions whether realistic brain function can be modeled via a simulated SC decline. After we verified that the presence of a decline in the interhemispheric connectomes in empirical SC dataset, we designed virtual brains, brain network models constrained by the imaging data of SC, to predict brain function from the empirical SC dataset (SC emp ) as well as one virtually aged participant (SC α ) (see Predicting functional connectivity: The Connectome-based Brain Network Model) and again, compared the derived (in-silico) functional data between each other as well as to the empirical (in-vivo) functional data (see Comparing estimated in-silico and in-vivo functional data) to ensure the SC changes of the virtual brains could reproduce age-related functional changes.
Predicting functional connectivity: the connectome-based brain network model The approach to virtually age a structural connectome provides the basis to test whether a specific modification in SC affects brain function in a causal sense. Therefore, as a preliminary investigation, we inspected the empirical SC datasets with regards to age and specifically tested whether the previously reported strong age-related interhemispheric SC decline  (Puxeddu et al. , 2020; Roland et al. , 2017)  is also applicable to the current dataset of older adults (see Fig. 2 ). Furthermore, we performed a virtual aging process on the 50 youngest participants (mean age, age-range) by homogeneously decreasing the interhemispheric connections. In particular, we applied a mask on the empirical connectome W 0 as follows: W = W 0 -αM inter * W 0 (3) where M inter is the antidiagonal mask, α is the normalized intensity of decrease and * represents an element-wise product (see Fig.
Specifically, for each dataset we conducted a parameter sweep or grid search for the global parameter in the range G = [1. 5 -3. 2] with sampling interval ΔG = 0. 05 and the noise variance in the range σ 2 = [0. 01 -0. 05] with Δ σ 2 = 0. 01 to determine the optimal couple (G, σ 2 ) for which SI is maximal. The direct numerical integration for the nodal equations (2) was performed via Heun-stochastic integration implemented in the Virtual brain open-source code. For each region, we integrated the equations with variable time steps (from 0. 005 ms to 0. 0005 ms) for min and we downsampled the variables r i (t), V i (t) to the sampling frequency fs = 100 Hz. The time step was variable to adapt the integration process and avoid numerical errors or instabilities and the neural mass field was filtered via the Balloon-Windkessel model  (Friston et al.
, 2000)  to emulate fMRI time series with repetition time TR set to ms. For each model fit (the 649 empirical and 20 × 50 = 1000 virtually aged connectomes), we obtained a G modulation index which is representative of how much the SC influences the brain dynamics. Before investigating if the SC-FC causal link was related to behavioral factors such as age, sex and cognitive performance, we compared the in-silico brain dynamics to in-vivo data assuring that a variety of dynamic and static properties of functional covariates are indeed realistic.
The simulations were performed with a G that was distributed according to a uniform prior (truncated between 1. 5 and 3. 2), such that the final posterior estimation resulted in: p(G SBI ⃒ ⃒ X empirical ) = p(X simulated |G SBI ). As highlighted by the formula, the SNPE was repeated on each subject, trained on the simulated data and the final estimation of G was obtained by maximizing the log likelihood of the features of the empirical BOLD data under the model. The SNPE provides the probabilistic statistical relation between G and the empirical data, and the final estimate of G SBI was considered as the maximum value of the estimated posterior distribution. For each individual SBI, the noise variance was set to the optimized value obtained via the parameter sweep. The quality of SNPE was evaluated as the Pearson correlation between G SBI and G Sweep as well as the trend of G SBI with age.
Quantification and statistical analysis To test whether we could actually reproduce the dynamical and statical properties of FC, we estimated linear trends of the homotopic FC and FCD variance difference features in the functional empirical data, the age-simulated data and the alpha-simulated data. Specifically, we performed Pearson correlations between the functional summary statistics and age or the alpha masking variable, respectively. The model-derived tethering between SC and FC (G modulation index) was then statistically related to both demographic and behavioral factors. First, we performed an ANCOVA indicating whether the G index is overall age-or sex-related (corrected for education). Furthermore, to analyze age-related trajectories of the SC-FC linkage, we performed partial correlations between the G modulation index and age (corrected for sex and education).
This was repeated after stratifying the group by sex, and the partial correlations (corrected for education) were statistically compared (Fishers-Z) indicating whether age-related G modulation index changes are different in males as compared to females. Lastly, we performed a Pearson's correlation to test the relation between the G modulation index of the virtually aged dataset and the alpha-variable. According to various aging theories  (Festini et al. , 2018; Reuter-Lorenz and Park, 2014) , changes in the SC-FC tethering might explain the variability of cognitive performance during the aging process. To test this hypothesis, we performed ordinary-least squares (OLS) multivariate regressions for each of the 16 cognitive scores, testing the relation between the cognitive performance and the G-coupling index, while correcting for age, sex and education.
The virtual aging brain model replicates age-related functional changes The summary statistics of the simulated data of the BNMs constrained by the individual SC emp (Fig. 2(B), (E) ) display a declining agetrend conforming to the empirical data (Fig. 2(A), (D) ), which indicates a fading interhemispheric communication and dynamical variability. Similarly, the summary statistics emerging from the virtually aged BNMs (constrained by SC α ) display declining trends with increasing α, similarly to the empirical dataset as well as the BNMs constrained by the SC emp. We also investigated the landscape of brain fluidity (σ 2 full ) for the different G values of the parameter sweep and for different age of the SC emp (Fig. 2 (G)), as well as for different α values applied to a sample target out of the 50 subjects (Fig.
, 2016). As a result of our framework, the simulated functional data of the BNMs, constrained by the empirical SC, displayed a declining age-trend in FC comparable to the empirical data: i. e. we observed lower homotopic FC with older age. When constraining the BNMs by homogeneously manipulating SC, i. e. when we decreased the interhemispheric SC, we also observed declining trends in FCD variance with increasing ɑ, again in line with the empirical data and the empirically constrained BNMs. Firstly, this mechanistically supports the hypothesis that interhemispheric SC serves as a basis for homotopic FC  (Mollink et al. , 2019; Roland et al. , 2017). Secondly, FCD variance has previously been reported to be a proxy of brain fluidity, which has been linked to cognitive flexibility  (Naik et al.
Data and code availability The code can be found in the following link (https://github. com/ ins-amu/virtual_aging_brain). The code for the Virtual Brain is also public available at the following link (https://github. com/the-virtual-br ain/tvb-root), while learning material to master the platform can be found at the TVB hub website (https://hub. thevirtualbrain. org/). The Structural Connectivity data are available on EBRAINS platform at the following DOI (https://doi. org/10. 25493%2F6640-3XH), while the fMRI time series and the cognitive scores are available upon reasonable request by sending an email to the lead contact or to SC. After the creation of a personal account and the approval of its request, any EBRAINS user could download the data by accepting the EBRAINS data users agreement.