Document-Level Machine Translation with Large-Scale Public Parallel Corpora (2024)

Abstract: Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a sentence level. This is primarily due to the severe lack of publicly available large-scale parallel corpora at the document level. We release a large-scale open parallel corpus with document context extracted from ParaCrawl in five language pairs, along with code to compile document-level datasets for any language pair supported by ParaCrawl. We train context-aware models on these datasets and find improvements in terms of overall translation quality and targeted document-level phenomena. We also analyse how much long-range information is useful to model some of these discourse phenomena and find models are able to utilise context from several preceding sentences.

heading: 1 Introduction
text: Machine translation has traditionally been framed as a problem of translating source text to target text one sentence at a time. However, depending on the languages and the content of the text being translated, it is often the case that a sentence is impossible to translate well in isolation – that further contextual information is required. Maruf et al. (2021) summarise several discourse phenomena that are impossible for sentence-level machine translation systems to deal with – including anaphoric pronouns, lexical cohesion, deixis, and ellipsis. There are also features like grammatical gender, number, style, and formality, that can sometimes not be determined from the individual sentence but are dependent on surrounding context. Therefore, it has been clear for many decades (Bar-Hillel, 1960) that a machine translation system cannot translate some sentences without the ability to capture other linguistic cues from context. Läubli et al. (2018) also
showed that while sentence-level neural machine translation can appear high-quality out of context, human evaluators had a much stronger preference for human translation when evaluating translation at the document level.
As a result, there have been many efforts to incorporate document context into neural machine translation. What almost all of these methods have in common is that they require parallel training data with document context.
ParaCrawl (Bañón et al., 2020) produced largescale parallel corpora and the released data includes information about the URLs from which the sentences were extracted, but the released corpora were only sentence-level. We use raw webpage text publicly available from ParaCrawl along with the officially released sentence-level corpora to assemble large-scale document-level parallel corpora for several language pairs. We release our code to generate the document-level datasets1 as well as the datasets in five selected language pairs2.
We then validate the usefulness of our datasets by training context-aware translation models for all of these language pairs, and find that models that are aware of target context perform better than sentence-level baselines, often in terms of overall translation quality, but more significantly when evaluated with respect to targeted discourse phenomena. We experiment with varying the amount of preceding context available to these contextaware models at test time, and find that while information from the immediately previous sentence is most useful – as is intuitively obvious, longer-range context up to a certain extent can also help models translate phenomena like anaphoric pronouns more accurately.
1https://github.com/Proyag/ParaCrawl-Context 2https://huggingface.co/datasets/Proyag/
paracrawl_context. More language pairs may be released in the future; it requires a resource-intensive but simple process of running our code on any ParaCrawl language pair.
13185

heading: 2 Related Work
text: Many attempts to utilise document context in machine translation have introduced specialised architectures to encode context (Tu et al., 2018; Jean et al., 2017; Kuang et al., 2018; Maruf and Haffari, 2018; Voita et al., 2018; Miculicich et al., 2018) alongside source sentences.
Some methods like Junczys-Dowmunt (2019); Sun et al. (2022); Post and Junczys-Dowmunt (2023) eliminate sentence boundaries altogether and use a standard transformer architecture to translate an entire chunk of text as a single sequence. However, unless they retain some sentence markers like in Junczys-Dowmunt (2019) or Tiedemann and Scherrer (2017), these models can be difficult to evaluate with our existing evaluation paradigms and metrics due to the dependence on sentencelevel test sets. In those cases, we often have to rely on sentence-splitting heuristics and alignment methods just to be able to compute a sentence-level metric on the model outputs. As a result, our work chooses a method to translate with one sentence at a time as input, but with document context provided as a separate additional input. This allows the model to benefit from context information while still being simple to evaluate with existing metrics and test sets.
There are few existing parallel corpora of significant size that retain document metadata – examples are Europarl (Koehn, 2005) which had only around 2M sentences for the largest language pairs; CzEng (Kocmi et al., 2020; Bojar et al., 2016) containing 61M sentence pairs with document annotation along with more than 100M synthetic sentence pairs, but only for eng↔ces; OpenSubtitles (Lison and Tiedemann, 2016); and News Commentary (Kocmi et al., 2023), where the latter two datasets are relatively large corpora for several language pairs but restricted in domain.
The CCAligned corpus (El-Kishky et al., 2020) includes hundreds of millions of comparable document pairs across many languages, from which sentence-level datasets were extracted and released. A dataset with sentence pairs and corresponding document contexts was not created; however, it should be possible to extract a similar dataset as the one we present here from the available data released by CCAligned which includes documents, URLs, and sentences.
Closest to our data, Al Ghussin et al. (2023) used publicly available parallel document metadata
from ParaCrawl3 to extract aligned paragraphs of text, and used these paragraphs as a proxy for documents. Even though their extracted datasets were at a relatively small scale due their use of only a subset of ParaCrawl data and strict filtering, they observed clear improvements in targeted evaluations of document-level translation phenomena.
Post and Junczys-Dowmunt (2023) showed that using only monolingual documents and backtranslating (Sennrich et al., 2016) them sentence by sentence to produce synthetic document pairs can surprisingly produce better results than using actual document pairs to train a document-level model. Their results, however, were mostly on unreleased private data, and their comparison could not be reproduced on public data precisely because of the absence of public datasets of adequate size.
Another recent orthogonal approach is to use large language models’ inherent ability to model long context to perform document-level translation (Wang et al., 2023; Zhang et al., 2023; Karpinska and Iyyer, 2023) with no or very few parallel training examples. While this paradigm is gaining popularity, it is yet to be comprehensively explored, and the need remains to have large-scale datasets of parallel text with document context.

heading: 3 Dataset
text: At the time of its release, ParaCrawl (Bañón et al., 2020) was the largest publicly available sentencelevel parallel corpus for most of the languages it supported. The ParaCrawl corpus mining process included steps to match documents that were estimated to be translations of each other, from which sentences were extracted and aligned, but unfortunately, the released corpora did not preserve document context or structure, and only contained isolated sentence pairs along with the source URLs they were originally extracted from.
However, separately, a lot of the raw text crawled from the web was also released4 as languageclassified base64-encoded text with their corresponding URLs. Therefore, we were able to match the webpage contents to their URLs in the sentencelevel parallel corpora to recover the corresponding documents.
To build document-level parallel datasets from these sources of data, we chose five language pairs – Czech (ces), Polish (pol), German (deu), French
3https://www.statmt.org/paracrawl-benchmarks/ 4https://paracrawl.eu/moredata
(fra), and Russian (rus), all paired with English (eng) – and used the following method:
1. Extract the source URLs and corresponding sentences from the TMX files from ParaCrawl release 95 (or the bonus release in the case of eng-rus). Each sentence is usually associated with many different source URLs, and we keep all of them. 2. Match the extracted URLs with the URLs from all the raw text data and get the corresponding base64-encoded webpage/document, if available. 3. Decode the base64 documents and try to match the original sentence. If the sentence is not found in the document, discard the document. Otherwise, keep the 512 tokens preceding the sentence (where a token is anything separated by a space), replace line breaks with a special <docline> token, and store it as the document context. Since some very common sentences correspond to huge numbers of source URLs, we keep a maximum of 1000 unique contexts per sentence separated by a delimiter ||| in the final dataset. 4. Finally, we compile three different files per language pair – a dataset with all sentence pairs where we have one or more source contexts, one with all sentence pairs with target contexts, and a third dataset with both contexts.
Even though the TMX files have source URLs for all released sentences, this process was lossy
5https://paracrawl.eu/releases
due to a few different reasons: • ParaCrawl was compiled from a number of
separate crawls or “collections”, and there were inconsistencies in how URLs were formatted in intermediate steps. We employed some basic heuristics to match as many URLs as possible, such as removing http:// and https:// and trailing slashes before matching, but there is a still a chance some URLs were missed in this process. • Data from CommonCrawl was not duplicated in the released raw text from ParaCrawl. To avoid re-downloading huge amounts of data, any URLs that were present in CommonCrawl but not in the other collections are missing from our dataset. • There were many instances where the original sentence could not be found in the contents of a webpage corresponding to its source URL. This is most likely due to the same URLs being crawled at different times and finding dynamic or possibly entirely changed content.
Due to the existence of multiple matched documents for some sentences in the datasets, source and target contexts for a sentence pair may not be aligned. However, approximately 99.9% of all extracted sentence pairs have exactly one source/target context, which implies that the contexts should be aligned in most cases. Further filtering is recommended if aligned contexts are required, with the simplest option being to remove the subset of sentence pairs with more than one matched context.
The sizes of our extracted datasets are shown in Table 1. Some samples can be found in Appendix A, and the full datasets are publicly available at https://huggingface.co/ datasets/Proyag/paracrawl_context.

heading: 4 Document-level Translation Models
text: To evaluate the usefulness of our datasets, we train document-level translation models using only our datasets. Even though higher-quality documentlevel training data exists at a smaller scale, we choose to train our models only on data from ParaCrawl in order to accurately evaluate the quality and utility of our data and to make a fair comparison with our sentence-level baselines.
At training time, for each input example, we first sample one context out of up to 1000 that are present in the document-level dataset. To ensure
that the models are capable of using variable-length context at test time, we uniformly sample a context length l from {1, . . . , 256} for each training example. We then retain at least l tokens from the preceding context, possibly exceeding the limit to avoid mid-sentence splits. This context sampling was implemented using a custom pipeline6 in the Sotastream toolkit (Post et al., 2023). We then use the source sentence as the main model input, the sampled context as a second input (see detailed model architecture in Section 4.1), and the target sentence as the target model output.
We train separate models for each language pair using either source or target context information. While the model is always provided ground-truth context at training time, this is not always available in the case of target context at test time, so we test using both ground-truth target context and real predicted output context. However, we note that one of the most common use cases of machine translation is in the context of computer-assisted translation (CAT) tools, where translators can see preceding context but typically machine translate and post-edit one sentence at a time, as a result of which gold-standard target context is available for each sentence.

heading: 4.1 Model Architecture and Training
text: We use the dual-encoder transformer architecture from Junczys-Dowmunt and Grundkiewicz (2018) but without tied parameters between the two encoders, implemented in the Marian framework (Junczys-Dowmunt et al., 2018). In other words, we modify a standard transformer encoder-decoder model (Vaswani et al., 2017), which takes the source sentence as input and produces the target sentence as output, to add a second encoder which takes additional source/target context as input. This is similar in spirit to Zhang et al. (2018), but we do not incorporate the context encoding into the source encoding, instead directly feeding both encodings to the decoder. As shown in Figure 1 of Junczys-Dowmunt and Grundkiewicz (2018), the decoder has two stacked cross-attention sub-layers to attend to the two encoder contexts. We use default transformer-big hyperparameters. This choice of architecture is less complex than the specialised architectures described in Section 2, but still allows for separating the current sentence and context in-
6https://github.com/Proyag/sotastream/blob/ custom_pipelines/sotastream/pipelines/sample_ from_fields_pipeline.py
puts, giving us greater control over evaluation and interpretability compared to models which translate an entire large chunk of text at a time. Moreover, the addition of the second encoder automatically accounts for the need for extra model capacity to encode the document context.
We train context-aware models for the following language pairs: eng→deu, eng→fra, eng→rus, eng→ces, and pol→eng.
We train our models with dynamic batch size to make optimal use of GPU memory. We train all models on 4 or 8 Nvidia A100 or 3090 GPUs, using gradient accumulation to ensure that the average effective batch sizes are approximately equivalent in each case. We validate every 50 million target tokens for eng→rus and every 500 million target tokens for all other language pairs, early stopping when cross-entropy calculated on the validation set does not improve for 10 consecutive validations.

heading: 4.2 Test Data for Evaluation
text: To assess the translation quality of our models, we perform two kinds of evaluation – general machine translation quality metrics, and using contrastive evaluation to measure the accuracy of the models on targeted discourse phenomena.

heading: 4.2.1 General Translation Quality Metrics
text: We compute standard sentence-level quality metrics – BLEU (Papineni et al., 2002) using the sacreBLEU implementation7 (Post, 2018) and COMET8 (Rei et al., 2022) – on the following WMT test sets, all of which were released with document metadata: WMT22 eng→deu and eng→ces (Kocmi et al., 2022), WMT23 eng→rus (Kocmi et al., 2023), WMT20 pol→eng (Barrault et al., 2020), and WMT15 eng→fra (Bojar et al., 2015).

heading: 4.2.2 Contrastive Evaluation
text: Contrastive test sets consist of input text and translations which appear correct at the sentence level, but may be wrong given more context. Models are evaluated by their ability to assign higher probability to the sentences that are correct in context. We evaluate our models on a few different contrastive test sets which measure the following types of discourse phenomena for specific language pairs:
• Anaphoric pronouns: The ContraPro test sets for eng→deu (Müller et al., 2018) and
7BLEU|#:1|c:mixed|e:no|tok:13a|s:exp|v:2.4.0 8Specifically wmt22-comet-da
eng→fra (Lopes et al., 2020) translation evaluate the accuracy of pronoun translation where the source English sentence does not contain enough information to determine the correct pronoun in the target language and context is required to generate the correct translated pronoun. One part of the DiscEvalMT test set (Bawden et al., 2018) also evaluates anaphoric pronoun translation in eng→fra. • Deixis and ellipsis: Good Translation Wrong in Context (GTWiC) (Voita et al., 2019) is a collection of contrastive test sets to evaluate a number of discourse phenomena in eng→rus translation, among which are deictic expressions, verb phrase ellipses, and correct inflection of nouns which depend on elided verbs. • Lexical choice: The DiscEvalMT contrastive test set from Bawden et al. (2018) tests lexical choice in eng→fra translation where the choice of certain words/phrases in translation is ambiguous without context information. It also tests lexical cohesion, i.e. the repetition of translated entities when the entity is repeated on the source side. An eng→ces extension of the lexical cohesion subset was created by Jon (2019). GTWiC also has a similar subset to test lexical cohesion in eng→rus.

heading: 5 Results and Analysis
text: We train sentence-level and document-level models in a few different configurations for comparison. Our baseline is a standard sentencelevel transformer-big model trained on all of the ParaCrawl parallel data for a given language pair.
We also train sentence-level baselines trained on the subsets of sentence pairs for which source or target contexts could be extracted, thus ensuring a fair comparison in terms of the number and content of training examples. Finally, for each language pair, we train two different document-level models: one which is aware of source context and one which uses target context. We further test the target context model in two different scenarios: using original ground-truth context for each test example and using the actual model output from previous sentences within a document as target context.

heading: 5.1 Effect on Overall Translation Quality
text: One of the ways we evaluate our documenttranslation models is simply in terms of overall sentence-level translation quality metrics. The results are summarised in Table 2.
We find that while source context does not seem to benefit overall translation quality, or at least not in a way that is reflected in these metrics, using the ground-truth target context generally improves translation quality over the baseline using the same number of training examples, i.e. “Subset - target” compared to “Gold target context” in Table 2. This improvement is not observed for eng→rus, probably due to the small number of training examples in the subset of sentences with target context not being enough to train a high-quality contextaware model. The sentence-level baseline using the full set of ParaCrawl sentence pairs (“Sentencelevel”) out-performs the context-aware models for the smaller language pairs, benefitting from having a much larger number of training examples.
A machine translation system in an environment where translated output is post-edited sentence by sentence, such as in CAT tools, has access to ground-truth target context for every line, and can thus benefit from the improved translation quality of the context-aware model. However, a fully automated document-level translation pipeline does not have this information available. To reproduce this scenario, we also try using actual model predictions from previous sentences within a document as target context (“Predicted target context” in Table 2), and find that this does not yield the same improvement as using ground-truth context. This could be explained as a manifestation of exposure bias (Ranzato et al., 2016) due to the models only being trained on ground-truth contexts and not being robust enough to accurately use the relatively noisy predicted context, resulting in errors being propagated through the context. In some cases, the difference may not even be an obvious error, but could instead be related to domain/style hints that are available in the original context to guide the translation but are lost in the context predicted by the model. While models can be made more robust against the propagation of errors through preceding context using methods like scheduled sampling (Bengio et al., 2015) to expose some generated context to the model during training, the loss of contextual hints is more difficult to remedy.

heading: 5.2 Accuracy on Contrastive Test Sets
text: We also perform evaluations on selected contrastive test sets for some language pairs, as mentioned in Section 4.2.2. Each example in a contrastive test set has a source sentence and source/target context along with two or more possible outputs, one of which is correct. We use our models to score all the possible outputs and say the model gets an example right if it assigns higher probability to the correct output than to the other options. We then calculate accuracy over the entire test set.
While the contrastive test sets include source context, we report results in this section only on our target context-aware models since, consistent with Table 2, we find that our source context-aware models are unable to outperform sentence-level baselines. Since each contrastive test set is different and designed for specific languages, we discuss them separately in this section.

heading: 5.2.1 ContraPro (eng→deu and eng→fra)
text: We use ContraPro (Müller et al., 2018) to evaluate the accuracy of pronoun translation for our eng→deu models. This test set contains examples of sentence pairs where the source English sentence contains the pronoun it which needs to be translated into one of es, sie, or er in the target German. The pairs are designed so that provided context information is required to determine the correct translation of the pronoun. Each example has two translations which are both apparently correct at the sentence level but one of them uses the wrong pronoun in context. Our models are not required to generate translations, only to score each alternative output given the source sentence and ground-truth target context. If the model is able to take the context into account, it should assign higher probability to the option with the correct pronoun.
A similar test set created by Lopes et al. (2020) evaluates the same phenomenon in eng→fra translation, where the English it is translated to il or elle in French and they is translated to ils or elles.
We find that while our eng→deu sentence-level model has an accuracy of 0.507, i.e. approximately random chance, the model with target context scores 0.785 when provided 5 sentences of preceding context. This shows that the model learns to accurately disambiguate the correct choice of pronouns using the context information.
For eng→fra, we find that the sentence-level model already achieves a reasonably high accuracy
of 0.815, due to the fact that the antecedents of the pronouns are located within the same sentence in approximately 43% of the examples in this test set. Using our target context-aware model, we still see an increase in accuracy to 0.824 given a single sentence of preceding context, but no further improvement with longer context.
Effect of Context Size Figure 1 shows the effect of the amount of context that is exposed to the eng→deu model on its ability to accurately translate the anaphoric pronouns in ContraPro. We find that a single sentence of target context is enough to significantly increase the accuracy of pronoun translation beyond a sentence-level baseline’s 0.507, and context longer than 3 sentences does not make much of a further difference in terms of total accuracy. This makes intuitive sense, since the antecedent of a pronoun is most often in the immediately preceding sentence and only very rarely more than 2 or 3 sentences away. However, for the subset of 442 examples (out of 12000) where the antecedent distance is greater than 3, the accuracy increases from 0.709 for the baseline to 0.908 for the context-aware model, which indicates that our model is in fact able to use long-range information to disambiguate pronouns.
5.2.2 Good Translation Wrong in Context (eng→rus)
Voita et al. (2019) introduced a collection of test sets for contrastive evaluation of a range of discourse phenomena in eng→rus translation. The test set is thus divided into 4 subsets:
• Deixis: These examples are related to gender and formality marking in Russian that are absent in the source English. The model needs to use context to translate these deictic words or phrases correctly. • Ellipsis: These examples have elliptical constructions in the English text that cannot be elided in Russian, so the translation needs to expand the ellipsis. There are two kinds of ellipsis-related errors targeted here: where the target text has wrong morphological inflection due to missing information from the source ellipsis, and where the wrong verb is generated for a verb phrase ellipsis. • Lexical cohesion: These examples evaluate the ability of the model to ensure that named entities that are repeated in the source are translated consistently in the output. Models
Model / Context Length
GTWiC Accuracy
Deixis Ellipsis
LC Infl. VP
Sentence-level 0.5 0.5 0.058 0.458 Trg context/1 0.586 0.5 0.07 0.468 Trg context/2 0.654 0.494 0.07 0.472 Trg context/3 0.692 0.5 0.074 0.472
Table 3: Accuracy of our target context-aware model on the GTWiC test sets with varying number of sentences of target context. “LC” denotes lexical cohesion. While performance on deictic expressions improves steadily with more context, lexical cohesion only improves very marginally, and verb phrase (VP) ellipsis accuracy remains very low.
need to be aware of preceding context to translate cross-sentential repetitions consistently. Unlike ContraPro, contrastive examples in GTWiC have several incorrect translations and one correct translation for each given source sentence and context. A test example is considered correct if our translation model scores the correct translation higher than all of the incorrect translations. We report accuracies separately for each GTWiC test set evaluating different phenomena.
The performance of our target context-aware model on the GTWiC test sets is reported in Table 3. We observe that the context-aware model is significantly more capable of translating deictic expressions accurately. However, we find that it does not perform well on the ellipsis test sets, with verb phrase ellipsis accuracy surprisingly being worse than chance, and improvements on lexical cohesion are also marginal. This is possibly because the models need both source and target context to be able to model these phenomena accurately. For example, it is difficult for the model to be aware that an entity should be repeated if is not aware that both the preceding source and target contexts had occurrences of the same entity.
A maximum of 3 context sentences is available per example in GTWiC, and we once again find that having more target context can be useful for the model to translate deictic expressions correctly, but the benefits diminish as the model usually gets adequate context from the last one or two sentences.

heading: 5.2.3 DiscEvalMT (eng→fra and eng→ces)
text: The DiscEvalMT eng→fra contrastive test sets (Bawden et al., 2018) evaluate two document-level
translation phenomena: • Anaphora: Similar to ContraPro, this test set
also contains examples where correctly generating a pronoun in the target French requires the model to use context information about the antecedent. • Lexical choice: Some of these examples contain an ambiguous word in the source and the model needs to be able to use context information to disambiguate the correct sense of the word and translate it correctly. The rest of the examples test models’ ability to consistently translate a repeated word or phrase. The eng→ces extension of DiscEvalMT (Jon, 2019) only includes the lexical choice test set.
These test sets also have two alternative translations for each input sentence and context, and our models are expected to score the correct option in context higher than the other option.
We can see in Table 4 that while our documentlevel models do not score very highly on this benchmark, they still out-perform the sentence-level baseline. Similar to GTWiC, for the lexical cohesion test sets, we believe that the models would perform better if they were able to access both source and target contexts, since they are otherwise unaware of the repeated word or phrase.

heading: 6 Conclusion
text: We release large-scale document-level parallel corpora in five language pairs extracted from the ParaCrawl datasets in an effort to mitigate the dearth of publicly available machine translation training data with document context. We also open-source code to enable the community to compile such datasets in more language pairs. Due to both the ParaCrawl pipeline and our code be-
ing open-source, it is theoretically possible to create document-level datasets for any supported language by crawling the web, and not just those already released by ParaCrawl.
While we treat any preceding text at the same URL as “context”, it is often the case that these are completely unrelated to a given sentence, such as UI elements, boilerplate text, or entirely unrelated content on the same webpage. Future work should also explore filtering these datasets to retain only genuine contextual information, which is likely to be much more useful to the model, although even content that is not strictly document context may help guide translation through indirect domain or style cues.
Our document-level translation experiments show that models aware of target context improve in terms of overall translation quality as well as in terms of some targeted discourse phenomena compared to a sentence-level baseline. We show that machine translation can benefit from multiple sentences of preceding context to accurately translate discourse phenomena like anaphoric pronouns, although very long-range context is rarely useful.

heading: 7 Limitations
text: Relevance of context Our work assumes that any extracted text preceding a given sentence on a webpage is relevant “document context” for that sentence. However, it is likely in many cases that the extracted context is unrelated to the sentence, since most webpages are not formatted as a coherent “document”. As a result, the dataset often includes irrelevant context like lists of products, UI elements, or video titles extracted from webpages which will not be directly helpful to document-level translation models.
Unaligned contexts For sentences with multiple matching contexts, the source and target contexts may not always be aligned. However, as mentioned in Section 3, the vast majority of sentence pairs have exactly one source/target context, and should therefore have aligned contexts. We recommend filtering on this basis if aligned contexts are required.
Availability of both contexts Our models are all trained with either source or target context being available to the models, but for some documentlevel phenomena like lexical consistency of repeated named entities, it is probably necessary for the model to be aware of both source and target
context. Our datasets make it possible for future work to extract training data with both contexts and train such models.
Model quality Our models are trained only on noisy ParaCrawl data and tested on high-quality WMT data. While there are much smaller but relatively high-quality document-level training datasets available, all our experiments were conducted only on ParaCrawl data to test the quality of the datasets without being influenced by other data. As a result, these models are not necessarily the strongest possible translation models, but they are useful to fairly and clearly compare document-level machine translation against sentence-level models.
Language coverage ParaCrawl was focused on European Union languages with only a few “bonus” releases for other languages. Moreover, most of the corpora were for English-centric language pairs. Due to the high computational requirements to extract these corpora, our work further chose only a subset of these languages, resulting in corpora for only a few European languages, some of them closely related. Given the availability of raw data and tools to extract such corpora for many more languages from all over the world, we hope the community is encouraged to build such resources for a much larger variety of language pairs. Documentlevel translation phenomena also vary widely by source and target language, so such experiments for more languages is left for future work.

heading: 8 Ethical Considerations
text: Harmful content The main released corpora from ParaCrawl were filtered to remove sensitive content, particularly pornography. Due to pornographic websites typically containing large amounts of machine translated text, this filtering also improved the quality of the resulting corpora. However, when we match sentences with their source URLs, it often happens that an innocuous sentence was extracted from a webpage with harmful content, and this content is present in our document contexts. We may release filtered versions of these corpora in the future, pending further work to filter harmful content at the document level.
Eurocentricity The Eurocentric nature of our work is remarked upon in Section 7. Due to most large-scale publicly available parallel corpora being English-centric, almost all machine translation research remains English-centric, at the cost of the
majority of the world’s language users. This limits both the generalisability and usefulness of a lot of research. We hope the release of more data and tools helps the community expand these efforts to more languages.

heading: Acknowledgements
text: This work was funded by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546].
Some of the computations described in this research were performed using the Baskerville Tier 2 HPC service (https://www.baskerville.ac.uk/). Baskerville was funded by the EPSRC and UKRI through the World Class Labs scheme (EP/T022221/1) and the Digital Research Infrastructure programme (EP/W032244/1) and is operated by Advanced Research Computing at the University of Birmingham.
We acknowledge computational resources provided on Karolina by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).

heading: A Samples from Released Datasets
text: In this appendix, we present some examples from the datasets to demonstrate the format and content of the extracted data. The contexts have been truncated here due to space constraints, but they still illustrate the usefulness of the context to the translation of the sentence. Note that while these examples have been hand-picked and generally show useful context, we still see some noise in the contexts like text fragments in the wrong language or the presence of many short uninformative lines.
eng→fra with target context Source sentence: Its entwined cobbled pathways and network of tunnels underground are interesting to look round. Target context: EXCURSIONS À PARTIR DE PRAGUE <docline> | Croisière sur la rivière Vltava avec dîner | Shuttle d’aéroport | <docline> . . . . . . <docline> Il s’agit d’une mine d’or exposée en Bohème, avec des châteaux magiques et des petites villes éparpillées dans la campagne, et touchée par les forêts denses de la chaîne de
montagne Šumava le long de la frontière avec l’Autriche. La petite ville de Tábor est charmante, et elle a été habitée par les taborites au quinzième siècle. Target sentence: Ses chemins pavés entrelacés et son réseau de tunnels souterrains sont intéressants à observer.
pol→eng with target context In this example, we see some noise in the form of Polish text fragments appearing in the English context. Source sentence: Stypendium można przeznaczyć na dowolny cel. Target context: START 2020 recruitment launched - Fundacja na rzecz Nauki PolskiejFundacja na rzecz Nauki Polskiej <docline> . . . . . . <docline> The principal criteria in the competition are the quality and originality of the candidate’s scientific accomplishments to date, as well as his or her single most important research achievement. In recent years the amount of the one-year stipend has been PLN 28,000. Target sentence: The stipend may be used by the laureate for any purpose.
eng→deu with source context Source sentence: The evening will bring clouds with rain or sleet. Source context: °C <docline> 2 °C <docline> 2 °C <docline> 2 °C <docline> 2 °C <docline> 1 °C <docline> Air pressure <docline> 1014 hPa <docline> . . . . . . <docline> Tomorrow <docline> In the early morning it will be mainly cloudy, but mostly dry. Before noon clouds with rain or sleet will dominate. Target sentence: Die Mittagszeit bringt wechselhaftes Wetter mit ab und zu etwas Regen.
eng→fra with both contexts This example contains broken sentence fragments, but is a good example of the context being required to disambiguate a pronoun in the target French – the translation of “it” to “la” requires context about the grammatical gender of the antecedent. Source sentence: Previously, it appeared only below 1,600 metres. Source context: aside to give to my sister.” <docline> The family used coffee revenues to pay his sister’s schooling, and his brother’s. Mr. Zikusoka followed in his father’s footsteps so that
he too could provide for his family. When he got married in 2005, he received a half-hectare of farmland and decided to grow coffee. <docline> . . . . . . <docline> He notes that coffee leaf rust disease, which usually affects coffee at altitudes lower than 1,400 metres, has now surfaced at 1,800 metres above sea level. <docline> Coffee berry disease has also shifted to higher altitudes, and is attacking crops at 1,800 metres above sea level. Target context: sa production de café n’a cessé de baisser en raison de maladies et d’insectes nuisibles. Une maladie fongique appelée flétrissement du café a envahi son exploitation, et les perceurs de la tige de café ont attaqué ses caféiers. De nombreux autres agriculteurs ont subi la crise du flétrissement dans le district de Mukono, l’une des principales régions productrices de café en Ouganda. <docline> . . . . . . <docline> Il note que la rouille orangée du caféier qui touchait généralement le café cultivé à des altitudes inférieures à 1 400 mètres est maintenant apparue dans les localités situées à 1 800 mètres au-dessus de la mer. <docline> La maladie du fruit du caféier s’est également déportée vers les altitudes plus hautes, et est en train d’attaquer les cultures situées à 1 800 mètres au-dessus du niveau de la mer. Target sentence: Autrefois, on ne la trouvait qu’aux endroits situés en dessous de 1 600 mètres.

heading: Future Work
text: We may release filtered versions of these corpora in the future, pending further work to filter harmful content at the document level. Eurocentricity The Eurocentric nature of our work is remarked upon in Section 7. Due to most large-scale publicly available parallel corpora being English-centric, almost all machine translation research remains English-centric, at the cost of the
majority of the world’s language users. This limits both the generalisability and usefulness of a lot of research. We hope the release of more data and tools helps the community expand these efforts to more languages