Recovering document annotations for sentence-level bitext (2024)

Abstract: Data availability limits the scope of any given task. In machine translation, historical models were incapable of handling longer contexts, so the lack of document-level datasets was less noticeable. Now, despite the emergence of long-sequence methods, we remain within a sentence-level paradigm and without data to adequately approach context-aware machine translation. Most large-scale datasets have been processed through a pipeline that discards document-level metadata. In this work, we reconstruct document-level information for three (ParaCrawl, News Commentary, and Europarl) large datasets in German, French, Spanish, Italian, Polish, and Portuguese (paired with English). We then introduce a document-level filtering technique as an alternative to traditional bitext filtering. We present this filtering with analysis to show that this method prefers context-consistent translations rather than those that may have been sentence-level machine translated. Last we train models on these longer contexts and demonstrate improvement in document-level translation without degradation of sentence-level translation. We release our dataset, PARADOCS, and resulting models as a resource to the community.1

heading: 1 Introduction
text: Since the early days of statistical methods, machine translation has been centered within a sentencelevel paradigm. N-gram based approaches, which typically obey sentence-boundaries, were the predominant machine translation method and did not effectively use the wealth of information contained across contexts (Mariño et al., 2006). Later, newer neural techniques, such as Transformers (Vaswani et al., 2017b), became popular and have been shown to be effective at handling longer sequences (Beltagy et al., 2020; Sun et al., 2022; Post and
1https://huggingface.co/datasets/ jhu-clsp/paradocs
Junczys-Dowmunt, 2023). During this time, researchers have periodically considered contextaware machine translation in training methodologies (Voita et al., 2018), evaluation sets and metrics (Jiang et al., 2022; Vernikos et al., 2022; Müller et al., 2018), and its unique ability to address discourse phenomena which are otherwise impossible to correctly translate without context (Voita et al., 2019; Bawden et al., 2018). To present, little work has been done to address the most obvious hurdle: a lack of document-level training data.
Many cornerstone datasets were created by finding known sources of professionally translated documents (Koehn, 2005; Kocmi et al., 2023; Bañón et al., 2020). These parallel documents were then sentence-segmented and aligned with a document-based alignment technique. When more data was demanded, these datasets proved insufficient, so data curators moved towards global mining—treating web crawls as a bag of sentences and searching for the most similar sentence in the target language (Schwenk et al., 2021b,a; El-Kishky et al., 2020a). This removes document order and makes document-reconstruction impossible.
Some datasets, such as ParaCrawl, exist as a sentence-level resource but have been constructed in a way more amenable to documentreconstruction. This work confronts this resource gap by providing document-level annotations for News Commentary, Europarl, and the unfiltered ParaCrawl (Bañón et al., 2020) data. Our contributions are:
1. the reconstruction of documents from three large, popular datasets, illustrated in Figure 1;
2. the implementation of a document-level filtering technique as an alternative to traditional bitext filtering which destroys document-level metadata;
3. analysis that shows this filtering prioritizes context-consistent translations;
9876
Art is trying to become science. Die Kunst versucht, Wissenschaft zu sein. It deals with philosophy, sociology and psychology, and desires to be a pedagogical resource too. It wants to be political, yet maintain the charisma of the underground (that is, by being non-political). It wants to be a commodity. Sie beschäftigt sich mit Philosophie, Soziologie, Psychologie und möchte eine Quelle für die Pädagogik sein. Die Kunst will politisch sein, aber sich das Charisma des unpolitischen Undergrounds erhalten. Sie will Ware sein.
…. Culture without qualification is fraud. <P> Art is trying to become science. It deals with philosophy, sociology and psychology, and desires to be a pedagogical resource too. It wants to be political, yet maintain the charisma of the underground (that is, by being non-political). It wants to be a commodity. … … Eine Kultur ohne Attribut ist Betrug. <P> Die Kunst versucht, Wissenschaft zu sein. Sie beschäftigt sich mit Philosophie, Soziologie, Psychologie und möchte eine Quelle für die Pädagogik sein. Die Kunst will politisch sein, aber sich das Charisma des unpolitischen Undergrounds erhalten. Sie will Ware sein. …
Culture without qualification is fraud. Eine Kultur ohne Attribut ist Betrug.
Original English Monolingual
E/–DE ParaCrawl Bitext
Original German Monolingual
21 • 35 • 2033-1794
21 • 34 • 1927-2031
21 • 33 • 1829-1925
21 • 32 • 1796-1827
20 • 31 • 1756-1794
20 • 34 • 2175-2193
20 • 33 • 2080-2173
20 • 32 • 1971-2078
20 • 31 • 1929-1969
19 • 30 • 1891-1927
Par. • Sent. • Chars. Par. • Sent. • Chars.
Figure 1: An example from ParaCrawl. The existing bitext has no contextual information. A model is trained to produce “Sie” (a feminine pronoun) from “It” without appropriate context. We restore this information by finding text in the corresponding monolingual dumps, and add document, paragraph, sentence, and character offset metadata.
4. results showing that models trained on this data are better at translating document-level phenomena without degrading sentence-level performance;
5. the public release of this data, PARADOCS, and these models.

heading: 2 Related Works
text: Three areas of research are relevant to this work: sentence-level bitext mining, document-level bitext mining, and context-aware machine translation.
Sentence-level Mining is the current default for most of the largest parallel datasets. In the most recent WMT translation task (Kocmi et al., 2023), parallel training data (i.e., ParaCrawl or WikiMatrix (Bañón et al., 2020; Schwenk et al., 2021a)) tends to be sentence-level. Similarly, of the top ten corpora on OPUS (Tiedemann, 2009; Schwenk et al., 2021b; El-Kishky et al., 2020b, 2021), which make up over 93% of their entire collection, eight2 are sentence-level (Schwenk et al., 2021b; Fan et al., 2021) and comprises over 72% of the data, with the exceptions being OpenSubtitles (Lison and Tiedemann, 2016) and DGT3 which have rather specific domains.4 Many of these datasets are constructed via the global-mining technique, which
2NLLB, CCMatrix, MultiCCAligned, ParaCrawl, XLEnt, MultiParaCrawl, LinguaTools-WikiTitles, CCAligned
3https://ec.europa.eu/jrc/ en/language-technologies/ dgt-translation-memory
4These statistics current as of February 5, 2024, according to their website: https://opus.nlpl.eu/
indexes all sentences using some semantic-based hashing using LASER (Schwenk and Douze, 2017) and aligns based on similarity matches irrespective of document-boundaries.
Document-level Mining is representative of early datasets where document-alignment was already assumed and sentence-alignment could be done via simple features such as sentence lengths (Gale and Church, 1993). This technique was used to create original datasets such as Europarl (Koehn, 2005). Other techniques are constrained by document order when aligning sentences (Varga et al., 2005; Sennrich and Volk, 2011; Thompson and Koehn, 2019). These have been used to create more recent corpora such as News Commentary (Kocmi et al., 2023). ParaCrawl (Bañón et al., 2020) initially aligns documents; however, the preprocessing before the final release destroys any of this preserved document-structure. CCAligned (ElKishky et al., 2020b) has a middle-ground methodology which first aligns documents determined by both a combination of URL pairs and document similarity, followed by a LASER-based alignment. CCAligned releases data in rough document-order; however there are no annotations for document boundaries or other context labeling.
Recent work has also explored using ParaCrawl as a source for document-level data due to the original alignment being constrained by documents (Al Ghussin et al., 2023). To extract documents, they extract paragraphs by using a subset of the data for which WMT released aligned document-level data in German–English as a document-alignment
benchmark task. This work diverges from this idea in considering the full ParaCrawl release which is a significantly larger portion of data and also expands the language set.
Context-Aware Machine Translation concerns itself with any form of integrating additional context to translate. There are a handful of well-studied discourse phenomena, such as gendered-pronoun translation, which are impossible to translate without incorporating this information (Wicks and Post, 2023; Lopes et al., 2020; Voita et al., 2019; Müller et al., 2018; Bawden et al., 2018). Further, Transformers are entirely capable of incorporating longer contexts as input (Sun et al., 2022). Thus, the field of context-aware translation has either involved modifying the input framework to better incorporate the signal of context-features (Lopes et al., 2020; Tan et al., 2019; Miculicich et al., 2018), or studying alternative uses of training data (Post and Junczys-Dowmunt, 2023; Yu et al., 2020).
The recent introduction of Large Language Models (LLMs) has also had a swath of studies that show these LLMs are better-situated for documentlevel machine translation as they naturally have significantly large context-windows (Wang et al., 2023; Petrick et al., 2023; Robinson et al., 2023; Kocmi et al., 2023). LLMs provide a desirable path towards document-level translation; however with their novelty and training pipeline opacity, it is still unknown what types of data elicits this performance in order to replicate on different sets of languages or smaller scales.

heading: 3 Document Annotation and Reconstruction
text: Released bitext files rarely have document annotations necessary to create contextual inputs for context-aware training. In the case of Europarl, News Commentary, and ParaCrawl, document ids are released which point to the original monolingual files. We use these monolingual files to reconstruct the original parallel document structure.
We construct annotations for each original aligned segment in the bitext. Each segment has an individual source and target segment which receive separate annotations dependent on their original document. Following the reconstruction process described in Section 3.1, we define a pipeline (Section 3.2) to extract contexts from this data. Finally, we describe a novel document filtering method (Section 3.3) that can be used to filter out low-
quality documents instead of only low quality sentences. With each progression of these steps, one can whittle the dataset smaller while improving the relative quality of documents.
All filtered splits, and the original unfiltered data, are publicly released to facilitate further research into context-aware machine translation, and document-based data selection and filtering methods. The sizes of each dataset with the increased filtering is displayed in Table 1. We explore reconstruction as an alternative to re-alignment of the original documents since it is less computationally intensive and re-uses existing annotations from published datasets. Future work can circumvent the need for reconstruction by preserving document metadata while data mining.

heading: 3.1 Reconstruction
text: If the end goal is to extract “context”, it is only necessary to determine whether two source-target pairs are consecutive in the respective source and target documents. Given an ordered list of segments5 from the unfiltered bitext, and the original web-crawled document, we can align each segment to the original document via exact string matching. Illustrated in Figure 1, this determines the index spans for each segment. These indices identify where precisely the segment occurs in the original monolingual document. This is relevant to identify the correct context as some sentences may have been left unaligned during the sentence-alignment process. We annotate the spans for all source and target segments in our data. For each segment in the RAW bitext, we list (with respect to the monolingual document):
• the paragraph index;
• the sentence index, determined by applying a Moses Sentence Splitter (Koehn et al., 2007);
• the starting character index, after normalizing whitespace;
• the ending character index, after normalizing whitespace;
• the probability of the language id according to NLLB’s fasttext LID model;
• the duplication count: the number of times this segment was repeated in ParaCrawl.6
5We use “segment” to refer to a given source or target which roughly equates to one sentence but may vary.
6Only ParaCrawl has significant boilerplate text so we
Consecutive sentences are defined such that the starting character index is two more than the ending index of the previous segment. For instance, in Figure 1, “Art is trying to become a science.” has a period that ends at index 1827 and “It” of the following sentence starts at 1829. This indicates there is a single whitespace token separating these two segments at index 1828 and they were originally consecutive; thus, they belong to the same context.
In Table 1, we report the size of the data we reconstruct by RAW. This data precludes approximately one-third of the ParaCrawl data for which the monolingual source is CommonCrawl. The CommonCrawl monolingual data is only accessible by querying servers which make recovering metadata annotations computationally difficult with respect to network issues and latency. We leave this section of ParaCrawl unannotated.

heading: 3.2 Context Extraction
text: The benefit of sentence-level bitext creation is that high-quality sentence-pairs are not discarded simply because the remaining document is poorly aligned. This allows for much higher rates of alignment, and a larger overall dataset. Unfiltered ParaCrawl documents (defined by a URL pair) are often poorly aligned, or have extraneous portions of documents in one or both languages. We still want
leave News Commentary and Europarl unannotated.
to keep a paragraph of context even if the boilerplate text is of low quality. To achieve this, we define document-breaking criteria. As we iterate over the annotated data, we accumulate context within document boundaries. When a given segment fails to meet some criteria, we break the preceding and proceeding contexts into sub-documents. Consider an article which may have been loosely translated. Some paragraphs may be literally translated while others may have been paraphrased or given additional context to make it more understandable to the audience. In a given paragraph, when one segment is unaligned, we break the paragraph into two sub-documents: those preceding the unaligned segment and those proceeding it These sub-documents can be still be linked by their parent document, but can be treated as independent for the sake of context extraction. In our experiments, we break on three conditions:
1. a segment is unaligned (minimum subdocument length is two)
2. the language id probability is less than 0.5 (as predicted by NLLB’s fasttext)
3. the duplication count is more than 100—this divides documents on boilerplate texts which have high frequency and do not contribute much meaningful contextual information.
The resulting dataset is titled DOCS and statistics can be seen in Table 1.

heading: 3.3 Document Filtering
text: A key issue of web-crawled bitext is that much of it was machine translated (Thompson et al., 2024). Sentence pairs that are well translated will still circumvent sentence-level filtering methods such as LASER. When these pairs sneak through, any resulting model will be nothing more than a mixture of distilled MT models from throughout the history of the internet. As machine translated sentences on the internet are likely to have been translated at the sentence-level, this makes any resulting translation particularly prone to making errors when when translating context-phenomena (Müller et al., 2018; Bawden et al., 2018). Thus, it is necessary to filter out these errors before training.
Peter et al. (2023) showed that quality estimators (QEs), which notably do not require a reference, are capable of distinguishing fine-grained quality differences necessary for filtering. Additionally, SLIDE (Raunak et al., 2023) showed that these same quality estimators can discriminate between context-consistent and context-inconsistent translations—as one might see when translating each sentence individually. We combine these two ideas to propose a document-filtering methodology.
SLIDE works by creating a series of contextchunks by sliding a window across the document. Each window is evaluated by a QE system and the scores are averaged in order to create a documentlevel score. Raunak et al. (2023) evaluates a combination of window and stride sizes. They find some effectiveness starting at a minimum window size of three. We use a window of three and a stride of one for our scoring. We also chose to use the CometKiwi QE model (Rei et al., 2022) as it is consistently a high performing model in these works.
The initial dataset is large, and much of it is low quality. We rank documents with this scoring technique, and experiment with three different filtering cutoffs, top 75%, 50%, and 25%, scored at the sub-document level (as described in Section 3.2). They are described as LOOSE75%, MEDIUM50%, and STRICT25%, respectively, in Table 1.

heading: 4 Source Data
text: To produce our dataset, we select source data from three large, publicly-available datasets that initially used document-level alignment.
ParaCrawl (Bañón et al., 2020) prioritizes a high-recall alignment so the original unfiltered data is orders of magnitude larger than the official release, but has an inferior quality. For instance, ParaCrawl v9 en-de is approximately 278M lines of cleaned sentences in arbitrary order, but an additional “RAW” file is available that contains nearly 10B of uncleaned sequential sentence alignments.7 The official release has undergone the aforementioned sentence-level filtering (deduplication, similarity score filtering, etc) that removes the original context. Fortunately, ParaCrawl releases an original unfiltered (RAW) data version. This equates to each source–target pair, a document id, and a pointer to the original monolingual source.8
News Commentary (Kocmi et al., 2023) is a smaller, albeit cleaner newswire dataset released annually for WMT training data.9 The released versions do maintain document order, but documents are not labeled.
Europarl (Koehn, 2005) was produced from the Proceedings of the European Parliament which is obligatorily translated into a handful of European languages. Europarl is typically n-way parallel which makes it ideal for machine translation despite the specific domain. The most recent version (v10) releases document ids; however, it is not available in all languages. We produce the alignment ourselves with the accompanying tools.

heading: 5 Experimental Design
text: In order to show the viability of constructing a dataset in such a fashion, we need to show two things: (1) a contextual model is at least as good at sentence-level translation quality as a sentencelevel model and (2) a contextual model outperforms sentence-level models when considering contextbased phenomena.

heading: 5.1 Baselines
text: We consider two types of sentence-level models for baselines. For each data filtering level, we additionally train sentence-level models without concatenation. This allows for comparison on models
7Most of these segments are justifiably discarded during preprocessing due to low quality which is why the v9 release is substantially smaller.
8The monolingual sources for two-thirds of ParaCrawl was released at https://paracrawl.eu/moredata.
9With special thanks to Barry Haddow who was kind enough to deliver us copies of the intermediate processing steps so we could recreate these annotations
trained on the same quantity and distribution of data. Additionally, we produce a new dataset that is filtered at the sentence level instead of the document level. Using the entirety of the original data, we filter using traditional bitext filtering pipeline. After deduplication, we remove sentences with: empty lines, more than fifty-percent punctuation, irregular frequencies of characters based on language histograms (Fan et al., 2020), uneven length ratios (greater than 1.5) of source-target, less than 0.5 of target language according to NLLB’s fasttext and langid.py, and those below a 0.85 LASER score (Schwenk and Douze, 2017). The resulting dataset is described as SENTENCES in Table 1.

heading: 5.2 Training
text: All models trained in this paper are trained with the Marian NMT (Junczys-Dowmunt et al., 2018) toolkit. We train context-aware models with a simple concatenation strategy (Tiedemann and Scherrer, 2017). We also train on both contexts and sentences during training via SOTASTREAM which mixes two data streams during training (Post et al., 2023). The first stream which samples from the PARADOCS data, concatenates documents up to ten sentences or 256 tokens (whichever is lesser) and inserts an <eos> token to separate segments. The second stream pulls from a supplementary dataset composed of preprocessed sentence-level bitext. These streams are mixed with a 1:1 ratio. These models are trained with typical next-token predic-
tion and no further alterations. The models are Transformers (Vaswani et al., 2017a) with 12 encoder layers and 6 decoder layers. We use a feed-forward dimension of 16, 384 and an embedding size of 1024. We train a single sentencepiece vocabulary (Kudo and Richardson, 2018) for each language pair trained on the supplementary data with a shared vocabulary size of 64k. All model iterations uses the corresponding vocabulary. The effective batch size is 500k tokens and one logical epoch is 1B tokens. We train for 10 logical epochs for a total of 10B tokens/20k updates. For evaluation, we use the model with the lowest cross-entropy loss per token on the FLORES200 dev set (NLLB Team, 2022; Goyal et al., 2021; Guzmán et al., 2019).
Supplementary Data is used in addition to the our contextual data. This ensures that the model can translate both sentences and documents, and is not burdened by a lack of language coverage from high quality sentence pairs mined via other techniques (i.e., global mining that produces high quality datasets such as CCMatrix). This supplementary data is controlled across experiments. We filter this bitext in the same way we filter the SENTS baseline described in Section 5.1.

heading: 5.3 Evaluation
text: We evaluate two-fold: sentence-level translation quality and the ability to address context-dependent phenomena. The former can be addressed by regu-
lar machine translation test sets. We choose to use FLORES200 test sets (NLLB Team, 2022; Goyal et al., 2021; Guzmán et al., 2019) as they are available in all of the languages in consideration. We additionally evaluate on the most recent WMT test set available for that language pair. For these evaluation sets, we report BLEU (Papineni et al., 2002) scored using SACREBLEU (Post, 2018). We additionally report COMET (Rei et al., 2020) in Appendix A but acknowledge that those scores may be inflated from the COMET-based filtering method.
In order to evaluate the translation of contextdependent phenomena, we turn to CTXPRO (Wicks and Post, 2023) which releases evaluation sets in these languages to evaluate three phenomena: gender, formality, and auxiliary verbs. Briefly, these handle the translations of English ‘it’ to gendered languages; the translations of English ‘you’ to languages with a T-V distinction; and the translation of English auxiliary verbs in elided sentences such as ‘I do.’ These are impossible to consistently translate correctly without context. We report accuracy scores as calculated by the CTXPRO PyPI package.
To translate, we employ two inference designs. First, for all models we translate using single sen-
tences as input. Second, for the contextual models, we additionally evaluate them under contextual inference. For each input, we concatenate up to ten sentences or 256 tokens (whichever is lesser) of preceding context. This parallels the contextual data during training. The last segment is then split (determined by <eos>) and used as the predicted translation.

heading: 6 Results
text: In general, we find that context-aware models trained with the PARADOCS data outperform any model trained without contextual inputs across evaluation metrics. Further, we find that the document-level filtering method is able to improve performance even as data quantity diminishes— indicating a higher quality of data selection. In Table 2, we display BLEU scores as an evaluation of general translation ability and subsequently in Table 3, we display CTXPRO accuracy scores as an evaluation of the ability to translate contextdependent phenomena. We additionally perform paired bootstrap resampling statistical tests on the evaluation sets to understand whether these differences are meaningful (p < 0.05) and comment
where appropriate (Koehn, 2004).

heading: 6.1 Effects of Document-Level Filtering
text: In Table 2, we show BLEU performance of all models. We compare the contextual models with the sentential baselines. Within each inference setting, performance steadily, and consistently improves between the models trained on the noisiest data (DOCS) and the cleanest (STRICT25%). This is true even when not using the contextual information available as shown in the sentence-level inference setting of the contextual models. This is reinforced by the same trend in CTXPRO scores displayed in Table 3. We note for BLEU scores, there is a small plateau or deterioration between the MEDIUM50% and STRICT25% data filtering stages and suspect this is a critical trade off point between having high-quantity, high-quality sentence translations (that are poorer-quality contextual translations) and having low-quantity, high-quality contextual translations. This is supported by the fact that we do not see this trend across the CTXPRO scores.

heading: 6.2 Sentence-Level Translation Performance
text: We also find that by training a model with contextawareness, the model does not lose the ability to translate stand-alone sentences and in many cases, marginally improves. This is evident by comparing the the models trained under each paradigm (Sentence, and Context Training), when only given sentences during inference. In Table 2, this is evident as we see little-to-no difference between the models trained with and without context when they translate individual sentences. In some cases, we see marginal improvement from the model trained with context. This indicates that training a machine translation within a context-aware paradigm is no worse than training one without.

heading: 6.3 Context-Awareness Boosts BLEU Performance
text: Further, we identify benefit from leveraging additional context to translate, even when translating datasets which are not dense with contextdependent phenomena, represented by WMT test sets and Flores200. Contextual models that are given preceding context during inference get consistent small gains in BLEU compared to their analogous sentence-level models in Table 2. In the larger language pairs (e.g., German and French), we additionally found that the top performing models were statistically outperforming all of the
sentence-level models however the statistical differences degraded with the smaller language pairs.

heading: 6.4 Translation of Context-Based Phenomena
text: Finally, we show that these context-trained models are more effective at translating context-dependent phenomena than their sentence-level counterparts. When considering both Gender and Auxiliaries, the improvement is evident. We do find the translation of Auxiliaries is still unfortunately low, but may be due to a relatively low rare number of occurrences in the dataset. We also see that in the case of formality, performance is marginally better, though relatively unaffected. We hypothesize this is due to the particularities of formality translation, rather than exemplification of the dataset, though acknowledge that alternatives may be necessary to target this particular phenomena. When investigating the statistical differences, we note specifically that in all cases, the context-aware models are statistically superior than their sentence-level counterparts specifically for the auxiliary class. This is particularly noteworthy as the auxiliary task is the hardest task: the potential correct answer is open to the set of all verbs rather than a simple one-of-three-genders problem as we see with formality. We also note there is less statistical support for the formality class. This speaks to the difficulty of this problem and may instead indicate that data is not the simplest solution to approach formality translation.

heading: 7 Analysis of Document Filtering
text: Document-level filtering is rare, and to our knowledge, has not been studied. We demonstrate that our SLIDE–CometKiwi filtering improves contextual performance, but this may be an indication of better sentence-level translations and not better document-level translations. For a documentlevel filtering method to hold up against webcrawled data, it should not only filter out poor quality translations, but should also filter out contextinconsistent translations. By context-inconsistent, we refer to translations that were produced at the sentence-level. This is roughly synonymous with machine translated text in a web-crawl setting; however, there is no robust machine translation detection methodology to the best of our knowledge.
We instead turn towards a proxy measure to determine if context-inconsistent translations are filtered out. After scoring each document as de-
scribed in Section 3.3, we can rank documents sequentially based off these SLIDE-CometKiwi scores. We should find more context-consistent translations in the upper percentiles of this distribution and fewer at the lower-end.
The ctxpro toolkit, which generated these evaluation sets, also identifies when there are ambiguous uses of gendered pronouns. As the toolkit is intended to be high precision, low recall, it is extremely likely to not identify ambiguity in pronouns when the translations are incorrect or inconsistent. Thus, we can use the ctxpro toolkit as a proxy measure for context-consistent translations.
We identify all examples of gendered-pronouns in our data according to the ctxpro toolkit, and investigate where SLIDE-CometKiwi ranks them in relation to the remaining documents. We show percentages in Table 4. We discriminate between intersentential—where the antecedent is in the same sentence—and intrasentential—where the antecedent is in a previous sentence. The former is quite easy to correctly translate for a sentence-level model while the latter is impossible. We further distinguish by the gender of the pronoun—where neuter is the majority class.
As shown in Table 4, most of the identified examples occur in the 1st or 2nd quartile. Further, we see that this is especially true for the feminine pronouns (a quintessential minority class for pronouns). Conversely, neuter pronouns have a more uniform distribution across quartiles. We hypothesize this indicates there is more machine translated texts in the lower quartiles—as they would have incidentally correctly translated neuter pronouns— and less in the top. More specifically, the most challenging translation for machines, ambiguous pro-
nouns with an intrasentential feminine antecedent, are mostly ranked in the top quartile. We assume this also means the top quartile was translated by humans.

heading: 8 Conclusions
text: Research in contextual machine translation is hampered by the lack of document annotations on parallel data. We augment three large popular MT datasets (ParaCrawl, News Commentary, and Europarl) with this information, creating a document-level dataset, PARADOCS. We introduce a document-level filtering method to apply to this data in lieu of traditional context-destroying sentence filtering methods. Simple, context-aware machine translation models trained on this data have shown to be better at machine translation in both general performance as measured by WMT test sets as well as targeted performance—measured by the ability to correctly translate discourse phenomena. We release the data as a resource to the community.10

heading: 9 Limitations
text: As mentioned in this work, document reconstruction is particularly constrained by the original dataset processing. Many languages have only been processed with the global mining technique (i.e., CCMatrix) and ParaCrawl notably only supports European languages. This works also assumes there exists many well-translated parallel documents in web-crawled corpora. Not only is this not true for many language pairs, but there is recent evidence to suggest that the more multi-way parallel data is, the more likely it was machine translated (Thompson et al., 2024).
This also significantly constrains the amount of data. In our smallest setting, English–German (an extraordinarily high-resource language pair) is still limited to 42M lines. If we were to extend this to lower-resource languages, we would be limited to perhaps a few thousand lines which are unlikely to make any meaningful difference in performance.

heading: 10 Acknowledgements
text: We would like to thank the reviewers for their discussion and feedback. We additionally would like to thank Barry Haddow, Marc Marone, Neha Verma, and Joe McKnight for their resources, technical discussions, and support.
10https://huggingface.co/datasets/jhu-clsp/paradocs

heading: A Appendix

heading: Future Work
text: Future work can circumvent the need for reconstruction by preserving document metadata while data mining