RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Characterizing intertextuality between Corpora
Our intertextuality measure is simply the cosine similarity of a pair of verse embeddings from a multilingual embedding model. For a given set of ground-truth references, we can also compute baseline similarities by randomly swapping one of the verses with another from the same chapter 3 3 3 Maintaining the same chapter ensures that false pairs likely remain upon the same topic, as opposed to choosing a random verse from anywhere in the Bible. . The ratio of the average intertextuality similarity to the average baseline similarity
can be used to compare the degree of intertextuality across different sets of translations.

Intuitively, a ratio much larger than one (1) indicates strong intertextuality, whereas anything less than one indicates that supposedly intertextual verses are not more similar than random pairings. When comparing changes in intertextuality ratio across translation, we compute the 95% confidence interval via bootstrapping. Specifically, we resample the original data with replacement 10,000 times, recalculating the ratio for each resample.

Note that this method relies upon having access to ground-truth references — or suspected references — and would likely be too crude a method to discover novel instances of intertextuality without extensive threshold tuning. Instead, we use this measure to ascertain the degree of intertextuality within a set of texts known to be intertextual. We can then use this measurement to characterize changes in intertextuality across the same set of texts in translation.

We compute intertextuality ratios for all original, human, and machine-translation texts, distinguishing the sets of references that are internal to a testament ( within ) and that cross between them ( across ). This distinction allows us to consider whether Christian writing is particularly referential to the Jewish Testament, or if it became so through the effects of translation. Christian theologians throughout history have often underscored the continuity of the Christian and Jewish testaments van der Waal ( 1980 ) , and human translators may have sought to emphasize this continuity in their translations. The full tables of these ratios can be found in Table 4 .
Data
We use three primary sources for our textual analysis: the Translator’s Amalgamated Hebrew Old Testament (TAHOT)
and Greek New Testament (TAGNT) 4 4 4 www.STEPBible.org , as well as a digitized copy of the Septuagint (LXX) 5 5 5 https://sourceforge.net/projects/zefania-sharp/files/Bibles/GRC .
The TAHOT is based on the Leningrad Codex, the oldest complete extant version of the Hebrew Old Testament. The TAGNT consolidates the Greek New Testament text from multiple early extant editions, and these are both compiled by Bible scholars at Tyndale House in Cambridge, UK, and released as part of the STEP Bible project 6 6 6 We release a formatted version of STEP Bible’s data on the Huggingface Hub. DOIs: 10.57967/hf/4174 , 10.57967/hf/4184 . . The Septuagint is the earliest Greek translation of the Hebrew Old Testament, completed by Jewish scribes in the few centuries preceding the events of the New Testament. 7 7 7 The complex history of Biblical scribal tradition means that almost all modern English translations use a versification system which at many points differs from the versification in the original Hebrew (cf. Genesis 31:55 in English translations is considered Genesis 32:1 in the Leningrad Codex). For consistency, we align all documents to use the English versification system across all experiments.

For modern human translations, we use the Johns Hopkins University Bible Corpus McCarthy et al. ( 2020 ) for the five languages in Table 2 , each of which include both testaments.

To independently evaluate our method, we use a benchmark corpus for intertextuality provided by Burns et al. ( 2021 ) detailing intertextual references in Classical Latin literature. Specifically, it contains 945 references curated by subject matter experts connecting Valerius Flaccus’ Argonautica I to earlier and contemporary Roman authors.
Translation
To compare the effects of human and machine translation,
we employ Cohere’s multilingual model Aya23 8 8 8 We use the model version with 8B parameters. Aryabumi et al. ( 2024 ) to translate all of the original manuscripts into the five languages of varying resource levels from Table 2 . Aya23 is chosen for this task as it has been shown to outperform other multilingual models of similar, and sometimes larger, sizes for machine translation Aryabumi et al. ( 2024 ) , but is small enough to be practical for academic research settings with limited compute power. Full pre-processing, prompting, and post-processing details may be found in References .
We report translation quality scores using the COMET metric Rei et al. ( 2020 ) in Table 3 , providing references (human-translated text in the target language), predictions (machine-generated text in the target language), and sources (original text in the original language).
Gold standard for intertextuality
For ground-truth information about which passages are truly interlinked, we use a dataset of Bible cross-references Owens ( 2023 ) . According to the dataset’s documentation, the initial data was seeded largely from the Treasury of Scripture Knowledge Torrey and Canne ( 1982 ) , an authoritative compilation of cross-references from prominent Biblical scholars over many centuries, which was then cleaned to remove duplicates and concatenate separate entries for adjacent references. Finally, the references were opened to crowd-sourcing annotation for voting on relevant connections.

We limit consideration to verse-to-verse links that connect passages from different books and can be resolved in all manuscripts. We disregard ordering by
summing the votes for both directions between a pair of verses, and use a vote threshold of 50 to consider a reference valid. 9 9 9 We independently verify that 96.0\% of the cross-references in our dataset with at least 50 votes are attested in an online version of the Treasury of Scripture Knowledge https://www.tsk-online.com/ . This produces a total of 2183 references: 548 are entirely within the Jewish testament, 961 within the Christian, and 674 that span them.
We differentiate these two cases with the qualifiers within , meaning within the same testament, and across , meaning across the two testaments.

Orig.

Human

NMT
Experiments
Benchmark Corpus : First, we evaluate our method on a benchmark corpus for intertextuality between Valerius Flaccus’ Argonautica I to earlier and contemporary Roman authors. We calculate an intertextuality ratio of 1.55, 95% CI [1.53,1.56], indicating that our method succeeds at characterizing known intertextuality at the corpus level.

Translation Quality : Table 3 shows translation scores from the Hebrew Old Testament, Greek Old Testament, and Greek New Testament into five target languages. English and Turkish consistently achieve the highest scores across all manuscripts, with English translations ranging from 61.2 to 72.6, and Turkish from 65.4 to 68.2, suggesting strong translation quality for these language pairs. In contrast, translations into Marathi show the lowest scores, ranging from 26.5 to 29.8, likely due to the complexity of translating between less common language pairs. These results establish a valuable benchmark for evaluating translation quality for underrepresented languages in historical texts.
Analyzing Intertextuality in Translation
Table 4 shows that there is a higher degree of intertextuality across the New Testament and the Hebrew Bible compared to intertextual references within each book.

The degree to which intertextuality is preserved is highest for the English translation and lowest for Marathi. Human translations consistently show higher levels of intertextuality.

As suggested by McGovern et al. ( 2024 ) , we indeed see that human translations over or underemphasize the intertextuality present in the text, whereas machine translations provide a neutral baseline, based on these results.
We can look closer at the translation effects by sorting intertextual pairs according to the absolute shift in similarity. Table 5 shows the original Greek, the human English translation, and the unconstrained machine translation for one such pair, between the Epistle to the Hebrews and the Book of Isaiah. The pair of verses has strong similarity in the original Greek (0.332), but this is nearly doubled by the human English translation (0.656). The highlighted Greek word, hamartion , typically translated as sin , occurs in Hebrews but not Isaiah, yet the latter’s human translation makes a point of using the term. Surface-level lexical decisions like this, and presumably many less direct choices, lead to uncalibrated translations that reinforce the received interpretation.
Future work
We plan to address the persistent issue of misalignment in parallel Bible corpora. Even in scholarly editions of digitized texts, misalignment is persistent. However, by applying the alignment methodology proposed by Craig et al. ( 2023 ) , we could unify alignment for research purposes. Finally, we leave to future work exploring larger narrative contexts by examining narrative episodes instead of verse-level intertextuality.
Limitations
In this work, we generate machine translations working from the oldest extant manuscripts of the Biblical texts. However, most translations present in the JHUBC were not translated directly from ancient manuscripts but instead work from English translations, which themselves were often translations of the Greek texts. So direct comparisons of the human translations and machine translations in this work should be treated with caution.
Acknowledgments
We would like to thank Matt Post for very helpful discussions about machine translation evaluation, as well as scholars at Tyndale House, especially Ellie Wiener and Caleb J. Howard, for useful discussions about Biblical Hebrew. Thanks to Andrew Caines for his helpful comments on previous drafts.
Hope McGovern’s work is supported by the Woolf Institute for Interfaith Relations and the Cambridge Trust.
Biblical intertextuality. The highlighted middle verse shows the intertextual reference from the New Testament to the Hebrew Bible, establishing a connection between the drying up of the Euphrates River and Moses parting the Red Sea. Both are instances of divine intervention in the context of a body of water. However, intertextuality here not only establishes a semantic parallel between two events, but it also emphasizes the difference. The passage from Exodus is moment of the divine judgment that leads to safety, whereas the drying up of the Euphrates is a preparation for the final judgment of the world.
Languages by family. Summary of languages used in this study: each has a full, aligned human translation of both the Jewish and Christian texts.
The sizes are reported from Tang et al. ( 2020 ) training data and reflect the variety of resource-levels.
COMET scores . Top-scoring translation for each source manuscript is in bold text. Second top-scoring translation is in italics.
Intertextuality ratios for source manuscripts and their human translations. Ratios within, and where possible between, testaments, for the Septuagint and TAGNT (Greek), TAHOT (Hebrew), and five human translations with 95% CI.
Overemphasized Intertextuality by Human Translation. The intertextuality from Hebrews 8:12 to Isaiah 43:25 is amplified by the human translator’s decision to render different words as "sin". The machine translation abstains from this and restores the original distance, but loses coherence.