# Unraveling the Synoptic puzzle: stylometric insights into Luke's potential use of Matthew

Sophie Robert Hayek, Jacques Istas, Frédérique Rey

<figure><img src="image_1.png" /></figure>

## To cite this version:

Sophie Robert Hayek, Jacques Istas, Frédérique Rey. Unraveling the Synoptic puzzle: stylometric insights into Luke's potential use of Matthew. Fourth Conference on Computational Humanities Research (CHR 2023), Dec 2023, Paris, France. hal-04827223

HAL Id: hal-04827223

https://hal.science/hal-04827223v1

Submitted on 9 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Unraveling the Synoptic puzzle: stylometric insights into Luke's potential use of Matthew*

Sophie Robert Hayek<fnref n="1" />,<fnref n="2" />*, Jacques Istas<fnref n="2" /> and Frédérique Rey<fnref n="1" />

**Abstract**

The literary sources behind the three canonical Synoptic Gospels, namely Luke, Matthew and Mark, have long intrigued scholars because of the Gospels striking similarities and notable differences in their accounts of Jesus's life. Various theories have been proposed to explain these textual relationships, including common oral witnesses, lost sources or communities possessing each other's works. However, a universally accepted solution remains elusive. Leveraging advancements in statistics, data analysis, and computing power, researchers have begun treating this as a statistical problem and quantitatively measuring the likelihood of the different theories based on verbal agreements and stylometric features. In this paper, we rely on a very recent Machine Learning based approach to solve the synoptic problem. We use Machine Learning classifiers two-sample tests, a novel approach relying on the analysis of the success rate of binary classifiers to identify whether two samples are drawn from the same distribution, to detect differences in sources within Luke's Gospel and variations in the edition patterns of Markan material between Matthew and Luke. This analysis is done on a pericope-per-pericope basis, defined as thematic units encompassing teachings or narrative episodes. The results suggest significant dissimilarities in style and edit distance, indicating that the double and triple material within the Gospel of Luke likely originate from different sources. This suggests that Luke derived his triple tradition from Mark and not from Matthew. Despite the necessity of cautious interpretation due to the size of the dataset, our study thus offers substantial evidence supporting the theory of Luke's dependency on Mark's material for his triple tradition and makes the two-source hypothesis, which suggests that Luke did not have access to Matthew's work, the most likely explanation based on our methodology.

**Keywords**

synoptic problem, New Testament, stylometry, digital humanities;

## 1. Introduction

The striking similarities, along with the notable differences, between the different accounts of Jesus' life related in the three canonical Synoptic Gospels, usually referred to as the Gospel of Luke (Lk), Matthew (Mt) and Mark (Mk), have led scholars since Antiquity to speculate on the order of their composition and the literary sources available to each community the Gospels came from. Over the years, several theories have been proposed to explain this phenomenon,

CHR 2023: Computational Humanities Research Conference, December 6 – 8, 2023, Paris, France

✉ sophie.robert@univ-lorraine.fr (S. Robert Hayek); jacques.istas@univ-grenoble-alpes.fr (J. Istas); frederique.rey@univ-lorraine.fr (F. Rey)

ID 0000-0003-4359-9124 (S. Robert Hayek); 0000-0003-0028-4931 (F. Rey)

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

CEUR Workshop Proceedings (CEUR-WS.org)

<footnote n="1">¹Laboratoire Ecritures, Université de Lorraine, France</footnote>

<footnote n="2">²Laboratoire Jean Kuntzmann, Université Grenoble Alpes, France</footnote>

<footnote>*Corresponding author.</footnote>

<footnote>†These authors contributed equally.</footnote>
including the idea of shared oral witnesses [1], the existence of now-lost common sources [2],
and the possibility of the community having access to each other's works [3, 4]. The debate
remains heated and no solution has been universally accepted as the most likely one.
With the recent advances in statistics and data analysis, as well as the exponential advances
in computing speed, several researchers have suggested treating this problem as "*a kind of
problem in Synoptic arithmetic*" [5, p. 202] and have attempted to provide some quantitative
measurements regarding the likelihood of each theory, either by studying the verbal agree-
ments, *i.e.* occurrences of the same words in the same context in two or more Gospels, or the
stylometric features of the text.

In this paper, we suggest a novel approach based on the use of a Machine Learning classifier
two-sample tests to detect different sources within the Gospel of Luke and different edition
patterns of the Markan material between Matthew and Luke. We compare statistically the
distribution of the stylometric features and show that the theory of Luke and Matthew being
independent, also known as the two-source hypothesis, is the most likely given our methodol-
ogy.

The major contributions of this study are:

* The introduction of a novel paradigm by conducting stylometric analysis at the pericope level rather than focusing on individual verses when comparing the Synoptic Gospels.

* The use of a very recent statistical approach, Machine Learning-based two-sample test, to identify stylistic differences within double and triple material of Luke's Gospel.

* The application of this Machine-Learning based approach to characterize the edits made by Luke and Matthew when it comes to the Markan material.

The results of our study allow us to lean towards the two-source hypothesis as being the most
likely one given the stylometric data of Luke's Gospel.

This paper is structured as follow. We give in section 2 more insights regarding the synoptic
problem and the major solutions suggested over the history of the Gospels. Section 3 presents
works related to ours which rely on statistical methods to propose solutions to the synoptic
problem. Section 4 presents our selected stylometric approach and its results when applied to
the SBL Greek New Testament (SBLGNT) text [6] are presented in section 5. We then conclude
our work and give some insights into our further works in section 6.

## 2. Motivation: the synoptic problem and the two-source hypothesis

Solving the synoptic problem consists in providing a theory that describes the relationships
between the three canonical Synoptic Gospels (Matthew, Mark, and Luke) by analyzing their
similarities and differences. If the potential influence of oral traditions on early Christian teach-
ings cannot be overlooked [1] to explain the parallels between the gospels, the often *verbatim*
wordings and the overall alignment of pericopes indicate a common written source. The peri-
copes can be categorized as *single tradition* (found in only one Gospel, such as the different
accounts of Jesus's birth), *double tradition* (present in Matthew and Luke, such as the teachings
in the Sermon on the Mount and the Sermon in the Plain in Mt 5-7 // Lk 6:17-49), or *triple*
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Graphical representation of the Two-Source Hypothesis and the Farrer-Goulder Model. Square in dashed lines indicate an hypothetical source that has never been found.</figcaption></figure>

tradition (found in all three Gospels, often with variations in order and context, such as the healing of the paralytic found in Mt 9:1–8, Mk 2:1–12, and Lk 5:17–26). The distribution of the different material per verse number and per pericope is available in table 1. Traditionally, the Augustinian model [7] proposed that Matthew was written first, followed by Mark as a summary of Matthew, and finally Luke<fnref n="1" />. However, the past two centuries of textual criticism have challenged this model, starting with Griesbach’s hypothesis [8] that Matthew was the initial Gospel, which was later used by Luke, while Mark combined elements from Luke and Matthew [3, 9]. The dominant theories in the 20th century presuppose Markan priority, suggesting that Mark’s Gospel was written first and served as a source for the two other canonical evangelists. When assuming Markan priority, two competing theories currently co-exist: the *Farrer-Goulder hypothesis* (FGH) [4, 10] proposes that Luke had access to both Mark and Matthew, drawing material from both and the *Two-Source Hypothesis* (2SH or Q hypothesis) [2, 11], now widely accepted, which posits that Luke and Matthew utilized a source composed of sayings of Jesus without any narrative material, called Q (abbreviation of *Quelle*, German for source), along with Mark’s Gospel. Matthew and Luke consequently derived their double tradition material from Q and their triple tradition material from Mark. Figure 1 provides a schematic representation of these two theories. As Farrer underlines in [4], the Q theory “wholly depends on the incredibility of St Luke having read St Matthew’s book”, as this theory relies on the assumption that Luke deliberately altered key sayings of Jesus, such as the Lord’s prayer (Mt 6:9–13, Lk 11:2–4). In this study, we presuppose Markan priority and leverage stylometric arguments to determine the likelihood of Luke having read Matthew’s Gospels and determine which theory is better supported by the evidence gathered from the stylometric features.

### 3. Related works

The use of statistical data in the analysis of the synoptic problem goes back to at least the end of the nineteenth century, when Hawkins’ *Horae Synopticae* was published in 1899 [13]. Ever since the recent advances in computing power and statistical algorithms, the Synoptic Gospels have received wide attention from statisticians.

The research can be separated into two wide categories: the study of *verbal agreements* [14,

<footnote n="1">¹Followed by John. Due to the complexity and ongoing research in the field of the relationship between Johannine literature and the Synoptic Gospels, this article does not delve at all into this subject matter.</footnote>
**Table 1**

Distribution of pericopes across genres and traditions. Our results vary from the now widely accepted statistics provided by Honoré [12] as we are working on a per pericope basis instead of on a per verse basis and do not take into account the number of verses per pericope.

<table><caption>(a) Pericope repartition across tradition</caption><thead><tr><th>Book</th><th>Tradition</th><th>%</th></tr></thead><tbody><tr><td rowspan="4">Lk</td><td>Lk-Mk</td><td>3</td></tr><tr><td>Lukan</td><td>22</td></tr><tr><td>Double</td><td>22</td></tr><tr><td>Triple</td><td>54</td></tr><tr><td rowspan="4">Mk</td><td>Lk-Mk</td><td>5</td></tr><tr><td>Markan</td><td>5</td></tr><tr><td>Mt-Mk</td><td>8</td></tr><tr><td>Triple</td><td>82</td></tr><tr><td rowspan="4">Mt</td><td>Matthean</td><td>13.96</td></tr><tr><td>Mt-Mk</td><td>6</td></tr><tr><td>Double</td><td>23</td></tr><tr><td>Triple</td><td>57</td></tr></tbody></table>

<table><caption>(b) Honoré's statistics on a verse basis</caption><thead><tr><th>Book</th><th>Tradition</th><th>%</th></tr></thead><tbody><tr><td rowspan="4">Lk</td><td>Lk-Mk</td><td>1</td></tr><tr><td>Lukan</td><td>35</td></tr><tr><td>Double</td><td>23</td></tr><tr><td>Triple</td><td>41</td></tr><tr><td rowspan="4">Mk</td><td>Lk-Mk</td><td>3</td></tr><tr><td>Markan</td><td>3</td></tr><tr><td>Mt-Mk</td><td>18</td></tr><tr><td>Triple</td><td>76</td></tr><tr><td rowspan="4">Mt</td><td>Matthean</td><td>20</td></tr><tr><td>Mt-Mk</td><td>10</td></tr><tr><td>Double</td><td>24</td></tr><tr><td>Triple</td><td>46</td></tr></tbody></table>

12, 15, 16, 17, 18], consisting in counting the number of occurrences of the same word being present in two Gospels within the same narrative framework, either in its inflected or lemma-tized forms, and stylistic analysis of the Gospels for source detection. Much of the last fifty years of research has relied on the study of these agreements and J.C. Poirier has provided a very thorough survey of the main verbal agreement work in [19]. He however concludes his study with little hope regarding the feasibility of relying on verbal agreements to "find a final solution to the synoptic problem", as the "statistical studies have too often amounted to coded expressions of their user's commitments".

Another possible approach relies on the analysis of the style of the Gospels, either using *Correspondance Analysis* as suggested by A. Linmans in [20] and D. Mealand in [21, 22, 23]. Notably, Mealand provides several different approaches to leverage stylistic analysis to provide insights regarding the likelihood of the 2SH: he uses in [21] *Correspondance Analysis* and *Discriminant analysis* to separate Lukan material into three sources (Q, Lukan and Markan) and adds cluster analysis and *Generalized Linear Models*, while accounting for genre (which was not the case in his Lukan study), to perform a similar analysis on Matthew's Gospel. The author finds that the results of his study mostly confirms the 2SH as he is able to correctly classify material for both Gospels. Linmans is even more prudent, and concludes that when accounting for genre, sources can no longer be clearly distinguished and that the whole case still hangs in the balance [20], even when considering parallels between the Gospels. In spite of Poirier's doubts, we expect that the newest advances in data science, able to capture more efficiently complex patterns in data, will bring some new lights into the synoptic problem.

Our contribution to the current state of the art of statistics and the Synoptic Gospels is three-folds: (1) we leverage more recent advances in data science and move on from Mealand's and Linmans Linear Discriminant Analysis [23, 20] to classifier two sample tests using Random Forests; (2) we work at the pericope level instead of at the verse level or sliding pools of words;
(3) we focus not only on stylometric analysis, but take into account the editing patterns of Luke and Matthew, which takes a step further the approach suggested by Linmans when he uses Principal Correspondance Analysis for parallels.

# 4. Methods

## 4.1. Dataset

Data from the *SBL Greek New Testament* (SBLGNT) edition and its morphological parsing and lemmatization, provided by MorphGNT [6], were used for the study. Given its wide reception and its on-going critical work, we assumed the SBLGNT text to be the "correct" one and did not consider the different variants, which will be included into some of our further works, where we hope to show the impact of considering different base texts on stylometry analysis. The verses were grouped into pericopes based on synoptic parallels suggested by the landmark work of K. Aland [24]. These pericopes were labeled as triple (Mt-Lk-Mk), double (Mt-Lk), Matthean, Lukan, or Markan. There were a total of 276 distinct pericopes, with 613 different variations of these pericopes: 237 from Luke, 154 from Mark, and 222 from Matthew. The triple tradition consisted of 127 pericopes, and the double tradition has 50 pericopes. Additionally, we differentiated between narrative and sayings pericopes, as their genre can have a significant impact on style as underlined by Linmans [20], Mealand [23] and Oaks [25]: we manually classified the pericopes across *sayings* (including prophecy and parable) and *narratives* (including Passion material, controversy and miracle stories). We acknowledge that some of this labeling can be considered arbitrary as some of the parable contain large narrative materials (such as the *Parable of the Prodigal Son*, Lk 15:11-32), and further consideration regarding the impact of this classification will be included in future works.

## 4.2. Defining the stylistic features

To perform the stylistic comparison between the different pericopes, we decide to focus our study using only non-significant language patterns, instead of focusing on content word frequency with metrics like *tf-idf* computed on the corpus as a whole: as the pericopes relate the same stories, they naturally tend to use the same vocabulary which could bias a distance metric. To focus solely on style, we only keep features that do not bring any meaning to the sentence other than its grammatical and logical structure.

In total, 103 metrics were designed, based on recent advances in stylometry [26, 27, 28], that can be roughly divided into three categories: (1) *grammatical and morphological features*, which consists in computing the ratio of inflects, *Part Of Speech* (POS), verb conjugations, sentences length, punctuation, capitalization ...; (2) *function word frequencies*, consisting in conjunctions, generic words such as temporal markers ...; (3) *dialog features*, which consists in counting the number of occurrences of words indicating dialog, basing ourselves on the classification provided in [29]. The exhaustive list of all the computed metrics are available in table 6 of the appendices. The 103 features are then computed for each pericope, each pericope being projected into a 103-dimensional space.
**Table 2**

Examples of computations of stylistic differences on a subset of features for a triple tradition pericope (The Healing of Peter's Mother In Law, pericope 37 of Kurt Aland's classification, Mt 8:14–15, Mk 1:29–31, and Lk 4:38–39)

<table><thead><tr><th colspan="4">(a) Input</th><th colspan="4">(b) Output</th></tr><tr><th>Book</th><th># Words</th><th># Proper Nouns</th><th># καί</th><th>Book</th><th># Words</th><th># Proper Nouns</th><th># καί</th></tr></thead><tbody><tr><td>Mt</td><td>33</td><td>2</td><td>6</td><td>Mt-Mk</td><td>-15.00</td><td>-3.00</td><td>-1.00</td></tr><tr><td>Mk</td><td>48</td><td>5</td><td>7</td><td>Mt-Lk</td><td>-8.00</td><td>0.00</td><td>3.00</td></tr><tr><td>Lk</td><td>41</td><td>2</td><td>3</td><td>Lk-Mk</td><td>-7.00</td><td>-3.00</td><td>-4.00</td></tr></tbody></table>

### 4.3. Computing stylistic difference

The stylistic difference between two pericopes is then defined as the difference between two stylometric vectors. The closer the style transformation, the smaller the differences in each variable, and vice versa. For pericopes from the triple tradition, there are two distances per pericope, and from the double, there is one distance per pericope. Single tradition material does not have any distance, as it has no reference point within other Gospels by definition. Stylistic difference vectors for pericope number *k* and feature *i* will be denoted as $\Delta^k(A - B)_i$, for $A, B \in \{Mt, Lk, Mk\}, A \neq B$. The gospel of Mark is used as a reference as both tested theory posits Markan priority.

For example, focusing only on feature 1 corresponding to the number of words in the pericope of The Healing of Peter's Mother In Law, which is within the triple tradition (Mt 8:14–15, Mk 1:29–31, and Lk 4:38–39, pericope 37 according to Aland's classification), Mark uses 48 words, while Luke uses 41 and Matthew uses 33: this results in a stylistic difference for feature 1 of $\Delta^{37}(Mt - Mk)_1 = -15$, $\Delta^{37}(Lk - Mk)_1 = -7$ and $\Delta^{37}(Mt - Lk)_1 = -8$. This difference is then computed for every of the 103 stylistic features and we give an example of the computation of these vectors in table 2: for example, we can see that when editing the Mother In Law Pericope, Luke and Matthew both decided to reduce the number of proper nouns to 2 and to shorten the number of words in the pericope. If similar editing choices are observed regularly, then this would give evidence that editing was not performed independently and vice versa.

Because the reduced space can get sparse (as some pericopes have had little change), we choose to apply Principal Component Analysis [30], selecting the number of components required to explain at least 90% of the variance of the dataset.

### 4.4. Using Random Forests classifiers for two-samples testing

Working with such a large feature set poses a challenge due to the multivariate nature of the stylistic vectors, which would require multivariate response models if we were to work using a classic statistical framework: a potential solution, as suggested by Mealand in [23], is to employ Generalized Linear Models with a multivariate target variable. However, Mealand's decided to limit its model to only five dimensions. Given the extensive input space in our study, such models would not provide reliable results as we would have to compute one statistical text per feature, leading to hundreds of statistical test that would increase the risk of family-wise error rate.

To tackle this issue and still work in a multi-dimensional space, we use the approach suggested
by Lopez-Paz et al. in [31], who suggest using the prediction of classification of Machine Learn-
ing (ML) models as a two-samples hypothesis test in order to assess if two samples are from
different distributions, with the following heuristic: if a ML model can discriminate more accu-
rately between the observed samples than between randomly drawn ones, then the two classes
must have different distributions. More formally, we compare two samples, P and Q. If the null
hypothesis (P = Q) is true, a binary classifier's accuracy on a held-out subset should be around
chance-level. Conversely, if the alternative hypothesis (P ≠ Q) is true, the classifier's accuracy
will be higher than chance-level, leading to rejection of H₀. As our classifier, we use Random
Forests [32], an ensemble-based learning method. Random Forests help mitigate overfitting
in low-sample scenarios by training CART [33] trees independently on bootstrapped samples.
This introduces randomness, preventing a single tree from dominating the ensemble. Final
predictions are then based on the most frequent labels among the individual trees' predictions.

**4.5. Tested hypotheses**

To enhance the comprehensiveness of our study, we adopt a dual approach. We begin by ex-
amining the stylistic variations within the Lukan text between the double and triple traditions,
aiming to identify distinct sources (as discussed in the next paragraph 4.5.1). We then propose
a novel approach by investigating the interdependence among the edits made by different au-
thors (as outlined in paragraph 4.5.2).

**4.5.1. Source detection for stylometry**

To decide which scenario is the most likely, one can inquire if the styles vary differently across
the triple and the double material: if so, one could assume that Luke has taken his double and
triple material from two different sources, like the 2SH posits. It would however not completely
invalidate the FGH, as one could argue that Luke could have taken all of his triple tradition ma-
terial from Mark, but would mean that we cannot observe any *editorial fatigue*<fnref n="2" /> as argued by
Goodacre in [10], where Luke style would match Matthew's instead of following Mark's.
To assess the stylometric difference across the sources, we design three different stylometric
hypotheses to be evaluated using the test methodology described in subsection 4.4. These tests
are named using the convention **Stylo.n**, *n* ∈ {1, 2, 3} and their alternative hypotheses and impli-
cations in the synoptic problem are summarized in table 3.

**Stylo.0** measures style differences between sayings and narrative in the triple and double tradition of Luke's gospel to evaluate the relevance of our selected features. If we cannot detect significant stylometric changes, it implies our features may not be reliable for detecting subtle variations like distinct sources. **Stylo.1** tests the style difference between the double and triple traditions using a ML model. Successful classification would indicate that the style varies differently across the two traditions, supporting the idea that Luke sourced his double and triple material from separate origins, thus validating the 2SH. **Stylo.2** acts as a sanity check, considering the genre of the classified contents. The goal is to build a model that accurately classifies

<footnote n="2">²As defined by Goodacre in [34], "*Editorial fatigue is a phenomenon that will inevitably occur when a writer is heavily dependent on another's work. In telling the same story as his predecessor, a writer makes changes in the early stages which he is unable to sustain throughout.*"</footnote>
<table><caption>**Table 3**
Summary of performed tests for comparison of style</caption><thead><tr><th>Test ID</th><th>Compared samples and size</th><th>Alternative hypothesis</th><th>Has Luke read Matthew ?</th></tr></thead><tbody><tr><td>Stylo.0</td><td>Narrative pericopes in Luke (120 samples) vs Sayings pericopes in Luke (115 samples)</td><td>There is a strong difference in style between narrative and sayings in Luke.</td><td>N/A</td></tr><tr><td>Stylo.1</td><td>Double tradition material in Luke (50 samples) vs Triple tradition material in Luke (127 samples)</td><td>There is a strong difference in style between double and triple material in Luke.</td><td>No</td></tr><tr><td>Stylo.2</td><td>Double tradition sayings in Luke (45 samples) vs Triple tradition sayings in Luke (49 samples)</td><td>There is a strong difference in style between sayings in double and triple material in Luke.</td><td>No</td></tr><tr><td>Edit.1</td><td>Stylistic distance between Luke and Mark (127 samples) vs Stylistic distance between Matthew and Luke (127 samples)</td><td>Luke editing of Mark differs significantly from Matthew's for the triple tradition.</td><td>No</td></tr><tr><td>Edit.2.1</td><td>Stylistic distance between Luke and Mark (127 samples) vs Stylistic distance between Matthew and Mark (127 samples)</td><td>Luke is significantly closer in style to Mark than to Matthew on the triple tradition.</td><td>No</td></tr><tr><td>Edit.2.2</td><td>Stylistic distance between Luke and Mark on narrative pericopes (49 samples) vs Stylistic distance between Matthew and Luke on sayings pericopes (49 samples)</td><td>Luke editing of Mark differs significantly from Matthew's for the triple tradition (sayings)</td><td>No</td></tr><tr><td>Edit.2.3</td><td>Stylistic distance between Luke and Mark on narrative pericopes (78 samples) vs Stylistic distance between Matthew and Luke on narrative pericopes (78 samples)</td><td>Luke editing of Mark differs significantly from Matthew's for the triple tradition (narrative)</td><td>No</td></tr></tbody></table>

sayings common to Matthew and Luke, as well as those found in Mark, to ensure that the model does not oversimplify classification by labeling double tradition material as "sayings" and triple tradition material as "narratives."

### 4.5.2. Comparison of edits

The other complementary approach that we suggest in this study is the comparison of the editorial choices made by Luke and Matthew of Markan material: if these patterns are different, then one could assume that they were made independently. We design two different statistical tests relying on a two-samples classifier, **Edit.1** and **Edit.2**, that we summarize in table 3. **Edit.1** aims at comparing the distance between the triple tradition between the three Gospels in order to understand where Luke's triple tradition comes from. If we make the hypothesis that a smaller variation in style means a similar source, then the distance between Luke-Matthew
should be significantly closer than the distance between Luke-Mark over the triple tradition if Luke had access to Matthew. Otherwise, one can assume that Luke took his triple tradition from Mark, which would lead to weakening the FGH and be consistent with the 2SH, as Luke and Matthew are expected to perform their editing independently. **Edit.2** aims at measuring the independence of Matthew's and Luke's use of Mark: the idea is to see if it is possible to distinguish an editing pattern between sources. If we can find no significant difference from the way Luke edits Mark than the way Matthew edits Mark, one could assume that the edition process did not happen independently and Luke had access to Matthew's triple tradition. These tests are then extended to narrative and sayings data, as sayings of Jesus could be expected to be less edited and are more likely to be taken verbatim.

## 4.6. Implementation

The Random Forest classifiers, as well as the PCA, is taken from sklearn [35]. We use 100 trees in the Random Forest and use the Gini criterion to measure the homogeneity of each split. We re-implemented the method described in [31] for the classifier two-samples tests. All our data and our pipeline to obtain these results are available at https://github.com/metz-theolab/chr_2023.

# 5. Results

The results of the stylometric tests (Stylo.n) and edit tests (Edit.n) are available in table 4.

<table><caption>**Table 4**

Result of stylometric source detection in Luke</caption><thead><tr><th>ID</th><th>Alternative hypothesis</th><th>p-value</th></tr></thead><tbody><tr><td>Stylo.0</td><td>Lukan material differs in style significantly between sayings and narrative contents</td><td>0.0</td></tr><tr><td>Stylo.1</td><td>Double and triple tradition differs in style significantly</td><td>0.03</td></tr><tr><td>Stylo.2</td><td>Double and triple tradition sayings differ significantly</td><td>0.05</td></tr><tr><td>Edit.1</td><td>Luke is significantly closer in style to Mark than to Matthew on the triple tradition.</td><td>0.02</td></tr><tr><td>Edit.2.1</td><td>Luke editing of Mark differs significantly from Matthew's for the triple tradition.</td><td>0.07</td></tr><tr><td>Edit.2.2</td><td>Luke editing of Mark differs significantly from Matthew's for the triple tradition (sayings).</td><td>0.45</td></tr><tr><td>Edit.2.3</td><td>Luke editing of Mark differs significantly from Matthew's for the triple tradition (narrative).</td><td>0.19</td></tr></tbody></table>

## 5.1. Stylometric source detection in Luke

**Stylo.0: Difference in style between narrative and sayings material in Luke** The test results demonstrate a significant ability of our stylistic features to differentiate between narrative and sayings content within the entirety of Luke's pericopes (p-value < 0.001). When employing a Random Forest classifier using 75% of the data as train and 25% as test, and stratification based on observed genre, the classifier achieves an accuracy score of 81% on unseen pericopes (95% over the complete dataset). These results indicate that our stylistic features effectively enable the classification of pericopes based on their style. When analyzing the
misclassified pericopes, they tend to fall along the genre borderlines, with the 6 misclassified sayings containing many narrative elements (The Parable of the Rich Man and Lazarus, the Day of the Son of Man, On Riches and the Rewards of Discipleship, The Parable of the Good Samaritan, The Parable of the Wicked Husbandmen, The Pharisee and the Publican) and the 7 misclassified narratives featuring discourses (The Beelzebub Controversy, The Rich Young Man, John the Baptists's Messianic Preaching, The Pharisees Seek a Sign, First Preaching Tour of Galilee, Peter's Denial Prediction and The Third Prediction of the Passion). This high accuracy reaffirms the validity and relevance of the devised features for the remainder of our study.

**Stylo.1: Difference in style between double and triple tradition material in Luke** The two-samples test conducted on the classifier reveals a highly significant distinction (p-value of 0.03) between the styles of the double and triple traditions. Utilizing a Random Forest classifier with a test-train split ratio of 0.25, we achieve an accuracy of 91% for unseen pericopes and 97% across the entire dataset, successfully identifying material from both the double and triple traditions. This disparity in style strongly supports the hypothesis that Luke drew his material from two distinct sources for the double and triple traditions, aligning with the 2SH. While proponents of the Farrer's model may argue that Luke derived the double tradition material from Matthew and the triple tradition material from Mark, this interpretation does not align well with the concept of editorial fatigue proposed by Goodacre, as our tests indicate that Luke disregarded Matthew's style in his handling of the Matthean edition. However, the study of the confusion matrix available in table 5b shows that the unbalanced dataset affects the accuracy per class (there are 127 triple tradition pericopes and 50 double tradition). Further works of ours will include adding some techniques for small unbalanced dataset [36], to enhance the quality of our prediction model.

**Stylo.2: Difference in style between double and triple tradition sayings in Luke** As shown in the previous paragraph, the stylometric features differ strongly depending on the genre of the pericope, and we need to make sure that the results of the test **stylo.1** are not only a measure of the difference of style, as the double tradition is mostly composed of sayings. However, the results of test **Stylo.2**, with a p-value of 0.05, shows a significant style difference between sayings from the double and triple tradition, allowing us to confirm the stylistic difference between material, even when accouting for genre. A random forest using a test-train split ratio of 0.25 yields an accuracy score of 75% on unseen data and 92% of the whole dataset. The confusion matrix is available in table 5c and the small sample size once again affects the power of the evaluation of the classifier on test data: the error is however balanced across the double and the triple tradition.

## 5.2. Difference in edition behavior

**Edit.1: Where did Luke take his triple tradition from?** The two-sample classifier tests reveal a statistically significant distinction (p-value of 0.02) between the stylistic similarity of Luke and Mark compared to Luke and Matthew. Luke and Mark exhibit a closer style (with a combined total distance of 50.46) than Luke and Matthew (with a combined total distance of 72.81). When a Random Forest classifier is trained using standard test/train fitting (with a 0.25
<table><caption>**Table 5**
Confusion matrix of Stylo.n tests</caption><thead><tr><th colspan="3">(a) Confusion matrix of Stylo.0</th><th colspan="3">(b) Confusion matrix of Stylo.1</th><th colspan="3">(c) Confusion matrix of Stylo.2</th></tr><tr><th>Reality Predicted</th><th>Double</th><th>Triple</th><th>Reality Predicted</th><th>Double</th><th>Triple</th><th>Reality Predicted</th><th>Double</th><th>Triple</th></tr></thead><tbody><tr><td>Double</td><td>21</td><td>6</td><td>Double</td><td>8</td><td>2</td><td>Double</td><td>9</td><td>3</td></tr><tr><td>Triple</td><td>5</td><td>27</td><td>Triple</td><td>2</td><td>33</td><td>Triple</td><td>3</td><td>9</td></tr></tbody></table>

ratio of test and train split), it achieves an accuracy of 79% when classifying unseen samples and a 95% accuracy across the whole dataset. The significant proximity observed between Luke and Mark, and the distance from Matthew to Luke, suggests that Luke most likely drew his triple material from Mark rather than Matthew. This finding once again contradicts the notion of editorial fatigue, as proposed by Goodacre, which would have resulted in a less pronounced difference in stylistic distances. We believe it is thus unlikely that Luke relied on Matthew's text to copy his triple tradition material, which supports the plausibility of the 2SH.

**Edit.2: Independence of Matthew's and Luke's use of Mark** All statistical tests conducted to evaluate the variation between Matthew's and Luke's edits of Mark have yielded inconclusive results, with p-values greater than 0.05. However, it is worth noting that the p-value of Edit2.1 is relatively low at 0.07, suggesting a tendency to reject the hypothesis that Luke and Matthew edit Mark in a similar manner. This inclination leads us to consider the hypothesis that Luke disregards Matthew's edits and implies that Luke did not consult Matthew's use of the Markan material. When examining sayings or narrative material, our study encounters limitations in drawing definitive conclusions due to the combination of high p-values and low statistical power, particularly when the effect size is small. The available dataset consists of only 156 triple tradition narrative pericopes and 96 limited triple tradition sayings pericopes. Consequently, our test subset comprises merely 39 and 24 samples, respectively, considering a 0.25 test-train splitting ratio. Given these constraints and until further works consisting on using methods for unbalanced dataset, we are unable to confirm or dismiss the notion of a difference in editing style between Luke and Matthew: we have to rely on the other performed tests, which all point to the 2SH.

## 5.3. Discussion

The two facets of our analysis analysis seem to point towards the 2SH: the stylometric tests show that the double and triple tradition differ significantly in terms of style, even when considering sayings (Stylo.1 and Stylo.2). Even though one must remain careful on results on smaller datasets, such a difference seems to confirm that the double and triple material come from different sources, and thus that Luke took his triple material from Mark and not from Matthew. This use of Mark's material by Luke is further confirmed by the distance stylistic tests (Edit.1), which shows that Luke's style is significantly closer to Mark's than to Matthew's.
Whenever comparing the editing style, the test **Edit.2** tends to show that Matthew and Luke use the Markan material differently, even though the test p-value and power is not sufficient to definitely conclude regarding the difference in material edition. We believe our features aggregated on the pericope level might be too global and we will investigate in further works more refined features, such as the verbal agreements, the addition and the deletion orders and use Deep Learning embedding models such as LSTM [37].

Whenever considering previous studies, our results generally align with results from other statistical studies of the synoptic Gospels: Mealand demonstrated (prudently) a difference in word usage on Matthew's Gospel into a Matthean, double, and triple tradition, and the Linear Discriminant Analysis he runs on Luke's Gospel also shows a division in double and triple tradition. As there is to our knowledge no stylometric study of the edition of patterns between Matthew and Luke, we cannot compare our results to previous studies, but future works will compare the feature importance of the Random Forests to the manual works performed by scholars to characterize Matthew and Luke's handling of Markan material.

# 6. Conclusion and further works

As a conclusion, our work adds yet another argument towards the likelihood of the 2SH model. We however want to emphasize that our intention is not to offer a definitive answer to the synoptic problem, but rather to introduce innovative analytical tools that can assist in presenting the most plausible solution to the problem through style analysis, even though numerous variables other than style, such as pericope orders, must be considered to provide a comprehensive answer.

Our further studies will include adding additional features better able to characterize source usage, such as verbal agreements, and see how it affects the results of the edition tests. We will also work on augmenting the number of samples, either by simulation through bootstrapping or by considering moving windows across the different verses, to increase the statistical power of our study. We will also take into account variants instead of taking the SBLGNT as the final text of the Gospels.

# 7. Acknowledgements

This work was funded by the SCRIBES (Biblissima+ Grant) and the ANR SHERBET (CE38-2023) projects.

# References

[1] J. D. Dunn, The Oral Gospel Tradition, Wm. B. Eerdmans Publishing Co., 2013.

[2] B. H. Streeter, The Four Gospels: A Study of Origins, Treating of the Manuscript Tradition, Sources, Authorship & Dates, Macmillan and Co., 1930. (revised).

[3] W. Farmer, The Synoptic Problem: A Critical Analysis, Mercer University Press, 1976.

[4] A. Farrer, On Dispensing With Q, Blackwell, 1955. In D.E.Nineham (ed.): Studies in the Gospels: Essays in Memory of R.H. Lightfoot.
[5] A. Schweitzer, The Quest of the Historical Jesus, Dover Publications, 2005.

[6] J. K. Tauber, MorphGNT: SBLGNT Edition. Version 6.12 [Data set], 2017. URL: https://github.com/morphgnt/sblgnt.

[7] Augustine, The Harmony of the Gospels (De Consensu evangelistarum), Kessinger Publishing, 2015.

[8] B. Orchard, T. Longstaff, J. J. Griesbach: Synoptic and Text - Critical Studies 1776--1976, volume 34, SNTS Monograph Series (Cambridge University Press), 1978.

[9] A. McNicol, D. Dungan, Beyond the Q Impasse: Luke's Use of Matthew: A Demonstration by the Research Team of the International Institute for Gospel Studies, Bloomsbury, 1996.

[10] M. Goodacre, The Case Against Q: Studies in Markan Priority and the Synoptic Problem, Trinity Press International, 2002.

[11] C. Weisse, Die evangelische Geschichte kritisch und philosophisch bearbeitet, Breitkopf und Hartel, 1838.

[12] A. Honoré, A Statistical Study of the Synoptic Problem, Novum Testamentum 10 (2/3) (1968) 5–147.

[13] J. Hawkins, Horae Synopticae: Contributions to the Study of the Synoptic Problem, Clarendon Press, 1899.

[14] J. O'Rourke, Some Observations on the Synoptic Problem and the Use of Statistical Procedures, Novum Testamentum 16(4) (1974) 272–277.

[15] C. Carlston, D. Norlin, Once More – Statistics and Q, Harvard Theological Review 64(01) (1971) 59–78.

[16] A. Abakuks, The Synoptic Problem and Statistics, Chapman & Hall, 2015.

[17] S. Mattila, Negotiating the Clouds around Statistics and "Q": A Rejoinder and Independent Analysis, Novum Testamentum 16(2) (2004) 105–131.

[18] T. R. Rosché, The Words of Jesus and the Future of the 'Q' Hypothesis, Journal of Biblical Literature 79 (1960) 210–220.

[19] J. Poirier, Statistical Studies of the Verbal Agreements and their Impact on the Synoptic Problem, Currents in Biblical Research 7(1) (2008) 68–123.

[20] A. Linmans, Correspondence Analysis of the Synoptic Gospels, Literary and Linguistic Computing 13(1) (1998) 1–13.

[21] D. Mealand, Correspondence Analysis of Luke, Literary and Linguistic Computing 10(3) (1995) 171–182.

[22] D. Mealand, Measuring Genre Differences in Mark with Correspondence Analysis, Literary and Linguistic Computing 12(4) (1997) 227–245.

[23] D. Mealand, Is there Stylometric Evidence for Q?, New Testament Studies 57(4) (2011) 483–507.

[24] K. Aland, Synopsis Quattuor Evangeliorum: Locis parallelis evangeliorum apocryphorum et patrum adhibitis edidit, Württembergische Bibelanstalt Stuttgart, 1964.

[25] M. Oakes, Literary Detective Work on the Computer, John Benjamin Publishing Company, 2014.

[26] R. Forsyth, Feature-Finding for Text Classification, Literary and Linguistic Computing 11 (1996) 163–174.

[27] J. Grieve, Quantitative Authorship Attribution: An Evaluation of Techniques, Literary and Linguistic Computing 22(3) (2007) 251–270.
[28] J. Savoy, Machine Learning Methods for Stylometry, Springer, 2020.

[29] F. Baskevitch, Le vocabulaire du son en grec ancien - étude sémantique et éty-
mologique, 2016. Accessed on the 07/07/2023 at https://www.academia.edu/28172654/Le_
vocabulaire_du_son_en_grec_ancien_ étude_sémantique et_étymologique.

[30] I. Jolliffe, Principal Component Analysis, Springer Series in Statistics, 2002.

[31] D. Lopez-Paz, M. Oquab, Revisiting classifier two-sample tests, International Conference on Learning Representations (ICLR) (2017).

[32] L. Breiman, Random Forests, Machine Learning 45(1) (2001) 5–32.

[33] L. Breiman, J. Friedman, C. Stone, R. Olshen, Classification and Regression Trees, Chapman and Hall/CRC, 1984.

[34] M. Goodacre, Fatigue in the synoptics, New Testament Studies 44 (1998) 45–58.

[35] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine Learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830.

[36] S. Narwane, S. Sawarkar, Machine Learning and Class Imbalance: A Literature Survey, Industrial Engineering Journal 12 (2019).

[37] S. Hochreiter, J. Schmidhuber, Long Short-term Memory, Neural computation 9 (1997) 1735–80.
<caption>**Table 6**</caption>

103 computed features

<table><thead><tr><th>(a) Grammatical features</th><th>(b) Words with computed frequencies</th></tr></thead><tbody><tr><td>pos N</td><td>πατήρ</td></tr><tr><td>number S</td><td>κύριος</td></tr><tr><td>case G</td><td>ἀμήν</td></tr><tr><td>case N</td><td>Χριστός</td></tr><tr><td>gender F</td><td>θεός</td></tr><tr><td>gender M</td><td>ἐαυτοῦ</td></tr><tr><td>avg word length</td><td>ἀν</td></tr><tr><td>unique word ratio</td><td>ἀλλά</td></tr><tr><td>stop words ratio</td><td>ἀπό</td></tr><tr><td>is dialog</td><td>ἄρα</td></tr><tr><td>capitalized words</td><td>δέ</td></tr><tr><td>substring comma</td><td>δή</td></tr><tr><td>substring dot</td><td>διά</td></tr><tr><td>substring median</td><td>ἔτι</td></tr><tr><td>substring kai</td><td>ἐγώ</td></tr><tr><td>avg len comma</td><td>ἐκ</td></tr><tr><td>avg len dot</td><td>ἐν</td></tr><tr><td>avg len median</td><td>ἐπί</td></tr><tr><td>nbr words</td><td>εἰ</td></tr><tr><td>pos RA</td><td>γάρ</td></tr><tr><td>gender N</td><td>ῆ</td></tr><tr><td>pos D</td><td>καί</td></tr><tr><td>pos P</td><td>κατά</td></tr><tr><td>pos C</td><td>μέν</td></tr><tr><td>pos RP</td><td>μετά</td></tr><tr><td>pos RR</td><td>μή</td></tr><tr><td>pos V</td><td>ό</td></tr><tr><td>pos A</td><td>@a</td></tr><tr><td>person 3</td><td>δε</td></tr><tr><td>person 2</td><td>τε</td></tr><tr><td>tense X</td><td>ύ</td></tr><tr><td>tense A</td><td>ύ</td></tr><tr><td>active M</td><td>ύδε</td></tr><tr><td>active P</td><td>περί</td></tr><tr><td>active A</td><td>σύ</td></tr><tr><td>mode P</td><td>σύν</td></tr><tr><td>mode I</td><td>ύπέρ</td></tr><tr><td>mode S</td><td>ύπό</td></tr><tr><td>mode N</td><td>ύ</td></tr><tr><td>number P</td><td>ύστε</td></tr><tr><td>case D</td><td>ἐάν</td></tr><tr><td>case V</td><td>παρά</td></tr><tr><td>case A</td><td>πάλιν</td></tr><tr><td>pos RD</td><td>αὐτός</td></tr><tr><td>person 1</td><td>υίος</td></tr><tr><td>tense P</td><td>ήμέρα</td></tr><tr><td>tense I</td><td>σήμερον</td></tr><tr><td>pos RI</td><td>αὖριον</td></tr><tr><td>pos X</td><td>ἀησοῦς</td></tr><tr><td>tense F</td><td></td></tr><tr><td>mode D</td><td></td></tr><tr><td>mode O</td><td></td></tr><tr><td>tense Y</td><td></td></tr><tr><td>pos I</td><td></td></tr></tbody></table>