Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselines
Authors: Stephen Bothwell
Justin DeBenedetto
Theresa Crnkovich
Hildegund Muller
David Chiang
Venue: EMNLP 2023 Main (2023)
Keywords: rhetorical parallelism
sequence labeling
NLP
Latin
Chinese
resource
TL;DR: This paper provides a complete introduction for the task of rhetorical parallelism detection, including discussions of two datasets, four metrics, and many modeling baselines.

Abstract: Rhetoric, both spoken and written, involves not only content but also style. One common stylistic tool is $\textit{parallelism}$: the juxtaposition of phrases which have the same sequence of linguistic ($\textit{e.g.}$, phonological, syntactic, semantic) features. Despite the ubiquity of parallelism, the field of natural language processing has seldom investigated it, missing a chance to better understand the nature of the structure, meaning, and intent that humans convey. To address this, we introduce the task of $\textit{rhetorical parallelism detection}$. We construct a formal definition of it; we provide one new Latin dataset and one adapted Chinese dataset for it; we establish a family of metrics to evaluate performance on it; and, lastly, we create baseline systems and novel sequence labeling schemes to capture it. On our strictest metric, we attain F$_1$ scores of $0.40$ and $0.43$ on our Latin and Chinese datasets, respectively.

Parallelism is organically employed in many ar- gumentative structures (e.g., "on the one hand on the other hand ...", making parallelism a rou- tine rhetorical figure. The rhetorical impact of Introducing Rhetorical Parallelism Detection: A New Task with Datasets, Metrics, and Baselines
Stephen Bothwell University of Notre Dame sbothwelend. edu
Justin DeBenedetto
Villanova University
justin.
Theresa Crnkovich
Hildegund Müller University of Notre Dame (tcrnkovi, hmuller, dchi ang) edu
David Chiang
Abstract
quotidie dicimus hoc, every day we say this
Rhetoric, both spoken and written, involves only content but also style. One mon stylistic tool is parallelism: the juxtaposi- tion of phrases which have same sequence of linguistic (e.g., phonological, syntactic, se- mantic) fcatures. Despite the ubiquity of paral- lelism, the field of natural language processing has seldom investigated it, missing chance to better understand the nature of the structure, meaning, and intent that humans convey. To address this, we introduce the task of rhetori- cal parallelism detection. We construct a for- Latin
et quotidie facimus. and every day do [it]
et quotidie fit in nobis. and every day [it] is done in us
Figure 1: Example three-way parallelism from Augus- tine's Sermones (Sermon 181, Section 8). Solid lines connect words with multiple linguistic features in com- mon; dashed lines indicate weaker connection.
mal definition of it; provide dataset and one adapted Chinese dataset for it; we establish a family of metrics to evaluate per- formance on it; and, lastly, create baseline systems and novel sequence labeling schemes capture it. On our strictest metric, we attain Fi scores of 0.40 and 0.43 Latin and Chinese datasets, respectively.
these structures is apparent even for audiences not schooled in classical rhetoric. Recognition of paral- lelism, then, important for grasping the structure, meaning, and intent that humans wish to convey; thus, the computational modeling of parallelism is both an interesting and practical problem.
However, because parallelism can occur so many levels finguistically--ae often with no lexical modeling parallelism computationally is difficult. As a first foray into studying the problem of rhetorical parallelism detection (RPD) and en- abling others to study it, we present in this paper multiple public datasets, metrics, and models for it. For one dataset, turn to the Latin sermons of Augustine of Hippo (354-430). Augustine had been trained as rhetorician and had taught the craft of secular rhetoric before his conversion Christianity. However, upon becoming preacher, he consciously replaced the adorned style of late ancient rhetoric with a style streamlined toward speaking well with diverse North African congre- gations. Parallelism frequently marked this style. Augustine did not just employ parallelism stylis- tically, however; he also theorized about it. In his work De Doctrina Christiana, Augustine described three rhetorical styles, or genera dicendi ("ways of
1 Introduction
Ueni, uidi, uici,' or, "I came, I saw, I conquuered' -why is this saying so memorable? One reason is the high degree of parallelism it exhibits in its morphology (each verb is first-person, singular, per- feet tense), phonology (cach verb starts with /w/ and ends with fit/), and prosody (each verb has two long syllables with accent on the first). In similar sayings (as in Fig. 1), syntax and semantics also contribute. The related elements generally occur in either the same order or in an inverted order.
'Popularly attributed to Julius Caesar (Suetonius Tranquil- lus, 1993). In Classical Latin, and r were graphical variants of one letter; as consonant, was pronounced /w/. Following common practice, we write it as u. We also write and as

---

speaking") (Augustine of Hippo, 1995); from these, he characterized the genus temperatum ("moderate style") by highly-parallel passages. Thus, his ser- mons are ideal for studying parallelism. In addition, his theory implies that parallelism detection may be useful for automatic stylistic analysis. Already, it has been used to analyze discourse structure (Gué- gan and Hernandez, 2006), summarize documents (Alliheedi, 2012), identify idioms (Adewumi et al., 2022), evaluate student writing (Song al., 2016), study political speech (Tan al., 2018) and detect fake news (Gautam and Jerripothula, 2020).
phonology, morphology, synlax, semantics, even style. The features could be the same (e.g., the repeated sounds in ueni, uidi, uici) or diametrically opposed (e.g., the opposite meanings in a time to be born and a time to die). These features typically occur in the same order or the opposite order.
Suppose that we have document w, which is sequence of tokens ... Wn- A span of w is a pair (i. where
S I n, whose contents are the tokens wi... wj. We say that two spans (i1, fi) 1) and (12, j2) are overlapping if they have at least one token in common-that is, i s $ ji or i2 S 4 S j2. Meanwhile, they are nested if i S is S S S or i2 s i S j S j2.
Parallelism detection may also have broader NLP applications. In syntactic parsing, in the sen- tence I saw a man with a mustache and a woman with telescope, the reading where telescope modi- fies saw is all but ruled out because it would violate parallelism. Thus, modeling parallelism could as- sist in syntactic disambiguation. Parallelism is also vital disfluency detection, as speakers tend to maintain prior syntactic context when amending verbal errors (Zayats and Ostendorf, 2019).
A parallelism is a set of two or more non- overlapping spans whose contents are parallel. We call these spans the branches of parallelism. An example of complete parallelism with three branches given in Fig. RPD, then, the task of identifying all of the parallelisms in a text.
Let P be a set parallelisms. Then P falls into one of three categories:
Toward such ends, we establish the task of rhetor- ical parallelism detection (RPD, Section 2). We create one dataset from Augustine's 's sermons and adapt another consisting of Chinese student essays (Section 4), establish evaluation metrics (Section 5), and investigate baseline models (Section 6). By au- tomatically learning and exploiting relationships among linguistic features, we achieve roughly a 40% F1 score on both datasets' test sets (Section 7).
is flat if all the branches of every parallelism in P are pairwise non-overlape con-overlapping.
nested some pair of its branches nest and no pair of its branches overlap without also being nested.
3. P is overlapping if it contains some pair of branches which overlap.
2 Task Definition
Moving away from the sentence-level conceptions of Guégan and Hernandez (2006) and Song al. (2016), we formalize the task of rhetorical par- allelism detection (RPD). Broadly, we deem se- quences to be parallel if they meet two conditions:
Our Augustinian Sermon Parallelism (ASP) dataset (see Section 4) is nested, and the Paibi Student Es- say (PSE) dataset (Song et al., 2016) is flat. Al- though our baseline models only predict flat sets of parallelisms, evaluations include nested ones.
1. Locality: Parallel structures should be within temporal proximity of one another for two re- lated reasons. First, structures that are close by are more likely to be intentional rather than incidental in the mind of the speaker/author. Second, for parallel structures to be rhetor- ically effective, they must be consecutive enough to be recalled by the listener/reader.
Related Work
In this section, we connect two subareas of NLP to RPD in order to display their relation to and influence on its development.
3.1 Automated writing evaluation
Arguably, the main research area that has explored RPD is automated writing evaluation (AWE). Since its inception (Page, 1966). AWE has aimed to re- duce instructor burden by swiftly scoring essays;
2. Linguistic Correspondence: Parallel struc- tures should have some linguistic features in common, which could include features of
'Song et al. dataset was not originally named hut has heen given name through its inclusion in this work.
"See chapters 4.20.40; 20.44; 21.47f.

---

Dataset
Documents
Sections
Tokens
Branched Tokens
Branches
Parallelisms
ASP
80
477
156,
19. 701
651 (39)
0.062 (14)
PSE-I
409
3,855
203
25,
2, 153 (0)
786 (0)
Table Dataset frequency statistics for both the datasets featured in this paper. "Sections" are defined as natural partitions of documents, such as divisions determined by editors (ASP) natural paragraphs (PSE-I). "Branched Tokens" refer to all tokens involved in a branch. Parenthesized values refer to the nested subset of a given quantity.
Parallelisms per Section
Branches Parallelism
Branch Distance (Tokens)
Branch Size (Tokens)
NIO (Branches)
% Pairs with No LO
Dataset
ASP
4.32 3.22
2.26 = 0.68
2.54 2.29
4.24 = 2.72
0.24 = 0.19
24.17% 2.30%
PSE-1
1.92 1.47
2.74 +: 0.75
2.41 = 4.00
11.86 + 7.09
0.31 0.15
Table 2: Dataset derived statistics. All figures but the last one are means with standard deviations. The lexical overlap (LO) metries are normalized with respect to the union of branches' token multisets.
however, especially with the involvement of neural networks, it seems to be limited in its pedagogical benefit because of its inability give sufficient feedback (Ke and Ng, 2019; Beigman Klebanov and Madnani, 2020).
DD has frequently been framed as a sequence labeling task. Some previous models use schemes which adapt BIO tagging (Georgila, 2009; Osten- dorf and Hahn, 2013; 3: Zayats al., 2014, 2016), while others (Hough and Schlangen, 2015) have proposed novel tagging schemes on the basis of prior theory (Shriberg, 1994) and the Switchboard corpus's ' notation (Meteer and Taylor, 1995), aug- menting tags with numerical links to indicate rela- tionships between reparanda and repairs.
Concerning RPD, Song el al. explored parallel structure as a critical rhetorical device for evaluat- ing writing quality (Song et al., 2016). Construing parallelism detection as sentence pair classifica- tion problem, they achieved 72% F1 for getting entire parallelisms correct their dataset built from Chinese mock exam essays via random for- est classifier with hand-engineered features. Sub- sequent work used RNNs (Dai et al., 2018) and CNNs with some custom features (Liu et al., 2022). Such work has also been applied in the IFlyEA as- sessment system to provide students with stylistic feedback (Gong et al., 2021).
We employ elements of both types of tagging schemes: we directly adapt BIO tags to paral- lelisms, and, like Hough and Schlangen (2015), we also augment tags with numerical links to indicate relationships between branches (see Section 6.
Datasets
This paper presents two datasets for RPD: the Au- gustinian Sermon Parallelism (ASP) and Paibi Stu- dent Essay (PSE) datasets. We first describe steps taken for the individual datasets in Sections 4.1 and 4.2 before discussing shared preprocessing steps and data analyses in Sections 4.3 and 4.4.
Compared to that work, formalization of RPD permits token-level (as opposed sentence- level) gramularity for parallel structure. In line with this, we approach RPD terms of sequence label- ing instead of classification. Moreover, we provide the first (to our knowledge) public release of token- level parallelism detection data (see Section 4).
4.1 Augustinian Sermon Parallelism Dataset
3.2 Disfluency detection
The ASP dataset consists of 80 sermons from the corpus of Augustine of Hippo (Augustine of Hippo, 2000a,b). 4 Our fourth author, an expert classi- cist and Augustine scholar, labeled these sermons for parallel structure using our annotation scheme. This scheme involves labeling branches and linking them to form parallelisms. We further distinguish between synchystic (same order) and chiastic (in-
RPD closely resembles and is notably inspired by disfluency detection (DD). DD's objective is gener- ally to detect three components of speech error: the reparandum (the speech to be replaced), the interregnum (optional intervening speech), and the repair (optional replacing speech) (Shriberg, 1994). Because the reparandum and repair often relate in their syntactic and semantic roles, they (as with parallelisms) share many linguistic features.
*The ASP dataset is freely available at https: //gi thub. htt//yyyoolooos/Auusttino wwg.sssiiiiii--eeenon.. allelisms

---

(a) ipse panis
esuriret
1 satietas
Sitiret
I1 airtus
so.that itself NOM:S hungeri NaMeety:NMM:sG thiSStI Strength:NOM:SG
infirmanetur weakenn:PFV:SB/V.33G ".. that bread itself might hunger, satiety might thirst, strength might be weakened, health might be wounded, life might
sanitas
udneruretur beG wound:1FV:SBIv:35G
In [ uita
moraretur life: dic:Ipry:sevv33s
11
die?'
I
mi m
I 2M
EN 1
xiãocão gēng
tiānkōng geng lán shēnghuó gêng mčihão
grass more green sky more blue life more good
"The grass is greener, the sky is bluer, and life is better."
Figure 2: Examples from the ASP and PSE datasets. Blue square brackets demarcate each branch, and numbering
indicates the parallelism which each branch belongs. We provide word-by-word English translation, gloss some morphological features in branches, and present an idiomatic translation. Example (a) is from Sermon 207,
Section 1. Example (b) is from Essay 1, Paragraph 5, For more ASP dataset examples, see Figs. 7 and
verted order) parallelisms. For more details on annotation scheme, see Appendix A; for details a verification of the dataset's quality through an inter-annotator agreement study, see Appendix C. An example Augustine's use of parallelism is presented in Fig. 2(a). In this subordinate clause, Augustine builds up a five-way parallelism. This parallelism not only boasts shared morphology and syntax through series of subject-verb clauses, but it also presents stylistic parallelism. Each clause uses the rhetorical device of personification, pre- senting an object or abstract idea as an agent, and this device is paralleled to emphasize it. As this example shows, Augustine frequently layers lin- guistic features to compose parallelisms.
times defined as having at least three branches (al- though the PSE dataset has many examples of two- branch parallelisms). One such three-way paral- lelism is given in Fig. 2(b), where both lexical repe- tition of the comparative adverb H (gèng) and syn- tactic parallelism help the sentence's three clauses to convey its message a spirited manner.
We obtained 409 of the original 544 essays from the authors. The essays are annotated for inter-paragraph parallelisms, intra-paragran paral- lelisms, and intra- sentenc parallelisms. To adapt them for our use, we first excluded inter-paragraph parallelisms, as these do not satisfy our criterion of locality. Then, for each intra-sentence parallelism that had been annotated, our annotators marked the locations of the parallel branches. For more details about this annotation process, see Appendix A. We call this version of the dataset PSE-I, or the version only having parallelisms inside paragraphs. We re- tained the tokenization generated by the Language Technology Platform (LTP) (Che et al., 2010).
To preprocess the ASP data, we remove all capitalization, following previous work Latin word embeddings (Bamman and Burns, 2020; Burns al., 2021). We also employ the LatinwordTokenizee from the CLTK library to tokenize our data (Johnson et al., 2021).
4.2 Paibi Student Essay Dataset
4.3 Data collection and preprocessing
The PSE dataset was created by Song et al. (2016) to improve automatic essay evaluation." 6 They col- lected a set of mock exam essays from Chinese senior high school students. Two annotators then marked parallel structure in these essays. Annota- tors labeled parallelism on the sentence rather than span level. On this task, they achieved an inter- annotator agreement of 0.71 according to the statistic (Carletta, 1996) over a set of essays.
Both datasets had annotations collected for them via the BRAT annotation tool (Stenetorp et al.. 2012). In applying the parallelism annotations to the data, both datasets excluded punctuation from parallelisms. This was done to reduce annotator noise brought about by the inconsistent member- ship of punctuation in parallelisms.
A fter preprocessing each dataset, we split both into training, validation, optimization, and test sets. To do so, attempted keep the ratio of tokens inside parallelisms to tokens outside parallelisms as even as possible across each set, assuring that the sets were distributed similarly. For more details on our data splitting, see Appendix E.
In Chinese, a H (päibī, parallelism) is some-
5The terms synckystic and chiastic used by analogy with the traditional rhetorical terms synchysis and chiasmus. 6The PSE dataset and its PSE-1 variant are freely available at https: www.githu...ooo.tt.eeeeeooe com/Mytholoooss//// -Essass.

---

Normalized Lexical Overlaps
Suppose that we have a document w, as above, and reference set G and a hypothesis H of parallelisms in w. An evaluation metric for RPD should check both that the branches in H have the same spans as those in G and that the branches are grouped into parallelisms in H as they are in G. Given these criteria, we follow prior work in coreference resolution (CR) to establish our met- rics. Coreference resolution involves detecting entity of interest and linking all mentions related to that entity together; in similar manner, our task links a set of spans together. As in the Constrained Aality-Aligmentt F-Measure metrics (Luo, 2005), we express our metrics as bipartite matching be- tween G and H. We use the Kuhn-Munkres (also known as "Hungarian") algorithm (Kuhn, 1955; Munkres, 1957) to maximally align parallelisms between G and H. To do this, we must define two functions: score and size.
1000
26%
800
2.3%
20%
Frequency
600
200
6%
3%
-1% 11% +1%
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9 1.0
Percentile Bins
Figure 3: Histogram normalized lexical overlap for all paired branches in the ASP dataset. Rounded per- centages indicating the amount of data represented in each bin (of size 1) are placed above each bar.
4.4 Dataset statistics
The function score(p1, (aee(pp.p2)) p2), where pt and p2 are parallelisms, determines how well pt matches P2, and the function size(p), where p is a par- allelism, bounds the score that can merit with another parallelism. These functions must sat- isfy size(p) 0 and 0 < score(p1, P2) min(size(pi), size(p2)).
To summarize our datasets, we provide a numerical and statistical overview in Tables and 2.
One factor we wanted to examine was the type of similarity that parallel branches exhibit; are their similarities surface-level or more linguisti- cally complex? To measure this, we compute the normalized lexical overlap (NLO) of all pairs of re- lated branches. Treating each branch as a multiset tokens, we apply the formula
With those in place, we can find the bipartite matching with maximum weight m,
Σ
m = max
score(pI.P2)
NLO(bi.b)) |b, O bzl
(PI.P2)
|bi U bal
where the maximization is over matchings M be- tween G and H. Having m, we define metrics the likeness of precision (P), recall (R), and the Fi-score (Fi) as follows:
where b1. b2 are related branches. This value is 1 if b, = b2, and it is 0 if b, n nb
In Fig. 3, we depict histogram of the ASP dataset's NLO across all related branch pairs (The PSE-I dataset's histogram is similar, but it peaks between 0.2 and 0.3 and has few pairs between and 0.1.) This histogram shows that the vast major- ity of related branch pairs are frequently lexically dissimilar from one another; most are below 0.6 NLO, showing that it is very common to have at least half the words in parallelism be different from one another. Thus, serted, parallelisms exploit many linguistic features produce an effect; turn, any method for RPD should try to capture these relationships.
m
P =
Σ P size(p)
m
R
Σ Epec size(p)
2
F 1/P+1/R
The simplest choice of size and score is
task definition as-
size(p)
if p1 P2
score(p1, P2) if p1 P2-
5 Evaluation Metrics
We call this the exact parallelism match (EPM) metric, where each parallelism in H must perfectly match one G to gain credit.
Next, we describe a general family of evaluation metrics for RPD and specify one instance of it.

---

As an example of this metric, consider the pas- sage given in Fig. 4. Suppose that the depicted parallelism-call it Pr-is the only parallelism in G. Furthermore, suppose that the model proposes two hypotheses, H = (P1, P2), where P2 is a paral- lelism that does not overlap with P1- Then
number of word-to-word hops get from b2 's start to bj 's end. A branch distance link is the (negative) number of branch-to-branch hops to get from b to b1- If P contains the interlocking parallelisms P1 = {(1,3), (7,9) and P2 = {(4, 6); (10, 12)), then the token distance between (1, 3) and (7,9) 9) is
4, while the branch distance is -2.
score(p1.P1) = score(p1. =0
It is important to form tag sequences that help the model to learn better, diverging decisions from the dominant majority class O. Therefore, in addition to adding links, we include three additional tag types, each of which is exhibited in Fig. 4.
From this, we derive the maximally-weighted matching, M = {(p), p1)). and = 1. Then
1 P
2 R
Fi --=3 3'
First, the M tag replaces O tags that occur for tokens that occur in the middle of consecutive branches in a parallelism. Lines 2 and of Fig. 4 ex- hibit this: four non-branch tags become M instead of O because they are between related branches. This shift obstructs a model from predicting single- branch parallelisms by providing a separate path- way to seek linked branches, as O - BC (where f stands for a link) can never occur in the data.
Appendix B introduces some alternative choices of score and size that give partial credit for imperfectly-matching hypothesized parallelisms.
6 Models
In this section, we propose some models for auto- matic RPD as a starting point for future research. Our models here treat RPD as a sequence label- ing problem, using several novel variants of BIO tagging (Ramshaw and Marcus, 1995).*
Second, the E tag replaces franch-ending I tags to indicate the end of a branch. Adding this tag removes the I - O transition. This change may courage model be more sensitive branch's endpoint; because most parallelisms are more than two tokens long, branches largely must transition from B to either I or E, and must eventually be selected before returning to O.
6.1 Tagging schemes
Branches are substrings, like named entities in NER, and parallelisms are sets of related substrings, like disfluencies in DD. Variants of BIO tagging have been successful for these sequence labeling tasks (Zayats et al., 2016; Reimers and Gurevych, 2017), so they are a natural choice for RPD as well. In this scheme, the first word of branch is tagged B; the other words of a branch are tagged I; words outside of branches are tagged O.
Third, the tag replaces tags in non-initial branches, where an initial branch is the first branch of a parallelism that occurs in a document. When paired with M tags, tags help to promote a se- quence of transitions which do not include O as a likely candidate until two branches have concluded. While the second example of Figure 4 displays this, Figure 5 shows this behavior across ASP's entire training set. Treating a sequence of tags as a linear chain, we tally all node transition pairs. When com- pared to the BIO chain of tags (left), we can see that --> O no longer exists in the BIOMJ chain (right). Only non-initial branch tokens (i.e., Bd or J) can transition to O. Through its supported transitions. the altered tagset actively enforces the constraint that a parallelism must have at least two branches.
However, BIO tagging does not indicate which branches are parallel with which. In each paral- lelism, for each branch b except the first, we aug- ment the tags with link, which is a number that indicates the previous branch with which is parallel. We propose two linking schemes.
Suppose we have consecutive parallel branches bi (11. and b2 = (f2, j2) with ji < iz. A to- ken distance link, akin to links used in DD (Hough and Schlangen, 2015), is ji 12; it is the (negative)
We accompany this paper with the lelism library. It implements each of our metrics and provides sample fornats for evaluating RPD. It available on PyPl and at https: / //gi thub. com/ caam/Mthhlla ogos/pyraaleliss.
With these three new tags, we form eight tagsets-BIO. BIO, BIOE, BIOJ, BIOM, BIOJE, BIOME, BIOMJ, and BIOMJE-an and apply each link type to them, forming sixteen tagging schemes. These tagging schemes constitute all parallelisms
*Our main modeling and results repository is available at https: / thub. com/My cooythossIIntrr RPRO. Our mod- els are implemented in PyTorch (Paszke et al., 2019) and are initially based on code by Robert Guthric (2017).
in each dataset: each level of nesting is an addi-

---

Latin English
quotidie facimus Every day say this and every day
quotidic dicimus hoc
quotidic fit
nobis
we do [it]
and every day [it] is done
BIO-Token
B
I
O
O
B-3
I
O
O
B-3
I J I E
BIOMI-Token
B
I
M
M
B-3
J
M
M
B-3
Q
BIOME-Branch
B
E
M
M
B-1
E
M
M
B-1
O
Figure 4: The example of Fig. from ASP annotated with various tagging schemes, For line 1, bolded words are part of the parallelism, and all contiguous units represent branches. For line 2, items in brackets indicate words not directly present (but implied) in the original text. For lines 3-5, "Token" and "Branch" refer to link types.
Tag Heatmap
Tag Heatmap
<START>
19
663
<START>
19
663
B
0
0
1.4K
0
0
0
1.4K
0
Preceding Word
Preceding Word
0
0
3.2K
1.4K
10
0
7.3K
10
3.2K
M
BX
1.8K
BA
1.8K
3
375
4.0K
1.4K
1.4K
0
1.8K
171K
682
0
0
1.4K
0
0
170K
682
<STOP>
<STOP>
<START>
BA
<STOP>
<START>
BΛ
<SOOP>
Following Word
Following Word
Figure 5: Two heatmaps tabulating tag transition frequencies across all (two) strata ASP's training set. Darker colors indicate larger values; text colors change for readability. The <START> and <STOP> tokens refer to section beginnings and ends. The left and right heatmaps are for the BIO and BIOMJ tagsets, respectively.
tional layer, or stratum (pl. strata), of tags.
(Vaswani et al., 2017) encoder layer. We tried three embedding options. The first op- tion was to learn embeddings from scratch. The second option was to use frozen word2vec embed- dings (Mikolov et al., 2013). We selected 300- dimensional embedding built from Latin data lem- matized by CLTK's LatinBackoffLeen emmat izer (Johnson et al., 2021) trained by Burns al. (2021). The third option was to employ frozen embeddings from a BERT namel. Latin BERT (Bam- man and Burns, 2020) for the ASP dataset and Chi- nese BERT with whole word masking (Cui et al., 2020, 2021) for the PSE-1 dataset.
6.2 Architecture
Inspired by prior success in sequence labeling for named entity recognition, we employ a conditional random field (CRF) output layer (Lafferty et al., 2001) in combination with various neural networks (Huang et al., 2015; Ma and Hovy, 2016; Lam- ple et al., 2016). Our general architecture (Fig. 6) proceeds in five steps. It embeds tokens (words subwords) as vectors; it blends token-level em- beddings into word-level embeddings; it encodes the sequence of embeddings to incorporate more contextual and task-specific information; maps encodings to tag space with a linear layer; it uses a CRF to compute a distribution over tag sequences. Each model examined in Scction 7.2 is vari- ation this paradigm. We tested total of six models. Following work in NER, we selected BiL.STM-CRF baseline (Huang al., 2015; Ma and Hovy, 2016; Lample et al., 2016). We also tried exchanging the BiLSTM for a Transformer
For both the Transformer encoder and the BERT embeddings, we applied WordPiece tokenization (Wu et al., 2016) by reusing the tokenizer previ- ously trained for Latin BERT (Bamman and Burns, 2020) with the tensor library (Vaswani et al., 2018). We employed operations (termed blending functions) to combine subword repre- sentations into word representations. The choice of blending function did not heavily impact our

---

trained for the ASP and PSE-I datasets, totaling 768 and 384 experiments, respectively.
CRF
Each trial's model trained for up to 200 epochs using Adam (Kingma and Ba, 2015) and gradient clipping with an L2 norm of (Pascanu et al., 2013). We stopped training early if the model's F1 score did not improve for 25 epochs the validation using the maximum branch-aware word overlap (MBAWO) metric (defined in Appendix B). Each trial was assessed on the optimization set. We de- noted the trial with the highest MBAWO score on this set per setting as that setting's s best run. We evaluated each best run on the test set.
Liincar
Encoder
(BiLSTM, Transformer, none)
Blender
(take-first, sum, mean)
Embedding (learned, word2vec, BERT)
Figure 6: Generic Encoder-CHe CRF architecture and per- level options examined. Shapes roughly indicate rela- tive changes in representation sizes.
7.2 Results
To compare the performance across models, we determined the best result for each model, as shown in Table 3. "Best" is defined as the experiment with the highest Fj score on the test set across all attempted settings. We also highlight a few factors which generally improved performance: the embeddings, encoders, and tagsets selected. 10
results, so we defer discussion of them to Ap- pendix D.3. For other models, we did not use subwords, and no blending function was needed.
To reduce the average sequence length seen by models, each sequence presented is section rather than whole document. For the ASP dataset, each sermon's division into sections was imposed by later editors of the texts. Meanwhile, for the PSE-I dataset, we split the data based on para- graph divisions. When sections remain longer than BERT'S 512-token limit, we split sections into 512- token chunks and concatenate the output sequences after the embedding layer. As noted in previous CR work (Joshi al., 2019), this approach is superior merging overlapping chunks.
In terms of embeddings, BERT embeddings vastly improved performance. As Table 5 depicts, BERT-based models exceeded every other non- BERT model by at least 0.2 F1 on the ASP dataset. We attribute their superiority the contextual rep- resentations produced by BERT. This is supported by the fact that both Burns al. 's embeddings and Latin BERT's embeddings were mainly trained on the same Internet Archive dataset (Bamman and Crane, 2011; Bamman and Smith, 2012).
We discuss other design choices in Appendix D.
In terms of tagging schemes, we wanted to know whether any schemes performed significantly better than the others. We applied the Friedman ranked- sums test (Friedman, 1937, 1940). Although this test is usually used compare multiple classifiers across multiple datasets (Demšar, 2006), we in- stead took results across single dataset (ASP) and sampling procedure (our hyperparameter search process) and considered each collection of best model F, scores as sample. Because the Fried- test is nonparametric, arising from the fact that model performance al- ready causes F1 scores differ heavily.
7
Experiments
In this section, we describe our experiments (Sec- tion 7.1 and present our results (Section 7.2).
7.1 Experimental design
For each dataset, we performed hyperparameter searches over several model architecture and tag- ging scheme combinations. We judged these as the elements which would likely primarily govern task performance. For the ASP dataset, we trained six possible architectures described in Section 6.2 with each of the sixteen tagging schemes described in Section 6. Meanwhile, for the PSE-I dataset, ran the three BERT-based architectures with each of the same sixteen tagging schemes. The hyperpa- rameters for each trial were chosen using random search (Bergstra and Bengio, 2012) described in Appendix F. Eight trials per configuration were
circumvent issues
With p < 0.05, we find the differences between samples to be significant (p < 0.001). We then
'We opted for MBAWO instead EPM because concerned that the stricter EPM metric would overly penalize incremental progress in capturing correct tokens.
10 an error analysis and a full catalogue of our results and best model hyperparameters, see Appendices G to

---

Dataset
Model Components
Tagging Scheme
Results
Embedding
Encoder
Tagset
Link Type
Precision
Recall
Learned (Word)
BiLSTM
BIO
Token
0.16
0.13
0.14
word2vec
BiLSTM
BIOMJE
Branch
0.24
0.12
0.16
Learned (Subword)
Transformer
BIOMU
Token
0,06
0.07
0.07
ASP
Latin BERT
BIOMJ
Token
0.51
0.33
0.40
Latin BERT
STL Transformer
BIOMJ
Token
0.47
0.34
0.39
Latin BERT
BIOME
Branch
0.44
0.31
0.37
Chinese BERT
BIOM
Branch
0.30
0.42
0.35
PSE-1
Chinese BERT
BiLSTM Transformer
BIOMU
Token
0.38
0.51
0.43
Chinese BERT
BIOME
Token
0.40
0.36
0.38
Table 3: Best test results, rounded the second decimal place, datasets according the EPM metric for each Encoder- CRF configuration. The best result in each column is written in boldface.
Tagset
Link
Avg. Rank (L)
Embedding Encoder
(ASP)
Fi (PSE-I)
Token
9.17
Learned (W) BiLSTM
0.08 0.03
BIO
Branch
9.92
word2vec BiLSTM
0.08 0.03
Token
9.83
Learned (SW) Transformer
0.02 0.02
BIOE
Branch
12.75
BERT
0.26 0.07
0.24 0.07
Token
11.83
BERT BiLSTM
0.32 + 0.04
0.36 0.04
BIOJ
Branch
11.08
BERT Transformer
0.28 + 0.05
0.30 0.05
Token
6.83
BIOM
Branch
6.50
Table 5: Best test set result averages over each Encoder- CRF configuration for EPM. "W" and "SW" stand for "word" and "subword", respectively. The BERT ver- sion used corresponds to each dataset's language.
Token
11.00
BIOJE
Branch
13.25
Token
9.17
BIOME
Branch
4.67
Token
3.33
BIOMJ
Branch
4.33
BIOMUE
Token
5,50
on average better than its Transformer (or encoder- less) variant. One possible reason for BiLSTMs' superiority may be that the subtask of predicting distance-based links benefits from LSTMs' ability to count (Weiss et al., 2018; Suzgun al., 2019b,a)
Branch
6.83
Table 4: Average rank of each tagging scheme's best model for all architectures across the ASP test set. The lowest (i.., best) result is bolded.
8 Conclusions and Future Work
used post-hoc Nemenyi test (Nemenyi, 1963) via the scikit-pon scikit-posthocs library (Terpilowski, 2019) to determine which tagging scheme pairs achieve significantly different results. With p < 0.05, only one pair of tagging schemes significantly differs: BIOMJ-TD and HOOJE-BD--- best and worst schemes, according to the average ranks of each scheme's Fj scores (as presented in Table 4). Given our low sample count, we suspect that further sam- plcs might show the superiority of certain schemes. With this in mind, we tentatively recommend any M-based tagging schemes, especially in combina- tion with either of the E or tags, for use.
In this paper, we introduced the task of rhetorical parallelism detection. We described two datasets, an evaluation framework, and several baseline mod- els for the task. Our baselines achieved fairly good performance, and we found that BERT embeddings. HRLSTM-basse encoder, and M-inclusive tag- ging scheme were valuable modeling components. We identify few directions for future work. The models described here have a closed tagset, so they cannot predict links for distances not seen the training data; modeling links in other ways might be more effective. Our models only predict flat parallelisms; approaches for nested NER (Finkel and Manning, 2009; Wang et al., 2020) may be viable direction for extending the Encoder-CRF paradigm toward this end. Finally, applying this work's methods to detect grammatical parallelism might enhance tasks like syntactic parsing, auto- mated essay evaluation, or disfluency detection.
In terms of encoders, BiLSTMs generally out- performed Transformers. Although non-BiLSTM models achieved peak performance in Table 3, the average performance by BiL.STM was consistently higher. Table depicts this regardless the type embedding used: each BiL.TM model performs

---

9 Limitations
that could either add to the available Chinese data based on an interest in parallelism detection as a sentence-level or clause-level subtask (Xu et al., 2021; Zhu et al., 2022) as well as work performing narrower, more restricted versions of word-level parallelism detection (Gawryjolck, 2009) or chias- mus detection (Dubremetz and Nivre, 2015, 201 8), in English. Future work can potentially expand RPD to the datasets provided by these works.
This work introduces a novel task in the form of rhetorical parallelism detection (RPD). Because it is novel, it is innately exploratory. Although it establishes approaches for building and organiz- ing datasets, for evaluating performance at various granularities, and for constructing models to cap- ture parallel structure, does not do all these per- feetly. Thus, it should not be the last word on the topic. In the following paragraphs, we highlight elements of this work which could be improved. First, this work puts forth two datasets for RPD: the Augustinian Sermon Parallelism (ASP) and the Paibi Student Essay (PSE) (Song et al., 2016) datasets. We annotated both datasets-the former from scratch and the latter on the basis of prior annotations. As is noted in Appendix C, our anno- tation scheme was not perfect.
Third, some modeling decisions were made suit the needs of our problem and provide base- lines without exploring the larger space of pos- sibilities. Namely, as described Section 6.2 and Appendix D.3, we use a blending function to combine subwords into words so that the appropri- ate number of tags would be produced. However, could have also eschewed the CRF altogether and attempted to follow BERT in selecting every word's s first tag as its representative tag (Devlin et al., 2019). Future work could further and more deeply investigate the search space of model archi- tectures to improve performance on this task.
For the ASP dataset, achieved 0.4124 EPM Fj score (although higher branch-based and
scores) between our two an- notators on bootstrapping experiments. Our scores indicate moderate agreement, but the meaningful disagreement implies that our guidelines could be sharpened.
Acknowledgements
We thank Wei Song for working with us not only provide data used in previous work but also to pre- pare it for public release. We thank Meng Jiang and Juliana, Joshua, and Joseph Chiang for refining the annotations on the PSE dataset. We thank John Lalor for his suggestions for our inter-annotator agreement study. Finally, we also thank our anony- mous reviewers and Brian DuSell, Darcey Riley, Ken Sible, Aarohi Srivastava, and Chihiro Taguchi for their comments and suggestions.
For the PSE-I dataset- version of the PSE dataset including our annotations-wee did not perform an inter-annotator agreement study. We did not consider it necessary because we were already overlapping with prior annota- tors' conclusions, but one could argue that the sentence- and span-level annotation tasks differ enough to warrant separate studies.
This research was supported in part by an FRSP grant from the University of Notre Dame.
Second, regarding the ASP dataset, we acknowl- edge that the use of Latin as foundation for this task limits its immediate applicability to modern NLP contexts. We believe that the ASP dataset is readily applicable to tasks such as authorship attribution and authorship authentication, which both have precedent in Latin (Forstall and Scheirer, 2010: Kestemont al., 2016; Kabala, 2020; Cor- bara et al., 2023) and frequently employ stylistic features in their approaches. Moreover, we be- lieve that ASP can aid distant reading approaches (Moretti, 2000; Jockers, 2013) in permitting the an- notation of large corpora with stylistic annotations.
References
Tosin Adewumi, Roshanak Vadoodi, Aparajita Tripa- thy. Konstantina Nikolaido, Foteini Liwicki, and Marcus Liwicki. 2022. Potential idiomatic expres- sion (PlE)-English: Corpus for classes idioms. Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 689-696, Mar- seille, France. European Language Resources Asso- ciation.
Mohammed Alliheedi. 2012. Multi-document Summa- rization System Using Rhetorical Information. Mas- ter's thesis, University of Waterloo, Waterloo, On- tario, Canada.
On the other hand, the inclusion of the PSE dataset for RPD provides modern language for which this task is available. Moreover, late into this work, we became aware of a vein of literature
Ron Artstein and Massimo Poesio. 2008. Survey ar- ticle: Inter-coder agreement for computational lin-

---

guistics. 596.
Computational Linguisties, 34(4):555-
Wanxiang Che, Zhenghua Li, and Ting Liu. 2010. LTP: A Chinese language technology platform. In Coling 2010: Demonstrations, pages 13-16, Beijing, China. Coling 2010 Organizing Committee.
Saint Augustine of Hippo. 1995. De doctrina Chris- tiana. Oxford Early Christian texts. Clarendon, Ox- ford.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes, 2018. The best of both worlds: Combining recent advances in neural machine translation. In Proceed- ings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pu- pers), pages 76-86, Melbourne, Australia. Associa- tion for Computational Linguistics.
Saint Augustine of Hippo. 2000a. Sermones: Part 1. volume 12 of Saint Augustine: Opera Ommia- Cor- pus Augustinianum Gissense. Electronic Edition. In- teLex Corp., Charlottesville, Virginia, U.S.A.
Saint Augustine of Hippo. 2000b. Sermones: Part 2, volume 13 of Saint Augustine: Opera Omnia Cor- pus Augustinianum Gissense. Electronic Edition. In- telexx Corp., Charlottesville, Virginia, U.S.A.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey F. Hin- ton. 2016. Layer normalization. arXiv: 10016666006000
Stanley F. Chen and Joshua Goodman. 1998. An em- pirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard University, Cambridge, Massachusetts.
David Bamman and Patrick J. Burns 2020. Latin BERT: A contextual language model for classical philology. ArXiv:2009. 10053.
Jacob Cohen. 1960. coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1))37-46.
David Bamman and Gregory Crane. 2011. Measuring historical word sense variation. In Proceedings of the 11th Annual International ACM/IEEE, Joint Con- ference Digital Libraries, JCDI. 11, pages 1-10, New York, NY, USA. Association for Computing Machinery.
Silvia Corbara, Alejandro Moreo, and Fabrizio Sebas- tiani. 2023. Syllabic quantity patterns as rhythmic features for Latin authorship attribution. Journal of the Association for Information Science and Technol- ogy, 74(1):128-144
David Bamman and David Smith. 2012. Extracting two thousand years of Latin from a million book li- brary. Journal on Computing and Cultural Heritage, 5(1).
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin. Shi- jin Wang, and Guoping Hu. 2020. Revisiting pre- trained models for Chinese natural language process- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing: Findings, pages 657-668, Online. Association for Computational Linguistics.
Beata Beigman Klebanov and Nitin Madnani. 2020. Automated evaluation of writing 50 years and counting. In Proceedings of the 58th Annual Meet- ing the Association for Computational Linguisties, the
pages 7796-7810, Online, Association for Computa- tional Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021, Pre-training with whole word masking for Chinese BERT. IEEF Transactions on Audio, Speech and Language Processing.
James Bergstra and Yoshua Bengio. 2012. Random scarch for dyper-parameter optimization. J. Mach. Learn. Res., 13:281-305.
Patrick J. Burns, James A. Brofos, Kyle Li, Pramit Chaudhuri, and Joseph P. Dexter. 2021. Profiling of intertextuality in Latin literature using word em- beddings. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4900 4907, Online. Association for Computational Linguistics.
Yange Dai, Wei Song, Xianjun Liu, Lizhen Liu, and Xinlei Zhao. 2018. Recognition of parallelism sen- tence based on recurrent neural network. In 2018 IEEE 9th International Conference on Software En- gineering and Service Science (ICSESS), pages 148- 151.
Janez Demšar. 2006. Statistical comparisons of clas- sifiers over multiple data sets. Journal of Machine Learning Research, 7(1::1-30.
Jean Carletta. 1996. Assessing agreement on classifica- tion tasks: The kappa statistic. Computational Lin- guistics, 22(2):249-254
Jacob Devlin, Ming- Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171 Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Noe Casas, Marta R. Costa-jussà, and José A. R. Fonol- losa. 2020. Combining subword representations into word-level representations in the Transformer archi- tecture. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: Student Research Workshop. pages 66-71, Online. Association for Computational Linguistics.

---

Marie Dubremetz and Joakim Nivre. 2015. Rhetori- cal figure detection: The case of chiasmus. In Pro- ceedings of the Fourth Workshop on Computational Linguistics for Literature, pages 23-31, Denver, Col- orado, USA. Association for Computational Linguis- tics.
similarity degree. 11th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 281 281-288, Trento, Italy. Associa- tion for Computational Linguisties.
Robert Guthrie. 2017. DeepLeaminggorNN.Phre torch: An IPython Notebook tutorial on deep learn- ing for natural language processing, including struc- ture prediction.
Marie Dubremetz and Joakim Nivre. 2018. Rhetorical figure detection: Chiasmus, epanaphora, epiphora. Frontiers in Digital Humanities, 5.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian er- for linear units (GELUs). arXiv:1606.0841
Jenny Rose Finkel and Christopher D. Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Nat- ural Language Processing. pages 141-150, Singa- pore. Association for Computational Linguistics.
Julian Hough and David Schlangen. 2015. Recurrent neural networks for incremental disfluency detec- tion. In Proc. Interspeech 2015. pages 849-853.
George Hripesak and Adam Rothschild. 2005. Agreement, the f-measure, and reliability in infor- mation retrieval. Journal of the American Medical Informatics Association : JAMIA, 12(3):29--988.
Christopher Forstall and Walter Scheirer. 2010. Fea- tures from frequency: Authorship and stylistic anal- ysis using repetitive sound. Journal of the Chicago Colloquium on Digital Humanities and Computer Science, 1(2).
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidi- rectional LSTM-CRF models for sequence tagging. CoRR, abs/1 abs/1508.0115 508.01991.
Milton Fiienmmnn 1937. The use of ranks to avoid the
assumption of normality implicit in the analysis of variance. Journal of the American Statistical Associ- ation, 32(2000)675-701.
Matthew Lee Jockers. 2013. Macroanalysis: Digital Methods and Literary History. Topics in the Digital Humanities. University of Illinois Press, Urbana.
Milton Friedman. 1940. A comparison of alternative tests of significance for the problem of rankings. The Annals of Mathematical Statistics, 11(1):86-99 -2.
Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd Cook, Clément Besnier, and William J. B. Mattingly. 2021. The Classical Language Toolkit: An NLP framework for pre-modern languages. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th Interna- tional Joint Conference on Natural Language Pro- cessing: System Demonstrations, pages 20-29, On- line. Association for Computational Linguisties.
Akansha Gautam and Koteswar Rao Jerripothula. 2020. SGG: Spinbot, Grammarly and GloVe based fake news detection. In 2020 IEEE Sixth International Conference on Multimedia Big Data (BigMM), pages 174-182.
Jakub Jan Gawryjolek. 2009. Automated annotation and visualization of rhetorical figures. Master's thesis, University of Waterloo, Waterloo, Ontario, Canada.
Mandar Joshi, Omer Levy, Luke Zettlemoyer. and Daniel Weld. 2019. BERT for coreference reso- lution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP), pages 5803-5808, Hong Kong, China. Association for Computational Linguistics.
Kallirroi Georgila. 2009, Using integer linear program- ming for detecting speech disfluencies. In Proceed- ings of Numan Language Technologies: The 2009 Annual Conference of the North American Chap- ter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 109-112, Boulder, Colorado. Association for Computational Linguistics.
Jakub Kabala. 2020. Computational authorship attri- bution in medieval Latin corpora: The case of the Monk of Lido (ca. 1101-08) and Gallus Anonymous (ca. 1113-17). Language Resources and Evaluation, (4<1):25-56..
Jiefu Gong, Xiao Hu, Wei Song, Ruiji Fu, Zhichao Sheng, Bo Zhu, Shijin Wang, and Ting Liu. 2021. IFlyBA: A Chinese essay assessment system with automated rating, review generation, and recommen- dation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Nat- ural Language Processing: System Demonstrations, pages 240-248, Online. Association for Computa- tional Linguistics.
Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. Proceed- ings of the Twenty-Eighth International Joint Con- ference Artificial Intelligence, IJCAI-19, pages 1300-6308 International Joint Conferences on Ar- tificial Intelligence Organization.
Mike Kestemont, Justin Stover, Moshe Koppel, Folgert Karsdorp, and Walter Daclemans. 2016. Authenti- cating the writings of Julius Caesar. Expert Systems with Applications, 63:86-96.
Marie Guégan and Nicolas Hernandez. 2006. Recog- nizing textual parallelisms with edit distance and

---

Diederik P. Kingma and Jimmy Ba. 2015, Adam: method for stochastic optimization. In 3rd Inter- national Conference ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1 5(1):32-33
Learning Representations.
Peter Bjor Nemenyi. 1963. Distribution- Multi- ple Comparisons. Ph.D. thesis, Princeton Univer- sity, Princeton, New Jersey, USA.
Klaus Krippendorff. 2019. Reliability, fourth edi- tion, chapter 12. SAGE Publications, Inc., Thousand Oaks.
Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochas- tic language modelling. Computer Speech Lan- guage, 8(1):1-38.
H. W. Kuhn. 1953. The Hungarian method for the as- signment problem. Naval Research Logisties Quar- lerly. 2(1-2):83-97
Toan Q. Nguyen and Julian Salazar. 2019, Transform- ers without tears: Improving the normalization of self-attention. In Proceedings of the 16th Interna- tional Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguis- tics.
John Lafferty. Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Prob- abilistic models for segmenting and labeling se- quence data. Departmental Papers (CIS).
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 260-270, San Diego, California. Association for Computational I Linguistics.
Mari Ostendorf and Sangyun Hahn. 2013. A sequential repetition model for improved disfluency detection. In Proc. Interspeech 2013, pages 2624-2628.
Ellis B. Page. 1966. The imminence of ... grad- ing essays by computer. The Phi Delta Kappan, 47(5)2238223..
J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):159-174
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1310-1318, Atlanta, Georgia, USA. PMLR.
Guanghui Liu, Lijun Fu, Bo Yu, and Li Cui. 2022. Au- tomatic recognition of parallel sentence based on sentences- innercss-inttractionn CNN and its application. In 2022 7th International Conference on Computer and Communication Systems (ICCCS), pages 245-250.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killcen, Zeming Lin, Natalia Gimelshein. Luca Antiga, Alban Desmaison. Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Py- Torch: An imperative style, nugh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 32, pages 8024-8035. Curran Asso- ciates, Inc.
Xiaoqiang Luo. 2005. On coreference resolution per- formance metrics. In Proceedings of Human Lan- guage Technology Conference and Conference on Empirical Methods in Natural Language Processing. pages 25-32, Vancouver, British Columbia, Canada. Association for Computational Linguisties.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNS-- CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1064-1074, Berlin, Ger- many. Association for Computational Linguistics.
Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using transformation-ue baseed learning. In Third Workshop on Very Large Corpora, pages 82-94.
Marie W. Meteer and Ann A. Taylor. 1995. Dysfluency annotation stylebook for the Switchboard corpus.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word represen- tations in vector space. In Ist International Con- ference Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedngs.
for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages #38-348. Copenhagen, Denmark. Association for Computational Linguistics.
Elizabeth Shriberg, 1994. Preliminaries to Theory of Speech Disfluencies. Ph.D. thesis, University of California, Berkeley, Berkeley, California, USA.
Franco Moretti. 2000. Conjectures on world literature. New Left Review, 1(1):54-68.

---

Wei Song, Tong Liu, Ruij Fu, Lizhen Liu, Hanshi Wang, and Ting Liu. 2016. Learning to identify sentence parallelism in student essays. In Proceed- ings of COLING 2016, the 26th International Con- ference on Computational Linguistics: Technical Pa- pers, pages 794-803, Osaka, Japan. The COLING 2016 Organizing Committee.
Robert Kern, Eric Larson, C J Carey, IIhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, lan Hen- riksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SeiPy 1.0: Fundamental algorithms for scien- tific computing in Python. Nature Methods, 17:261- 272.
Fábio Souza, Rodrigo Frassetto Nogueira, and Roberto de Alencar Lotufo. 2019, Portuguese named entity recognition using BERT-CRF. arXiv:1909.1644
Jue Wang, Lidan Shou, Ke Chen, and Gang Chen. 2020. Pyramid: A layered model for nested named en- tity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5918 15185592, Online. Association for Computational Linguistics.
Pontus Stenetorp, Sampo Pyysalo, Goran Topić. Tomoko Ohta, Sophia Ananiadou, and unnichii Tsu- jii. 2012. BRAT: A web-based tool for N-P-sssisted text annotation. In Proceedings of the Demonstra- sions Session at EACL 2012, Avignon, France. Asso- ciation for Computational Linguistics.
Qiang Wang. Bei Li. Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong. and Lidia S. Chao. 2019. Learning deep Transformer models for ma- chine translation. In Proceedings the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1810-1822, Florence, Italy, Associa- tion for Computational Linguistics.
Gaius Suetonius Tranquillus. 1993. De Vita Caesarum Libri VIII, volume of Opera. B. G. Teubner.
Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019a. LSTM networks can perform dynamic counting. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 54, Florence. Associa- tion for Computational Linguistics.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite pre- cision RNNs for language recognition. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Pa- pers), pages 740-745, Melbourne, Australia. Asso- ciation for Computational Linguistics.
Mirac Suzgun, Yonatan Belinkov, and Stuart M. Shieber. 2019b. On evaluating the generalization of LSTM models in formal languages. In Proceedings of the Society for Computation Linguisties (SCiL) 2019, pages 277-286.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey. Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rud- nick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv: 1 1609.0814 184.
Chenhao Tan, Hao Peng, and Noah A. Smith. 2018 "You are no Jack Kennedy": On media selection of highlights from presidential debates. Proceedings of the 2018 World Wide Web Conference, WWW '18, pages 945 955, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committec.
Maksim Terpilowski. 2019. scikit- posthocs: Pairwise multiple comparison tests in Python. The Journal of Open Source Software, 1636):1169.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Fran- cois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018 Tensor2Tensor for neural machine translation. CoRR, abs/ 100/10003.746...
Guowei Xu, Wenbiao Ding, Weiping Fu, Zhongqin Wu, and Zitao Liu. 2021. Robust learning for text classi- fication with multi-source noise simulation and hard example mining. In Machine Learning and Knowl- edge Discovery Dutahases. Applied Data Science Track, pages 285-301, Cham. Springer International Publishing.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Ilaia Polosukhin, 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Vicky Zayats and Mari Ostendorf. 2019, Giving atten- tion to the unexpected: Using prosody innovations in disfluency detection. In Proceedings of the 2019 Conference of the North Americun Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 86 95,95, Minneapolis, Minnesota. As- sociation for Computational Linguistics
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Eygeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones,
Vicky Zayats, Mari Ostendorf. and Hannaneh Ha- jishirzi. 2016. Disfluency detection using a bidi-

---

rectional LSTM. In Proc. Interspeech 2016, pages 2523-2527.
For the ASP dataset, we developed an annotation scheme consisting of BRAT'S entities and relation- ships. An entity could be a ParallelArm or one of ChiasmA and ChiasmB. A relationship could be Parallel, linking two ParallelArm entities. This relationship indicates synchystic parallel struc- ture. A relationship could also be Chiasm, linking ChiasmA and ChiasmB and labeling chiastic structure. Our two annotators, both for the initial data collection and the inter- anter-annotator agreement study (see Appendix C), could nest annotations; however, they could not overlap them.
Victoria Zayats, Mari Ostendorf, and Hannaneh Ha- jishirzi. 2014, Multi-domain disfluency and repair detection. Proc. Interppeech 2014, pages 2907- 2911.
Dawei Zhu, Qiusi Zhan, Zhejian Zhou, Yifan Song. Jiebin Zhang, and Sujian Li. 2022. ConFiguRe: Ex- ploring discourse-level Chinese figures of speech. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3374-3385, Gyeongju, Republic of Korea. International Com- mittee on Computational Linguisties.
Meanwhile, for the PSE-I dataset, we made a few changes. Because we were mostly interested in annotating sentences that were already marked as parallel in the dataset, did away with the distinc- tion between synchystic and chiastic parallelisms. Instead, all branches of parallelism were tagged as Branch entities and connected with Parallel relationships. Annotators were also allowed to use a dummy entity type to tag sentences if they were deemed not to be parallel, but this was never done. The five annotators were told to focus on marked sentences in BRAT, but they were allowed to con- nect these sentences unmarked sentences in the same paragraph if they noticed a parallelism.
A Annotation Procedures
Section 2, we described two major guidelines which directed our study parallel structures. Here, we list the specific criteria which were used by our annotators to locate parallelisms. Moreover, we describe our annotation scheme in more detail. We begin by enumerating general criteria in Ap- pendix A. before discussing specific applications of these criteria in Appendix A.2.
A.1 General Guidelines
In general, branches of parallelisms were tagged from the first identical (or similar) word to the last identical (or similar) word; as such, each branch was maximally represented. Branches were de- tected, paired, and combined based upon whether they exhibited at least two of these criteria:
B Additional Metries
In Section 5, we constructed a framework for defin- ing parallelism-related metrics. We defined one such metric for exact parallelism matches between parallelisms. However, the score and size func- tions allow for finer granularity. Below, we de- fine three additional metrics which examine paral- lelisms in terms of their branches, their words-per branch, and their words altogether.
They contained identical number of words (or were within approximately two words of one another).
They had identical syntactic structure.
First, we describe the maximum parallel branch match (MPBM) metric. MPBM focuses on branches. It retains sense of parallel structure by requiring that at least two branches are shared between the hypothesis and reference parallelisms to produce any score. The functions of MPBM are:
They had two or more pairs of words of an identical grammatical form in identical order.
They had two or more pairs of words that are lexically identical, are synonyms, are cog- nates, or are antonyms.
They had least two words that have pho- netically similar ending (e.g., rhyme).
size(p) =Ip score(p1, P2) has hasParrBraachh)).p))) (p1 Pzl
They were short distance from one another (i.e., they had few intervening words). The ac- ceptable margin was again around two words.
where
hasParBranch(g,r)) [[lg O 2]
A.2 Specific Applications
As mentioned earlier, we used the BRAT annotation tool (Stenetorp et al., 2012) to form our datasets.
Next, we proceed to metrics which examine parallelisms terms their words (which we

---

use interchangeably with tokens below). Let
C Inter-Annotator Agreement Study
P be a parallelism. We define words(p) Uu.nepll..... as the set of all token positions in those branches; this formulation dissolves dis- tinctions between branches for a parallelism. We also define branchedWords(p)) = !fi.....j (i E p} as the set of all token positions in branches divided up into set per branch. This formulation preserves branch-level distinctions.
To provide some measure of the quality of our data, we performed an inter-annotator agreement study on the ASP dataset. Due to constraints on time and expertise, we recruited our third author, an experi- enced classicist and Latin teacher, to revisit eight sermons using our annotation scheme. To address the limited annotated data, we computed boot- strapped estimate for agreement across the whole dataset. After obtaining an initial computation of inter-annotator agreement, we ran 1,000 trials obtain a 95% confidence interval for our agreement scores. The number of items sampled was equiva- lent to the total number of matchings made in the initial computation.
The first word-level metric is the maximumm branch-aware word overlap (MBAWO) metric. The objective of this metric is similar to MPBM: if at least two distinct branches are matched in a parallelism, then such a matching should obtain some score. However, this metric is less strict than MPBM in that only words from two distinct branches need to match. In short, if at least two branches in hypothesis parallelism have words that match with at least two branches in a reference parallelism, then the match merits nonzero score based on the number of tokens matched.
Because the space of possible spans and links be- tween spans is prohibitively large, we could not use more traditional inter-annotator agreement metries such as Cohen's x (Cohen, 1960) or Krippendorff's o (Krippendorff, 2019). Instead, we used our own metrics as defined in Section 5 and Appendix B. However, following the conclusions of Hripesak and Rothschild that the F1 measure converges the number of negative pairs grows increas- ingly large, we can treat our F₁ scores in a similar manner to chance-corrected agreement statistics.
To compute this metric, we define (W(1) P2) branchedWords(pp P1) x branchedWords (p2); that is, W pairs all branches of one parallelism with all branches of another parallelism in terms of their word indices. Then define score and size as:
spze(p) |words()))
As final preprocessing step before matching annotations, we corrected them with two strategies to reduce noise. We felt that these strategies were warranted after discussion unearthed two vague points in our guidelines. We first describe our is- sues with the guidelines below, and then we clarify the annotation cleaning done afterward.
score(p1,
max W (PiPP)))
hasParWord(W)
Σ (WW
1p 2nnp p2l
where
hasParWord W) =I[ ][wordMatches W) 2]
1. Conjunctions: It was unclear whether con- junctions meaningfully contributed to par- allelism. For some parallelisms, they may have been syntactically necessary-bu did that mean that they should be part of the paral- lelism? We decided that conjunctions should be included if every branch of the parallelism has one; otherwise. they should be omitted. This allows polysyndeton to be recognized while avoiding the requisite but incidental con- nections created by some syntactie structure.
wordMatches(W)
Σ
Tlp p2l 0]
(WW
MBAWO calculates an internal maximally- weighted bipartite matching for token overlap; how- ever, it only values those matches which correspond to parallel structure in the reference data. It en- forces that through the function wordMatches.
Finally, we define the maximun word overlap (MWO) metric. MWO is at the opposite extreme of EPM; does necessarily control for the presence of parallel structure. Even so, its ability to measure model's capacity to find any parallel words may be useful. Its functions are as follows:
2. Interlocking Paralletisms: Second, it was difficult determine the best way parti- tion parallel structure into branches. This was especially the case with longer parallelisms that involved multiple clauses. Take Fig. 7
size(p) = |words(p))
score(p). P2) = [words(pi) O words(p2))

---

Scoring Object
Actual Fí
Sample Mean and Deviation
Confidence Interval
Parallelism
0,4615
0.4124 = 0.0303
[0.4088, 0.4160]
Branch
0,5084
0,5080 = 0.0310
[0.5043, 0.5117|
Branched Words
0.5906
0.5973 0.0322
[0.5935, 0.6012] [0.6092, 0.6167]
Words
0.6136
0.6130 0.0312
Table 6: Aggregated actual and sample inter-annotator agreement scores. All values are rounded to the fourth decimal place and range between 0 and 1. Higher is better for the actual Fj and the sample mean; lower is better for the standard deviation. Smaller confidence intervals indicate greater stability of our estimate.
Latin inanis auro,
plenus deo; [is] full of God; [he who] lacks every transient supply. [is] full of the Lord's favor.
inanis omni transitoria facultate, plenus sud domini uoluntate.
English [he who] lacks gold,
Interpretation
Stratum
Pr
P1,2
1
P1,4
Stratum
Interpretation
Stratum Stratum 2
Pi,1
P1,2
P2.1
P2.2
P3.1
P3.2
Interpretation
Stratum Stratum
Pr.1
P1,2
Figure 7: Selection of possible interpretations of parallel structure of a passage from Augustine's 's Sermones (Ser- mon 177, Section 4). Clauses are aligned for the Latin and English the top the table. Each subsequent section of the table relays an interpretation of parallel structure; each Pr. indicates branch j of parallelism i.
as an example. Without clear rules, there are multiple ways to produce parallel structure:
ing from a maximal interpretation if the paral- lelisms intentionally use different features.
To address these issues in original annota- tions, we automatically altered the annotations two bases. First, if all branches of a parallelism were preceded by a conjunction, then conjunctions were included in all branches; otherwise, no con- junctions were permitted, and any present were re- moved from annotations. Second, we collapsed in- terlocking parallelisms into larger branches if they were adjacent. If these larger branches were part of nested parallelism, and such branches produced parallelism on an earlier stratum, we removed these larger branches entirely. Although these alterations did cause gains annotation agreement, they were all between roughly 0.02 to 0.03 Fj-
Interpretation considers each clause its own branch.
Interpretation 2 considers each pair of clauses, which juxtapose inanis ("with- out, lacking, empty") and plenus ("filled, full"), single parallelism; moreover, it also recognizes the similar syntactic relationships in the individual clauses.
Interpretation only annotates the first stratum of the prior interpretation.
Ultimately, we ruled that the third interpre- tation was best. It maximally captured the parallel structure in these clauses. Although it is true that each pair of clauses has its own re- lationships (e.g., same number of words, same syntactic structure), these relationships also contribute to the larger structure they are con- tained within, as each inanis and plenus takes noun of the same case (ablative) as its object. Thus, these connections are still captured in the third interpretation. We only advise differ-
We present our inter-annotator agreement results in Table 6. Across the board, these scores fall below traditional standards (i.e., above 0.67 at the very least) for good chance-coneetedd reliability in con- tent analysis (Carletta, 1996). However, as noted by Artstein and Poesio, acceptable inter-annotator agreement values remain disputed; the earlier work by Landis and Koch denotes the range between 0.4 and 0.6 as moderate agreement and the range

---

between 0.6 and 0.8 as substantial agreement. Regardless of the exact standard that we follow, we do believe that the disagreement displayed by this inter-annotator agreement study is meaningful and in part due to some vagueness in our original guidelines. We have already updated our guidelines provide greater specifications for future annota- tors, and we are currently working on improving our data with these guidelines. We hope to release an enhanced version of our dataset in future work.
The Kneser-Ney replacement probability for the ASP dataset, given the current data splits, is ap- proximately 0.6529.
D.2 Transformer Modifications
We apply the Transformer encoder in some of our models with a slight change from the usual imple- mentation. In line with previous work which sees minor improvements in model performance (Chen et al., 2018; 8; Wang et al., 2019; Nguyen and Salazar, 2019), we swap the traditional order of layer nor- malization and skip connections. Applying the notation of Nguyen and Salazar, use PreNorm in the incorporation of the residual connection:
D Model Idiosyncrasies
In Section 6.2, we go over our general Encoder- CRF model architecture, and describe some of the major components of models. Due to space considerations, some details were omitted from that discussion. We discuss such details here.
X/+1 = LayerNorm(xx + F((x))
where / is the layer's index, F represents the layer itself, and LayerNorm represents the layer normal- ization operation (Ba al., 2016).
D.1 Training <UNK> in BILSTM-CRE's
In Encoder-CRF models which apply word-level vocabulary, one necessary issue concerns the han- dling of unknown words. Usually, such models maintain an <UNK> token for such cases. However, if the <UNK> token not seen during training, the model will not know how to use it and may confuse it with other tokens seen during training.
D.3 Subword Blending
As previously mentioned, we WordPiece to- kenization with our models which either have a Transformer encoder or Latin BERT embeddings. Using subword- tokens presents an issue for a word-level tagging problem: in what way should the model process subwords to generate word-level tags? The original BERT paper, which tackles NER, does not use CRF and instead lets the tag classification of each word's first subword repre- sent the whole word (Devlin et al., 2019).
Various strategies have been employed to ad- dress this problem. Some decide that all words seen less than & times, where k is some preset value, be- come <UNK> tokens (Reimers and Gurevych, 2017). Others select singlctons-word words which only appear once in the training data-and alter them to become <UNK> during training with predetermined fixed probability (Lample al., 2016). We follow the latter strategy, but we augment so that we can adapt the probability to the dataset at hand and can avoid choosing an arbitrary value.
We take inspiration from BERT's methods and others' approaches to handling subword-to-word relationships (Souza et al., 2019; Casas et al., 2020) by putting blending layer into the architecture. As mentioned in Section 6.2, the blending layer's ob- jective is combine subword-level encodings into word-level encodings. We define three variations:
We propose singleton replacement probability analogous to Kneser-Ney smoothing (Ney et al., 1994). We compute the frequency of each type in the dataset. Then, tally the number of types which occur exactly once, n1, and the number of types which occur exactly twice, n2. Here, we use notation from previous work where n represents the number of word types and the subscripts repre- sent a conditional frequency on that count (Chen and Goodman, 1998). Finally, for the singleton replacement probability Pr, the equation:
Take-First: In this approach, BERT and select the first subword of each word as its representative. (Note that, when convenient, we abbreviate this variant as "tf".)
mimic
Mean: In this approach, we take the mean of all subword encodings per word. This is akin to subword-aware mean pooling layer.
Sum: In this approach, we sum all subword encodings for a word to represent it. This rep- resentation may allow for a model to recog- nize the accumulation of multiple subwords.
B1
Py + 2nz

---

Dataset
Split
Optimal Ratio
Inner Tag Ratio
Outer Tag Ratio
Training
0.7
0.7021
0.7020
Validation
0.1
0.0982
0,1008
ASP
Optimization
0.1
0.0946
0,0980
Test
0.1
0.1051
0.0992
Training
0.7
0.6947
0.7007
Validation
0.1
0.1026
0.0998
PSE-1
Optimization
0.1
0.1005
0.0997
Test
0.1
0.1022
0.0998
Table 7: Data split ratio results. All ratios are rounded the fourth decimal place.
Dataset
Compared Splits
Inner p- Value
Inner Statistic
Outer p-Value
Outer Statistic
Training
Validation
0.5797
0.5722
0.6284
0.4990
Training
Optimization
0.8347
-0.2162
0.7557
-0.3236
Training
Test
0.7210
0.3668
0,4727
0.7405
ASP
Validation
Optimization
0.5705
-0.5816
0.5631
-0,5942
Validation
Test
0.8557
-0.1848
0,9601
0.0509
Optimization
Test
0.6751
0.4290
0.4980
0,7038
Training
Validation
0.7476
-0.3236
0.6212
-0.4972
Training
Optimization
0.9533
-0.0588
0.8240
0.2235
Training
Test
0.9882
-0.0148
0.2588
1.1381
PSE-1
Validation
Optimization
0.8291
0.2166
0.5779
0.5588
Validation
Test
0.7987
0.2560
0.2526
1.1540
Optimization
Test
0.9720
0.0353
0.5651
0.5780
Table 8: Data split Welch's r- T-Tsst results. All p-values and statistics are rounded to the fourth decimal place.
Dataset
Split
Parallelisms
Branches
Branched Tokens
Tokens
Training
1448
3264
13833
94740
Validation
1935
13580
ASP
Optimization
191
1863
13196
Test
485
2070
13440
Total
2062
4651
19701
134956
Training
541
1478
17736
168848
Validation
87
236
2618
24152
PSE-I
Optimization
77
219
2565
24064
Test
81
220
2610
24139
Total
786
2153
25529
241203
Table 9: Data split results per relevant object count. Note that "branched tokens" refers to all tokens in branches not the number tokens in general.
E Data Splitting Approach
counts of its outer and inner tags." !1 A tag is an outer tag if it is O M; otherwise, it is inner tag. Then, we attempted to place file in each split. If placing that file in a split minimized the mean- squared error (MSE) among the splits, then the file was placed in that split. To be specific, we averaged the MSE for inner and outer tags to compute the final MSE. We repeated this process for every file.
As noted in Section 4, parallelisms are not evenly distributed over sections. Sections are also not evenly distributed over documents. Because we performed evaluation in terms of documents, we wanted to apply a simple heuristic to guarantce that the data splits would fairly distribute tags.
As further heuristic, we sorted the files after their inner and outer tag counts were computed. We did so by the number inner tags from maximum
To do this, used a straightforward approach optimize split creation. To apply the approach, we supplied with set of ratios which govern the quantity and relative size the splits. We began by taking each file in the dataset and computing
I1 As in the main paper, we created strata of tags for cach datasct equal to the maximum nesting depth of a branch.

---

minimum, breaking ties by the number of outer tags. This was in line with our objective balance out the parallelisms and branches seen across splits: the ordering allowed for files which have a greater effect on MSE to be placed first, thereby letting files with less tags smooth out error gradually.
F.1 Selected Hyperparameters
Table 11 provides information about what hyper- parameters are varied per model. Due to space considerations, use number of abbreviations; these are supplied in Table 10.
To verify our procedure, we provide the result- ing mean-squared error for our splits. We also used Welch's s r-test for this purpose. For each pair of splits created, we ran this test twice with per-file counts of the inner and outer tags. In this case, we do not want the test to report significance; rather, in showing that the difference between the distri- butions is insignificant (e.g., p 2 0.05), we showing that the splits are divided up such that they could have been drawn from the same distribu- tion. We use the SciPy implementation of Welch's (-test for this purpose (Virtanen et al., 2020).
F.2 Designated Hyperparameter Spaces
Table 12 lists the hyperparameter spaces which we used during the random search process. While some of the above hyperparameters are relatively self-explanatory in their functionality, we provide explanation for those which may be less clear otherwise warrant further explanation:
Activation Function: The PyTTrch (Paszke et al., 2019) implementation of the Trans- former (Vaswaní et al., 2017) provides inte- grations for both ReLU and GeLU (Hendrycks and Gimpel, 2016) as activation functions. We uphold that support in our modifications (as presented in Appendix D.2) to see whether they have any major effects on our results.
F Hyperparameter Search Spaces
For our hyperparameter search experiments in Sec- tion 7.2, we performed random hyperparameter search (Bergstra and Bengio, 2012) of eight trials per setting. However, because we are proposing a new task, we did not have prior literature on suc- cessful hyperparameters which we could draw upon in order to define hyperparameter spaces. Thus, to perform these experiments, we created our own.
Dimensionalities: For all embedding and hid- den dimensionality spaces, let the search space be in powers of and every value averaged between adjacent powers of two. We used this approach to explore a decent por- tion of the search space without leaving big gaps between larger powers. Furthermore, we differed the spaces for BiLSTMs and Trans- formers to adapt more to precedent while still allowing some exploration. In particu- lar, took inspiration from some hyperpa- rameter LSTM searches in NER (Reimers and Gurevych, 2017) and the general application of the Transformer (Vaswani et al., 2017).
These hyperparameter search experiments were governed by three main elements:
1.
The set of hyperparameters which are selected for variation. Although many hyperparame- ters could be varied, only some were chosen to direct the search toward hyperparameters which were suspected to be meaningful.
Depth: In initial trials, we found that high depth values prevented the models from learn- ing meaningful information. Thus, instead of setting maximum of 6 encoder layers per the initial Transformer implementation (Vaswani al., 2017), we lowered it to 4.
2. The space itself from which hyperparameters were drawn, as this determines all possible values which the hyperparameter can take.
3. The set of constraints which are imposed upon the random trials. Naturally, we want to pre- vent any trial from being duplicated; however, there other inductive biases which may guide our search.
Learning Rates: In the NER literature, a va- riety of learning rates have been used (along- side variety of optimizers). Values from 1 (Huang al., 2015) to .015 (Ma and Hovy, 2016) .01 (Lample al., 2016) vari- cty of manually-tuned others (Reimers and Gurevych, 2017) have been proposed. Due to the differences in our task, however, we examine a wider range on the basis those
In the subsections below, we expound upon each these elements and provide tables which summa- rize pertinent information regarding them.

---

Abbreviation
Full Hyperparameter Name
AF
Activation Function
BF
Blending Function
HS
Hidden Size
IS
Input Size
LR
Learning Rate
Lemma
Lemmatization
Table 10: List of all hyperparameter abbreviations.
Embedding
Encoder
Varied Hyperparameters
Learned (Word)
BiLSTM
Depth, HS, IS, LR, Lemma
word2vec
BiLSTM
Depth, HS, LR AF, BF, Depth, Heads, HS, IS, LR BF, LR
Learned (Subword)
Transformer
(Chinese/Latii)) BERT
(Chinese/Latin) BERT
BiLSTM
BF. Depth, HS, LR AF, BF, Depth, Heads, HS, LR
(Chinese/LLati) BERT
Transformer
Table 11: Listing of all hyperparameters varied per model.
Hyperparameter
Space
Activation Function
(ReLU, GeLU)
Blender
[mean, sum, take-first]
Depth
[1, (,,3,,44)
Heads
(1, 4, 8)
Hidden Size (BiLSTM)
(32, 48, 64, 96, 128, 192, 256, 384, 512)
Hidden Size (Transformer)
(256, 384, 512, 768, 1024, 1280, 1536, 1792, 2048)
Input Size (BiLSTM)
[64, 96, 128, 192, 256, 384, 512, 768)
Input Size (Transformer)
(128, 192, 256, 384, 512, 768, 1024)
Learning Rate Lemmatization
(0.0001, 0.0002....,...
(True, False]
Table 12: Collection of hyperparameter search spaces.
values. Previous experience in using Adam (Kingma and Ba, 2015) has models strug- gle with high learning rates, hence the choice of varying between lower ones.
sult, we direct the random search process in some cases by forcing trials to meet set of constraints. The main constraint we supplied was for the Baased models. This constraint was the word-level dimensionality compression constraint, which assured that HS z IS. The hidden state is in- tended here to compress and learn how to word embedding information. This constraint is applied implicitly for the model with BERT embeddings and the BiLSTM encoder, as every dimensionality in the search space was at or below 768.
For all such hyperparameter search spaces, sampled from each uniformly.
F33 Hyperparameter Trial Constraints
Although we could theoretically generate any combination of hyperparameterss in our randomly- sought trials, it is not necessarily the case that any combination will be fruitful. With wide enough search space, it is possible that random search will avoid discovering regions of the search space where performance is approximately maximal. As re-
G Error Analysis
The main body of this paper, and especially Sec- tion 7.2, highlights major hyperparameters for our

---

Parallelism
Branch
Word
Total
FP
FN
FP
FN
FM
FP
FN
FM
(by Sermon)
Sermon 24
8
22
3
2
0
2
0
40
Sermon 175
7
13
0
0
1
1
25
Sermon 177
11
39
2
0
1
56
Sermon 188
2
4
20
Sermon 207
2
14
Sermon 211
19
0
25
Sermon 219
0
0
0
0
0
9
Sermon 222
0
0
0
0
2
Sermon 271
0
5
0
0
0
0
0
5
Total (by Category)
45
119
9
3
2
9
196
Table 13:
presentation our error analysis categorization results on ASP's validation set with the model with
the highest Fj score from Section 7.2.
Encoder- CRF architecture which improve perfor- mance. In this section, we attempt to provide some insight about where our Encoder-CRF mod- els could still improve by describing the kinds of errors that these models make. We focus the ASP dataset in this section, this dataset de- signed for word-level parallelism detection.
words, a hypothesis and reference are only catego- rized once if they have some overlap; otherwise, they are categorized independently. We provide a more detailed, category-by-cates this scheme in our main code repository.
overview of
We present the results of our error analysis cat- egorization in Table 13. From this table, we see that the predominant cause of enror comes from the highest level of granularity: parallelisms. Con- versely, we see that other kinds of error are quite infrequent. As a result, we can surmise that these Encoder-CHi models are relatively all-or-nothing in their sequence labeling approach. They are pro-
To elucidate relevant and constructive model be- havior, we defined a of categories for this error analysis. That set of categories can be divided into a three-by-three grid on the basis of three error granularities and three error types. Following our metrics, these granularities are at the levels par- allelisms, branches, and words. Mcanwhile, two of our error types derive from classification етror types: false positive (FP) and false negative (FN).
ficient at getting entire parallelisms correct, and largely down to the boundaries, when they detect parallelism. However, their ability to confidently detect them in full needs
Our third error type, false mixture (FM), is combination of the FP and FN which applies to branches and words. Because model can both predict branch or word that a parallelism does contain (false positive) and omit branch or word that a parallelism contains (false negative), both error types are simultancously possible for these granularities. Conversely, a parallelism cannot both be entirely missing and entirely distinct from refer- ence data; thus, the granularity-tyne combination of parallelism and false mixture is not possible. This leaves us with a of eight categories.
work.
To illustrate some of these error categories, we provide set of three examples in Fig. 8. Fig. 8(a) presents the most common case: the false negative parallelism. Fig. 8(b), in turn, presents its false positive version. Finally, Fig. 8(c) shows what a false mixture looks like.
Starting with Fig. 8(a), we find two-branch parallelism from ASP. Both branches contain two clauses spanning four words, are headed by sub- jects of the same case and number, and apply verbal forms in their first clauses with the same lemma (promitto). Augustine plays up contrasts by juxta- posing human beings and God as well as verbs for belief (crederes) and doubt (dubitas). The moods used to define the conditional structures of each branch, being contrafactual and factual, respec- tively, further highlight this contrast. In short, these
We went over ASP's validation set with this an- notation scheme, comparing hypothesized paral- Jelisms from the model with the highest secheit-scoringg Fi score from Section 7.2 and reference parallelisms. We added a category for every matched hypothe- sis parallelism and reference parallelism. other

---

homo promitterer
crederes
deus
G promiseuPP,AACTT:B//3.3.G hwt. belleve.IFF..ACTTHBBYYY2 god:NOM:SG
promiltit promisceRS..ATT:ND:3355
et dubitay
11
and doubtrRs::CTT,NN:22S
"If human were promising, you would believe them; God promises, and you doubt Him?"
(b) ... quam illa quae ( (caeli
creatoren de caelo ... than that who heaven:GEN:SG maKer.ACC:SG from sky:AB1,,SG BBM.CDOw.PRF.AACHN:::33 terreno
deposuit
>1.er {
corpore terrae
induit
conditorem )
earthen:ABL;SC body:ABL;SG earth:GEN;SS than that which put the maker heaven down from heaven, and that which clothed the builder of the earth with carthen body; ...
builder:AcC;se
....
(c) fecit enim hocl per fideles do this through faiC OWD:ACC;PL ) a se ordimatas et christi iugo iam subditas. ACtho by oneself govern and christ yoke already subject
suos
1 (er christianos
SHOS
>1h (
(per
through christian:ACCIPL. OWN:ACC;PL
through
potestates
"For he did this through his own faithful, through his own Christians, through the authorities governed by him and subjected already the yoke of Christ."
Figure 8: Examples of erross for ASP dataset's validation set and the model with the highest F3 score from Section 7.2. Blue square brackets demarcate branches in the reference data, red triangular brackets demarcate branches in the hypothesis data, and numbering indicates the parallelism to which a branch belongs relative to its data source. Below each Latin sentence, we provide word-by-word English translation, gloss some morphological features in branches, and present an idiomatic translation. Example (a) is from Sermon 177, Section 11 and displays a parallelism-lest level false negative. Example (b) is from Sermon 207, Section and displays a parallelism- false positive. Example is from Sermon 24, Section and displays branch-level false mixture.
clauses contain many distinct cues as to their paral- lel nature. In spite of this, our model did not detect this pair. One possibility is that variations on token order due to function words (si, et) and commas caused the model to ignore the branch pair.
of two three-word branches centering around two prepositional phrases that share the word suos. The pair are followed by third clause that is not part of the parallelism due to its rather elaborate content and lack of suos (although, it contains the related form se). To speculate about the model's mistake, may have been drawn by lexical connections be- tween the second branch and the final clause (i.e., per with per, suas with se, and christianos with christi). However, because of the final clause's ad- ditional complexity, it then only selected couple words to pair with the second branch.
Next, Fig. 8(b) shows predicted parallelism that the reference data did not contain. On the one hand, there are aspects of this branch pair that make them seem parallel. The clauses juxtapose heaven (caeli, de caelo) and earth (terreno, terrae) in the same grammatical roles, Both also use verbs conjugated in the same manner, and each takes direct object in the same case (creatorem, conditorem) which are synonymous. On the other hand, the word or- der in these clauses calls their parallel nature into question; there is no clear alignment between the word orders in each clause (e.g.. the direct objects are the second and fifth words, respectively). As a result, it seems understandable both why the model was fooled by this sentence as well as why it was not annotated in the first place.
H Exhaustive Experimental Results
In this section, we display box plots which give an overview of all F, scores attained from each of the best models across all configurations explored during the hyperparameter search and all metrics. All box plots are created from 16 data points, as each model architecture variation (i.e., embedding- encoder combination) was examined across sixteen distinct tagging schemes. The ASP dataset's plots are presented in Figs. 9 to 12. Meanwhile, the PSE- I dataset's plots are presented in Figs. 13 16. We also provide tabular data and CSV data for all our results, additionally including precision and recall for all metrics, in our main code repository.
Finally, we provide an example of a branch-level false mixture in Fig. 8(c). In this branch-level false mixture, the model agrees with the reference par- allelism on its second branch. However, both ignores the reference's first branch and adds an additional branch after the second. The reference contains a relatively simple parallelism consisting

---

I Exemplary Model Hyperparameter Tables
In this section, we provide sets of tables which de- scrihe the hyperparameters used for each best trial catalogued in Appendix H. For the ASP dataset, organized the results into three tables-Tables 14 base upon what hyperparameters were var- ied. For the PSE-I dataset, we organized them similarly in Tables 17 and

---

EPM F1 Scores the Test Set (Groups: embedding; encoder)
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
Learned [Word]; BILSTM
Word: BILSTM
Learned Tansterr
Latin BERT: None
Latin BERT: BILSTM
Latin BERT: Transforer
Figure 9: Box plots presenting model-relative results on the ASP dataset's test set using the Exact Parallelism
Match (EPM) metric.
MPBM F1 Scores on the Test Set (Groups: embedding; encoder)
0.50
x
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
Learned [Word]; BILSTM
Word; BILSTM
Learned [
Latin BERT: None
Latin BERT; BiLSTM
Latin BERT: Transformer
Figure 10: Box plots presenting model-relatie results on the ASP dataset's test set using the Maximum Parallel Branch Match (MPBM) metric.

---

MBAWO F1 Scores the Test Set (Groups: embedding; encoder)
0.50
0.45
0.40
0.35
0.30
0.20
0.10
0.05
0.00
Learned [Word]; BILSTM
Word: BILSTM
Learned Tansterr
Latin BERT: None
Latin BERT: BILSTM
Latin BERT: Transforer
Figure 1I: Box plots presenting model-relative results on the ASP dataset's test set using the Maximum Branch- Aware Word Overlap (MBAWO) metric.
MWO F1 Scores on the Test Set (Groups: embedding; encoder)
0.55
0.50
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10
Learned [Word]; BILSTM
Word; BILSTM
Learned [
Latin BERT: None
Latin BERT; BiLSTM
Latin BERT; Transformer
Figure 12: Box plots presenting model-relative results on the ASP dataset's test using the Maximum Word
Overlap (MWO) metric.

---

EPM F1 Scores on the Test Set (Groups: embedding; encoder)
0.40
0.35
0.30
0.25
0.20
0.15
0.10
Chinese BERT; None
Chinese BERT: BILSTM
Chinese BERT; Trans mransformer
Figure 13: Box plots presenting model-relative results on the PSE-I dataset's test set using the Exact Parallelism
Match (EPM) metric.
MPBM F1 Scores on the Test Set (Groups: embedding; encoder)
0.55
X
0.50
0.45
0.40
0.35
0.30
X
0.25
Chinese BERT: None
Chinese BERT: BILSTM
Chinese BERT: Transformer
Figure 14: Box plots presenting rodel-relative results on the PSE-I dataset's set using the Maximum Parallel Branch Match (MPBM) metric.

---

MBAWO F1 Scores the Test Set (Groups: embedding; encoder)
0.65
0.60
0.55
0.50
0.45
Chinese BERT; None
Chinese BERT: BiLSTM
Chinese BERT: Transfore Pransformer
Figure 15: Box plots presenting model-relative results on the PSE-I dataset's test set using the Maximum Branch- Aware Word Overlap (MBAWO) metric.
MWO F1 Scores on the Test Set (Groups: embedding; encoder)
0.75
0.70
0.65
0.60
0.55
0.50
Chinese BERT: None
Chinese BERT: BILSTM
Chinese BERT: Transformer
Figure 16: Box plots presenting model-relative results on the PSE-I dataset's test set using the Maximum Word Overlap (MWO) metric.

---

Model Components
Tagging Scheme
Hyperparameters
Embedding
Encoder
Tagset
Link Type
DEPTH
HS
IS
LEMMA
LR
Token
384
512
N
0.0013
BIO
Branch
384
768
N
0.0008
Token
64
768
N
0.0021
BIOE
Branch
48
256
N
0.0039
Token
64
512
N 0.0076
BIOJ
Branch
384
384
N 0.0028
Token
128
384
N 0.0010
BIOM
Learned (Word)
Branch
128
128
N 0.0035
BiLSTM
Token
256
768
N 0.0063
BIOJE
Branch
2
256
256
Y 0.0034
Token
32
192
N 0.0045
BΙОМЕ
Branch
128
256
N 0.0082
Token
96
512
N 0.0052
BIOMJ
Branch
64
768
N 0.0047
Token
192
512
Y 0.0018
BΙOMJ
Branch
3
128
192
0.0085 Y
Token
4
96
300
Y 0.0028
BIO
Branch
256
300
Y
0.0040
Token
3
32
300
Y
0.0076
BIOE
Branch
128
300
Y
0.0029
Token
48
300
Y
0.0072
BIOJ
Branch
64
300
Y 0.0059
Token
256
300
Y 0.0028
BIOM
Branch
256
300
Y
0.0059
word2vec
BiLSTM
Token
32
300
Y
0.0035
BIOJE
Branch
48
300
Y 0.0049
Token
192
300
Y
0.0002
ВBОМE
Branch
32
300
Y
0.0032
Token
256
300
0.0035 Y
BIOMJ
Branch
96
300
Y
0.0058
Token
128
300
Y
0.0022
BOОMJ
Branch
192
300
Y
0.0028
Table 14: Best hyperparameters for non-BERT models with BiLSTM encoder for the ASP dataset.

---

Model Components
Tagging Scheme
Hyperparameters
Embedding
Encoder
Tagset
Link Type
AF
BF
Depth
Heads
HS
IS
LR
BIO
Token
ReLU
sum
3
4
384
192
0.0081
Branch
ReLU
mean
4
768
128
0.0076
Token
ReLU
tf
1
I
1280
192
0.0071
BIOE
Branch
GeLU
tf
2
8
384
768
0.0095
Token
GeLU
tf
2
4
512
1024
0.0052
BIOJ
Branch
GeLU
mean
8
2048
768
0.0043
Token
ReLU
tf
2
8
1792
192
0.0011
BIOM
Leamed (Subword)
Transformer
Branch
GeLU
mean
4
2048
256
0.0060
Token
GeLU
2
512
768
0.0082
BIOJE
Branch
ReLU
mean
2
384
128
0.0051
Token
GeLU
4
1280
384
0.0071
BIOME
Branch
GeLU
384
384
0.0056
Token
ReLU
512
384
0.0065
BIOMJ
Branch
GeLU
tf
2
384
512
0.0100
Token
ReLU
256
192
0.0087
BIOMIE
Branch
GeLU
1280
512
0.0100
Token
GeLU
sum
8
256
768
0.0090
BIO
Branch
ReLU
mean
4
1024
768
0.0086
BIOE
Taken
ReLU
sum
4
4
1280
768
0.0028
Branch
ReLU
768
768
0.0034
Token
ReLU
1792
768
0.0007
BIOJ
Branch
GeLU
4
1024
768
0.0021
Token
GeLU
1024
768
0.0035
BIOM
Branch
GeLU
2
1024
768
0.0035
Latin BERT
Transformer
Token
GeLU
1792
768
0.0048
BIOJE
Branch
GeLU
1536
768
0.0037
BIOME
Token
GeLU
mean
1024
768
0.0016
Branch
GeLU
sum
1536
768
0.0033
Token
ReLU
tf
384
768
0.0061
BIOMJ
Branch
ReLU
mean
1536
768
0.0014
BIOMJE
Token
ReLU
mean
256
768
0.0086
Branch
GeLU
mean
384
768
0.0045
Table 15: Best hyperparameters for models with a Transformer encoder on the ASP dataset.

---

Model Components
Tagging Scheme
Hyperparameters
Embedding
Encoder
Tagset
Link Type
BF
DEPTH
HS
LR
Token
sum
0.0055
BIO
Branch
tf
0.0010
Token sum
0.0012
BIOE
Branch sum
0.0055
Token tf
0.0077
BIOJ
Branch sum
0.0070
Token tf
0.0064
BIOM
Branch sum
0.0012
Latin BERT
Token sum
0.0063
BIOJE
Branch mean
0.0005
Token
mean
0.0045
BIOME
Branch mean
0.0021
Token
tf
0.0005
BIOMJ
Branch mean
0.0030
Token
sum
0.0001
BIOMJE
Branch mean
0.0061
Token mean
3
384
0.0007
BIO
Branch sum
3
96
0.0022
Token tf
4
48
0.0024
BIOE
Branch mean
3
512
0.0012
Token mean
3
256
0.0008
BIOJ
Branch sum
4
128
0.0066
Token
2
192
0.0048
BIOM
Branch tf
3
192
0.0073
Latin BERT
BiLSTM
Token mean
4
48
0.0017
BIOJE
Branch sum
3
128
0.0010
Token mean
2
64
0.0083
BIOME
Branch sum
4
128
0.0026
Token sum
3
96
0.0028
BIOMJ
Branch
tf
4
192
0.0013
Token
4
192
0.0055
BIOMJE
Branch
mean
2
96
0.0086
Table 16: Best hyperparameters for models with BERT based embeddings and a non- non-Tranfformer encoder on the
ASP dataset.

---

Model Components
Tagging Scheme
Hyperparameters
Embedding
Encoder
Tagset
Link Type
AF
BF
Depth
Heads
IIS
IS
LR
Token
ReLU
8
1280
768
0.0010
BIO
Branch
ReLU
sum
2
1792
768
0.0040
Token
ReLU
tf
I
256
768
0.0088
Branch
GeLU
tf
2
1280
768
0.0010
Token
ReLU
tf
4
8
1536
768
0.0063
BIOI
Branch
ReLU
tf
512
768
0.0078
Token
GeLU
tf
256
768
0.0100
BIOM
Chinese BERT
Branch
ReLU
8
1792
768
0.0094
Transformer
Token
GeLU
tf
1536
768
0.0016
BIOJE
Branch
ReLU
1536
768
0.0061
Token
GeLU
2048
768
0.0001
BIOME
Brunch
GeLU
1536
768
0.0086
Token
GeLU
768
768
0.0025
BIOMJ
Brunch
ReLU
1280
768
0.0076
Token
ReLU
512
768
0.0073
BIОМJE
Branch
GeLU
sum
768
0.0020
Table 17: Best hyperparameters for models with Transformer encoder on the PSE-I dataset.

---

Model Components
Tagging Scheme
Hyperparameters
Embedding
Encoder
Tagset
Link Type
BF
DEPTH
HS
LR
Token
tf
.0002
BIO
Branch mean
.0089
Token
.0063
BIOE
Branch
.0012
Token
mean
.0043
BIOJ
Branch
sum
.0072
Token
sum
.0020
BIOM
Chinese BERT
Branch mean
.0006
Token mean
.0050
BIOJE
Branch tf
.0028
Token
.0087
B
Branch tf
.0037
Token mean
.0092
BIOMI
Branch tf
.0017
Token tf
.0052
BIOMJE
Branch mean
.0066
Token mean
192
.0012
BIO
Branch
sum
2
96
.0019
Token
sum
3
32
.0025
BIOE
Branch
sum
2
64
.0043
Token
sum
2
512
.0002
BIOJ
Branch
mean
1
.0007 48
Token
3
128
.0080
BIOM
Chinese BERT
Branch mean
2
32
.0056
BiLSTM
Token mean
4
64
.0040
BIOJE
Branch sum
4
128
.0003
Token tf
4
32
.0016
BIOME
Branch tf
4
32
.0010
Token sum
3
128
.0004
BIOMJ
Branch sum
4
48
.0011
Token
tf
1
32
.0058
BIOMJE
Branch
mean
2
32
.0025
Table 18: Best hyperparameters for models with BERT based embeddings and a non- Tranrfasformer encoder on the
PSE-I dataset.


Reviews:
[{"review_id": "IBuYCWSS20", "reviewer": "Reviewer_J4Zn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review; we appreciate your kind words about the new task of rhetorical parallelism detection. We are glad that our spin on sequence labeling was well-received. \n\nConcerning the inclusion of more concrete information about the dataset in the paper, we absolutely agree with your suggestions. We had trouble finding space to include it in this draft, but we want to find space in the final version.\n\nIn previous drafts of this work, we performed some error analysis and categorized errors into three major types. Although this analysis was derived from an older baseline system, we nevertheless discuss it here to give an example of what one might expect in the final version.\n\n(1) From Sermon 212, Section 1:\nLatin: ... ut iam non aggrauet animam [nec ullam quaerat refectionem]1 , [quia nullam patietur defectionem]1 ...\nEnglish: \"... that no longer shall the body burden the soul nor shall it seek any restoration, since it suffers no defect ...\"\n\n(2) From Sermon 253, Section 5:\nLatin: <[transcendit nubes]1 et [transcendit sidera]1>1 , <[transcendit angelos]1>1 , <[transcendit omnem creaturam]1>1 <peruenit ad uerbum>1 , per quod facta sunt omnia .\nEnglish: \"He transcended clouds and transcended stars, transcended angels, transcended every creature, and he arrived at the Word, through whom all things were made.\"\n\n(3) From Sermon 186, Section 2:\nLatin: quid enim attenderunt , nisi quia <humana natura potuit in melius commutari>1 , <in deterius autem diuina non potuit>1?\nEnglish: \"What, namely, did they consider, except that human nature was able to be transformed into something better, but that divine nature was not able to be transformed into something worse?\"\n\nIn the examples above, square brackets indicate reference parallelisms (with a numbering indicating the parallelism ID of a given branch relative to the presented passage), whereas angle brackets indicate predicted parallelisms.\n\nIn Example (1), we find errors of the first type, in which the model does not predict a parallelism at all (a complete false negative). This was surprising because of the many similarities between the two annotated clauses, as their word order indicates a one-to-one alignment and the grammatical structure is the same. Moreover, the words *ullam* and *nullam* are antonyms (\"some/any\" and \"no/none\", respectively), and *refectionem* and *defectionem* are both phonetically alike while playing similar grammatical roles.\n\nIn Example (2), we find errors of the second type, in which a model predicts a parallelism in the same location as a reference parallelism, but the predicted parallelism differs from what was annotated. The model fails to bound the individual *transcendit* clauses joined by *et* (\"and\"), relying too heavily on punctuation for boundaries. It also adds the clause *peruenit ad uerbum*. This clause is grammatically like prior clauses with a similarly-conjugated verb and an accusative noun. However, the accusative noun is an object of a prepositional phrase cued by *ad*; furthermore, it clearly does not include the verb which links the prior clauses: *transcendit*. This error could be described as an \"alternate interpretation\"--a false positive that is not baseless but has issues.\n\nIn Example (3), we find errors of the third type, in which a model predicts a parallelism where none was annotated. However, the model still chooses text that has some merit. The first and second branches contrast *humana natura* with *diuina natura* (the latter *natura* being understood). Both clauses also focus on the same verb, using it both positively (*potuit*) and negatively (*non potuit*). Their substantive adjectives--*in melius* (\"into something better\") and *in deterius* (\"into something worse\")--also both parallel one another syntactically while contrasting semantically. In short, there are many similarities between these two clauses; however, the branches' word orders are scrambled, not quite fitting the mold of a parallelism. As a result, it is understandable why our annotator skipped it while our model did not.\n\nWe hope that this sample of more data and error analysis clarifies our data better and provides precedent for what one can expect from our paper!\n\nFinally, we agree that additional experiments to show the generalizability of this work would be of interest and, as you say, more convincing of our claims. We have a few ideas for fitting tasks (*e.g.*, Chinese grammatical error detection, Latin syntactic parsing) and, given the time, will look into an addition of this kind.\n\nThanks again for your time and care in reviewing our paper. We are happy to answer any other questions or hear out further suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper very clearly describes a problem, and then very effectively mobilizes machinery from pre-2020 NLP* to make a well defined machine learning task. The creativity in identifying parallels (kind of meta!) between this task and NER, disfluency detection, coreference resolution is clear, and it means that the metrics and models used are time-tested, well understood, and trustworthy.\nI am not an expert in this area but I presume that this dataset will be helpful to digital humanities practitioners in detecting rhetorical devices at a large scale, as well as for automatic essay scoring if it is clear that the essay writers have been encouraged to use parallelism.\n\n\n* I mean this in a very positive way!", "reasons_to_reject": "While this paper is very thorough in defining and presenting the RPD task, it is missing some contents that would be interesting and relevant in my opinion. For example, I think it is important for readers to get a 'feel' for a new task, so more examples demonstrating the different types of parallelism (phonology/morphology/syntax/semantics) and their prevalence in the two datasets would be valuable. Since a 40% F1 score is reported, an error analysis would be instructive for future researchers. The authors mention the potential usefulness for downstream tasks; a small demonstration on even a proxy task would help make this claim more convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces the task of Rhetorical Parallelism Detection. Noting that parallelism is a common rhetorical feature and its presence or absence is a useful indicator for tasks like essay scoring, syntactic parsing, and disfluency detection, the authors annotate a dataset of parallelism in two text corpora, one in Latin and one in Chinese. Detailed definitions of parallelism are provided, and a thorough description of evaluation methods is presented. A reasonable suite of baseline models is tested, showing that good performance is possible but there is room for improvement.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z2HcmLCCri", "reviewer": "Reviewer_FFhL", "summary": "", "questions": "Question A: You split your data into training, validation, optimization and test. What is the use of your validation set? I noticed you said in the paper that the optimization set is used for hyper-param tuning but no mention of the use of validation set.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review. We're glad that the majority of proposed elements in our paper (*e.g.*, datasets, metrics, models) were presented in a manner that was transparent and straightforward. For our rebuttal, we want to start by addressing some of the points of our work which were less clear--namely, those concerning training and tagging. \n\nFirst, regarding training, we split our data into four sets: training, validation, optimization, and test (as you mentioned). During training, the validation set was used to determine whether early stopping would occur. After each epoch, we computed MBAWO F1 scores over the validation set; if a model's F1 scores on this set did not improve over 25 epochs, we stopped training early. We recognize that we did not mention the validation set's purpose explicitly in the relevant subsection, and we will make this clearer in the final version.\n\nSecond, regarding tagging, the additional tags which we add comprise one of our paper's major novelties. As a result, we certainly want to improve the clarity of the designated section. One critical point to convey is that these tags are intended to divert the model's attention from the majority class of \"O\" tags. The conditional random field examines all bigram transition probabilities to determine the optimal series of transitions (and thus the optimal series of tags) over the entire input sequence. However, because \"O\" tags are involved in an overwhelming majority of transitions, a model could easily transition to an \"O\" tag and discard information about incomplete parallelisms due to the high probability of the transition \"O\" -> \"O\".\n\nThe \"M\" tag addresses this issue by decreasing the number of \"O\" tags overall, replacing them when an \"O\" tag is between any two related branches. The \"E\" tag prevents the \"I\" -> \"O\" transition by replacing the last \"I\" tag of any branch with an \"E\" tag. Lastly, when the \"J\" tag is paired with the \"M\" tag, it produces a closed loop of transitions for parallelisms. Each parallelism has an initial branch that starts with a \"B\" tag; this is followed by 0 or more \"I\" tags and 0 or more \"M\" tags; subsequently, there is a linked \"B\" tag (since each parallelism must have at least two branches) followed by 0 or more \"J\" tags. Only the linked \"B\" tag and the \"J\" tag potentially return to \"O\" tags, thereby enforcing that at least two branches must be found before the model exits the parallelism.\n\nAs a final note, we agree that our inter-annotator agreement ideally would be higher for our dataset. While agreement is better on the word level, achieving a value higher than 0.6 (considered \"substantial\" agreement by Landis and Koch), we would like to further refine the dataset. We hope that sharing our work with the wider research community can aid in enhancing its quality.\n\nWith all that said, we want to thank you again for your time, comments, criticisms, and question. We are happy to answer any further questions or take further suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Compared to previous work, the proposed RPD datasets allow token-level parallel structures. The RPD task is re-formulated as a sequence tagging task rather than a classification task, which is novel and interesting. These datasets (together with the new metrics) will be useful for researchers to explore parallelism detection.\n\nThe paper is well written and methods are described in great detail.", "reasons_to_reject": "The agreement between annotators is actually not high, this brings some doubts on the quality of the datasets. \n\nThe authors bring in additional tags (e.g., M, E, J), though experiments in Table 5 demonstrate the benefits, the description (line 404 - 431) is not easy to follow and comprehend. The motivation behind thus is not well illustrated.", "questions_to_authors": "Question A: You split your data into training, validation, optimization and test. What is the use of your validation set? I noticed you said in the paper that the optimization set is used for hyper-param tuning but no mention of the use of validation set.", "paper_topic_and_main_contributions": "This paper focuses on the detection of rhetorical parallelism in Latin and English. Specifically, this paper gives a precise definition of rhetorical parallelism detection (RPD) and proposes two RPD datasets in Latin and Chinese respectively. A bunch of baselines (word2vec, Bi-LSTM, BERT) are tested on proposed datasets. Moderate results demonstrate the challenging of this task. Additionally, this paper proposes a new evaluation metric for this sequence tagging task by calculating the proportion of correct alignments (aligned using Hungarian Algorithm).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "ODp0UWJZsx", "reviewer": "Reviewer_MynM", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and your meticulous examination of our paper! We're glad that you found it clear, comprehensive, and readily replicable.\n\nWe'll start by addressing your concerns about our experimental procedures. As described in our Sections 4.3 and 7.1, we divided the data into four sets: training, validation, optimization, and test. Our hyperparameter search involved eight random trials with each combination of model and tagset (the \"variants\" which you mention) in which hyperparameters were varied (as described in Appendix F). Each trial saw the training set during training, used the validation set for early stopping, and was tested on the optimization set. Then, the best trial of those eight on the optimization set was deemed the \"best\". After the hyperparameter search was concluded, only the best trials were evaluated on the test set. \n\nAs you mention, we used the MBAWO metric instead of the EPM metric for early stopping. Because EPM is a strict metric, we were concerned that we would overly penalize models for making incremental progress on capturing correct tokens, stopping their training procedures too early. Conversely, being a word-level metric, MBAWO captures that incremental progress more directly. An opposing argument could be made that relying on MBAWO might procure a model that overfits on finding individual words but not whole parallelisms. However, because both metrics innately require parallel structure to achieve a higher score, we can safely wager that MBAWO and EPM scores will correlatively improve with one another. However, because we were being exploratory with our new task, taking advantage of MBAWO's higher granularity seemed like a better fit.\n\nRegarding the hyperparameter search, our rationale behind the division between \"variants\" and \"hyperparameters\" was somewhat intuition-based. We felt that the model architecture and the selected tagset would be the most critical values in the hyperparameter search. Thus, we chose to examine them *en masse*. However, we agree that the number of results collected and presented is massive. We wanted to be able to recommend general sets of hyperparameters for future work so that other researchers would have some basis for selecting them in the future. Currently, this recommendation has only been partially achieved, perhaps being hampered by performing too wide of a search. If possible, we would like to expand upon our results section with more statistical and error analyses, thereby providing more justification for our vast results collection.\n\nTurning to your dataset concerns, you are correct that our lower inter-annotator agreement results raise doubts about our task. Inevitably, the only solution for this problem is further annotation studies with revised guidelines. Pertinent future work could involve collecting a larger dataset of parallelism annotations on a living language (*e.g.*, English) both to ascertain our ability to achieve higher performance on this task and to make this task more widely accessible. At the same time, Song *et al.*’s satisfactory annotator agreement score in their sentence-level dataset gives us hope that our token-level Latin dataset can attain a similar reliability. Releasing our data to the research community may aid in its refinement as we continue to improve it ourselves.\n\nLastly, we wanted to elaborate on potential uses of our dataset. In our paper, we mentioned (and hope to expand upon in the final version) a connection between our work and automatic stylistic analysis (lines 66-68). From a classicist's perspective, parallelism is an important feature of rhetorical Latin prose; its presence or absence forms an important criterion in the comparison of different texts, authors, or even eras of Latin prose writing and speaking. Although classicists closely study stylistic phenomena in individual texts, it is quite time-consuming to cover extensive corpora. (For instance, although our dataset consists of 80 sermons, there are over 800 in Augustine's extant corpus!) Thus, from an NLP perspective, our dataset can assist in a distant reading approach: by training models for rhetorical parallelism, data can be automatically annotated and used toward a diachronic study of rhetorical styles. Features extracted from the performance of RPD could also be used for tasks like authorship attribution, as a proclivity (or lack thereof) for parallelism may help to distinguish one author from another.\n\nThese uses, however, have their primary audience as those already studying Latin. We acknowledge your point that it may be difficult to have the wider NLP community invest its efforts in the Latin dataset when their work may concern other living languages. For those whose interest would primarily be on living languages (and who perhaps would endeavor to create data in those languages), our dataset can always assist in showing the generalizability of their proposed methods.\n\nWith that said, we want to thank you again for your thorough examination of our work and your careful commentary. If you have any more questions, concerns, or suggestions, please let us know--we'd be glad to hear them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear problem statement, motivation, methods, results, and discussion\n2. Plausibly broad interest, especially for applications of NLP to writing pedagogy\n3. Impressive scope of work, including a novel task and formalization, a large expert-annotated resource, and extensive modeling\n4. Methods and appendices are unusually comprehensive, detailed, and clear, which both supports replication and makes clear that great care was taken to anticipate and propose principled solutions to diverse technical challenges that arise in both annotation and modeling for this task.\n", "reasons_to_reject": "1. There is a seemingly arbitrary division made between \"hyperparameters\" (things like learning rate, state size, etc) that are tuned using random search on validation data and \"variants\" (choices about model architecture and tagging scheme) that are exhaustively evaluated on the test set. These variants also seem to me like hyperparameter choices that will need to be tuned in service of selecting a final working model for this task in any actual application. There are very many \"variants\" considered here, and as a result I think far too many models were evaluated on the test set (the appendices contain 15 pages of test set results). If this were primarily a modeling paper, I would consider this to be a fishing expedition and a serious flaw. However, this is primarily a resource paper, and this large set of models is framed as a \"baseline\" showing that the task is tractable to some extent. I still think it would have been better to treat the architecture and tagging scheme as additional hyperparameters and at most only report differences in validation set performance, but I don't think that what was actually done seriously undermines the main contribution.\n\n2. The inter-annotator agreement score for the Latin dataset is low (around 0.4 to 0.6 depending on the measure being tested). The authors are laudably frank about this, and about possible issues with the annotation guidelines that might have contributed to this outcome, and they promise to address this in future annotation efforts. Nonetheless, if rhetoric experts show such low agreement, it raises concerns both about construct validity for this task and about the amount of available headroom for improving on the baseline results for any groups that choose to take this up.\n\n3. I don't understand why a different performance metric was used for early stopping (MBAWO) vs. for the critical evaluations (EPM). Some justification would be helpful.\n\n4. The new resource is in two languages: Latin and Chinese. The Chinese dataset is a light reannotation of an existing dataset, so the bulk of the annotation contribution is for Latin. Although I recognize the historical importance of Latin in the study of rhetoric, I do find it strange that the authors chose to debut this task by concentrating on a language that few living humans use for communication, and I'm having a hard time seeing practical use cases or strong intrinsic motivation for the NLP community to invest in targeting the Latin component of this dataset (targeting Chinese is self-evidently valuable). Is the primary target use case something like writing pedagogy for classics students? In any case, revisions that help clarify the expected practical utility of the Latin portion of the dataset would be welcome.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes rhetorical parallelism detection (RPD) as a novel NLP task in which the objective is to identify sets of textual spans in natural language that constitute a parallelism as defined by rhetoreticians. To this end, the authors formalize the task and several plausible evaluation metrics, offer a novel expert-annotated dataset of naturally-occurring parallelisms in Latin and Chinese texts, identify and suggest solutions to important technical challenges (e.g., which tagging scheme to use), and report the performance of a diverse array of baseline models, the results of which show both substantial evidence of learning but also much potential for improvement, suggesting that this is a tractable but challenging task.\n\nI'm impressed by this paper and support publication. I do have some concerns that I elaborate on below.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes rhetorical parallelism detection (RPD) as a novel NLP task in which the objective is to identify sets of textual spans in natural language that constitute a parallelism as defined by rhetoreticians. To this end, the authors formalize the task and several plausible evaluation metrics, offer a novel expert-annotated dataset of naturally-occurring parallelisms in Latin and Chinese texts, identify and suggest solutions to important technical challenges (e.g., which tagging scheme to use), and report the performance of a diverse array of baseline models, the results of which show both substantial evidence of learning but also much potential for improvement, suggesting that this is a tractable but challenging task.

In general, the reviewers found RPD and the paper to be very interesting, well-motivated and clearly presented. The scope of the paper is quite wide, including a novel task and formalization, a large expert-annotated resource, and extensive modeling.

One of the main concerns brought up is the low IAA scores. These scores raise doubts about the quality of the dataset, the construct validity for the task, and how much space there is for improvement on a baseline for this task. The authors respond echoing the concerns and elaborating the possible reasons behind this; however, they suggest that releasing the dataset to the community will provide the opportunity for refinement. 

Additional questions or need for clarification brought up by reviewers were extensively addressed in the author response. I would expect any camera-ready version to incorporate these details and clarifications.
Decision: Accept-Main