That’s Optional: A Contemporary Exploration of "that" Omission in English Subordinate Clauses (2024)

Abstract: The Uniform Information Density (UID) hypothesis posits that speakers optimize the communicative properties of their utterances by avoiding spikes in information, thereby maintaining a relatively uniform information profile over time. This paper investigates the impact of UID principles on syntactic reduction, specifically focusing on the optional omission of the connector "that" in English subordinate clauses. Building upon previous research, we extend our investigation to a larger corpus of written English, utilize contemporary large language models (LLMs) and extend the informationuniformity principles by the notion of entropy, to estimate the UID manifestations in the usecase of syntactic reduction choices.

heading: 1 Introduction
text: Exploiting the expressive richness of languages, speakers often convey the same messages in multiple ways. A body of research on uniform information density (UID) puts forward the hypothesis that speakers tend to optimize the communicative effectiveness of their utterances when faced with multiple options for structuring a message. The UID hypothesis (Frank and Jaeger, 2008; Collins, 2014; Hahn et al., 2020) suggests that speakers tend to spread information evenly throughout an utterance, avoiding large fluctuations in the perunit information content of an utterance, thereby decreasing the processing load on the listener.
The UID hypothesis has been used as an explanatory principle for phonetic duration (Bell et al., 2003; Aylett and Turk, 2006), the choice between short- and long-form of words that can be used interchangeably, such as "info" and "information" (Mahowald et al., 2013), and word order patterns (Genzel and Charniak, 2002; Maurits et al., 2010; Meister et al., 2021; Clark et al., 2023). Our work studies how UID principles affect the phenomenon of syntactic reduction – the situation
where a speaker has the choice of whether marking a subordinate clause in sentence with an optional subordinate conjunction (SCONJ) "that" or leave it unmarked, as in "My daughter mentioned [that] he looked good". The only study that tested the UID hypothesis computationally in the context of syntactic reduction is Levy and Jaeger (2006), followed by Jaeger (2010), who studied the effect of multiple factors on the speaker choice of explicit or implicit "that" conjunction. Investigating sentences with main clause (MC, e.g., "My daughter mentioned") and subordinate clause (SC, e.g., "[that] he looked good"), connected by the optional SCONJ, the authors found that UID optimization was the most prominent factor affecting a speaker choice of "that" omission. Specifically, Jaeger (2010) investigated 6700 sentences extracted from the SwitchBoard spoken English dataset, and operationalized the UID principle by computing the surprisal (non-predictability) of the SC opening word (SC onset) using a statistical bigram language model computed from the corpus itself.
Our work studies the role of UID principle in syntactic reduction in multiple differing ways. First, we extend the investigation to a much larger corpus of informal written English collected from social media. Second, we use contemporary large language models (LLMs) to estimate the operationalizations of information uniformity in syntactic reduction, suggesting the robustness of our findings. Finally, inspired by the information-theoretic nature of UID and prior art (Maurits et al., 2010; Meister et al., 2021), we extend the SC onset surprisal UID manifestation with the notion of SC onset entropy – the information entropy of LLM distribution over SC opening word, conditioned on the main clause – factor that turns out to have a complementary and significant effect.
The contribution of this work is, therefore, twofold: First, we collect and release a large and diverse corpus of nearly 100K sentences, where
378
main and subordinate clauses are connected by the optional SCONJ "that".1 Second, we go above and beyond prior work by using transformer-based LLMs (Vaswani et al., 2017), thereby providing a sound empirical evidence for UID principles associated with syntactic reduction decision, shedding a new and interesting light on the manifestation of UID in spontaneous written language.

heading: 2 Dataset

heading: 2.1 Data Collection
text: Our dataset in this work was collected from the Reddit discussions platform. Reddit is an online community-driven platform consisting of numerous forums for news aggregation, content rating, and discussions. Communication on discussion platforms often resembles a hybrid between speech and more formal writing, and findings from spoken language may extend to the spontaneous and informal style of social media. As such, Reddit data has been shown to exhibit code-switching patterns, similar to those found in spoken language (Rabinovich et al., 2019). We, therefore, believe that this data presents a good testbed for our analysis.
Data Extraction We collected 2M posts and comments by over 20K distinct redditors spanning over 5K topical threads and years 2020–2022. We then split the data into sentences and filtered out sentences shorter than five or longer than 50 words. The remaining 487,614 sentences were parsed using the SOTA benepar syntactic parser, extracting two sentence types with main and subordinate clause, possibly connected by "that": (1) Explicit usage, as in "do you agree that his suggestion sounds better?" More specifically, we identified sentences where SCONJ "that" immediately follows the main verb, as with the main verb "agree" in the example above. A set of rules was devised for identifying relevant sentences, filtering out cases where "that" was used in roles other than SCONJ, such as demonstrative determiner ("I have never been to that part of the city"), demonstrative pronoun ("that is a beautiful view"), or relative pronoun, ("Ann is on the team that lost.").2 (2) Implicit usage, as in "my brother thinks [that] partners should always choose the former alternative", where SCONJ "that" could have been used
1All data and code are available at https://github.com/ ellarabi/uid-that-sc-omission.
2Due to its much lower frequency, we leave the investigation of "that" as a relative conjunction to future work.
but was deliberately omitted. The set of rules used for identifying these sentences is identical to the rules used for detection of explicit usages, except that we required the absence of "that" in the appropriate syntactic role. Appendix A.1 provides details on syntactic analysis and rules used to extract relevant sentences. Table 1 reports the details of the collected dataset.
Evaluation A random subset of 500 sentences split equally between explicit and implicit "that" usages was selected for manual evaluation by one of the authors of this paper. The evaluator was guided to check whether omitting "that" in explicit SCONJ cases would result in equally valid, meaning-preserving utterance, and vise versa – whether adding explicit "that" in places it was omitted, would not hurt the sentence fluency and semantics. 96.4% of the first sentence set were found valid, and 95.7% of the second sentence set. Invalid cases include mainly ungrammatical utterances and sentences in languages other than English.

heading: 2.2 Data Analysis
text: We next tokenized and lemmatized the sentences using the the spacy python package. Table 2 presents example sentences, taken verbatim from our dataset, with explicit and implicit usages of "that" conjunction. Note that sentences with the same verb lemma (e.g., "forget") show syntactic reduction in some cases but not in others.
Studying "that" omission in native and learner English, Olohan and Baker (2000) found that the optional usage of "that" conjunction typically follows reporting main verbs – such as "say", "think", "suggest". Our data largely supports this observation: while the total of 434 distinct main verb lemmas were found to precede the optional "that", roughly two thirds (64.7%) of all usages (or potential usages – omissions) are covered by the top-10 most frequent lemmas in the dataset. Additionally, different verbs exhibit different distribution of ex-
plicit and implicit usages: while "that" is omitted in the majority of cases following lemmas "think" and "guess", other lemmas, like "say", "know", "believe", and "realize" show more balanced behavior. Figure 1 presents the relative frequency of the top-10 most common lemmas in the dataset (bar height), and the split between explicit and implicit "that" SCONJ usages immediately following those main verbs. In particular, the findings in Figure 1 imply that the lemma alone does not carry sufficient predictive power about the potential syntactic reduction in subsequent subordinate clause.

heading: 3 Methodology
text: We define a set of factors that we were found to affect syntactic reduction choices (Levy and Jaeger, 2006; Jaeger, 2010), and further study the magnitude of their predictive power by casting the usecase as a classification scenario. We harness the power of contemporary LLMs for reliable computation of SC onset surprisal, as well as for computation of its complementary predictor: SC onset entropy. We define the following predictors:
Main clause (MC) length Previous work suggested that the conjunction is likely to be spelled out explicitly in longer sentences; in particular after
a longer main clause. This predictor is computed by the number of tokens preceding the (explicit or implicit) SCONJ. As an example, in the sentence "Do you realize [that] I’ve never actually seen him at the office?", MC length will be assigned 3.
Subordinate clause (SC) length Similar intuition suggests that the length of a subordinate clause (and more generally, the rest of the sentence) can be used as another predictor. In the example sentence above, SC length will be assigned 9.
Main verb frequency Jaeger (2010) found negative correlation between the main clause verb frequency and the tendency to spell out "that" SCONJ. We compute the frequency of main verbs in all sentences as their relative count in the entire corpus of over 480K sentences (see Section 2).
SC subject distance This predictor is defined as the number of words at the SC onset up to and including the SC subject. Multiple studies found positive correlation of this factor with the tendency to spell out SCONJ (Hawkins, 2001, 2004; Jaeger, 2010). We extract the SC subject using the nsubj annotation assigned by spacy’s dependency parser to the subordinate clause subject.
SC onset information density (ID) Levy and Jaeger (2006) and Jaeger (2010) computed this factor by using the simplest possible estimation, where the information of the SC onset is only conditioned on the main verb, and is operationalized by the notion of surprisal: –log p(SC onset | main verb). All counts (and probabilities) were calculated from the dataset at hand. Harnessing the power of modern pretrained LLMs, we define this predictor as the probability of SC onset, conditioned on entire main clause, namely –log p(SC onset | MC).
Notably, Levy and Jaeger (2006) trained the bigram model in a controlled setting where all "that" conjunctions had been omitted. Without this control, results may be circular, e.g., in cases where "that" is explicitly spelled out, the computation –log p(SC onset | MC) could be
self-evident because "that" is normally inserted between MC and SC onset (recall that SC onset denotes the opening word of the subordinate clause, "that" excluded). Since training a language model from scratch on corpora with omitted SCs is often impractical, we marginalize out the presence of "that", re-defining the SC onset surprisal to be:
–log ( p(SC onset | MC) + p(SC onset | MC ◦ "that") )
This refined definition of SC onset surprisal eliminates the need to re-train a language model on a corpus where the SC "that" had been omitted.
SC onset entropy We argue that the information density of the subordinate clause onset can be extended by the complementary notion of entropy – the expected value of the surprisal across all possible SC onsets: 𝐻(𝑝)=−Σ𝑖𝑝𝑖 ∗ log(𝑝𝑖); for a given main clause MC, 𝑝𝑖=𝑝(𝑤𝑖 |𝑀𝐶), where 𝑤𝑖 is the 𝑖𝑡ℎ word in the model’s vocabulary 𝒱. For a certain sentence prefix, entropy calculation involves the computation of the probability distribution over the model’s vocabulary 𝒱 for next word prediction. While the computation is practically impossible with a small corpus and an N-gram LM, this information is easily obtainable from pretrained LLMs. Although conceptually related, SC onset entropy and SC onset surprisal were found to be uncorrelated in our dataset: Pearson’s 𝑟 of -0.02 was found between these two predictors.
Other predictors Among additional factors investigated in prior studies are (1) SC onset frequency, (2) SC subject frequency, (3) the distance of the main verb from the SC onset, and (4) SC ambiguity ("garden path"). The first two factors were found to moderately correlate with SC onset surprisal (Pearson’s 𝑟=-0.57) in our experiments, and hence omitted from the predictor set – not a surprising finding given that in 84.5% of cases SC onset is also the SC subject. The third predictor turns irrelevant in our experimental setup, where SC immediately follows the main verb. Finally, and most notably, Jaeger (2010) manually annotated their sentence set for SC ambiguity ("garden path"), and found this factor non-predictive of "that" omission; we, therefore, refrain from using this predictor here due to the manual effort required for "garden path" annotation in our ample data.

heading: 4 Experimental Results and Discussion
text: Experimental Setup We use the OPT-125m autoregressive pretrained transformer model (Zhang et al., 2022), roughly matching the performance and sizes of the GPT-3 class of models, for computation of SC onset surprisal and entropy. Given a sentence prefix, we first extract next token logits and convert them to a probability distribution over the lexicon by applying the softmax function. SC onset surprisal was computed by applying the natural log on the SC onset token probability given the relevant sentence prefix. SC onset entropy was computed by applying the entropy equation (see Section 3) on the outcome probability distribution.3
Estimating the contextual surprisal (or entropy) per word with decoder LLMs operating at the subword level is hard; we, therefore, approximate these metrics by computing the surprisal (or entropy) over the subwords. Pimentel et al. (2023) show that this is practically equivalent to computing a lower bound on the true contextual measurements.
Finally, logistic regression is used as a predictive model due to its effectiveness and intrepretability.
Experimental Results Our main results are presented in Table 3. We report two scenarios: (1) all main verb lemmas preceding the SC are considered, and (2) only sentences with the most-frequent "think" main verb lemma are considered. Using these two different experimental setups, we test whether observations evident for the full set of main verbs, also emerge in a single main verb scenario. All predictors are standard-scaled for comparative analysis. The effectiveness of our predictors is supported by the considerable (in particular, much higher than chance) classification accuracy in both cases: 0.63 when using all main verbs, and 0.88 when using the "think" verb lemma only.
Analysis and Discussion Several observations emerge from the table: inline with prior studies, sentence length – manifested in both MC and SC – has significant positive effect on the explicit usage of "that" connecting the two clauses. One of the highest (absolute value) coefficients is assigned to SC onset surprisal, confirming the findings by Jaeger (2010). The UID hypothesis is further strengthened by the high (the highest in the all
3Experiments with larger OPT models and decoder models from additional model families resulted in similar findings, while less efficient (higher latency). We, therefore, adhere to our choice of advanced, yet relatively small, model.
MC verb lemmas case) coefficient assigned to SC onset entropy; that is, SC onset (non-)predictability can be viewed in a more holistic manner, where both the low predictability of the specific SC onset and the high entropy of the potential sentence continuation, carry over complementary and uncorrelated predictive power on syntactic reduction decision. The overall picture remains consistent in the scenario where the single lemma "think" is considered (albeit SC subject distance shows insignificant), implying the robustness of our findings.
Our main findings are further strengthened by the illustration in Figure 2. Kernel density estimation of SC onset surprisal with explicit "that" usages is shifted to the right (A), reflecting the lower predictability of SC onset in this cases compared to those where "that" was omitted. This observation stays sound when only "think" main verb is considered for experiments (B). Sub-figures C and D depict the complementary entropy plots – higher SC onset entropy in explicit "that" usages is mirrored by the right shift of the red line in both full main verb set and "think"-only cases.
The definition of surprisal inherently implies the correlation of SC onset surprisal with its frequency. Indeed, these two factors exhibit moderate negative correlation for both all lemma set and "think" lemma only (Pearson’s 𝑟 of -0.57 and -0.47, re-
spectively). Replacing SC onset surprisal with its frequency resulted in a slightly weaker regression model in our case, suggesting that surprisal introduces additional predictive power beyond frequency. While surprisal and frequency are highly correlated, they are typically associated with different psycholinguistic behaviours, and we leave a more thorough investigation for future work.

heading: 5 Conclusions
text: We study the UID hypothesis manifestation in syntactic reduction using a large, diverse and carefully compiled corpus of English sentences with explicit or implicit "that" subordinate conjunction. Harnessing the power of contemporary pretrained LLMs, we show that SC onset surprisal and entropy are the main factors affecting a speaker’s choice to spell out the optional conjunction "that".
Last but not least, a large body of linguistic literature has studied the conditions under which complementizers (like "that" subordinate conjunction) can or cannot be omitted (inter alia Erteschik-Shir (1997); Ambridge and Goldberg (2008)). We believe that future work in this field should better engage with this literature, incorporating insights for more linguistically-informed approach to the task of syntactic reduction analysis.

heading: 6 Ethical Considerations
text: We use publicly available data to study the manifestation of UID in syntactic reduction. The use of publicly available data from social media platforms, such as Reddit, may raise normative and ethical concerns. These concerns are extensively studied by the research community as reported in e.g., Proferes et al. (2021). Here we address two main concerns. (1) Anonymity: Data used for this research can only be associated with participants’ user IDs, which, in turn, cannot be linked to any identifiable information, or used to infer any personal or demographic trait. (2) Consent: Jagfeld et al. (2021) debated the need to obtain informed consent for using social media data mainly because it is not straightforward to determine if posts pertain to a public or private context. Ethical guidelines for social media research (Benton et al., 2017) and practice in comparable research projects (Ahmed et al., 2017), as well as Reddit’s terms of use, regard it as acceptable to waive explicit consent if users’ anonymity is protected.
We did not make use of AI-assisted technologies while writing this paper. We also did not hire human annotators at any stage of the research.

heading: 7 Limitations
text: We believe that the main limitation of this work is the relatively restrictive experimental setup of sentences used to study UID principles in syntactis reduction. As an example, additional syntactic setting of interest includes sentences where "that" is used as a relative conjunction, as in "the book [that] I read last week made me quite sad...". Due to its much lower frequency in our data, we leave the investigation of "that" omission before a relative clause to future work.
The current study also limits its set of main clauses to those where the SCONJ immediately follows MC verb, not considering cases like "My boyfriend has mentioned several times [that] we should approach this guy with the offer", where the main verb "mentioned" is separated from the SC onset "we" by the "several times" phrase. However, we have reasons to believe that similar findings would be evident in these scenarios, and plan to extend the research to those cases as well.

heading: Acknowledgements
text: We are grateful to Shuly Wintner for much advice during the early stages of this work. We are also
thankful to Alon Rabinovich for his help with the annotation effort for this study.

heading: A Appendices
text: A.1 Identification of Sentences with Optional "that" Subordinate Conjunction Figures 3 and 4 depict two parsing trees of sentences with explicit and implicit usage of "that" SCONJ, respectively. After parsing a sentence, a set of rules was applied for identification of cases where "that" is used (or could have been used) in the role of subordinate conjunction connecting main and subordinate clause. As mentioned in Section 2, the extraction process was tuned for accurate (over 95%) performance.

heading: Future Work
text: Due to its much lower frequency in our data, we leave the investigation of "that" omission before a relative clause to future work. The current study also limits its set of main clauses to those where the SCONJ immediately follows MC verb, not considering cases like "My boyfriend has mentioned several times [that] we should approach this guy with the offer", where the main verb "mentioned" is separated from the SC onset "we" by the "several times" phrase. However, we have reasons to believe that similar findings would be evident in these scenarios, and plan to extend the research to those cases as well