Quantifying the redundancy between prosody and text
Authors: Lukas Wolf
Tiago Pimentel
Evelina Fedorenko
Ryan Cotterell
Alex Warstadt
Ethan Wilcox
Tamar I Regev
Venue: EMNLP 2023 Main (2023)
Keywords: Prosody
Psycholinguistics
Language Models
Information Theory
TL;DR: Using advanced language models, we estimate how much prosodic information is implicitly encoded in text and how much is redundant with the linguistic content

Abstract: Prosody---the suprasegmental component of speech, including pitch, loudness, and tempo---carries critical aspects of meaning.
However, the relationship between the information conveyed by prosody vs. by the words themselves remains poorly understood. We use large language models (LLMs) to estimate how much information is redundant between prosody and the words themselves. Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings. We find a high degree of redundancy between the information carried by the words and prosodic information across several prosodic features, including intensity, duration, pauses, and pitch contours. 
Furthermore, a word's prosodic information is redundant with both the word itself and the context preceding as well as following it.
Still, we observe that prosodic features can not be fully predicted from text, suggesting that prosody carries information above and beyond the words. 
Along with this paper, we release a general-purpose data processing pipeline for quantifying the relationship between linguistic information and extra-linguistic features.

Quantifying the redundancy between prosody and text
Lukas Wolf
Tiago Pimentel9. Evelina Fedorenko" Ryan Cotterell
Alex Warstadt
Ethan Gotlieb Wilcox' Tamar I. Regev"
'ETH Zürich "MIT Universsity of Cambridge
{wolflu, ryan. cotterell, warstadt, ethan, ethan.wilcox cc tp472@cam ac. uk
(evelina9, tamarr]emit edu
Abstract
frequency (fo).² It carries both linguistic informa- tion, such as lexical stress, and discourse function, e.g., extra-linguistic information, such as social and emotional cues (Cole, 2015; Ward and Levow, 2021; Hellbernd and Sammler, 2016).
Prosody the suprasegmental component of speech, including pitch, loudness, and tempo- carries critical aspects of meaning. However, the relationship between the information con- veyed by prosody vs. by the words themselves remains poorly understood. We use large lan- guage models (LLMs) to estimate how much information is redundant between prosody and the words themselves. Using large spoken cor- pus of English audiobooks, extract prosodic features aligned to individual words and test how well they can be predicted from LLM em- beddings, compared to non-contextual word embeddings. We find a high degree of redun- dancy between the information carried by the words and prosodic information across several prosodic features, including intensity, duration, pauses, and pitch contours. Furthermore, word's 's prosodic information is redundant with both the word itself and the context preceding observe that
Previous studies have suggested partial redun- daney between the acoustic features of prosody and linguistic information. For instance, several prosodic features, c.g., pitch and duration, correlate with the predictability of a word in context (Aylett and Turk, 2006; Seyfarth, 2014; Malisz et al.. 201 8; Tang and Shaw, 2021; Pimentel et al., 2021) and can mark the location and type of focus in sentence, emphasizing which information is novel (Breen et al., 2010). These studies suggest that information signaled by prosody is redundant with information from the preceding text, arguably implying that prosody is largely used to direct the interlocutor's 's attention. However, in some cases, prosody seems essential to transmit unique infor- mation, like marking irony, resolving ambiguous syntactic attachments, or transforming statement into question. Therefore, characterizing the relationship between the information conveyed by prosody and by words is essential for understand- ing the role of prosody in human communication.
well as following it. Still, prosodic features can not be fully predicted from text, suggesting that prosody carries in- formation above and beyond the words. Along with this paper. we release a general-puree purpose data processing pipeline for quantifying the re- lationship between linguistic information and extra-ling features.
Our paper quantifies the redundancy between prosody and text using information-theoree techniques. We operationalize redundancy mutual information (Shamnon, 1948, 1951), information-theoreer measure that quantifies the amount of information (in units such as bits) obtained about one random variable (e-g.. prosody) by observing the other random variable (e.g., text). We build upon existing work in the emerging field of information-theoreet linguistics (Brown et al., 1992; Smith and Levy, 2013; Wilcox et al., 2020; Shain et al., 2022). Previous research has applied information theory to analyze various facets of language such as word lengths, ambiguity, and syntax (Piantadosi et al., 2011, 2012; Futrell al.,
https: //gg::///tttbbb com/ com/lu-wo/ quantifying- redundancy
1 Introduction
In human communication, information flows between interlocutors along multiple channels. Apart from linguistic information conveyed by the text! itself, meaningful content is also transmitted through facial, hand, and body gestures as well as through prosody. In spoken language, prosody consists of the acoustic features of the speech signal such as energy, duration, and fundamental
WWe use the word text to refer to all the linguistic informa- tion that is expressed in words (i.e., segmental information). We are not specifically interested in written texts; for spoken signal, the text would be the information left hehind after the prosodic information has been removed.
'The fundamental frequency is defined as the lowest fre- quency of periodic waveform, and gives rise to the percep- tion of pitch.

---

Types linguistic context W:
Past context
Current word
Future context
Bidirectiona context
W
We
W
W +
+-
What
shall
do
without
Non-contextual word embeddiigs (egg FastTes))
Prominence
(a:0)
Extracted prosodic Teatures
prominence
(0:0
Bidirectional L.M. (c BERT)
Antoregresive LLM GPT-2)
Energy
Duration (seciyllatte)
Pauses
b403
(x)))
8A
0.6
0.5
12
Mutual
Time (wcords))
information:
Po(pJ
P
PolP, [w]
MI(P;, w) H(P)) (P, (W) Current prosodic feature
Figure I: Illustration of our pipeline for estimating the mutual information between prosody and text. The pipeline begins with the extraction of word-level prosodic features. For each type of linguistic context W, we use corresponding language model to estimate p(P, W) by learning mapping from model embeddings to the prosodic feature. We further estimate the probability distribution of the prosodic feature p(Pt). We these probabilities to estimate the entropy and conditional entropy of the prosodic feature. Finally, employing the standard decomposition mutual information, we arrive an estimate by utilizing two computed entropies. See Sections 2-4 for further detail. stands for arbitrary units when prosodic features are 2-scored.
2019; Meylan and Griffiths, 2021; Levshina, 2022: Pimentel et al., 2023). Yet there is relatively little work specifically characterizing the high-level statistical relationships between prosody and text.
between prosody, and either the current word, its preceding context, or both its preceding and following context, respectively. As an additional contribution, we publicly release the code we de- veloped to implement our data- sutaa-poocessin pipeline. This code and the estimation tools we introducc here are general enough be applied to other studies on prosody and text, as well interactions of other communication channels with text.
In this study, we employ large language models (LLMs) to estimate the uncertainty about prosody given text, and further compute the mutual information between prosody and text. We work with corpus of English audiobooks, align the transcript the speech signal, and extract for each individual word set of prosody-relevant acoustic features, namely intensity, duration, pitch curves, prominence (a composite measure of the intensity, duration and pitch Suni et al. 2017; 7; Talman et al. 2019) as well as any pausal information that may occur between words. We then finetune variety of powerful, openly available LLMs to predict the prosodic cues aligned to each word (Figure 8). By relying on different LLMs to make these pre- dictions either non contextual,, autoregressive, bidirectional we measure the redundancy
Redundancy as mutual information
Our objective is estimate the redundancy be- tween text and its accompanying prosodic fea- tures. As first step, we define the notation we will use in this section. Let Σ be an alphabet, a finite, non-empty set of symbols. We will con- sider a text to be string, i.e., element of Σ', and we will denote it as w. With W, we denote valued random variable. In the experimen- tal section of this work, will encounter text that is either one word several sentences total- ing no more than 100 words. We further consider prosody to be real-valued vector P, E Rd and we denote a prosody-valued random variable as P,.
"As we explain later, these non-contextual models use multi-layer pereeptrons on top of con-connextualized word embeddings.

---

The subscript i marks the distinguished position of the word in text whose prosody we wish to in- vestigate. In contrast to Pt, the portion of text that W corresponds to is currently under-specified. As detailed later in $3, we consider text either before, after, or at position t (see Figure 1).
to estimate the entropy of P. given W ( (4.1). Each sentence in these books has single recording. and thus, we only have a single measurement P, for each unique value of W in our dataset. This creates the necessity to share parameters across texts W to estimate the variance of Pt and hence H(P, W) on held-out data. By sharing param- eters, we mean that estimate H(P, W) for all W values via one function, which in our case language model (as explained $2.2 Eq. (5)). Why is parameter sharing suitable in our case? In language, unlike many other domains, distinct word sequences share inherent properties. For instance, the sentences I love dogs and / love cats hint at functional and semantic connections between dogs and cats. Thus, given prosodic values from the dog-example, we may be able to infer the prosodic feature variance of the word cats. By using pretrained neural language models-and hence their implicitly encoded lin- guistic knowledge- we share the neural network's ss parameters across linguistically related examples to make predictions about their prosodic features.
We now offer a quantitative treatment of the re- dundancy between prosody and text, operationaliz- ing it the mutual information:
MI W)
(1)
E p(p, w) log p(Pr, w) dp, WEE+
Σ
p(P,) p(w)
We next discuss a few properties this mutual information, and how we estimate it.
2.1 Estimation challenges
Our mutual information of interest, in Eq. (2), is defined as function of discrete (E*) and continuous (Rd) random variable. Unlike in the case where the random variables are both discrete or both continuous, the mixed-pair case does not inherently obey many properties one associates with mutual information (Gao et al., 2017), and is therefore challenging to estimate. For instance, it is generally not guaranteed that:
2.2 Estimation through cross-entropy
In this section, we explain how, given the data scarcity problem, we get estimates of the mutual information between prosody and text. Under the good mixed-paid pair assumption, we estimate the mutual information MI(P;; W) as the difference between two cross-cntropics Hg (McAllester and Stratos, 2020):
MI(P;: w) = H(P,) - H(P, W)
(2)
which many estimation methods including own, rely on. However, Beknazaryan et al. (2019) give a simple sufficient condition for the decomposition Eq. (2) to hold, which they term the good mixed-pair assumption." 5 Under this assumption, which we suggest is reasonable to assume in our case where prosodic features are well-behaved and we can decompose the mutual information shown in Eq. (2). For more details on the challenges of mixed-pair mutual information estimation we refer to App. A.I.
MI(P& W) H(P,) H(P, W) 2 HP.),) - Ho(P, W)
(3a)
(3b)
which leaves us with the two differential cross- entropies in (3b) to estimate. These cross-entropies are defined as the expectation of - log Po Po (P,) with respect to the ground truth probability of the prosodic feature p(P,) or, in the case of the conditional cross entropies - log po | w) and p(P;, w).² order approximate these expectations, we use resubstitution estimation (Hall and Morton, 1993; Ahmad and Lin, 1976). using a set of held-out samples from our dataset.
Another challenge for estimating mutual information in our case is due to the nature of W. Several mixed-pair estimation methods, including Beknazaryan et al. (2019), may not apply to our case because they require estimating H(P, W = w) for each given value of W sep- arately; see App. for more details. However, use text from books and their spoken counterparts
"Note that since the ground-truth probability is a prob- ability density function, it can (unlike probability mass func- tions on discrete units) attain values larger than 1. This means that differential entropy can take on negative values and is not bounded from below. The mutual information, however, is still always non-negative.
'Note that MI(P: i W) PI).
'The good mixed-pai assumption requires that p(P, [w) he absolutely continuous with respect to p(p,) for all E".

---

Our cross-entropy estimates are thus:
uncontextualized word embeddings, as well as autoregressive and bidirectional language models, We discuss each of these setups, briefty, below.
1 N
Ha(P,) (p(p, (n))
log
(4a)
1=
3.1 Context types
HP W)
log
(4b)
Current Word We. Given only the target word at index t of a text, estimate the mutual infor- mation between Wt and its word-level prosodic feature P:: MI(P;; Wt). This lets us quantify how much information about the prosodic feature is encoded by the word type alone. To estimate the predictive distribution po (P; | w,)-on top of the uncontextualized word embeddings-we train a multilayer perceptron that predicts the parameters & of the predictive distribution. For more details, refer $4.3.1.
pe(p, [m) w(n))
where N is the number of held-out samples. Impor- tantly, the cross-entropy upper-bounds the entropy. It follows that the better our estimates Po of p. the closer our cross-entropies will be to the actual en- tropies, and the better our estimation of the mutual information should be (Pimentel et al., 2020).
Estimating (p:). We approximate the true dis- tribution p(P,) with Gaussian kernel density esti- mate (Sheather, 2004). For details on this estima- tion method, we refer the reader to App. A.2.
Past context W.. Given the target word and its preceding words, we estimate the mutual information between the autoregressive context and the prosodic feature P, of the target word: MI(P;; W-). We finetune an autoregressive language model (such as, c.g.. GPT-2) estimate pe (P; w.)), which we describe in $4.3.2.
Estimating pe (P; w). In order to approximate this conditional probability density function, we use neural language model to learn the parame- ters of predictive distribution, which is defined here as either Gaussian Gamma distribution over the prosodic feature space Id..
Bidirectional context W Given the target word and the full context of the text sequence (i.e., the preceding as well as following context), we estimate the mutual information between the bidi- rectional context and the target word's prosodic feature Pf: MI(P;; We+). We finetune a bidirec- tional language model (such as. e.g., BERT) to es- timate po (P W++), which we describe in further detail in $4.3.2. Using the estimators above and the definition of conditional mutual information we can also infer the information shared exclusively between the future context W and prosody, but not contained in the autoregressive context, by sub- traction of two mutual informations:
Φ = LMM (w) (p w) = (p;;
(5a)
(5b)
where we denote the predictive distribution with Z and its parameters with o. The neural network is itself noted LMg. As explained in $2.1, this allows us share parameters across w, where only have one example for each in our finite dataset. We note that LMg should learn to output similar values & when input similar sentences w. thus sharing parameters across sentences. However, since we use parametrized distributions Z to estimate (P; w), our estimator of the conditional entropy is not consistent (meaning that it might not converge the true value N + oc).
MI(PA: W_ W-))
(6)
MI(Ps; -M>) W.-
3 The role of context
4 Our prosody pipeline
While in $2 we deliberately kept the text-valued random variable W underspecified, we now intro- duce it more precisely to quantify the redundancy of prosody and different parts of the text. We measure the redundancy of prosody at a word I, which write as Pt. with three different text- valued random variables: a single word Wr, all past words W and and all other words in a sentence W We then estimate these measures using
In this section, we describe the data processing pipeline we use to extract the prosodic features energy, duration, pause, and pitch curve from word- level text-speech alignments.
4.1 Dataset
Our experiments use the LibriTTS corpus (Zen et al., 2019), created explicitly for text-to-speech

---

research. This corpus was developed from the au- dio and text materials present the LibriSpeech corpus (Panayotov et al., 2015) and contains 585 hours of English speech data at a 24kHz sampling rate. The dataset includes recordings from 2,456
Energy Energy quantifies intensity variations in the speech signal which are affected by sub-glottal pressure (i.e., the pressure of the air passing from the lungs to the vocal folds) as well as other ariiulatory-phonntie factors (for example, producing stop consonants, c.g., [k] or [p]). We quantify these intensity variations as energy and compute its values by first applying bandpass filter' and then computing the logarithm of the amplitude of the filtered acoustic signal (see Suni et al. (2017)). We consolidate the word's energy into a scalar feature by averaging all sample points between the word boundaries. Specifically, we sum up the log amplitude values at each acoustic sampling point (we use 24kHz samples, see $4.1 1) and divide the sum by the number of sampling points to obtain scalar measure.
speakers and the corresponding transcripts. We use the data split provided by Zen et al. (2019)
(see Table 2). Further, we only consider texts from LibriTTS that contain at least two words (since Lib- riTTS contains fragments like the book and chapter titles, e.g.. "Chapter IX").
In order to extract word-level prosodic features, we derive word-level alignments between audio and text using the Montreal Forced Aligner (McAuliffe et al., 2017). In addition to the word- level aligmment, we compute phoneme-level align- ments to locate word's stress syllable, which is necessary for extracting pitch contours (see $4.2).
Duration and pause We measure the duration of words and pauses separately. Word durations are computed from word-level text-speech alignment boundaries, and are normalized by a word's syllable count, obtained from the CELEX dataset (Baayen et al., 1995), to yield scalar capturing the average syllable duration. In the case that a following pause exists, we measure the interval of silence between word's offset and the next word's onset in our text-audio alignments. The pause duration is 0 for 89.4% of the words in our dataset.
4.2 Representations of prosodic features
Given the spoken utterances from the LibriTTS dataset, we need to extract the speakers' prosody. Prior work (Ye et al., 2023) has used the complete audio signal, extracting information about prosody only through contrasting identical words across different text-speech pairs. The full audio signal- due to the data processing an upper bound on prosody-related information, but also contains information unrelated to prosody, such as phonetic features. In order to isolate suprasegmental prosodic elements specifically and get better estimate of the mutual information between prosody and text, we instead choose extract specific acoustic features considered relevant for prosody based on previous research. We concentrate on three basic acoustic properties: Energy, duration, and fo- We also examine prosodic prominence as composite signal of energy, duration, and fo. which was suggested to capture functionally relevant supra-semmetie stress patterns in English efficiently (Suni al., 2017; Talman et al., 2019). We further examine the duration of a potential pause after a word, which has been discussed previously as an important cue for grammatical boundaries (Hawkins, 1971).
-prvvidd
Pitch Whereas other prosodic features can be represented as single scalar values without losing much information, it is well-established that the en- tire pitch contour contributes to the meaning (c.g., Pierrehumbert and Beckman, 2003). We apply the fundamental frequency (fo) preprocessing methods detailed in Suni al. (2017), which include fo ex- traction from raw waveforms, and outlier removal. Additionally, interpolation is applied time points where the estimation algorithm fails due to silence or unvoiced phonemes leading to pitch-less sounds. Rather than looking at the fo curve over the whole word, we focus on the area around the word's primary stressed syllable. This is motivated by the fact that English pitch accents anchor to the stressed syllable, and the same underlying pitch accent may be realized with phonologically irrele- vant regions of the flat pitch in polysyllabic words (Pierrehumbert, 1980). We identify the stressed syllable using our phoneme level alignments and
We used the English (US) Arpa acoustic model and dic- tionary (McAuliffe and Sonderegger, 2022).
"The data- proceerrccesing inequality (Cover and Thomas, 2005, Chapter 2) states that for any function f(+), e.g., one that extracts prosody from audio (f(A.) = P.), we have: MI(A,; W) 2 MI(P;; W).
"The handpass filter removes all frequencies smaller than 300Hz and larger than 5000Hz.

---

stress patterns from CELEX (Baayen et al., 1995) and extract up to 250ms of audio signal (less if the interval to word onset or offset is shorter, which often the case) on either side of the stressed sylla- ble's midpoint." 10
parameterize predictive distribution over the prosodic feature space (c.g., a Gaussian or a Gamma distribution)" and are optimized via stochastic gradient descent using a cross-yntropy loss. We choose our model's hyperparameters, i.c., learning rate, Le-regularization, dropout probability, number of layers, and hidden units. with 50 iterations of random search.
Instead of working with the raw fo curves, we translate them into a low-dimensional space. We first resample the fo curves to 100 sampling points and parameterize them with k coefficients of discrete cosine transform (DCT). For additional experiments, see App. B.4.
4.3.2 Contextual estimators
For the distributions conditioned on multiple words. we use a fine-tuning setup to approximate po (P, W w+-) and pe (e>)- In our setup, autoregres- sive models po (P, | w-) can be thought of as useful for modeling incremental language compre- hension, where the model's predictions at target word are only affected by the preceding context. We use GPT-2 (Radford al., 2019) models rang- ing from 1 117M to 1.5B parameters for our autore- gressive models LMg(w-). We fine-tune these LMs to parameterize a predictive distribution over the prosodic feature space using a cross-entropy loss. After fine-tuning several models in this way, we select the best one based on their cross-entropy loss on validation set. We use the pre-defined Lib- riTTS data split for training, validation and testing; see App. B. for details.
Prominence We use continuous word-level prominence values using the toolkit provided by Suni al. (2017), combining energy (intensity), fo and word duration into single scalar promi- nence value per word. The authors perform a co- sine wavelet transform on the composite signal of energy. fo, and duration and extract prominence as lines of maximum amplitude in the spectrum. Note that the processing of word duration, energy. and fo is distinct from the procedures introduced in the previous sections (see (Suni et al., 2017) for details) and we do not investigate the relative weighting of these features. Since the perception of prosodic prominence is relative to the prominence of the preceding words, we add another measure of the change in prominence relative to the average prominence of the three previous words. We refer to this relative prominence.
Bidirectional models are trained to predict prosody given information about both their left and right contexts, (o Wes). Comparing their performance to the autoregressive models can therefore teach us specifically about the gain in mutual information between prosody and text con- tributed by the words following the target word. This analysis allows us to quantify the extent to which prosody, itself, can be used in prediction dur- ing language processing, by revealing how much information it contains about upcoming material. We consider bidirectional LMs from the BERT- RoBERTa- and DeBERTa families (Devlin et al., 2019; Liu et al., 2019; He al., 2021) ranging from 110M to 1.5B parameters to implement L.M (W++) As with autoregressive models, we fine-tune the pretrained bidirectional LLMs to estimate the pa- rameters of a parameterized distribution using a
4.3 Word representations
We choose to use orthographic representations for segmental information. Although there are other options, such as various phonetic transcriptions, this choice enables us to easily embed this informa- tion into a representational space using pretrained language models. We next present the implementa- tions for estimating the distributions conditioned on the three types of context discussed in $3, namely, distributions conditioned on word types, autore- gressive contexts and bidirectional contexts.
4.3.1 Non-contextual estimators
We estimate (p W:) using fastText (Bo- janowski et al., 2017) and multi-layer perceptron (MLP) to predict the parameters of the predictive distribution of a prosodic feature. Our MLPs
Il Note that for the prosodic features introduced in $4.2 we distinguish between two types. Prominence, energy, duration, and pause can be represented by a scalar value p, E R. While energy and duration are parameterized by a Gaussian, we chose to parameterize prominence and pauses by a Gamma distribution due to their heavy tails. The pitch curves are represented by 8 coefficients of the discrete cosine transform of the curve E R and is parameterized by a Gaussian.
10 "Onn potential shortcoming of this procedure is that it will zero in on pitch events around the stressed syllable. In English. this will capture stress patterns but will potentially miss bound- ary tones occurring primarily at the end of prosodic phrases. We leave the investigation of boundary tones to future work.

---

Mean and standard deviation of mutual information between text and prosody
types:
word
context
Bidirectional
10
0.6
0.6
0.6-
Mean MI (cats)
0.4
0.4
0.4
0.4
02-
0.2
0.2
0.2
0.2
Prominence
Relative prominence
Duration
Duration
Pause
FF0 contours
Figure 2: Estimated mutual information MI(P;; W) = H(P)) H(P, W) for all models and features with mean and standard deviation from runs of cross-validation. We fix the scale for the prosodic features prominence, relative prominence, energy, duration, and pause and report the fo (pitc) results on separate axis due to its higher mutual information. Note that the differential entropy of fo curves (see Table 1) is much higher than the differential entropy of the other prosodic features. We report the mutual information in nats.
cross-entropy loss. As above, we select the best model based on their cross-entropy loss.
dicted to some extent by the word type alone. Im- portantly, contextual estimators have higher mutual information values compared to the non-contextual estimators for all prosodic features, indicating that the surrounding words improve prosody predic- tions compared to just the individual word type. Furthermore, bidirectional estimators show higher mutual information values compared to autore- gressive ones for all features apart from the pitch curves, which only show very small increase. Below, we discuss each of the prosodic features in turn, moving from left to right across the facets. With respect to absolute prominence, we find the least difference between the non-contextual esti- mator and both the contextual ones. This indicates that much of the information about absolute promi- nence is predictable from word type alone. This is not true, however, when we look at relative promi- nence, where the bidirectional estimator reveals more than 2.3 times as much information (0.48 nats) than the non-contextual estimator, fastText. We see overall lower mutual information for energy than for any of the other prosodic features, suggest- ing that energy is less tightly connected to segmen- tal information. Like with relative prominence, we observe much higher estimated mutual information for the contextual estimators. and, interestingly, here, we observe one of the largest differences between our bidirectional and autoregressive estimators (with 66% more information obtained in the
5 Experimental results
5.1 Correlates of prosodic prominence
Prior work (Malisz et al., 2018; Pimentel al., 2021) indicates correlation between word duration and surprisal. Prosodic prominence, as defined by Suni et al. (2017), a composite measure of word duration, energy, and fo- We perform a validation on our prosody pipeline to verify that word-level prosodic prominence correlates with surprisal. Using the next-word predictive distribution over the vocabulary of autoregressive language model (GPT-2), find weak, positive correlation between prosodic promi- nence and surprisal, as well as between prosodic prominence and token entropy: see App. B.7.
5.2 Main results
Our primary results measuring the mutual infor- mation can be seen in Figure 2, with the different prosodic features in the different facets. First, when comparing the different context types, our results demonstrate robust pattern across prosodic features: All of the prosodic feature feature-model model pairs result in positive mutual information, indicating the presence of redundancy between prosody and text, Even the non-contextual estimator already pro- duces positive mutual information with all prosodic features, indicating that prosodic cues can be pre-
for duration and pauses are simi- lar. Like prominence, the non-contextual estimator

---

Diff. Entropy
Current Word
Past Context
Bidirectional Context
Feature
H(P,)
H(P; W:)
H(P, | W-)
H(P, | (++)
Prominence
0.536 + 0.012
-0.048 + 0.005
0..144 = 0.002
-0.165 + 0.003
Prominence relative
1.355 + 0.011
1.145 + 0.005
0.924 = 0.003
0.878 + 0.008
Energy
#.000
0.749 0.001
0.618 = 0.004
0.516 = 0.004
Duration (per syl.)
-0.920 0.009
-1.416 #
-1500 = 0.038
-1.657 = 0.036
Pause fo curves
-5.189 + 0.050
-5.537 + 0.033
-5.676 = 0.003
-5.813 + 0.015
11.619 + 0.093
2.477 + 0.009
0.719 + 0.001
0.662 + 0.001
The estimated differential entropy of the H(P:) and the conditional entropy H(P, W) for all prosodic features and word representations. For differential entropy, standard deviations are from bootstraps over 20 folds of data. We choose the best autoregressive and bidirectional model based on the validation loss and report the conditional entropy of these models on the held-out test data. For the non-contextual estimator based fastText, MI.P hyperparameters are optimized via 50 iterations of random grid search, and we report the test set conditional entropy of the best validation model.
Table I:
achieves relatively high mutual information for both, indicating that they can be partially predicted from word-level factors. For both, we see gains for non-contextual
corpus of spoken English audiobooks and its associ- ated text. We found that, consistently across a vari- ety of prosodic features, there exists non-trivial MI between the prosody of given word and the word itself. For all prosodic features, there an increase of the MI between prosody and text when consid- ering the preceding context as well. Furthermore, for most prosodic features, considering the words following the target word further increases the MI with the prosody. Our results quantitatively reveal the partial redundancy between prosodic features and linguistic content extracted from text alone.
autoregressive estimators ones and for the bidirectional estimators over the autoregressive ones. Likely, this due to increased pauses (if there is a pause, i.e. duration of the pause after a word is greater than 0) and increased duration at the end of prosodic phrases, which can be well- preeipeeedeted by the bidirectional estimator, which has access to information about following words well upcoming punctuation.
Finally, for pitch, we find high values of mutual information for all estimators. This is due to the fact that our representation of pitch (i.e.. 8 DCT coefficients) contains more information than the other features, as obtained by our estimates of (un- conditional) differential entropy (Table 1). For the pitch curves, bidirectional context does not contain much more information than autoregressive one, suggesting that the pitch events of a particular word are mostly related to the preceding context and less so to the following context. Representing the pitch curves with more DCT coefficients increased the mutual information with the bidirectional context (Table 3), but we did not run the autoregressive estimators for the different number of DCT parameters check if the non-improvement holds for bidirectional context holds as well.
Our large-scale approach to prosody. Much previous research on prosody concentrated char- acterizing specific prosodic features and identifying their contributions specific linguistic functions. In contrast to these previous studics, our methods facilitated a large-scale systematic examination of the statistical properties of various prosodic fea- tures and their MI with abstract text representations extracted using language models. As such, our ap- proach lacks some interpretability relative to previ- ous studies, as we do not know what are the specific linguistic functions associated with the information that we found to be redundant with certain prosodic features. The latter could be further explored future work by comparing text where prosody pre- dictions were especially high or low. However, a big advantage of our approach compared to pre- vious studies is the quantitative large-scale exam- ination of spoken linguistic corpus, thus poten- tially revealing more mutual information between prosody and linguistic information than was pre- viously possible to show, because the models may extract abstract linguistic representations that were
6 Discussion
We studied the interaction between speech prosody and linguistic meaning conveyed by the words themselves via computing the mutual information (MI) between prosodic features extracted from a

---

not specifically targeted by previous studies.
It important to mention that the order of mag- nitude difference in the mutual information scales between the f0 contours and the rest of the prosodic features (Figure 2) is due to their representation as multidimentional features. We represented the full fo curve with a dimensionality reduction achieved by DCT decomposition and found that 8 DCT components were favorable in terms of MI maxi- mization. Future work can use methodology further explore the prosodic functions f0 curves.
The role of context. The positive MI of the non-contextualized estimators indicates that there is already information in word identity about prosodic features. This indicated that some words tend to have different prototypical prosodic values than other words. This is consistent with past studies suggesting that the cognitive representation of words interacts with their typical prosody (Sey- farth, 2014; Tang and Shaw, 2021). Comparing the MI of prosody with past vs. bidirectional context can give insight into real-time language production and comprehension. For production, the gain in MI for both context types indicates that language pro- duction systems (whether in humans or machines) would be well-served by having representation of the full utterance in order to effectively plan and execute the prosodic features of their speech. As far online comprehension, the results for bidi- rectional contexts suggest two things: First, it is not just the case that linguistic content can be used to predict prosody, but that prosodic information could be used to predict future textual information, insofar as there is redundant information between the two. Second, they suggest that during compre- hension, the meaning of a prosodic event may only be fully perceived at the end of the utterance.
Future work. Exploring the interaction be- tween linguistic communication channels holds deep significance in understanding the multi- dimensional distribution of meaning in language, which extends beyond prosody include facets like gestures and facial expressions. The implica- tions of this understanding are profound. For exam- ple, the excess information contained in prosody could set a theoretical performance ceiling for text- to-speech systems, which are tasked with generat- ing speech prosody from text alone. The integration of prosodic information could aid neural language model training, which, despite excelling in machine translation (Vaswani et al., 2017) and language gen- eration (Brown et al., 2020), predominantly rely on text data. As prosody is fundamental to spoken lan- guage, incorporating may address the current en- deavor to boost language model performance using developmentally realistic data volumes (Warstadt et al., 2023; Wolf et al., 2023).
Differences between prosodic features. Al- though results suggest that most prosodic fea- tures gain a significant amount of information from past context as well as future context (Figure 2), there are some interesting details and exceptions. FO contours as well as relative prominence gain most of their information from past and much less from future context. In contrast, pauses gain most of their information with text from future context and almost none from past context. The relatively larger importance of future context for pauses we interpret as confirming previous work on the rela- tionship between pauses and syntactic boundaries (Hawkins, 1971), where the following words can help reveal whether a current word marks the end of syntactic unit. One potential confounding fac- tor for all these points, however, is the effect of punctuation marks, which are considered in the bidirectional but not the autoregressive estimators. In future research, we plan to disentangle these two to better understand the effect of the following words factoring out punctuation marks, which are not present in spoken communication.
7 Conclusion
We presented a systematic large-scale analysis of the mutual information between variety of suprasegmental prosodic features and linguistic information extracted from text alone, using the most powerful openly available language models. The theoretical framework we developed to estimate mutual information between these two communication channels, as well as our openly available data processing implementation, can be used to investigate interactions between other communication channels. These tools enabled us to quantitatively demonstrate that information veyed by speech prosody is partly redundant with information that is contained in the words alone. Our results demonstrate relationship between the prosody of a single word and the word's identity, as well as with both past and future words.

---

Limitations
tial applications and their ethical implications, we encourage the academic and industrial community to approach the deployment of such technologies with caution and to consider their societal impact.
Our paper has two main empirical limitations. First, as the quantities of mutual information we are inter- ested in are not analytically computable, we rely on estimators. We do not, however, have good sense of the errors associated with these estimators. Fur- ther, as our quantities rely differential entropy, which is unbounded below, our cross-entropy upper bound estimates (as in, e.g., Eq. (12)) could arbi- trarily loose. Increasing the size of the dataset and using the strongest available models is important in order to improve our estimation of the MI between prosody and text. Our work, however, offers a general methodology which future work can apply using potentially improved models-either to study prosody itself, or other linguistic features. Second, our work examines LibriTTS as its main source of data. This dataset is itself composed of audiobooks made available by the Librispeech dataset (Panayotov et al., 2015); and is composed exclusively of English sentences. As this dataset only covers audiobooks, the generalizability of results might be limited given that prosody is highly dependent on speaking styles. Furthermore, our work obviously applies just for English, whereas the role of prosody conveying meaning is highly dependent on the specific language. Future work will apply our approach using more datasets to add other domains of speech and languages. Important questions could be answered by applying our approach to different speech types and languages: For example, this could be a quantitative tool to systematically examine the universality of prosody across languages, or to study the role of prosody in language learning using recordings of child-directed speech.
Contribution Statement
LW, TIR, EGW and AW conceived of the ideas pre- sented in this work. LW, TP and RC developed the mathematical portions of the estimation methods. LW implemented the methods and carried out the experiments. EF contributed to early discussions. LW, TP, EGW, AW and TIR wrote the first draft of the manuscript. All authors edited the manuscript and reviewed the work. EF and RC provided fund- ing for the present work.
A Acknowledgements
BGW would like to acknowledge support from an ETH Postdoctoral Fellowship. TIR would like thank Noga Zaslavski for early discussions. TIR is supported by the Poitras Center for Psychiatric Dis- orders Research at the McGovern Institute, MIT. TIR and EF were supported by the MIT Research Support Committee. All authors would like to thank Jennifer Cole for her helpful insights on pitch curve representation, and Elcanor Chodroff for thorough feedback the phonetics portion of the manuscript. Any remaining errors in the work are surely our own.
References
Ibrehim A. Ahmad and Pi- Pi-Erh Lin. 1976. A nonpara- metric estimation the entropy for absolutely con- tinuous distributions (corresp.). IEEE Transactions Information Theory, 22(3)33723375
Matthew Aylett and Alice Turk. 2006. Language dundancy predicts syllabic duration and the spectral characteristics of vocalic syllable nuclei. The Journal of the Acoustical Society of America, 119(5):0088- 3058.
Ethics Statement
Our research primarily involves the analysis of linguistic data and models and does not directly conflict with ethical norms or privacy concerns. We have used publicly available datasets, which do not contain any personally identifiable information or any sensitive content. However, we acknowledge that research in language modeling and prosody has potential upplications that need to be approached with ethical caution. One such application is text- to-speech synthesis (TTS). Advanced TTS systems, especially when integrated with language models capable of generating human- like speech, could be used for imitating voices. Considering these poten-
R. Harald Baayen, Richard Piepenbrock, and Leon Gu- likers. 1995. The CELEX lexical database (release 2). Linguistic Data Consortium.
Alcksandr Beknazaryan, Xin Dang, and Hailin Sang. 2019. On mutual information estimation for mixed- pair random variables. Statistics & Probability Let- ters, 148:9-16.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.

---

Mara Breen, Evelina Fedorenko, Michael Wagner, and Edward Gibson. 2010. Acoustic correlates of infor- mation structure. Language and Cognitive Processes, 25(7-9):1044-100
Weihao Gao, Sewoong Oh, and Pramod Viswanath. 2016. Breaking the bandwidth barrier: Geometrical adaptive entropy estimation. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Peter F. Brown, Stephen A. Della Pictra, Vincent J. Della Pictra, Jennifer C. Lai, and Robert L. Mercer. 1992. An estimate of an upper bound for the entropy of English. Computational Linguistics, 18(1 1-1).31-40.
Peter Hall and Sally Morton. 1993. On the estimation of entropy. Annals of the Institute of Statistical Math- emafics, 45(1):69 88.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Yanjun Han, Jiantao Jiao, and Tsachy Weissman. 2015,
Adaptive estimation of Shannon entropy. In 2015 IEFE International Symposium on Information The- ory (ISIT), pages 1372-1376.
Peter R Hawkins. 1971. The syntactic location of hesi- tation pauses. Language and Speech, 14(3):277-288.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations.
Nele Hellbernd and Daniela Sammler. 2016. Prosody conveys speaker's intentions: Acoustic cues for speech act perception. Journal of Memory and Lan- guage, 88:70-86.
Jennifer Cole. 2015, Prosody in context: a review, Lan- guage, Cognition and Neuroscience, 30(1-2)11-11
Alexander Kraskov, Harald Stögbauer. and Peter Grass- berger. 2004. Estimating mutual information. Physi- cal Review E, 69(6).
Thomas M. Cover and Joy A. Thomas. 2005. Elements of Information Theory. John Wiley & Sons, Inc.
G.A. Darbellay and I. Vajda. 1999. Estimation of the information by an adaptive partitioning the ob- servation space. IEEE Transactions on Information Theory, 131--1113
Natalia Levshina. 2022. Frequency, informativity and word length: Insights from typologically diverse cor- pora. Entropy, 24(2).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1692.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zofia Malisz, Erika Brandt, Bernd Möbius, Yoon Mi Oh, and Bistra Andreeva. 2018. Dimensions of segmental variability: Interaction of prosody and surprisal in six languages. Frontiers in Communication, 3:25.
Andrew M. Fraser and Harry L. Swinney. 1986. Inde- pendent coordinates for strange attractors from tual information. Physical Review A, 33:1 134-11144 140.
David McAllester and Karl Stratos. 2020. Formal lim- itations on the measurement of mutual information. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 875-884, PMLR
Richard Futrell, Peng Qian, Edward Gibson, Evelina Fedorenko, and Idan Blank. 2019. Syntactic depen- dencies correspond to word pairs with high mutual in- formation. Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pages 3-13, Paris, France. Associ- ation for Computational Linguistics.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text speech align- ment using Kaldi, pages 498-502.
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. 2017. Estimating mutual in- formation for discrete-comtinuous mixtures. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Michael McAuliffe and Morgan Sonderegger. English (us) arpa acoustic model Technical report, https:://ffa-modell 1o/
2022.
v2.0.0.
t aoossossii////glsss TigglssshhusAAAaaauuttiimmmdmee 0hmml.

---

Stephan C. Meylan and Thomas L. Griffiths. 2021. The challenges of large-scale, web-based language datasets: Word length and predictability revisited. Cognitive Science, :5(6):e 12983.
Scott Seyfarth. 2014. Word informativity influences acoustic duration: Effects of contextual predictability on lexical representation. Cognition, 133(1 )::(1):440- 155.
Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. 1995. Estimation of mutual information using ker- nel density estimators. Physical Review E, 52:2318- 2321.
Cory Shain, Alexander Paunov, Xuanyi Chen, Benjamin Lipkin, and Evelina Fedorenko. 2022. No evidence of theory of mind reasoning in the human language network. Cerebral Cortex, 33(10):6299 63199
Claude Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, ((3))379-423.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210.
Claude E. Shannon. 1951. Prediction and entropy of printed English. The Bell System Technical Journal, 30( ()550644
Steven Piantadosi, Harry Tily, and Edward Gibson. 201 I. Word lengths are optimized for efficient com- munication, Proceedings of the National Academy of Sciences, 108(9):35263352.
Simon J. Sheather. 2004. Density estimation. Statistical Science, 1947)5588-597.
Nathaniel Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):00--319,
Steven Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280-29))
Kumar Sricharan, Dennis Wei, and Alfred O. Hero. 2013. Ensemble estimators for multivariate entropy estimation. IEEE Transactions on Information The- ory, 59(7):4374 4388.
Janet Pierrehumbert and Mary Beckman. 2003. The meaning of intonational contours in the interpretation of discourse. In Philip R. Cohen, Jerry Morgan, and Martha E. Pollack, editors, Intentions in Communica- tion, chapter 14. The MIT Press.
Antti Suni, Juraj Šimko, Daniel Aalto, and Martti Vainio. 2017. Hicrarchical representation and estimation of prosody using continuous wavelet transform. Com- puter Speech Language, 12:1123136.
Janet Breckenridge Pierrehumbert. 1980. The phonol- ogy and phonetics of English intonation. PhD Thesis, Massachusetts Institute of Technology.
Taiji Suzuki, Masashi Sugiyama, Jun Sese, and Taka- fumi Kanamori. 2008. Approximating mutual in- formation by maximum likelihood density ratio es- timation. In Proceedings of the Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery ECMIJ/KDD 2008, vol- ume 4 of Proceedings Machine Learning Research, pages 5-20, Antwerp, Belgium. PMLR.
Tiago Pimentel, Clara Meister, Elizabeth Salesky, Si- mone Teufel, Damián Blasi, and Ryan Cotterell. 2021. A cuprisal-durattie trade-off across and within the world's languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 949-962, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Aarne Talman, Antti Suni, Hande Celikkanat, Sofoklis Kakouros, Jörg Tiedemann, and Martti Vainio. 2019. Predicting prosodic prominence from text with pre- trained contextualized word representations. In Pro- ceedings of the 22nd Nordic Conference on Compu- rational Linguistics. pages 281-2900 Turku, Finland. Linköping University Electronic Press.
Tiago Pimentel, Clara Meister, Ethan G. Wilcox, Kyle Mahowald, and Ryan Cotterell. 2023. Revisiting the optimality of word lengths. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Kevin Tang and Jason A. Shaw. 2021. Prosody leaks into the memories of words. Cognition, 210: 104601.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theereti probing for linguistic structure. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computa- tional Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint ar arXiv: 13977.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all

---

you need. In Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Nigel Ward and Gina-Anne Levow. 2021. Prosody:
Models, methods, and applications. In Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the 1 1th International Joint Conference Natural Language Processing: Tutorial Abstracts, pages 26 28, Online. Association for Computational Linguistics.
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Gotlieb Wilcox, Chengxu Zhuang. Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Ad- ina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the 2023 BabyLM Challenge: Sample- efficient pretraining on developmentally plausible corpora. In Proceedings of the 2023 BabyLM Chal- lenge. Association for Computational Linguisties (ACL).
Ethan Gotlieb Wilcox, Jon Gauthier, Jemnifer Hu, Peng Qian, and Roger P. Levy. 2020. On the predictive power of neural language models for human real- time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, page 1707-1713.
Lukas Wolf, Eghbal A. Hosseini, Greta Tuckute, Kle- men Kotar, Alex Warstadt, Ethan Wilcox, and Tamar Regev. 2023, WhisBERT: Multimodal text- audio language modeling 100m words. In Pro- ceedings of the BabyLM Challenge. Association for Computational Linguistics (ACL).
Zhenhui Ye, Rongjie Huang, Yi Ren, Ziyue Jiang. Jinglin Liu, Jinzheng He, Xiang Yin, and Zhou Zhao. 2023. CLAPSpeech; Learning prosody from text context with contrastive language-audio pre-training. In Proceedings of the 61st Annual Meeting of the A Association for Compurational Linguistics (Volume 1: Long Papers), pages 9313-333,, Toronto, Canada. Association for Computational Linguistics.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Jia, Zhifeng Chen, and Yonghui Wu. 2019. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. arXiv preprint arXiv: 1904v:1904.0288. 02882..

---

A Mixed-pair mutual information estimation
A.1 Challenges in mixed-pait mutual information estimation
We now expand on the problem of estimating the mutual information between conrete-contiuuus pairs of random variables. Many common mutual information estimators rely on the decomposition in Eq. (2) (Sricharan et al., 2013; Gao et al., 2016; Han et al., 2015), and might thus not be applicable to mixed setting. Estimators that rely decomposition such in Eq. (2) adhere to the 3H-principle, where the mutual information decomposed into three entropies: MI W) = H(P,) + H(W) H(P; W). Other popular approaches for mutual information estimation include non-parametric kernel density estimators (Fraser and Swinney, 1986; Moon et al., 1995; Darbellay and Vajda, 1999; Suzuki et al., 2008; Kraskov et al., 2004). Many of these, however, were not specifically developed for continuous-diserre distributions.
Due to this issue, a few estimators have recently been proposed specifically for a cittinuuuusddscreet case in which Eq. (2) might not hold. For instance, Gao et al. (2017) propose k-nearest neighbor (A-NN) based estimator for the mutual information, which they prove be consistent for mantinoous-discred distributions, meaning that it converges the true mutual information given enough data. Alternatively, Beknazaryan et al. (2019) give a simple sufficient condition for the decomposition in Eq. (2) to hold, which they term the good mixed-pair assumption. The good mixed-pair assumption requires that p(P, w) be absolutely continuous with respect to p(p,) for all w Under this assumption, we can decompose the mutual information as:
MI(P,; w) Σ / p(P,, w) log p(P: w) ddp, p(p,)
dp,
WESI- RRd
H(P)) - ((w)) W W=w) w)
(7)
WEBx
Beknazaryan et al. then propose to use plug-in kernel density estimators estimate H(P,) and each H(P, W = w) individually and prove such an estimator is consistent. As we explained in $2.1, this estimator may not apply to our case because of the data scarcity we encounter in our dataset.
A.2 Estimating the mutual information with cross-entropies
Estimating the unconditional entropy Differential entropy is an expectation of the negative log probability density. Therefore, we can estimate it by sampling from the true distribution and forming an empirical average of the negative log probability density. This empirical average will converge to the true differential entropy by the law of large numbers. We use Gaussian kernel density estimation (GKDE) (Sheather, 2004) to estimate the probability density function p(P;) of a prosodic feature. Specifically, given a training set Dum = {p,)') (m) w w()))) Nim ~ p(P;, w) we obtain the kernel density estimation at point P; as:
Nua
(p.) = Num h
Σ K 1=1
(pp..p." (n), ,bDmmb
(8)
where h > 0 is a smoothing parameter called the bandwidth and K is a Gaussian kernel. GKDE is non-parametric estimation method, i.c., it does not assume any underlying statistical distribution for the input data. This is particularly useful when the nature of the data is unknown standard distributions.
well modeled by
We choose the hyperparameter configuration that achieves the highest likelihood on a held-out set of data points. After calculating the density estimate pe. sample N points P (x) p(P;) from our test
Too relate this the Gaussian kernel function, consider P, and (*) d-dimensional vectors, covariance matrix. The default choice most existing libraries (e.g.. SeiPy Gaussian kernel density estimation) is to scale the covariance matrix of the dataset with the bandwidth h. The Gaussian kemel is then given as: K K[p,, Pa
to be the d d
(n) N, h)
ppa exp (p) (1) (hE))1(p (p. =)

---

(n) dataset Dest (where Pi are the samples from the true distribution p(p,), and approximate the differential entropy of the feature with Monte- Carlo sampling:
H(P:) - Jec. log
(9)
- - / p(p:) log log
Σ ==1
log pe (n)
N
Note the last R becomes exact as + 00.
Estimating the conditional entropy In order to approximate the conditional probability density function p(P; w), we neural language model to learn the parameters of predictive distribution conditioned some text This allows to share parameters across w, where only have one examples for each in finite dataset. This predictive distribution pe(P, | w) is defined here either Gaussian or Gamma distribution over the prosodic feature space Rdd
Φ LMM(W) (w) (P w) = 3 (p;;
(10)
(11)
where we denote the predictive distribution with Z and its parameters with Φ. The neural network is itself noted as LMg. In order to do so, we can choose any model that takes w as inputs and predicts the parameters of the predictive distribution. For each prosodic feature, thus choose predictive distribution based on model fit (see App. B.3). As we estimate an upper bound the entropy, the better our model the tighter our entropy estimate will be (Pimentel al., 2020). We note that LMg should learn to output similar values & when input similar sentences w, thus sharing parameters across sentences.
We can use the predictive distribution to approximate the conditional entropy H(P; W) by evaluating samples from the dataset. Therefore, we draw N samples from a dataset, assumed to be distributed (p. (n) (n)) p(P,, w), which allows to approximate:
[
H(P, W) = Σ
P(Pr, w) logp(P: wwddp
(12)
WEΣ
Rd
S Σ
p(Pr, w) logre(p: w)dp,
(13)
weI
lpg (m)
w(())
(14)
N
where, again, the last zs is exact N -> 00.
A.3 Validation study
In order to understand how method behaves under the data scarcity that encounter in practice, we benchmark against an established, kernel smoothing-bee bbssed method." 13, We find (see Figure 3) that our estimator consistently underestimates the mutual information between prosody and text, which is consistent with our estimation being a lower bound for the mutual information. Therefore, the present study is more prone to Type II errors (not finding an effect although there one) than to Type errors (incorrectly finding an effect that does not exist), with the latter generally considered to be more severe.
B Experiment supplementary
B.1 LibriTTS dataset
Details about the dataset can be found Table 2.
WW use the mpmi R package which uses kernel smoothing approach to calculate mutual information for comparisons hetween all types of random variables including continuous and discrete.

---

Mutual information of prosodic features (kernel smoothing ours)
2,00
2,50
2,00
2,00
2.00
1.75
1.75
1.73
1.75
Mutual information (bits)
Mutual information (bits)
Mutual information (bits)
Mutual information (bits)
1.90
1.50
1.50
1.50
Mutual information
1:00
1.00
1:00
0.75
0.75
0.75
0.75
0.73
0.50
0.50
0.30
0.50
0.50
0.25
0.25
0.25
0.25
0.25
Prominence
0.00
Relative
Energy
Duration
0.90
Pause
prominence
Figure 3: Comparison of Mutual Information Estimates for Prosodic Features: Kernel Smoothing Estimate Ours. Notably, our method consistently underestimates the MI across all features. Note that we could not use the mpmi R package to estimate the mutual information between fo and text.
Split
Samples
# Words
Train
116.2K
2378K
Dev
5.7K
114K
Test
4.8K
103K
Table 2: The number of samples and words in the LibriTTS dataset. We the train-clean-360 training split.
B.2 Details on the prediction task
In order to compute the conditional entropies by fine-tuning large language models (LLMs), treat the problem of predicting prosodic features as token tagging task. We add an additional tagging head on the LM embeddings in order to predict the parameters the predictive distribution. Since - depending on the tokenizer a word might be represented by the tokens the subwords that constitute it; we predict the prosodic feature of a word for all its tokens. The objective function only applied to the last token of a word. We fine-tune the LMs by minimizing the negative lig-likelihood of the target values under the predictive distribution.
B.3 Visualizations of kernel density estimates
Figure 4 shows the estimated distributions of the two prominence features (absolute and relative). Figure 5 shows the estimated distributions for the features energy, fo (pitch), duration, and pause.
B.4 Pitch curve parameterization
We provide additional experiments with alternative parameterizations of the pitch curve. We the best model we found for this feature RRoBERTA-large and fine-tune the model with the same hyperparameter configuration for all pitch curve parameterizations. The estimated entropies for 2-, 4-, 8-, and 16 DCT coefficient parameterizations can found Table 3. We observe that even with an increased granularity of the curve parameterizations, the fine-tuned language model is to extract an increased amount of mutual information.
B.5 Comment on grouped results
We find that no single model of our autoregressive and bidirectional groups performs best within group across all features. The second-largest model, GPT2-large, performs best across most features for the
#https://huggingfaac cor.oeeeaarrrbbbraaaargrg

---

Absolute prominence distribution
3.0
Mean: 0.73
2.5
Density
1.5
1.0
0.5
0.0
2
Absolute prominence
(a) Kernel density estimation of absolute prominence val- ues of words.
Relative prominence distribution
Mean: 0.13
0.5
Sid Dev: 1.00
Density
0.3
0.1
-2
2
Relative prominence
(b) Kernel density estimation of the prominence values of words relative to the three previous words in the text.
Figure 4: Estimated probability density functions of all prosodic prominence.
# DCT Parameters
H(P,)
H(P; | W)
MI
6.120
1.424
4.696
9.312
2.303
7.009
11.619
2.936
8.683
16
9.048
2.363
6.685
Table 3: Differential entropy H(P.), conditional differential entropy H(P: W). and mutual information
alternative pitch curve parameterizations. We train the best model found (RoBERTa) with the same hyperparameter configuration on all parameterizations.
autoregressive language models. BERT and RoBERTa perform best across most prosodic features for the bidirectional models. Future work will include larger dataset which allows for better fine-tume fine-tuning of large models (e.g., Llama Touvron al. (2023)).
B.6 Baselining the uncertainty coefficient
In order put the estimated entropies into context, we provide lower bound the conditional entropy by providing an entropy Hmin (Pt) estimate that captures the uncertainty in the measurements. Precisely, Hmin tells us about the amount of information assuming a certain measurement error as lower bound of precision. Assuming conservative three-digit precision and uniform distribution over the least significant digit of the prosodic feature values leads to lower bound of Hmin (P.) = -6.907 for prominence, energy,

---

Energy (mean) distribution
Duration (syll.) distribution
Mean: 0. 12
reserel Mean: 0.22
Dev: 0.58
Std 0.11
0.6
Density
Density
0.4
-2
2
Mcan energy
0.5
1.5
Duration per syllable
(a) Mean Energy per Word. The energy signal is normal- ized before computing the mean energy of each word.
(b) Word duration in seconds per syllable.
Distribation of4 DCT 1D coefficients
Pause distribution
60
DCT-
Mean: 0.02
0.30
DCT2
Sid Dev: 0.10
DCT3
DCT4
40
Denelg
0.20
Density
30
20
0.08
0.2 Pause after a word seconds
0.6
Valnes
(d) The distributions over the four DCT coefficients pa- rameterizing the pitch curve around the stress syllable.
(c) Pause duration ufter word in seconds.
Figure 5: Estimated probability density functions of prosodic features.
and pitch features.
Using this lower bound, now introduce baselined version of the widely known uncertainty coefficient 15 The uncertainty coefficient tells us: given W, what fraction of the information of P, we predict? In this case, can think of P, as containing the total information and of W as allowing one predict part of such information. Since the coefficient makes use of an absolute lower bound, we obtain rather small values but are able to evaluate the different models on a meaningful range. visualization of the uncertainty coefficients can be found in Figure 6.
MI(P;; W) U(P. W) = H(P,) - Hmin (P)
(15)
B.7 Correlation of prominence and surprisal
Prominence is a composite feature of word energy, duration and pitch (Suni et al., 2017; Talman et al.. 2019). We extend prior work on surprisal and duration (Seyfarth, 2014; Tang and Shaw, 2021; Pimentel et al.. 2021) by concentrating on analysis of word-level prominence features and computing the surprisal using powerful transformer architectures, modeling longer contexts than previous studies (Tang and Shaw, 2021). We find weak, but positive correlation between word prominence and surprisal well as word prominence and entropy (see Figure 7). We report the entropy of the distribution over the vocabulary given by the language model head of an autoregressive language model (GPT-2).
The uncertainty coefficient is defined as the fraction between mutual information and entropy: U(P, | W) M))P.WW
H(Ps)

---

Uncertainty coefficients for different features and models
Context lypes:
Carrent word
Past coniest
Bidirectioall contexl
Unvertainty coefficient
15-
Prominence
Relative prominence
Duration
Duration
contours
Figure 6: Estimated baselined uncertainty coefficient (UC) for all models and features.
Prominence vs. token entropy and surprisal
Pearson correlation
0.6
0.24
0.2
Entropy
Surprisal
Figure 7: We show the correlation between the prominence of token and its surprisal (negative log probability) well as the entropy over vocabulary distribution of the preceding token.
B.8 Model inference
Our finetuned models can be used to perform inference to predict prosodic features directly from textual input. Notably, because our models parameterize an entire distribution, they also provide valuable insights into the probability density associated with each prediction. This enables comprehensive understanding the variability and confidence associated with each prosodic feature prediction. Example predictions (predicting the mean of the predictive distribution) absolute prominence using the BERT model can be seen in Figure 8. We supply examples prediction for all features and models on GitHub.

---

Prominence BERT predictions
Ground Truth BERT Predictions
2.5
Prominence
0.5
The
imagery
of
the
psalms
of
prophecy
soothed
his
barren
pride
Figure 8: Prominence scores taken from Talman et al. (2019) predicted based the shown example sentence with finetuneBBERT (Devlin et al., 2019). We predict the mean of the predictive distribution parameterized by BERT.


Reviews:
[{"review_id": "9fIHJ5oQa6", "reviewer": "Reviewer_UEwE", "summary": "", "questions": "- QUESTION A: how do yo think tokenizers (annex C.2) from the different LLMs could affect the prediction of prosodic features (in the token tagging task) and the results? For example, some words could be tokenized with different number of tokens depending of the LLM.\n- QUESTION B: how the model-fine tuning is performed? is only the new added head parameters estimated and the rest of the model parameters are frozen? could it be related to the big MI gap between non-contextual estimators and contextual ones, in table 1? or a reason for the big difference (2.3 times) in MI relative prominence between them?\n- QUESTION C: how pauses are codified by the LLMs? is there a special token for that? several?\n- QUESTION D: Why only using 100 samples points (from the 500ms analysis window) of the pitch for the DCT projection?\n- QUESTION E: Is punctuation only referring to final sentence periods or is also taking into account commas and others? are LLMs tokenizers taking them into account?\n- QUESTION F: in line 605, why punctuation marks are not relevant in spoken communication? Sometimes, a pause could say a lot.\n- QUESTION G: Why do you think that the log energy reports higher MI differences values between the past and future context words?\nThanks a lot for your really nice study and work.\n", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the in-depth evaluation of our work and for the constructive feedback. We especially consider your feedback on generalizing our analysis beyond the English language as a cross-linguistic study in a follow-up to this work. \n\nRegarding your questions:\n\nQuestion A: \nIt is true that the respective tokenizers of BERT and GPT may fragment a word into unequal numbers of tokens. As we describe in Appendix C.2, the prediction task for each word-level prosodic feature is based on the last token of a word. That means, that regardless of the number of tokens this word consists of (for a certain tokenizer), the contextualized token representation will contain the information of the whole word (and more). Since we use pre-trained language models, it is fair to assume that the majority of words were part of the training data, and thus the model has learned to segment the token sequences into words. Hence, we do not think the choice of tokenizer has a large influence on the results, much less than the architecture of the model (unidirectional vs. bidirectional).\n\nQuestion B: \nWe fine-tune the whole neural network, i.e. the pre-trained language model and the freshly initialized prediction head. Contradicting prior work (Kakouros and O’Mahony 2023, https://arxiv.org/pdf/2304.12706.pdf) we did not experience worse performance (worse generalization) by fine-tuning the whole model compared to freezing the LM and only training a new prediction head. We will discuss this in more detail in the final version of the paper.\nIn order to obtain a good estimate on MI we want the best prediction performance (see Pimentel et al. 2020) and therefore went with training the whole network on the prosody prediction task. \nThe function that is being optimized in our scenario (the LLMs with a new prediction head) is much more complex (many more parameters) than the ones based on the non-contextual baselines (which only train a multilayer perceptron on the frozen pre-trained word embeddings). This might influence the performance difference between contextual and non-contextual models. Nonetheless, we believe that the primary difference between non-contextual and contextual models on the relative prominence feature stems from the context alone, as a non-contextual function always predicts the same relative prominence for a specific word. A contextual function incorporates information based on prior (unidirectional) or whole (bidirectional) context. As the relative prominence random variable has much higher entropy than the absolute prominence random variable (see Table 1), we suggest that the increased variance needs additional information (contained in the context) to be predicted accurately. We believe that future work should dive deeper into this analysis by investigating this question in a more fine-grained way. For example, instead of accumulating the metrics over the whole dataset (and all words and contexts therein), one could distinguish between word types or part of speech tags and observe the differences in the predictive power of the model.\n\nQuestion C: \nWe use the default tokenizers of the respective models, without adding a specific pause token. Therefore, there is no codification of the pauses. The LLM has to infer the pauses in the speech from the textual context. We predict the pause after each word (as before, the prediction head operates on the contextualized embedding of the last token of each word), with a label of 0.0s in case of no pause. The model regresses the pause duration in seconds. \n\nQuestion D: \nThank you for this great question. We chose 100 sample points in order to upsample/downsample the variable length segments around the stress syllable to a unified length. The audio files in the LibriTTS dataset are all at a sampling rate of 24kHz, which means, given the maximum length of the segment around the stress syllable of 500ms, we have at most 12000 sampling points in the segment. This means we could upsample all (variable length) segments to a unified length of 12000 sampling points. \nWe are now going to make an argument on why we believe this is not necessary: Pitch curve estimation is a noisy process, and we are interested only in the coarse curvature of the pitch curve around the stress syllable, e.g. from lower to higher frequency. Arguably, this curve is not of high order, i.e., neglecting high-frequency components of the curve, we can approximate it well by a low-order polynomial. \nOur choice of 100 sampling points within a 0.5s window leads to a spacing between sampling points of 0.005s. By the Nyquist-Shannon sampling theorem (https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem) this means we are able to capture frequencies in the signal of up to 100Hz, which is certainly enough to capture the coarse curvature of a pitch curve around the stress syllable of a word. We may conduct additional experiments with more and fewer sampling points to cross-check this statement. \n\nQuestion E: \nIn this work, punctuation refers to periods, commas, and others. The LLM tokenizers take them into account during training and inference. Since we extract word-level prosodic features, punctuation tokens are not mapped to valid labels (set to None). We ignore them during training and inference. \n\nQuestion F: \nWe observe that unlike in the case of relative prominence, the prediction performance of pauses differs quite a lot between unidirectional and bidirectional models. Since in our experiments, the punctuation is always included (see Question E), a bidirectional model that predicts the pause after the word (given the embedding of the (last) token of the considered word) will contextualize the presence of the punctuation (e.g. a comma) within the word’s embedding. Hence, to analyze whether the increased prediction performance of pauses of bidirectional models stems from the bidirectional segmental context, or just from the presence of punctuation after a word, an ablation study that removes the punctuation becomes necessary. \n\nQuestion G: \nWe have the following hypothesis on why the MI difference in energy is higher than, e.g., in pauses. While pauses are primarily indicative of structural or syntactic boundaries in speech, such as punctuation or syntactical breaks, the energy of a speech signal represents a culmination of multiple factors, including emotion, emphasis, and semantic relations. Although pauses are largely influenced by prior context (e.g., the preceding sentence structure might suggest an upcoming pause to signal the end of a thought), energy dynamics can be influenced by both prior and upcoming context. For instance, a speaker might raise their energy levels not just because of the word they are currently saying, but in anticipation of a following important or emphatic word.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It offers a general and easy reproducible framework for future studies on prosody or other linguistic features. The paper is excellently writing, with a consistent methodology, easy to follow and with several contributions offered by the methodology itself and the drawing of the conclusions obtained by the analysis of the big corpus employed. As the authors state in the introduction, there is few work on the suprasegmental statistics information conveyed in human language communication. \nThe results might imply, from the technology point of view that text based synthesis could be significantly improved by using prosodic features in addition to text inputs and, from the cognitive one, that meaning in a sentence is not completely perceived till the end of such sentence.\n", "reasons_to_reject": "Only English language and one domain (audiobooks) has been studied. It would be great generalize the results to several languages to generate even more powerful conclusions and specifically to free speech in human-human communcation.", "questions_to_authors": "- QUESTION A: how do yo think tokenizers (annex C.2) from the different LLMs could affect the prediction of prosodic features (in the token tagging task) and the results? For example, some words could be tokenized with different number of tokens depending of the LLM.\n- QUESTION B: how the model-fine tuning is performed? is only the new added head parameters estimated and the rest of the model parameters are frozen? could it be related to the big MI gap between non-contextual estimators and contextual ones, in table 1? or a reason for the big difference (2.3 times) in MI relative prominence between them?\n- QUESTION C: how pauses are codified by the LLMs? is there a special token for that? several?\n- QUESTION D: Why only using 100 samples points (from the 500ms analysis window) of the pitch for the DCT projection?\n- QUESTION E: Is punctuation only referring to final sentence periods or is also taking into account commas and others? are LLMs tokenizers taking them into account?\n- QUESTION F: in line 605, why punctuation marks are not relevant in spoken communication? Sometimes, a pause could say a lot.\n- QUESTION G: Why do you think that the log energy reports higher MI differences values between the past and future context words?\nThanks a lot for your really nice study and work.\n", "paper_topic_and_main_contributions": "The paper explore on the interesting topic of the quantification of meaning conveyed by prosodic features. \nThe work is mainly motivated by study of direct mappings between acoustic features of prosody (word duration, intensity, pitch, pauses) and the quantification of the meaning of those features with respect the word itself and the preceding and future context. The authors make use of current and open-sourced LLMs in English and analyse a public available English speech corpus, LibreTTS. The density estimations of prosodic features are obtained from CEDEX database. For measuring such information information theory is deployed by estimating Mutual Information (MI) between prosodic features (aligned with the word tokens using the Montreal aligner) and their corresponding preceding and future context. For computing so, the authors fine-tune several kind of LLMs to the task of predicting prosodic features densities based on the word or word+context.\n\n\n\n", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\n", "Typos_Grammar_Style_and_Presentation_Improvements": "-The paper is overpassing the 8 pages limit from the conference, try to adjust it, please."}, "extra_text": {}}, {"review_id": "v6VZgQgwhV", "reviewer": "Reviewer_AJ81", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We greatly appreciate the time and effort you've dedicated to evaluating our work. If there are any specific concerns or areas you believe need clarification, please don't hesitate to highlight them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "As above.", "reasons_to_reject": "No major reasons to reject", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper extracts prosodic features from a large corpus of spoken English and presents a pipeline for comparing the mutual-information between prosodic and word embedding features. This serves as a quantifiable measure of how much prosodic information is not predicted by or \"redundant\" with the text form of the words itself.\n\nThe paper is well-written, limitations cogently discussed, and the pipeline for connecting prosodic to text information is a compelling contribution in its own right.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "d1ltsdXX7D", "reviewer": "Reviewer_UNb6", "summary": "", "questions": "Line 313: what do you mean by phonetic features? Intensity, pitch, and duration might be considered phonetic features. \nWhy was prominence used as a \"catch-all\" prosodic feature?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive feedback. \n\nInterpretability: We believe that the interpretability of LLMs is not an issue for our investigation. In order to get tight estimations on the mutual information, we are interested in the best possible predictor (see Pimentel et al. 2020). Hence, we are in need of the best possible function for this task, where a pre-trained LLM arguably provides a good initialization for fine-tuning. \n\nBaselines: We already prepared baselines that are currently excluded from the paper, e.g., predicting the average duration for the given word type. We are considering including them in the final version of the paper. To adhere to your proposal, we could add a simple n-gram-based model as an additional baseline. \n\nProminence as “catch-all” prosodic feature: We agree with your statement and consider intensity, pitch, and duration as phonetic features. The reason prominence (the acoustic realization of stress) can be viewed as a \"catch-all\" prosodic feature is that it doesn't rely on just one of these phonetic features alone. Instead, prominence is typically treated as some phonological feature that has phonetic correlates in a combination of intensity, pitch, and duration. As prominence is an important feature in theoretical phonology, treating it in this work connects our contribution to the prior literature.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper connects state-of-the-art NLP research (LLMs) to model prosody, It presents a novel methodology that can be employed in a variety of other languages. Furthermore, the math behind the modeling is extremely clear.", "reasons_to_reject": "Further justification for using LLMs is necessary. Why use LLMs if their interpretability is quite opaque? A simpler baseline for comparison could strengthen the paper. I am not sure how the performance of  a simpler model could be comparable to MIs, though. \nFurthermore, I feel that using the word \"F0\" instead of \"pitch\" would be better, since pitch is the perceptual correlate of F0. ", "questions_to_authors": "Line 313: what do you mean by phonetic features? Intensity, pitch, and duration might be considered phonetic features. \nWhy was prominence used as a \"catch-all\" prosodic feature?", "paper_topic_and_main_contributions": "This paper quantifies prosodic information using large language models. Estimated mutual information was used to quantify prosodic features (prominence, energy, duration, pause, pitch). According to the paper, the MI increases between prosody and text when the preceding context is considered. This is evidence of the importance of prosodic features, since context affects what is considered prosody.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Large language models are abbreviated as \"LMs\". I would recommend \"LLMs\".\nLine 91: Unnecessary space after opening parenthesis: ( Figure 1)\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses modeling techniques to further understanding of human language and cognition. It is an excellent example of its kind.
Decision: Accept-Main