SlowGuess/ACL_2020_OCR huggingface dataset:
Speakers Enhance Contextually Confusable Words
Eric Meinhardt
La Jolla, CA, USA 92093
emeinhardt@ucsd.edu
Eric Baković
ebakovic@ucsd.edu
lbergen@ucsd.edu
Recent work has found evidence that natural languages are shaped by pressures for efficient communication — e.g. the more contextually predictable a word is, the fewer speech sounds or syllables it has (Piantadosi et al. 2011). Research on the degree to which speech and language are shaped by pressures for effective communication — robustness in the face of noise and uncertainty — has been more equivocal. We develop a measure of contextual confusability during word recognition based on psychoacoustic data. Applying this measure to naturalistic speech corpora, we find evidence suggesting that speakers alter their productions to make contextually more confusable words easier to understand.
A major open question in the study of natural languages is the extent to which pressures for efficient communication shape the online production choices of speakers or the system of forms and form-meaning mappings.Zipf (1936, 1949) famously noted that highly frequent words tend to be shorter and hypothesized that this could be explained in terms of pressures for efficient communication: the average cost of producing a word is lower than it would be otherwise.
More recent work has formalized hypotheses about the effect of communicative pressures on language usage and design using tools from information theory (Shannon 1948, Cover and Thomas 2012) and rational analysis (Anderson 1990, 1991). This work has found evidence that meanings are allocated to word types in a way that minimizes speaker effort (Piantadosi et al. 2011, 2012), and that this appears to be at least partly explainable by online production choices (Mahowald et al. 2013).
While this research offers evidence that lexicons and the production choices of speakers are
shaped by pressures for efficient communication, other work examining how much words and lexicons are shaped by pressures for ensuring effective communication in the face of noise and uncertainty has been more equivocal. This work has found evidence that words with greater neighborhood size or density — that is, words that have a greater number of similar-sounding neighbors — have faster onset of production, and have lower overall durations. Words with greater neighborhood density also take longer for listeners to recognize and comprehend, and have less acoustically distinctive vowels (Vitevitch 2002, Gahl et al. 2012; see Vitevitch and Luce 2016 for review).
This work provides a challenge for communicatively-oriented models of production: words with greater numbers of similar-sounding neighbors seem likely to be more confusable, and therefore speakers would be predicted to decrease the likelihood of noise by, e.g., increasing their duration. However, this work does not directly estimate word confusability, instead using neighborhood density or an acoustic similarity measure as a proxy. It remains possible that greater word confusability is associated with phonetic enhancement, and that a more direct measure of confusability would reveal this relationship.
In this paper, we present a measure of relative word confusability based on both a language model and psychoacoustic data, and we examine how well it predicts word durations in natural speech corpora. This measure differs from neighborhood density in three ways: 1) it is sensitive to edit type; 2) it considers words with edit distance greater than 1; and 3) it takes into account top-down expectations.
The structure of the paper is as follows. We first present a derivation of a Bayesian model of word recognition (broadly similar to Norris and McQueen 2008) that incorporates both linguistic context and a model of noise estimated from the
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1991-2002 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
gating data of Warner et al. (2014). We use this speech recognition model to define a measure of confusability, and apply this measure to content words in the NXT-annotated subset of the Switchboard corpus and in the Buckeye corpus (Calhoun et al. 2010, Pitt et al. 2005). We provide evidence that greater confusability is associated with longer duration.
A number of other studies have examined how language is shaped by pressures for communication in the presence of noise. Dautriche et al. (2017) examines whether the words of natural lexicons are dispersed, as would be predicted if these lexicons are optimized to prevent confusions between different words. This work finds that in fact lexicons exhibit clear tendencies towards being clumpier rather than dispersed.
The current study follows previous work in using the phenomena of reduction and enhancement to investigate whether communication is optimized for robustness to noise. Speech tokens that are produced with shorter than usual duration, or with parts omitted or made less distinctive, are said to be reduced, and those tokens produced with longer durations or produced more distinctively are enhanced.
Previous work has provided evidence that reduction and enhancement are influenced by contextual predictability. Words, syllables, and segments that are more contextually predictable tend to be reduced and those that are less contextually predictable tend to be enhanced (see e.g. Van Son et al. 1998, Van Son and Pols 2003, Jurafsky et al. 2001, Aylett and Turk 2004, 2006, Cohen Priva 2008, 2012, 2015, Seyfarth 2014, Demberg et al. 2012, Pate and Goldwater 2015, Buz et al. 2016, Turnbull et al. 2018; see Bell et al. 2009, Jaeger and Buz 2018 for reviews). According to a communicatively-oriented account, this is explainable as balancing efficiency against effectiveness: speakers economize on production cost the more that context facilitates accurate listener inference of the speaker's intent.
Other work has investigated the effects of environmental noise on speech production. This includes work investigating whether speakers modulate their productions in response to overt signals of communication difficulty, e.g. loud environments or talking to listeners who are children,
elderly, or non-native speakers (Lombard 1911, Uther et al. 2007, Picheny et al. 1986).
2 A model of word confusability
We propose a simplified model of word confusability, in which there are two factors that will make word  $v$  in context  $c$  more vs. less confusable. On the one hand, a listener who has observed context  $c$  has some 'top-down' beliefs and expectations about what  $v$  will be before the speaker produces any acoustics for  $v$ . On the other hand, once the speaker has produced acoustics for  $v$ , there will be (in general ambiguous) 'bottom-up' acoustic cues that will usually underdetermine what the speaker's choice of  $v$  actually was. The goal of the listener is then to combine their top-down expectations with their bottom-up observations to reason about which words are more vs. less likely to have been what the speaker intended. $^{1}$
We operationalize the perceptibility of word  $v$  as the probability that the listener accurately recovers this word in situations where the speaker uses it; the confusability of a word is inversely related to its perceptibility. If a speaker has a model of the expected confusability of a given word, they can then decide to lengthen or shorten their particular production of the word token, balancing listener comprehension and their own effort.
2.1 Model definition
To model the in-context confusability of word tokens, we model the task of word recognition as one of Bayesian inference, with the following underlying generative process for the speaker:
1. At some point in time, the speaker has already produced some existing sentential context  $c$ , consisting of a sequence of orthographic words. We assume for simplicity and tractability that the listener knows exactly what this context is at each timestep.
2. The speaker produces the current word  $v$  — e.g. cigarette. We model this as sampling according to a language model  $p_L$ :  $v \sim p_L(\cdot | c)$ .
3. The speaker determines the segment sequence  $x_{1:f} = (x_1, \dots, x_f)$  corresponding to their word choice. For example, the speaker will determine that the segments [sɪgəːt] correspond to the word cigarette.
<sup>1</sup>Note that of the two basic factors integrated here, previous probabilistic work on reduction has been limited to using only 'top-down' expectations.
In our corpora, there is a unique correct segment sequence for a given orthographic word. For ease of exposition, we therefore identify  $x_{1:f}$  with its corresponding orthographic form  $v$ . Abusing notation, we will write  $p_L(x_{1:f}|c)$  for the distribution over segmental forms induced by the language model.
4. The listener receives a segment sequence  $y_{1:f} = (y_1, \dots, y_f) - \mathrm{e.g.} \left[ \int \mathrm{i} g \mathrm{o} \mathrm{i} \varepsilon \mathrm{t} \right]$  ('shigarette')—drawn from a channel distribution  $p_N$  conditioned on the speaker's intended segment sequence:  $y_{1:f} \sim p_N(\cdot | x_{1:f})$ . This represents the effects of noise on the signal received by the listener.
The task of the listener is to then combine their observation (represented here by  $y_{1:f}$ ) with their prior expectations about which words are likely given the context. The listener tries to determine how likely each wordform in the lexicon is to have been the one intended by the speaker. Their posterior belief  $p_{\text{LISTENER}}$  about which segmental wordform  $x_{1:f}$  was intended is described by Bayes' rule:
$$
\begin{array}{l} p _ {\text {L I S T E N E R}} \left(x _ {1: f} \mid y _ {1: f}, c\right) (1) \\ = \frac {p _ {N} \left(y _ {1 : f} \mid x _ {1 : f}\right) p _ {L} \left(x _ {1 : f} \mid c\right)}{p \left(y _ {1 : f} \mid c\right)} (2) \\ = \frac {p _ {N} \left(y _ {1 : f} \mid x _ {1 : f}\right) p _ {L} \left(x _ {1 : f} \mid c\right)}{\sum_ {x _ {1 : f} ^ {\prime}} p _ {N} \left(y _ {1 : f} \mid x _ {1 : f} ^ {\prime}\right) p _ {L} \left(x _ {1 : f} ^ {\prime} \mid c\right)} (3) \\ \end{array}
$$
Suppose for example that the listener perceives  $y_{1:f} = [\mathrm{Ig}\vartheta \mathrm{I}\varepsilon \mathrm{t}]$ . Their beliefs about the lexicon  $p_L(X_{1:f}|C)$  will tell them that this is not a valid segmental wordform, but that [sigəIεt] is a valid wordform. Their beliefs about the noise distribution for the language  $p_N(Y_{1:f}|X_{1:f})$  tell them that  $x_{j} = [\mathrm{s}]$  is a plausible segment to be misperceived as  $y_{j} = [\int]$ ; together this suggests that a good explanation of their percept is the intended wordform  $x_{1:f} = [\mathrm{sig}\vartheta \mathrm{I}\varepsilon \mathrm{t}]$ .
Equation 1 allows us to measure how accurately the listener will be able to reconstruct the speaker's intended message, given a perceived segmental wordform  $y_{1:f}$ . However, this is not sufficient to determine the confusability of an intended wordform. In general, an intended wordform  $x_{1:f}$  may give rise to many different perceived wordforms  $y_{1:f}$  as a result of noise. In order to measure
its confusability, we therefore need to marginalize over the possible perceived segment sequences.
We define the contextual perceptibility of a segmental wordform  $x_{1:f}$  in context  $c$  to be the expected probability that the listener accurately recovers it:
$$
\begin{array}{l} \underset {y _ {1: f} \sim p _ {N} (\cdot | x _ {1: f})} {\mathbb {E}} p _ {\text {L I S T E N E R}} \left(x _ {1: f} \mid y _ {1: f}, c\right) (4) \\ = \sum_ {y _ {1: f}} p _ {\text {L I S T E N E R}} \left(x _ {1: f} \mid y _ {1: f}, c\right) p _ {N} \left(y _ {1: f} \mid x _ {1: f}\right) (5) \\ \end{array}
$$
The space of all possible channel strings  $y_{1:f}$  grows exponentially in sequence length  $f$ . However, each segment is only substantially confusable with a small number of other segments and the probability of more than a small number of channel errors is small. We therefore approximated Eq. 4 with a Monte Carlo estimator:
$$
\underset {y _ {1: f} \sim p _ {N} (\cdot | x _ {1: f})} {\mathbb {E}} p _ {\text {L I S T E N E R}} \left(x _ {1: f} \mid y _ {1: f}, c\right) \tag {6}
$$
$$
\approx \frac {1}{n} \sum_ {i = 1} ^ {n} p _ {\text {L I S T E N E R}} \left(x _ {1: f} \mid y _ {1: f} ^ {i}, c\right) \tag {7}
$$
$$
y _ {1: f} ^ {i} \sim p _ {N} (\cdot | x _ {1: f}) \tag {8}
$$
We choose  $n = 1000$  to balance the variance and computational feasibility of the estimator.
Finally, following the reasoning given in Levy (2005, 2008b), we take the negative logarithm of this quantity and arrive at a surprisal, which represents the contextual confusability of segment sequence  $x_{1:f}$  in context  $c$ :<sup>3</sup>
$$
\begin{array}{l} h \left(x _ {1: f} \mid x _ {1: f}, c\right) (9) \\ = - \log_ {y _ {1: f} \sim p _ {N} (\cdot | x _ {1: f})} \mathbb {E} _ {p _ {\text {L I S T E N E R}}} \left(x _ {1: f} \mid y _ {1: f}, c\right) (10) \\ \end{array}
$$
3 Materials and methods
We make use of two types of data: psychoacoustic gating data for estimating a noise model, and several corpora of natural speech for evaluating whether individuals increase the duration of more confusable words.
3.1 Words duration data
Word durations were analyzed separately in two spoken corpora of American English: the Buckeye Corpus of Conversational Speech (Pitt et al.
2This notation ignores homophony, though the model is in fact sensitive to this.
3Compare Equations 4-9 with Eq. VII of Levy (2008a), a study of sentence-level confusability.
2005) and the NXT Switchboard Annotations (Calhoun et al. 2010), a richly annotated subset of Switchboard-1 Release 2 (Godfrey and Holliman 1997).
The Buckeye Corpus contains about 300,000 word tokens, taken from interviews with 40 speakers from central Ohio. Word durations for the present study were taken from the timestamps provided for word-level annotations. Each word token had a broad transcription uniform across all instances of the word type and a second, token-specific close transcription created by a human annotator.
The Switchboard Corpus contains transcripts of telephone conversations between strangers. The NXT annotated subset includes about 830,000 word tokens from 642 conversations between 358 speakers recruited from all areas of the United States. Word durations for the present study were taken from the 'phonological word'-level timestamps; these were the result of annotator-checked and -corrected timestamps initially made by alignment software. Each phonological word was also associated with a segmental transcription that was uniform across all instances of the word type.
Exclusion criteria almost exactly follow Seyfarth (2014) for the reasons cited there. These criteria are mainly designed to exclude non-content words and words whose pronunciation is likely affected by disfluencies or prosodic structure. Our criteria only diverge in the following manner: Word tokens were excluded if the utterance speech rate (total number of syllables / length of the utterance in seconds) was more than 3 standard deviations from the speaker mean (vs. 2.5 in Seyfarth 2014). After exclusion criteria were applied, about 44,000 (4,900) and 113,000 (8,900) word tokens (word types) remained in the Buckeye and NXT Switchboard corpora, respectively.
3.2 Diphone gating data
The model of word confusability was based on the diphone gating experiment data of Warner et al. (2014). Participants listened to gated intervals of every phonotactically licit diphone of (western) American English and attempted to identify the full diphone they thought was being produced during the interval. Along with earlier work by some of the same researchers on Dutch (Smits et al. 2003, Warner et al. 2005), this represents by far the richest and most comprehensive acoustic confusion
matrix data of its kind.
Warner et al. (2014) identified all adjacent pairs of segments within and between words based on an electronic prouncing dictionary of about 20,000 American English wordforms. A set of approximately 2,000 phonotactically licit diphones were extracted from this transcribed lexicon. At least one stimulus nonsense word was created per diphone by inserting the diphone into an environment consisting of at most one syllable on the left and at most one syllable on the right.
A recording of each stimulus wordform was then marked up with (generally) six temporal gates. For each stimulus wordform, one recording was created for each gate, starting at the beginning of the original recording and going all the way up to a gate location, followed by a ramping procedure (rather than truncation or white noise) to avoid systematically biasing confusion data.
In each trial, participants heard a gated stimulus recording. If the recording included a preceding context, this context was displayed on the screen. The participant then selected the stimulus diphone they thought was in the recording (i.e. not including context).
From this response data, each gate of each stimulus diphone can be associated with a frequency distribution over response diphones. Only the response data for gates corresponding to the end of each segment of the diphone were used in the current study. For each of Buckeye and NXT Switchboard, the segment inventories of the gating data and of each speech corpus had to be projected down to a common set of segments. In each case, this involved collapsing the distinction in the corpora between syllabic and non-syllabic nasal stops. For reasons of data sparsity, the distinction between stressed and unstressed versions of any given vowel was also collapsed.
3.3 Language model
Our measure of contextual confusability uses a language model to compute the prior probability of a word in context. We estimate a language model from the Fisher corpus (Cieri et al. 2004), a speech corpus matched for genre and register to Buckeye and Switchboard. This corpus contains about 12 million (orthographic) word tokens taken from nearly 6000 short conversations, each on one of
4See Grosjean (1980) for reference on the gating paradigm.
about 100 topics.
We estimated n-gram models of several orders from the Fisher corpus using KenLM (Heafield 2011). The n-gram order was treated as a hyperparameter, and selected on the Training Set, as described below. An add-1 smoothed unigram model was also created from word frequencies in the Fisher corpus using SRILM (Stolcke 2002, Stolcke et al. 2011).
3.4 Channel model
The channel model describes the conditional distribution  $p_{N}(Y_{1:f}|X_{1:f})$  over what sequence of segments  $y_{1:f}$  a listener will perceive (e.g. [ɪgət], shigarette) given the full intended sequence  $x_{1:f}$  (e.g. [ɪgət], cigarette). We estimate this distribution using the diphone gating data in Section 3.2. We make the simplifying assumption that the channel distribution for segment  $y_{i}$  is conditionally independent of all other  $y_{j}$  ( $j \neq i$ ) given intended segments  $x_{i-1}, x_{i}, x_{i+1}$ .
By conditioning on adjacent segments, we can capture some effects of coarticulation on confusability. For example, nasals before oral stops are systematically likely to be misheard as having the same place of articulation as the stop:  $x_{1:f} = [\mathrm{anpa}]$  (alveolar nasal before labial stop) is more likely to be misperceived as  $y_{1:f} = [\mathrm{ampa}]$  (a labial nasal) than the reverse, and a confusion of [n] for [m] is comparatively less likely when [n] is between vowels as in [ana] (Ohala 1990).
For each gate  $g \in \{3,6\}$  and for each diphone  $x_{1}x_{2}$ , the response data from Section 3.2 induce a conditional frequency distribution over channel dphones  $f_{g}(y_{1},y_{2}|x_{1},x_{2})$ . These frequency distributions were smoothed by adding a pseudocount to every channel diphone in every distribution; the distributions were then normalized to define a smoothed pair of diphone-to-diphone channel distributions  $p_{g}(y_{1},y_{2}|x_{1},x_{2})$ . From the marginals of these distributions we constructed an approximation (Eq. 11) of the triphone-to-uniphone channel distribution via their geometric mean:
$$
\begin{array}{l} \widetilde {p _ {t}} (y _ {i} | x _ {i - 1}, x _ {i}, x _ {i + 1}) \\ \propto \sqrt {p _ {3} \left(y _ {i} \mid x _ {i - 1} , x _ {i}\right) \cdot p _ {6} \left(y _ {i} \mid x _ {i} , x _ {i + 1}\right)} \tag {11} \\ \end{array}
$$
With the simplifying assumption that only substitution errors are possible, we obtain a preliminary string-to-string channel model:
$$
\widetilde {p} _ {N} \left(y _ {1: f} \mid x _ {1: f}\right) = \prod_ {j = 1} ^ {j = f} \widetilde {p} _ {t} \left(y _ {j} \mid x _ {j - 1}, x _ {j}, x _ {j + 1}\right) \tag {12}
$$
We are primarily interested in using the channel model to define a ranking on the confusability of words, i.e. to determine which words are more or less confusable than others. This makes the channel model defined by Equations 11 and 12 not fully adequate.
The diphone gating data were collected in a laboratory setting with rates of noise lower than for naturalistic speech. As a result, when the noise model is estimated from this data, it implies the absolute rate of accurate perception (as defined by Equation 3) is close to 1 for most words. This makes it hard for the Monte Carlo estimator defined in Equation 7 to determine stable rankings of confusability. In order to estimate rankings in a more stable manner, we introduce a model hyperparameter  $0 < \lambda \leq 1$ , and define a new triphone-to-uniphone channel distribution by:
$$
\begin{array}{l} p _ {t} \left(y _ {i} \mid x _ {i - 1}, x _ {i}, x _ {i + 1}\right) (13) \\ = \left\{ \begin{array}{l l} \lambda \cdot \widetilde {p _ {t}} \left(y _ {i} \mid x _ {i - 1}, x _ {i}, x _ {i + 1}\right), & y _ {i} = x _ {i} \\ \beta \cdot \widetilde {p _ {t}} \left(y _ {i} \mid x _ {i - 1}, x _ {i}, x _ {i + 1}\right), & y _ {i} \neq x _ {i} \end{array} \right\} (14) \\ \end{array}
$$
Here  $\beta \geq 1$  is used to normalize the distributions; it is fully determined by  $\lambda$  for a particular distribution  $p_t(\cdot | x_{i-1}, x_i, x_{i+1})$ . The term  $\lambda$  is used to increase the noise rate in the channel distributions. Note that two important features of the original triphone-to-uniphone distributions  $\widetilde{p}_t$  are maintained in the new model. First, the ratios of outcome probabilities within a single triphone distribution remain the same:
$$
\frac {p _ {t} \left(y _ {i} \mid x _ {i - 1} , x _ {i} , x _ {i + 1}\right)}{p _ {t} \left(y _ {i} ^ {\prime} \mid x _ {i - 1} , x _ {i} , x _ {i + 1}\right)} = \frac {\widetilde {p} _ {t} \left(y _ {i} \mid x _ {i - 1} , x _ {i} , x _ {i + 1}\right)}{\widetilde {p} _ {t} \left(y _ {i} ^ {\prime} \mid x _ {i - 1} , x _ {i} , x _ {i + 1}\right)} \tag {15}
$$
for segments  $y_{i},y_{i}^{\prime}\neq x_{i}$ . Second, the relative probability of accurate perception is preserved across triphone distributions:
$$
\frac {p _ {t} \left(x _ {i} \mid x _ {i - 1} , x _ {i} , x _ {i + 1}\right)}{p _ {t} \left(x _ {i} ^ {\prime} \mid x _ {i - 1} ^ {\prime} , x _ {i} ^ {\prime} , x _ {i + 1} ^ {\prime}\right)} = \frac {\widetilde {p} _ {t} \left(x _ {i} \mid x _ {i - 1} , x _ {i} , x _ {i + 1}\right)}{\widetilde {p} _ {t} \left(x _ {i} ^ {\prime} \mid x _ {i - 1} ^ {\prime} , x _ {i} ^ {\prime} , x _ {i + 1} ^ {\prime}\right)} \tag {16}
$$
The new model maximally agrees with the experimentally estimated distribution, differing only in the absolute amount of noise implied.
<sup>5</sup>We do not use lower-perplexity neural language models due to intractability resulting from the normalizing constant in Equations 3 and 4.
<sup>6</sup>We stop short of utilizing a full triphone-to-triphone channel distribution for tractability.
The gating data does not provide information for estimating the probability of deletion or insertion errors.
The final string-to-string channel model is defined by:
$$
p _ {N} \left(y _ {1: f} \mid x _ {1: f}\right) = \prod_ {j = 1} ^ {j = f} p _ {t} \left(y _ {j} \mid x _ {j - 1}, x _ {j}, x _ {j + 1}\right) \tag {17}
$$
This new channel model has an increased noise rate, making it easier to estimate stable rankings of confusability across words.