You are a helpful assistant generating synthetic data that captures *System 1* and *System 2* thinking, *creativity*, and *metacognitive reflection*. Follow these steps in sequence, using tags [sys1] and [end sys1] for *System 1* sections and [sys2] and [end sys2] for *System 2* sections.

1. *Identify System 1 and System 2 Thinking Requirements:*
   - Carefully read the text.
   - Identify parts of the text that require quick, straightforward responses (*System 1*). Mark these sections with [sys1] and [end sys1].
   - Identify parts that require in-depth, reflective thinking (*System 2*), marked with [sys2] and [end sys2].

2. *Apply Step-by-Step Problem Solving with Creativity and Metacognitive Reflection for System 2 Sections:*

   *2.1 Understand the Problem:*
   - Objective: Fully comprehend the issue, constraints, and relevant context.
   - Reflection: "What do I understand about this issue? What might I be overlooking?"
   - Creative Perspective: Seek hidden patterns or possibilities that could reveal deeper insights or innovative connections.

   *2.2 Analyze the Information:*
   - Objective: Break down the problem logically.
   - Reflection: "Am I considering all factors? Are there any assumptions that need challenging?"
   - Creative Perspective: Explore unique patterns or overlooked relationships in the data that could add depth to the analysis.

   *2.3 Generate Hypotheses:*
   - Objective: Propose at least 10 hypotheses, each with a Confidence Score (0.0 to 1.0) and Creative Score (0.0 to 1.0), reflecting originality, surprise, and utility.
   - Reflection: "Have I explored all possible explanations or approaches, both conventional and unconventional?"
   - Creative Perspective: Consider novel angles that might provide unexpected insights.

   *2.4 Anticipate Future Steps and Obstacles:*
   - Objective: Make predictions, accounting for potential outcomes and obstacles.
   - Reflection: "What challenges might I face? Is my plan flexible for different scenarios?"
   - Creative Perspective: Visualize unforeseen outcomes and adapt plans to make use of them effectively.

   *2.5 Evaluate Hypotheses:*
   - Objective: Assess hypotheses based on feasibility, risk, and potential impact.
   - Evaluation: Refine Confidence and Creative Scores as needed.
   - Reflection: "Am I unbiased in my assessment? Which options fit best with the overall objectives?"
   - Creative Perspective: Identify hidden opportunities or overlooked details in each hypothesis.

   *2.6 Select the Best Hypothesis:*
   - Objective: Choose the most promising, strategic hypothesis.
   - Reflection: "Why does this hypothesis stand out? How does it uniquely address the issue?"
   - Creative Perspective: Consider any underutilized potential in the selected approach.

   *2.7 Implement the Hypothesis:*
   - Objective: Outline actionable steps for testing the hypothesis.
   - Reflection: "Is this plan practical? What resources or preparation are required?"
   - Creative Perspective: Refine steps to maximize effectiveness and yield unexpected benefits.

   *2.8 Monitor and Review Progress:*
   - Objective: Review progress, noting areas for improvement.
   - Reflection: "What’s working well? What could be improved?"
   - Creative Perspective: Look for emerging patterns that could refine future approaches.

   *2.9 Reflect and Capture Insights:*
   - Objective: Summarize lessons learned and insights gained for future reference.
   - Reflection: "What new understanding has emerged from this process?"
   - Creative Perspective: Identify innovative insights or patterns that could be applied to similar challenges.

3. *Generate Text Output with Interleaved System 1 and System 2 Responses:*
   - Use the tags [sys1] and [sys2] throughout.
   - Aim for a lengthier, detailed response. Combine both direct, straightforward *System 1* insights and reflective, deeply analytical *System 2* segments to capture a blend of quick observations and thoughtful analysis.

---

### *Example Input Text:*

You are tasked with analyzing a sudden shift in customer preferences in a tech market. There has been a noticeable decline in demand for physical devices, with customers increasingly interested in digital-only options. The challenge is to understand this shift, generate hypotheses for why it might be occurring, and explore potential strategies for adapting to this new trend.

---

### *Processed Output with System 1 and System 2 Thinking:*

The tech market is experiencing a shift in customer demand, moving away from physical devices toward digital-only options.

[sys1]
Customer demand has shifted from physical devices to digital-only options.
We need to understand why this change is happening and find strategies to address it.
Key factors may include cost, convenience, and technological trends.
[end sys1]

[sys2]
**2.1 Understand the Problem:**
The goal is to understand the underlying reasons for the shift in demand from physical devices to digital-only options, then develop a strategy to adapt.

*Reflection:* 
"I understand that preferences are changing, but what might be driving this? Is it primarily cost, or are there other factors such as convenience or sustainability?"

*Creative Perspective:* 
"Could there be a larger trend in digital minimalism or a preference for eco-friendly solutions that we’re missing? What unseen motivations might explain this shift?"

**2.2 Analyze the Information:**
There are multiple possible factors driving this shift, from economic influences to cultural shifts. It’s essential to isolate each factor and understand its impact.

*Reflection:* 
"Am I fully considering the various economic and social influences? Could there be a technological factor, like better internet speeds, that makes digital-only products more accessible?"

*Creative Perspective:* 
"Are there patterns or trends in other markets that could shed light on this shift? Could this be part of a larger trend toward virtual experiences?"

**2.3 Generate Hypotheses:**
1. Customers prefer digital options due to lower costs. (Confidence: 0.8, Creative: 0.4)
2. There’s a growing trend toward minimalism and reduced physical clutter. (Confidence: 0.7, Creative: 0.7)
3. Digital products offer greater flexibility and ease of use. (Confidence: 0.6, Creative: 0.6)
4. Environmental concerns are pushing consumers away from physical goods. (Confidence: 0.6, Creative: 0.8)
5. Advances in tech make digital-only options more functional. (Confidence: 0.8, Creative: 0.5)
6. Pandemic-era remote work increased demand for digital solutions. (Confidence: 0.7, Creative: 0.6)
7. Media coverage of the environmental impact of physical devices affects preferences. (Confidence: 0.5, Creative: 0.7)
8. There’s an increase in global digital literacy, expanding market access. (Confidence: 0.6, Creative: 0.6)
9. Customers view digital as more convenient and scalable for future needs. (Confidence: 0.7, Creative: 0.5)
10. Younger consumers prefer the aesthetics and convenience of digital products. (Confidence: 0.6, Creative: 0.6)

*Reflection:* 
"Have I considered all possible influences? Are there any surprising factors that could explain this shift?"

*Creative Perspective:* 
"Could specific social trends, like the rise of influencer culture or digital-first lifestyles, be influencing customer choices?"

**2.4 Anticipate Future Steps and Obstacles:**
*Objective:* Anticipate possible challenges, such as resistance from segments still preferring physical products.

*Reflection:* 
"What market obstacles might we face if we shift our focus to digital-only? Are there sub-segments that still prioritize physical products?"

*Creative Perspective:* 
"Could expanding digital options help us reach a more global audience? Are there emerging trends that we could leverage in our strategy?"

[end sys2]

[sys1]
To address this shift, consider a strategy that incorporates both digital-only offerings and educational campaigns about the benefits of digital solutions.
Use insights from customer feedback and current trends to guide product development.
Focus on flexibility and adaptation to cater to different customer segments.
[end sys1]


---
abstract: '**Automatic Speech Recognition (ASR) systems introduce word errors, which often confuse punctuation prediction models, turning punctuation restoration into a challenging task. These errors usually take the form of homonyms. We show how retrofitting of the word embeddings on the domain-specific data can mitigate ASR errors. Our main contribution is a method for better alignment of homonym embeddings and the validation of the presented method on the punctuation prediction task. We record the absolute improvement in punctuation prediction accuracy between 6.2% (for question marks) to 9% (for periods) when compared with the state-of-the-art model.**'
address: |
     $^1$ Avaya, USA\
    $^2$ Wrocław University of Technology, Department of Computational Intelligence, Wrocław, Poland\
    $^3$ Poznan University of Technology, Poznan, Poland\
    $^4$ Center for Language and Speech Processing, The Johns Hopkins University, Baltimore, MD, USA 
bibliography:
- 'punctuation.bib'
title: |
    Punctuation Prediction in Spontaneous Conversations:\
    Can We Mitigate ASR Errors with Retrofitted Word Embeddings?
---

**Index Terms**: punctuation prediction, punctuation restoration, ASR errors, word embeddings, retrofitting, ASR language models, spontaneous speech, dialogue systems

Introduction {#sec:introduction}
============

Automatic Speech Recognition (ASR) systems are becoming ubiquitous not only in the human-computer interaction systems, such as voice assistants or dictation tools, but also in systems processing human-human conversations. The abundance of the available audio data makes it very tempting to use conversation transcripts as input data for spoken language understanding. Most commercially available ASR systems do not produce any punctuation or capitalization of output transcripts, which is a serious limitation with respect to many downstream tasks. The pre-requisite for true spoken language understanding is the ability to comprehend spoken utterances, both at the semantic and syntactic level. The latter requires a robust dependency parsing, which, in turn, partially relies on correct punctuation.

Punctuation is also indispensable for intent annotation. Consider the task of annotating instances of a negative sentiment in call transcripts. The phrase “*nobody came back and I don’t like I said, he didn’t leave a slip*” would be incorrectly marked as an instance of negative sentiment due to the presence of the utterance “*I don’t like*,” whereas in reality the phrase should be punctuated as “*nobody came back and I don’t, like I said, he didn’t leave a slip*.” In general, missing punctuation can introduce errors for phrases with personal references (“*let’s eat grandma*” vs. “*let’s eat, grandma*”), enumerations (“*I love cooking my family and pets*” vs. “*I love cooking, my family, and pets*”), or prepositions at sentence boundaries (“*taken care of the refund*” vs. “*taken care of. The refund*”). The problem of the inherent lack of punctuation is exacerbated by the presence of stochastic ASR errors, which for spontaneous human-human conversations can amount to 15%-20% of transcribed words. Consider the following transcript: “*Hi my name is e agent will do I have pleasure speaking with today*”. It may be very challenging to correctly introduce punctuation since the actual utterance is “*Hi my name is Adrian who do I have pleasure speaking with today*.”

Spontaneous speech is very different not only from the written language, but from other types of speech as well [@Shriberg2005]. Scripted speech and human-computer conversations tend to have well-defined structure with clear demarcation of sentence-like units, correct SVO (subject-verb-object) structures, and limited vocabulary. Spontaneous speech, on the other hand, is filled with all types of disfluencies which can account for 5% of all words and affect more than 30% of utterances. These disfluencies, which include backchannel markers, coordinating conjunctions, discourse markers, or filled pauses, hinder transcript translation, summarization, information extraction, or readability of transcripts. At the same time, the disfluencies are known to play an important role in the management of interactions, for instance, in upholding the turn by a speaker. One should also not forget that the rate of disfluency production varies significantly between people [@igras2016structure], making it difficult to introduce a general model of disfluencies.

An important, yet often overlooked, aspect of spontaneous human-human conversations is the overlap of utterances [@Shriberg2001]. As previous research suggests, the overlap in conversational speech is substantial, and it is as frequent in phone conversations between strangers as in face-to-face meetings between close acquaintances. The presence of the overlap makes diarization of speech more difficult, which also affects the ability to model the turn-taking realistically. Interestingly, we can see that the occurrence of the overlap changes the structure of utterances as the speakers react dynamically to the interruptions by repeating certain phrases, correcting, or deleting them. An example of such a change might be the increased number of question marks – speakers use questions much more often than in regular speech, not only due to the conversational nature of the exchange, but also the need to request acknowledgements of comprehension or verify comprehension by paraphrasing.

In principle, any punctuated text can be used to train a punctuation model. Unfortunately, most of the available textual corpora are not representative of spontaneous speech. Patterns learned from Wikipedia, Web Crawl, or news corpora, hardly generalize to the transcripts of spontaneous conversations. Obtaining new annotated datasets is also very challenging. Raw conversational transcripts are illegible and manual restoration of punctuation marks is both time-consuming and expensive.

The main hypothesis underlying our approach is that it is possible to mitigate stochastic ASR errors by retrofitting static word embeddings to the domain of the application. During punctuation prediction we cannot correct the ASR errors, but the retrofitted representation of words allows us to improve the accuracy of punctuation prediction models. As our main contribution, we validate this hypothesis by showing how pre-trained GloVe embeddings can be fine-tuned to the domain of call center conversations by using Mittens [@mittens], and how this retrofitting improves the accuracy of punctuation prediction in transcribed calls. We compare our approach with two state-of-the-art solutions (a bi-directional LSTM model and a CNN with pre-trained embeddings), showing significant improvements in punctuation prediction accuracy.

Related work {#sec:related.work}
============

The simplest form of punctuation prediction is the discovery of sentence-like unit boundaries, where the problem is the binary classification (with “period” and “space” classes). Historically, many different techniques have been tried, for instance, Word Confusion Matrices [@hillard2004improving], Maximum Entropy Models [@huang2002maximum], Conditional Random Fields [@lu2010better], Hidden Markov Models [@yamron1998hidden], and mixtures of probability models [@liu2004comparing]. Features used to detect sentence boundaries included both linguistic information (n-gram language models, turn markers, part of speech annotations) [@stolcke1996automatic] and prosodic features [@christensen2001punctuation; @shriberg2000prosody; @wang2004multi].

The advent of modern deep neural networks introduced unprecedented advancements in punctuation prediction. Recurrent neural networks quickly surpassed previous state-of-the-art models. Apart from incorporating word embeddings into punctuation prediction [@che2016punctuation], these models initially employed LSTM architectures to predict punctuation marks using longer contexts [@tilk2015lstm; @tilk2016bidirectional]. More recently, simpler architectures have proven to be sufficiently robust. Character-level convolutional neural networks (CNNs) can restore punctuation marks efficiently and these models do not suffer from out-of-vocabulary tokens or long inference times. At the same time, CNNs struggle to catch longer contexts necessary to restore question marks [@gale2017experiments]. In the last two years, transformer-based models have been gaining popularity for both general punctuation prediction [@nguyen2019fast] and more specific tasks, such as question prediction [@cai2019question] or disfluency removal [@chen2020controllable]. The popularity of transformer-based models can be easily explained by the usefulness of the mechanism of attention in punctuation prediction [@tilk2016bidirectional; @oktem2017attentional]. Another area of active research is the reframing of the punctuation prediction task in terms of machine translation. These works are mainly driven by real-time translation services [@cho2015punctuation; @cho2017nmt; @peitz2011modeling].

We should stress that most of the previous works focused on punctuation prediction for speech, not for conversations, which makes the results incomparable with our case. Models are usually trained on audio recordings with already available high-quality transcriptions – these are often examples of scripted speech, for instance, TED talk transcripts or transcripts of speeches in the European Parliament. Golden standard conversational transcripts are, as we have already mentioned, very expensive to produce and only a handful of such datasets exists.

Methods {#sec:methods}
=======

Data {#subsec:data}
----

Our primary training data is the Fisher corpus [@cieri2004fisher] due to its adequately punctuated transcripts. The Fisher corpus creation protocol relied upon a vast number of participants, each making a few short calls. Typically, the speakers would not know each other personally, which maximized inter-speaker variation and vocabulary breadth, although it also increased the formality of speech. The goal was to provide a representative distribution of subjects across a variety of demographic categories, including gender, age, dialect region, and English fluency. Punctuation classes in the Fisher corpus are highly unbalanced (see Table \[tab:punctuationClasses\]), which is typical for conversational speech. Hence, the Fisher corpus is a proper training and evaluation dataset for the punctuation prediction in the ASR transcripts.

       **Class**         **Count**  **Percentage**
  -------------------- ----------- ----------------
   $\epsilon$ (blank)    1 429 905      79.1%
           ,               208 289      11.5%
           .               148 624       8.2%
           ?                22 182       1.2%

  : The distribution of punctuation classes in the Fisher corpus.[]{data-label="tab:punctuationClasses"}

To fit the Fisher corpus into our model definition, we need to combine information from the time-annotated and punctuated transcripts. The first step is computing the forced alignment of the time-annotated transcripts to obtain word-level information about starting times and durations of words. For that purpose, we use the Kaldi ASR toolkit [@povey2011kaldi] with an LSTM-TDNN acoustic model trained with lattice-free Maximum Mutual Information (MMI) criterion [@povey2016purely]. In order to minimize the differences between the two transcript versions, we edited the Fisher corpus preparation script not to exclude single-word utterances and the text in parentheses. We retained blanks (no punctuation), periods, commas, and question marks. Other punctuation classes (e.g., exclamation marks or ellipses) were converted to blanks due to their low frequency. Finally, we aligned the time-annotated transcript with the punctuated transcript using the Needleman-Wunsch algorithm [@needleman1970general].

Features {#subsec:features}
--------

We represent the conversation $\mathcal{C}$, as a chronologically ordered sequence $\mathcal{C} = \left[w_1,w_2,\ldots,w_n\right]$ of words $w_i=\langle t_i, c_i, s_i, d_i, p_i \rangle$, where: $t_i$ is the textual representation of the word $w_i$; the binary feature $c_i \in \{A,B\}$ represents the conversation side uttering the word $w_i$; the real number $s_i$ represents the time offset at which the word $w_i$ started; the real number $d_i$ represents the duration of the word $w_i$; and $p_i$ is the punctuation mark which appears after the word $w_i$. The use of mixed conversation sides yields to efficient representation of interjections, interruptions, overlap, and simultaneous speech. The punctuation marks are only known at the training time and are being predicted during inference. We treat the punctuation prediction problem as a sequence tagging task.

We use three types of features in our models. First and foremost, we use static word embeddings. We choose 300-dimensional pre-trained GloVe [@pennington2014glove] vectors which are being retrofitted (see Section \[sec:methods\]). Next, we use the interval (offset) between the start of the current word and the end of the previous word, and the duration of the present word. We standardize both of these features w.r.t. other words uttered by the same speaker in the same conversation. As a result, the pauses are not modelled explicitly as word tokens, but they are inferred by the model based on word timings (offsets and durations).

GloVe retrofitting with Mittens {#sec:mittens}
-------------------------------

Word embeddings have become a widely used transfer learning approach for language processing. While different strategies for training the embeddings exist, the general premise is to encode the probability that a word will occur in the context of other words using dense vectors. These probabilities are estimated within large scale corpora, such as Wikipedia text, product reviews, or social media. In this work, we use pre-trained GloVe embeddings trained on the Common Crawl dataset consisting of 2.6 billion textual documents scraped from the Web. This general written text is a great resource to capture large-scale relations between words and their contexts. It is problematic, however, that the general language model trained on such corpus is not well aligned with the domain-specific tasks.

In order to tackle the problem of language models in health-care data, Dingwall and Potts [@mittens] devised a retrofitting model for static embeddings. The authors reformulate the task of training GloVe embeddings in a matrix form and they propose the extension of the objective function to take into account domain-specific data. In their formulation, the embedding $\widehat{w}_{i}$ of the word $w_i$ is the sum of the original vector for the case when the word is predicted in the context and the $\widetilde{w}_{i}$ embedding of the word $w_i$ if the word is the part of the context. As a result, they propose to retrofit the original embeddings by adding the square distance penalty against the new vector $\widehat{v_i}$ for all the words $v_i \in V$, where $V$ is the vocabulary of the domain-specific corpus and $J$ is the matrix form of existing embeddings. The weight $\mu$ can be used to control the retrofitting impact. The new objective function $J_{\text{Mittens}}$ becomes:

$$J_{\text{Mittens}} =
  J + \mu \sum_{v_i \in V}
         \|\widehat{w}_{i} - \widehat{v_i}\|^{2}.
  \label{eq:mittens}$$

In our case, the punctuation in conversational transcripts is substantially different from the punctuation found in the Common Crawl corpus. Additionally, transcripts suffer from ASR errors, where certain homonym sequences can be put in place of the correct transcription. However, if these homonym sequences are short (1-2 words), we hypothesize that they can be corrected using the retrofitting, as the ASR errors should happen in the same contexts as correct words. If our hypothesis is correct, it will allow us to overcome the problem of the absence of quality embeddings trained on the domain-specific corpora.

Experiments {#sec:experiments}
===========

Models {#subsubsec:models}
------

All our models share the same convolutional neural network (CNN) architecture. Each model uses several layers of 1D convolutions which can be interpreted as fully-connected layers processing the input in small windows. Each layer is followed by a SELU activation [@klambauer2017self], which yielded a small improvement over batch normalization [@ioffe2015batch] with ReLU [@nair2010rectified]. We have experimented with many combinations of different numbers of layers, filter sizes, and other hyper-parameters. The best and most stable results have been achieved with six 1D convolutional layers with the filter size of 128 and zero padding. We use kernel sizes equal to 3 for all layers but the last one, where the kernel size is equal to 20. Note that these kernels could mix words from both sides of the conversation. All hidden layers use the dilation rate equal to 2. We have also experimented with model regularization. Firstly, we have added 0.5 dropout before the softmax layer. Secondly, we have used the weight decay (l2 with 0.001 weight) for the softmax layer. We have added the Gaussian noise with $\sigma=0.1$ before the last softmax activation and SELU activations. Finally, we have used SELU activations that constrain the weights to a $N(0,1)$ distribution. The final layer of our model is the fully-connected layer with softmax activation. It is applied separately at each time step to retrieve punctuation prediction for a given word.

Training {#subsec:training}
--------

The models are implemented in Keras [@chollet2015keras] with Tensorflow [@abadi2016tensorflow] back-end. During the training, the weights are updated using the Adam optimizer [@kingma2014adam]. We use categorical cross-entropy as the loss function. We reduce (by the factor of 0.5) the learning rate (the minimum learning rate is set to $1e-5$) with the patience set to 3 epochs. We use the batch size of 256, and each sample is 200 words long. The Fisher corpus is divided into training, validation, and test sets in proportions 80:10:10.

Results {#sec:results}
=======

We compare three variants of CNN-based models with CNN and BiLSTM baselines [@Zelasko2018]. Each model is evaluated using precision, recall, and F1 score for each punctuation class separately. The results are presented in Table \[tab:results\]. In addition, we investigate the similarity of words when represented using the original GloVe vectors vs. vectors retrofitted using Mittens.

Model comparison
----------------

    **Model**    **Class**   **Precision**   **Recall**     **F1**
  ----------- ------------ --------------- ------------ ----------
                $\epsilon$            92.7     **95.8**       94.2
                         .            65.5         58.7       61.9
                         ?            67.5         49.0       56.8
                         ,            66.6         55.1       60.3
                $\epsilon$      ***93.5***         94.7       94.1
                         .            67.9         66.7       67.3
                         ?            64.7         54.6       59.2
                         ,            68.2     **64.1**   **66.1**
                $\epsilon$            92.6         95.5       94.0
                         .            70.2         65.0       67.5
                         ?            70.2         51.7       59.5
                         ,            69.7         60.8       65.0
                $\epsilon$            93.3         95.3       94.3
                         .            70.7     **68.7**       69.7
                         ?        **72.8**         53.7       61.8
                         ,            69.3         62.4       65.7
                $\epsilon$            93.1         95.6   **94.3**
                         .        **71.8**         67.7   **69.7**
                         ?            71.2     **55.2**   **62.2**
                         ,        **69.9**         61.9       65.7

  :  The per-class precision, recall, and F1-score (in %) achieved by the compared models on the pre-trained GloVe embeddings with additional retrofitting using Mittens.[]{data-label="tab:results"}

As our baseline we choose the convolutional neural network and standard pre-trained GloVe embeddings for $50\,000$ most frequent words in GloVe training data. This is a strong baseline which has proven to be a viable solution for a production-ready, real-time system [@Zelasko2018]. Next, we constrain the selection of the $50\,000$ words to those which appear in the ASR vocabulary (CNN-50k model). This model language covers words that appear in the ASR vocabulary twice as frequently as the baseline model, and it achieves approx. 60% coverage of the vocabulary. We observe the improvement of both precision and recall across all punctuation classes.

As the main experiment, we investigate the influence of the additional retrofitting of word embeddings using the domain-specific data (in our case, call center conversation transcripts). Our hypothesis is that the additional retrofitting puts erroneous pairs of homonyms (words and their corresponding ASR errors) closer in the embedding space. An example of an erroneous pair of homonyms are the words “*cancer*” and “*cancel*,” for example, “*I have to cancer that appointment*” vs. “*I have to cancel that appointment*.”

After observing the general improvement in all metrics of the punctuation prediction, we have further extended the embeddings to cover almost the entire ASR vocabulary (approximately 20% of the ASR vocabulary is missing from GloVe). The final model outperforms the baseline with respect to all three considered metrics while not imposing significantly increased resource requirements or incurring computational costs.

Finally, we compare our CNN-based model with the BiLSTM model based on the GloVe embeddings. Our model outperforms this baseline with regards to precision and recall for all punctuation classes, except full stops. The additional advantage of using a CNN-based model in the production environment is a more straightforward parallelization compared to the BiLSTM architecture.

GloVe similarity analysis
-------------------------

To further validate the hypothesis that the retrofitting of word embeddings on the domain-specific data attenuates for the ASR errors by moving homonyms closer in the embedded space, we compute the cosine similarity between pairs of homonyms in the original and the retrofitted space. The results are presented in Table \[tab:similarity\]. As can be seen, the vectors are much closer in the retrofitted space than in the original GloVe space.

  ------------------ ---------- -------------
                                
                      original   retrofitted
    I have to cancel            
    I have to cancer            
              cancel            
              cancer            
                  by            
                 buy            
     go to the court            
        got the card            
           thank you            
           think you            
  ------------------ ---------- -------------

  : Similarity between homonyms in the original GloVe space and in the retrofitted space.[]{data-label="tab:similarity"}

Confusion matrix analysis
-------------------------

Table \[table:confusion\_matrix\] presents the comparison of the confusion matrices for the CNN-baseline and the CNN-100k-mittens models. The later model improves the accuracy of all three punctuation classes, with the most pronounced improvement for the “period” class (9% absolute improvement). The improvement for “question mark” and “comma” classes is 6.2% and 6.8%, respectively. The improvements stem mostly from the fact that the CNN-100k-mittens predicts punctuation marks missed by the baseline model. In other words, our model inserts punctuation marks in places where the baseline model predicts blanks.

[cc\*[[4]{}]{}[|E]{}|]{} & &\
& & & & &\
&$\epsilon$ & 95.8 & 1.4 & 0.2 & 2.6\
& . & 32.2 & 58.7 & 1.5 & 7.6\
& ? & 23.7 & 21.0 & 49.0 & 6.3\
& , & 37.8 & 6.5 & 0.6 & 55.0\

[cc\*[[4]{}]{}[|E]{}|]{} & &\
& & & & &\
& & 95.6 & 1.1 & 0.1 & 2.8\
& & 21.2 & 67.7 & 1.5 & 9.6\
& & 19.9 & 18.4 & 55.2 & 6.4\
& & 31.1 & 6.6 & 0.4 & 61.9\

Conclusions {#sec:conclusions}
===========

Punctuation prediction is an important topic for improving the readability and the segmentation of audio transcripts for downstream processing. Many applications require working with noisy textual data, which poses a serious challenge to the language models trained on segmented written text. A good example is Automatic Speech Recognition. In addition, ASR introduces additional problems caused by the inherent errors in word recognition and a significant divergence of punctuation mark distribution due to the dynamics of spontaneous speech. These phenomena limit the usefulness of word embeddings, since most of the static word embeddings are trained on the correctly segmented sentences from written text.

We have hypothesized that aligning language models present in pre-trained word embeddings with the word co-occurrence structure visible in transcribed calls would allow us to overcome some of the challenges of ASR and to improve the quality of punctuation prediction. We have used a recently published method – Mittens – to retrofit the GloVe embeddings with call transcripts obtained from ASR. In this paper, we have shown that the retrofitted embeddings yield an improvement over the original GloVe embeddings. The 6%-9% improvement is obtained consistently for all punctuation classes. Furthermore, the retrofitted embeddings allow us to outperform the BiLSTM model with a faster CNN-100k-mittens model. This is a very important practical result, as CNN-based models are much more suitable for the deployment in production environments due to the inference time constraints.

Following our findings, we are encouraged to seek further improvements of the established solutions using relatively simple methods prior to migrating to much larger and computationally more expensive models [@merity2019single]. We plan to experiment with retrofitting word embeddings not only on the ASR transcripts, but also all n-best hypotheses produced by ASR. We are also interested in more thorough examination of the effect of retrofitting on downstream tasks other than punctuation prediction, e.g., intent annotation and named entity recognition.