Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It? (2024)

Abstract: Large Language Models (LLMs) can generate text by transferring style attributes like formality resulting in formal or informal text. However, instructing LLMs to generate text that when spoken, is more intelligible in an acoustically difficult environment, is an underexplored topic. We conduct the first study to evaluate LLMs on a novel task of generating acoustically intelligible paraphrases for better human speech perception in noise. Our experiments in English demonstrated that with standard prompting, LLMs struggle to control the non-textual attribute, i.e., acoustic intelligibility, while efficiently capturing the desired textual attributes like semantic equivalence. To remedy this issue, we propose a simple prompting approach, prompt-and-select, which generates paraphrases by decoupling the desired textual and non-textual attributes in the text generation pipeline. Our approach resulted in a 40% relative improvement in human speech perception, by paraphrasing utterances that are highly distorted in a listening condition with babble noise at signal-to-noise ratio (SNR) −5 dB. This study reveals the limitation of LLMs in capturing non-textual attributes, and our proposed method showcases the potential of using LLMs for better human speech perception in noise.1

heading: 1 Introduction
text: Paraphrase generation is the task of rephrasing a sentence while retaining its meaning (Bhagat and Hovy, 2013). Humans perform paraphrasing in spoken conversations, to enable their listeners to perceive spoken messages as intended (Bulyko et al., 2005; Bohus and Rudnicky, 2008). Motivated by human speech production strategies, paraphrasing has also been applied to speech synthesis systems, to enhance the quality, naturalness (Nakatsu and
1Our code and data are available at https://github. com/uds-lsv/llm_eval_PI-SPiN.
White, 2006; Boidin et al., 2009), and intelligibility of synthetic speech, especially in challenging acoustic conditions (Zhang et al., 2013). Recent explorations on why certain sentences are more intelligible than their paraphrases showed that, the observed intelligibility gain in a noisy listening environment is attributed to the rephrasing, which introduces more acoustic cues that survived the masking effect of the noise (Chingacham et al., 2021, 2023). In other words, the enhanced speech perception with paraphrasing is driven by noiserobust acoustic cues.
The potential of paraphrasing is however, seldom used to build human-like spoken dialogue systems that are adaptive to human listeners’ perception errors in noise, presumably due to the limited investigations to generate paraphrases that are acoustically more intelligible in a noise condition. Prior studies relied on human annotations to identify the ideal paraphrase among a set of candidates (Nakatsu and White, 2006; Zhang et al., 2013; Chingacham et al., 2023), with little discussion on generating intelligible paraphrases. This raises the question of how to generate text that is semantically equivalent to and acoustically more intelligible than the given input sentence, for a noisy environment. We refer to this task as Paraphrase to Improve Speech Perception in Noise (PI-SPiN).
This task is particularly interesting in the context of generative LLMs, which have shown incredible performance in natural language generation (NLG) tasks such as paraphrase generation and dialogue generation (Radford et al., 2019; Wei et al., 2022; Li et al., 2024). Moreover, recent studies have demonstrated LLMs’ capability to control text generation for a wide range of style attributes like sentiment, syntax, formality, and politeness (Zhang et al., 2023; Sun et al., 2023a). PI-SPiN differs from those controllable text generation problems, as it aims to generate text that satisfies the desired textual attributes (e.g., semantic equivalence), in
1
standard prompting
addition to the non-textual attribute (i.e. acoustic intelligibility), which is hard to describe textually.
To explore the potential of LLMs in PI-SPiN, we proposed to evaluate LLMs’ inherent capability to generate acoustically intelligible paraphrases, without any model fine-tuning. Through standard prompting methods like zero-shot learning (ZSL), we found that the model was able to capture textual attributes, while consistently struggling to improve acoustic intelligibility. We also observed that increasing the description of the desired non-textual attribute in the prompt only confuses the model, and it may even lead to a deterioration in textual attributes that were achievable otherwise.
To effectively utilize LLMs for generating acoustically intelligible paraphrases, we propose a simple approach called prompt-and-select, which guides paraphrase generation by introducing the desired non-textual attribute in a post-processing step (see Figure 1). It is a two-step process beginning with prompting the LLM to generate multiple candidates and then selecting the best candidate based on acoustic intelligibility, which is hard to capture in textual mode alone. By conducting a human evaluation with native English listeners, who have no hearing impairments, we verified that the LLM-generated paraphrases via prompt-and-select approach are indeed more intelligible than original sentences, in a listening environment with babble noise at SNR −5 dB.2
2See definitions of babble noise and SNR in Appendix A.
Our main contributions are as follows:
• We conduct an elaborate study on the evaluation of LLMs on a novel task called PI-SPiN.
• Our results illustrate the weakness of standard textual prompting to control a non-textual attribute – acoustic intelligibility.
• Our proposed approach prompt-and-select is an effective solution to generate paraphrases that are more acoustically intelligible.

heading: 2 Related Work
text: Acoustic Intelligibility. Speech perception has been a long-standing research topic in speech science (Kalikow et al., 1977; Luce and Pisoni, 1998; McArdle and Wilson, 2008), which contributed towards a better understanding of human (mis)hearing. More specifically, several human perception experiments were conducted to investigate the intelligibility of speech tokens such as vowels (Pickett, 1957; Cutler et al., 2004), consonants (Weber and Smits, 2003; Jürgens and Brand, 2009), words in isolation (Luce and Pisoni, 1998; Clopper et al., 2010; Wilson and Cates, 2008), words in context (Kalikow et al., 1977; Uslar et al., 2011; Chingacham et al., 2021), especially in noisy environments. While several studies showcased the influence of linguistic characteristics such as predictability (Kalikow et al., 1977), syntactic complexity (Uslar et al., 2011; Carroll and Ruigendijk, 2013; van Knijff et al., 2018), and lexical features (Luce and
Pisoni, 1998; McArdle and Wilson, 2008), on the intelligibility of utterances in noise, there is limited explorations in utilizing the linguistic potential to improve acoustic intelligibility in noise.
We share the motivation to improve speech perception in noise using paraphrases with early studies (Cox and Vinagre, 2004; Nakatsu and White, 2006; Zhang et al., 2013; Chingacham et al., 2023). Nakatsu and White (2006) proposed to train a reranker to select the paraphrases that are more likely to sound natural, when synthesized. They generated multiple paraphrases for each sentence mainly by modifying the word order and replacing a few lexical units in the original sentence. On the other hand, Zhang et al. (2013) proposed an objective measure to distinguish the intelligibility difference among paraphrases that are of the same syntactic type, thereby restricting the type of sentential paraphrases. More recently, Chingacham et al. (2023) investigated the potential of improving intelligibility by considering a larger set of paraphrase types, which are generated using modern paraphrasing models. However, our work is distinct from theirs as we explore LLMs’ inherent ability to generate acoustically intelligible paraphrases.
LLM Evaluation. Given the rapid growth of LLMs such as ChatGPT and GPT-4 (OpenAI, 2023), there has been a surge of research interest towards a holistic evaluation of their capabilities (Chang et al., 2024). Recent studies have attempted to examine their performance across diverse tasks, such as machine translation (Hendy et al., 2023; Zhu et al., 2023), text summarization (Yang et al., 2023; Pu and Demberg, 2023), etc; and also aspects of multilinguality (Lai et al., 2023b; Ahuja et al., 2023) and multimodality (Bang et al., 2023). Close to our work, there have been a few studies looking into the controllable generation ability of LLMs. Lai et al. (2023a) explore the potential of ChatGPT as a text-style transfer evaluator. Sun et al. (2023b) present a systematic study on ten controllable generation benchmarks. Notably, their control factors are derived from the language perspective (e.g., sentiment and number), whereas our work pioneers the investigation of the potential of LLMs as an acoustically intelligible paraphrase generator.

heading: 3 PI-SPiN Task Description
text: Typically, the paraphrase generation task focuses on generating text that is semantically equivalent
to the given input text. However, the PI-SPiN task aims at generating text that is semantically equivalent to, as well as, acoustically more intelligible than the original input text, in an adverse listening condition.
For example, consider the following paraphrase triplet (s1, s2, s3) from the Paraphrases-in-Noise (PiN) dataset3 (Chingacham et al., 2023):
s1: “i was raised in a generation we did need all those things.”
s2: “we did need all those things when i was a child.”
s3: “we did need all those things when i was young.”
s1 is a sentence retrieved from a spoken corpus, while s2 and s3 are outcomes of a paraphrase generation pipeline. Though all sentences are semantically equivalent to each other, they exhibited a significant difference in acoustic intelligibility under noise. More precisely, when these sentences were uttered in a difficult listening condition with babble noise at an SNR of −5 dB, humans perceived s2 with fewer errors in perception compared to s1, while s3 was perceived much worse than s1. The better intelligibility of utterances can be attributed to both linguistic features like predictability (Kalikow et al., 1977), syntactic structure (Uslar et al., 2013), as well as acoustic features like the underlying sounds of the utterance (Luce and Pisoni, 1998). In the more recent investigations on the intelligibility difference among paraphrases (Chingacham et al., 2023), it was shown that the better intelligibility of s2 in such high noise environments is mainly driven by the noise-robust acoustic cues that are defined by both the constituting sounds as well as the noise signal. PI-SPiN aims to generate paraphrases (like s2) that are likely to improve human speech perception in such noisy conditions.
Speech intelligibility in noise is better when sentences are simple (Carroll and Ruigendijk, 2013), shorter (Coene et al., 2016), and linguistically more predictive (Valentini-Botinhao and Wester, 2014). However, the intelligibility of an utterance in noise is not only driven by its underlying text. The perception is also influenced by the acoustic cues that survived the masking effect of the background noise (Cooke, 2006). Hence, PI-SPiN is a
3See Appendix B for more samples.
text generation task, that involves both textual attributes like semantic equivalence and a non-textual attribute that captures the noise-robustness of an utterance.
To generate the acoustic realization of a sentence, we used the Tacotron2 text-to-speech (TTS) system, which demonstrated performance on par with that of a professional voice talent (Shen et al., 2018). More specifically, we used the Tacotron2 model4 pre-trained on the LJSpeech dataset by SpeechBrain (Ravanelli et al., 2021). Further, to create the noise-distorted signals, the clean audio signals underwent a noise-mixing procedure using an open-sourced tool, audio-SNR.5 The babble noise from the NOISEX-92 dataset (Varga and Steeneken, 1993) was mixed with clean audio at SNR−5 dB. To determine whether the generated text satisfies the desired outcome, we primarily relied on automatic metrics, which are discussed in detail in the following section.

heading: 4 Experimental Setup
text: Model. For all our experiments, we used ChatGPT6 (Ouyang et al., 2022), which is one of the most popular LLMs. It has shown impressive performance on paraphrase generation with textual style attributes, while its ability on acoustically intelligible paraphrasing remains unclear. We adopt default parameters (temperature=1.0, top_p=1.0) for the API calls.
Dataset. The evaluation dataset consists of 300 short sentences, which are spoken in a conversational scenario. The dataset is created by filtering out sentences with 10 to 12 words from the top 1000 lines of the speech corpus, Switchboard (Godfrey et al., 1992).
Metrics. Human evaluation is the gold standard for most text-generation tasks. However, human evaluation is expensive and time-consuming, which limits the scale of evaluation. Thus, we perform an automatic evaluation of the whole evaluation dataset and a human evaluation of a subset of the dataset. For automatic evaluation, we employed a range of metrics, which determine the semantic equivalence between the input and output texts, as well as, the linguistic and acoustic features that contribute to the acoustic intelligibility in noise.
4https://huggingface.co/speechbrain/ tts-tacotron2-ljspeech
5https://github.com/Sato-Kunihiko/audio-SNR 6Version: gpt-3.5-turbo
1. Semantic equivalence. Semantic Textual Similarity (STS) measures how similar two texts are in terms of their meaning. In the past, several STS scores were proposed (Bär et al., 2012; Han et al., 2013). More recently, Zhang et al. (2020) proposed BERTScore, which has shown encouraging results in correctly identifying the semantic equivalence/distance between two texts. For all our evaluations, the STS score is the BERTScore-f1 calculated using the distilled BERT model (Sanh et al., 2019). The higher the STS value, the better the semantic equivalence between two texts.
2. Lexical deviation. Lexical deviation (LD) shows to what extent two texts are similar or different in terms of their surface form. The difference in wording between the two texts is particularly interesting for paraphrase generation. Bandel et al. (2022) showed that the deviation in the linguistic forms of paraphrases is one of the critical factors that decides its quality – high-quality paraphrases exhibit high LD, as well as, high STS as they differ lexically, yet maintain the semantics. As defined in Liu and Soh (2022), we used the overlap in lexical tokens of the uncased lemmatized form of two texts to capture the lexical deviation between the input sentence and the model-generated paraphrase. The higher the LD score, the more difference in paraphrased wording.
3. Utterance length. It is a textual attribute that influences acoustic intelligibility, as it was observed that shorter sentences introduce fewer misperceptions in noise (Chingacham et al., 2023). Though paraphrases of shorter lengths are more likely to be perceived correctly, shorter paraphrases may risk missing some semantics of the original text. Hence, it is critical to evaluate utterance length along with semantic equivalence. To measure utterance length in terms of phonemes (i.e. PhLen), we used a grapheme-to-phoneme model7 to generate the phonemic transcript of a sentence. Further, to compare the length within each input-output pair, the pairwise ratio of PhLen is calculated by dividing the length of the model output by that of its input sentence (denoted as PWR-PhLen). Thus, when the model-generated text is similar to the input text, PWR-PhLen value is close to 1.0, while a value much less than 1.0 reflects that the model-generated text is considerably shorter than the original text.
7https://pypi.org/project/g2p-en/
4. Linguistic predictability. Several studies in the past have shown that when lexical tokens are more predictable from the context, word misperceptions are less likely to occur (Kalikow et al., 1977; Uslar et al., 2013; Valentini-Botinhao and Wester, 2014; Schoof and Rosen, 2015; Bhandari et al., 2021). Thus, we considered the perplexity (PPL) score determined by a pre-trained language model, GPT28 (Radford et al., 2019) to estimate the linguistic predictability of a sentence. To compare the linguistic predictability among input and output texts, the pairwise ratio of the perplexity is calculated by dividing the PPL of model-generated text by the input sentence PPL (denoted as PWR-PPL). Higher PPL scores indicate lesser linguistic predictability. Thus, a PWR-PPL value less than 1.0 indicates that the model-generated text is more predictable than the input text.
5. Acoustic Intelligibility. The acoustic intelligibility of an utterance in a noisy environment is primarily driven by the acoustic cues that survived the energetic masking of the noise – utterances with better noise-robust acoustic cues are better perceived in noise (Cooke, 2006; Tang and Cooke, 2016). We use the Speech Intelligibility (SI) metric, STOI (Taal et al., 2010), to capture the acoustic intelligibility of an utterance. STOI is a non-textual attribute, as it measures the mean correlation of short-time envelopes between the clean and noisy audio signals of an utterance. The higher the STOI score, the higher the noise-robustness of an utterance. Similar to other pairwise ratios, the pairwise ratio of STOI (PWR-STOI) is calculated by dividing the STOI of model-generated text by the input text STOI. Thus, PI-SPiN aims at generating paraphrases with PWR-STOI values above 1.0 indicating that the model output is acoustically more
8Version: distilgpt2
intelligible than the input sentences. All pairwise ratios range between 0.0 and +∞, while STS and LD range between 0.0 and 1.0. For the evaluation, we report each of these metrics, averaging across the evaluation dataset.

heading: 5 Evaluating LLMs for PI-SPiN
text: In our experiments, an LLM is prompted to generate a paraphrase for each input sentence in the evaluation set with a prompt template: {prompt prefix} + {input text}. In the following section, we described two prompting methods that we employed and evaluated for the task.

heading: 5.1 Standard Prompting
text: In this setting, the model is prompted to generate an intelligible paraphrase given an input sentence in a zero-shot manner. As shown in Table 1, we investigate three types of prompts, which describe the desired attributes with different granularity: low (pzsl−low), medium (pzsl−med), and high (pzsl−high). With the increasing number of taskspecific tokens in the prompt, the task description is more detailed. Prompts are designed by including keywords like ‘paraphrase’ and ‘intelligible’ that represent the desired outcome. Additionally, a few more tokens like ‘10-12 words’ and ‘spoken-styled’ were used in the prompt to ensure that the generated paraphrase adheres to the length and style of input sentences. We hypothesize that with additional task-oriented tokens in the prompt, the model will steer the paraphrase generation by optimizing the intelligibility.
Results and Analysis Table 2 summarizes the results of all three prompts that we used in standard prompting. We observed that ChatGPT can generate high-quality paraphrases as indicated by high scores for semantic equivalence and lexical
deviation (i.e. STS and LD). More importantly, we found that the length of paraphrases generated by the prompt pzsl−med (PhLen = 42.08) is considerably shorter than those generated with the prompt pzsl−low (PhLen = 50.67), indicating the effectiveness of additional keywords in pzsl−med to control a textual attribute – length. However, the non-textual attribute, acoustic intelligibility (i.e. STOI) of model-generated paraphrases is not significantly different from their corresponding input sentences as reflected by the PWR-STOI scores being not significantly different from 1.0. Furthermore, paraphrases generated with a detailed task description in pzsl−high, also resulted in a similar observation – LLM struggles to improve the non-textual attribute while controlling textual attributes appropriately.
9See Appendix C for the absolute scores of different metrics.
Compared to pzsl−low and pzsl−med, pzsl−high resulted in worse performance, indicated by considerably longer output texts despite prompting to control length (PWR-PhLen = 1.250) and output texts that are linguistically less predictive (PWRPPL = 1.236). It is also reflected in a higher lexical deviation (LD = 0.723) at the expense of lower textual similarity between input and output (STS = 0.837). To have a deep understanding of its behavior, we conducted a qualitative analysis as shown in Table 3. We noticed that the additional context of the non-textual attribute confused the model in understanding the task objective and resulted in model hallucination. In sum, using standard prompting may not effectively elicit the model’s ability to generate paraphrases with the intended non-textual attribute, which is beyond the model’s comprehension.10
10In Appendix D, we also conducted a preliminary study

heading: 5.2 PAS: Prompt-and-Select
text: Prior studies on dialogue generation (Boidin et al., 2009; Nakatsu and White, 2006; Weston et al., 2018) have demonstrated the utility of a simple yet effective pipeline of controlling text generation in two steps: first generating a candidate set of dialogues, and then selecting the best candidate based on the task requirement. Similarly, we proposed to decompose the current task into a two-step process: (1) prompt the LLM to generate multiple output texts that are semantically equivalent to the input text and (2) select the best candidate based on the acoustic intelligibility.
Our approach is similar to the prompt-andrerank method proposed in (Suzgun et al., 2022). However, our approach deviates from theirs mainly in two ways: (1) instead of using beam search at the decoding phase, we propose to utilize the potential of an LLM to generate multiple (n) candidates that exhibit the desired textual attributes and (2) the best candidate selection is based on a metric (i.e. PWR-STOI) that represents a non-textual attribute, which is not considered in prior studies.
For the first step of paraphrase generation, we perform zero-shot prompting with an appropriate task description, pzsl−med. Thus, pzsl−med is the prompt that generates exactly one candidate and involves no selection; it is also referred to as ppas(n=1). However, to generate multiple paraphrases (eg: n = 6), the prompt statement can be simply modified to include the n value, as shown below
• Generate 6 simple, intelligible, and spokenstyled paraphrases with 10-12 words for the
on in-context learning, suggesting that demonstrations are not helpful in capturing the non-textual attribute.
11See Appendix C for the absolute scores of different metrics with varying numbers of candidates.
given input sentence: {input text}
Following the creation of the candidate set, STOI scores are calculated for all model-generated text as well as the input text, by first synthesizing the clean utterances and then mixing babble noise at SNR −5 dB. Finally, the candidate with the highest PWR-STOI is selected as the model output.
Results and Analysis We begin our analysis by comparing the results of standard prompting (n = 1) with the PAS approach, involving 6 candidates (n = 6). As shown in Figure 2a, PAS showcased a high quality of paraphrase generation as indicated by high STS and high LD, similar to the standard prompting setup. Similarly, Figure 2b illustrates that other textual attributes like linguistic predictability (PWR-PPL = 1.056) and utterance length (PWR-PhLen = 1.192) of the PAS approach resulted in similar outcomes of the standard prompting method – output texts are a bit longer than input texts, while their linguistic predictability scores are similar. Importantly, compared to the standard prompting, the prompt-and-select approach yielded a noticeably high PWR-STOI (µ = 1.074, p < 0.05), which is significantly above 1.0. This indicates that the model-generated text is considerably more intelligible than their corresponding input sentences in the given noise condition. We can see more clearly from Figure 2b that PAS (n = 6) leads to a relative improvement of 8.4% in PWRSTOI compared to the standard prompting (n = 1). Our findings suggest that introducing the desired non-textual attribute in a post-processing step is a potential framework to generate desired text with multi-modal behavior.
This raises a follow-up question of whether generating more candidates in the first step further improves the overall PWR-STOI of generated para-
phrases. To this end, we modify the number of candidates (n) in the prompt statement to double the candidate pool size. We found that by increasing the candidate set, there is an improvement in acoustic intelligibility. However, when n is increased from 6 to 12, there was only a limited improvement of 1.6% in PWR-STOI. On the other hand, we observed that textual attributes like linguistic predictability and lexical deviation are not significantly different under varying n values.
Interestingly, the pair-wise ratio of sentence length slightly increased, with more choices in the candidate selection; however, the overall PWRPhLen in this approach is still below the standard prompting setup with no tokens to control length (pzsl−low). Increasing n from 6 to 12 slightly reduced the overall semantic equivalence between the model input and output paraphrase. This indicates that the choice of n introduces a trade-off between the improvement in acoustic intelligibility (PWR-STOI) and the overall semantic equivalence (STS) and one has to choose n considering this trade-off between the gain in non-textual attribute and the need for semantic equivalence.

heading: 5.3 Human Evaluation
text: In addition to the evaluation with automatic metrics, we also conducted a human evaluation to verify whether the model output in the PAS setup (using ppas(n=6)) is indeed more intelligible than their corresponding input sentences. For the human perception experiment, we created two subsets of the evaluation dataset of 300 pairs: random30 and top30. random30 is a set of 30 pairs randomly selected from the evaluation dataset, while top30 is the top 30 input-output pairs that exhibited the larger improvements in STOI scores.
We followed the experiment design of Chingacham et al. (2023) to capture the human speech perception of an utterance in a (noisy) listening setup. After synthesizing the noisy utterances of each sentence using a TTS (Shen et al., 2018) and a noise-mixing tool (audio-SNR), participants were
asked to listen and transcribe each sentence. Every utterance in the dataset was listened to by six different participants. For each listening instance, the edit distance between the phonemic transcriptions of the actual and transcribed text is measured to determine the rate of correct recognition. Furthermore, the sentence-level intelligibility (Sent-Int) of each utterance is calculated by averaging the correct recognition rates exhibited by the six listeners.
The perception experiment was conducted with 24 native English listeners with no hearing impairments (14 females and 10 males; average age = 30.71). After data collection, we calculated the pairwise ratio of sentence-level intelligibility (PWR-Sent-Int) by dividing the Sent-Int scores of the output paraphrase by their corresponding input sentence. A mean score of PWR-Sent-Int significantly above 1.0 indicates that the model-generated paraphrase is significantly more intelligible than the input sentence, in a given listening condition.
Results and Analysis As illustrated in Table 4, top30 items signify that the model-output paraphrases have considerably improved the human perception in a noisy listening condition. We observed that the overall human speech perception of model-output paraphrases (Sent-Int = 0.66) was considerably higher than the input sentences (SentInt = 0.47), introducing a 40% relative gain in the overall intelligibility. This is also reflected in the PWR-Sent-Int score that is significantly above 1.0.
We observed the PWR-Sent-Int of random30 is not significantly above 1.0, even though the PWRSTOI is significantly above 1.0. With further analysis of two subsets, we found that the mean STOI of input sentences in top30 (µ = 0.507) is significantly less than random30 (µ = 0.561). This means that random30 consists of sentences that are better intelligible in noise. Also, we observed a strong negative correlation (r = −0.442, p < 0.001) between the STOI of input sentences and the gain in intelligibility (PWR-Sent-Int), which highlighted the limited benefits of paraphrasing
input sentences in random30. However, top30 consists of all input sentences, which are more likely to benefit from paraphrasing in noisy listening conditions and they reflected the same in the human evaluation. We conclude with the observation PAS is a simple yet effective solution to alleviate the struggles of LLM to generate text with textual and non-textual attributes, without model fine-tuning.

heading: 6 Conclusion
text: In this work, we evaluate LLMs on acoustically intelligible paraphrase generation for better human speech perception in noise. Our results demonstrate the limitations of LLMs in controlling text generation with a non-textual attribute – acoustic intelligibility. To alleviate the struggles of LLMs in generating text that satisfies both textual and non-textual attributes, we proposed a simple yet effective approach called prompt-and-select. With human evaluation, we found that when the original utterances are highly prone to misperceptions in noise, prompt-and-select can introduce 40% of relative improvement in human perception. We hope the findings of this work inspire further explorations to control LLMs’ text generation with different real-world context cues, thereby building more human-like agents. For future work, we could consider two approaches to improve LLMs on this task: 1) fine-tuning LLMs with a large parallel dataset consisting of sentences and their corresponding intelligible paraphrases, and 2) incorporating the acoustic representation of the utterances to control the paraphrase generation.
Limitations
The proposed “prompt-and-select” approach relies on the efficacy of STOI scores to identify the best candidate which is more likely to be perceived correctly in noise. In other words, this approach requires a metric that accurately estimates the desired non-textual attribute. This could be a limitation for problems that require human annotations for candidate selection. Additionally, the current approach introduces an overhead in computation and inference time, due to multiple generations and STOI calculation that involves speech synthesis and noise-mixing procedure. Further investigations are required to study the trade-off between the benefits of paraphrasing and the cost of additional resources. Moreover, our study only evaluated ChatGPT, one of the representative LLMs, due to budget and re-
source constraints. We believe that a holistic evaluation covering more open-source models, such as Mistral (Jiang et al., 2023) and Llama 3 (Meta, 2024), will be beneficial to deepen our understanding of LLM capabilities.
Ethics statement
In this work, generative LLMs are evaluated for a new task without model fine-tuning. It is an impactful step to democratize LLMs for research facilities with limited data and computing resources. We conducted a human evaluation on Prolific, ensuring that all participants were paid (9 GBP) for their service, considering the recommended minimum wage per hour in the UK, in 2023. Also, we ensured to provide an inclusive environment for our participants in the perception experiment, providing non-binary options to mark their gender identity.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their valuable feedback. This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project-id 232722074 – SFB 1102.

heading: B More Samples from the PiN Dataset
text: In Table 8, we provide more paraphrase triplets from the PiN dataset.

heading: C Absolute Scores
text: We provide absolute scores for different evaluation metrics in Table 5 in addition to their pairwise ratios.
D In-context Learning
Prior research has shown that LLMs can efficiently learn to control text generation with demonstrations and perform better than just providing a task description (Brown et al., 2020). Thus for the incontext learning (ICL) setup, the input prompt is modified to include a set of exemplars that represent the desired model behavior. In other words, to instruct the model to generate acoustically intelligible paraphrases in an ICL setting requires a set of sentences and their corresponding paraphrases that are acoustically more intelligible in a noise condition.
To provide the best in-context demonstrations, we created another set of 300 short sentences from the Switchboard corpus excluding those in the evaluation set. Then, their corresponding paraphrases were generated by prompting ChatGPT with pzsl−med. Following speech synthesis and noise mixing with babble noise at SNR −5 dB, we identified the top 5 pairs that exhibited a larger pairwise difference in STOI scores. Further, the sentences within each pair were rearranged in such a way that the second sentence is always better intelligible than its paired paraphrase. Further, the sentences within each demonstration pair were concatenated with a token (eg: ‘=>’) and embedded with pzsl−low for in-context learning. Table 6 represents the exact prompt statement ( picl) that we used for the in-context learning.
Results and Analysis As shown in Table 7, the model learned to generate paraphrases, similar to those given as examples. Compared to the zero-shot learning with minimal task description (pzsl−low), the model in the ICL setup (picl) gener-
ated texts that are semantically more similar and lexically less divergent from the input sentences. More interestingly, the model also learned to optimize the desired textual attributes like length (PWRPhLen) and linguistic predictability (PWR-PPL) of generated paraphrases, even in the absence of prompt tokens to explicitly control those features. Nevertheless, the demonstrations are still not helpful in controlling the non-textual attribute. We observed that the acoustic intelligibility scores of output sentences were not significantly different from their input sentences (PWR-STOI = 0.997). Once again, this shows the inability of the LLM to generate acoustically intelligible paraphrases, even though it captures textual attributes from the given exemplars.

heading: Future Work
text: For future work, we could consider two approaches to improve LLMs on this task: 1) fine-tuning LLMs with a large parallel dataset consisting of sentences and their corresponding intelligible paraphrases, and 2) incorporating the acoustic representation of the utterances to control the paraphrase generation. Limitations
The proposed “prompt-and-select” approach relies on the efficacy of STOI scores to identify the best candidate which is more likely to be perceived correctly in noise. In other words, this approach requires a metric that accurately estimates the desired non-textual attribute. This could be a limitation for problems that require human annotations for candidate selection. Additionally, the current approach introduces an overhead in computation and inference time, due to multiple generations and STOI calculation that involves speech synthesis and noise-mixing procedure. Further investigations are required to study the trade-off between the benefits of paraphrasing and the cost of additional resources. Moreover, our study only evaluated ChatGPT, one of the representative LLMs, due to budget and re-
source constraints. We believe that a holistic evaluation covering more open-source models, such as Mistral (Jiang et al. , 2023) and Llama 3 (Meta, 2024), will be beneficial to deepen our understanding of LLM capabilities