A growing body of work on adversarial examples has identified that for machine-learning (ML) systems that operate on high-dimensional data, for nearly every natural input there exists a small perturbation of the point that will be misclassified by the system, posing a threat to its deployment in certain critical settings BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . More broadly, the susceptibility of ML systems to adversarial examples has prompted a re-examination of whether current ML systems are truly learning or if they are assemblages of tricks that are effective yet brittle and easily fooled BIBREF9 . Implicit in this line of reasoning is the assumption that instances of ”real" learning, such as human cognition, yield extremely robust systems. Indeed, at least in computer vision, human perception is regarded as the gold-standard for robustness to adversarial examples.Evidently, humans can be fooled by a variety of illusions, whether they be optical, auditory, or other; and there is a long line of research from the cognitive science and psychology communities investigating these BIBREF10 . In general, however, these illusions are viewed as isolated examples that do not arise frequently, and which are far from the instances encountered in everyday life.In this work, we attempt to understand how susceptible humans' perceptual systems for natural speech are to carefully designed “adversarial attacks.” We investigate the density of certain classes of illusion, that is, the fraction of natural language utterances whose comprehension can be affected by the illusion. Our study centers around the McGurk effect, which is the well-studied phenomenon by which the perception of what we hear can be influenced by what we see BIBREF0 . A prototypical example is that the audio of the phoneme “baa,” accompanied by a video of someone mouthing “vaa”, can be perceived as “vaa” or “gaa” (Figure 1 ). This effect persists even when the subject is aware of the setup, though the strength of the effect varies significantly across people and languages and with factors such as age, gender, and disorders BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 .A significant density of illusionable instances for humans might present similar types of security risks as adversarial examples do for ML systems. Auditory signals such as public service announcements, instructions sent to first responders, etc., could be targeted by a malicious agent. Given only access to a screen within eyesight of the intended victims, the agent might be able to significantly obfuscate or alter the message perceived by those who see the screen (even peripherally).Illusionable instances for humans are similar to adversarial examples for ML systems. Strictly speaking, however, our investigation of the density of natural language for which McGurk illusions can be created, is not the human analog of adversarial examples. The adversarial examples for ML systems are datapoints that are misclassified, despite being extremely similar to a typical datapoint (that is correctly classified). Our illusions of misdubbed audio are not extremely close to any typically encountered input, since our McGurk samples have auditory signals corresponding to one phoneme/word and visual signals corresponding to another. Also, there is a compelling argument for why the McGurk confusion occurs, namely that human speech perception is bimodal (audio-visual) in nature when lip reading is available BIBREF20 , BIBREF21 . To the best of our knowledge, prior to our work, there has been little systematic investigation of the extent to which the McGurk effect, or other types of illusions, can be made dense in the set of instances encountered in everyday life. The closest work is BIBREF22 , where the authors demonstrate that some adversarial examples for computer vision systems also fool humans when humans were given less than a tenth of second to view the image. However, some of these examples seem less satisfying as the perturbation acts as a pixel-space interpolation between the original image and the “incorrect” class. This results in images that are visually borderline between two classes, and as such, do not provide a sense of illusion to the viewer. In general, researchers have not probed the robustness of human perception with the same tools, intent, or perspective, with which the security community is currently interrogating the robustness of ML systems.For the McGurk effect, we attempt an illusion for a language token (e.g. phoneme, word, sentence) $x$ by creating a video where an audio stream of $x$ is visually dubbed over by a person saying $x^{\prime }\ne x$ . We stress that the audio portion of the illusion is not modified and corresponds to a person saying $x$ . The illusion $f(x^{\prime },x)$ affects a listener if they perceive what is being said to be $y\ne x$ if they watched the illusory video whereas they perceive $x$ if they had either listened to the audio stream without watching the video or had watched the original unaltered video, depending on specification. We call a token illusionable if an illusion can be made for the token that affects the perception of a significant fraction of people.In Section "Phoneme-level experiments" , we analyze the extent to which the McGurk effect can be used to create illusions for phonemes, words, and sentences, and analyze the fraction of natural language that is susceptible to such illusionability. We thereby obtain a lower bound on the density of illusionable natural speech.We find that 1) a significant fraction of words that occur in everyday speech can be turned into McGurk-style illusions, 2) such illusions persist when embedded within the context of natural sentences, and in fact affect a significant fraction of natural sentences, and 3) the illusionability of words and sentences can be predicted using features from natural language modeling.We began by determining which phoneme sounds can be paired with video dubs of other phonemes to effect a perceived phoneme that is different from the actual sound. We created McGurk videos for all vowel pairs preceded with the consonant // as well as for all consonant pairs followed by the vowel // spoken by a speaker. There are 20 vowel phonemes and 24 consonant phonemes in American English although /ʤ/ and /ʒ/ are redundant for our purposes. Based on labels provided by 10 individuals we found that although vowels were not easily confused, there are a number of illusionable consonants. We note that the illusionable phoneme pairs depend both on the speaker and listener identities. Given Table 1 of illusionable phonemes, the goal was then to understand whether these could be leveraged within words or sentences; and if so, the fraction of natural speech that is susceptible.We sampled 200 unique words (listed in Table 2 ) from the 10,000 most common words in the Project Gutenberg novels in proportion to their frequency in the corpus. The 10k words collectively have a prevalence of 80.6% in the corpus. Of the 200 sampled words, 147 (73.5%) contained phonemes that our preliminary phoneme study suggested might be illusionable. For these 147 words, we paired audio clips spoken by the speaker with illusory video dubs of the speaker saying the words with appropriately switched out phonemes. We tested these videos on 20 naive test subjects who did not participate in the preliminary study. Each subject watched half of the words and listened without video to the other half of the words, and were given the instructions: "Write down what you hear. What you hear may or may not be sensical. Also write down if a clip sounds unclear to you. Not that a clip may sound nonsensical but clear." Subjects were allowed up to three plays of each clip.We found that watching the illusory videos led to an average miscomprehension rate of 24.8%, a relative 148% increase from the baseline of listening to the audio alone (Table 3 ). The illusory videos made people less confident about their correct answers, with an additional 5.1% of words being heard correctly but unclearly, compared to 2.1% for audio only. For 17% of the 200 words, the illusory videos increased the error rates by more than 30% above the audio-only baseline.To create a predictive model for word-level illusionability, we used illusionable phonemes enriched with positional information as features. Explicitly, for each of the 10 illusionable phonemes, we created three features from the phoneme being in an initial position (being the first phoneme of the word), a medial position (having phonemes come before and after), or a final position (being the last phoneme of the word). We then represented each word with a binary bag-of-words model BIBREF23 , giving each of the 30 phonemes-with-phonetic-context features a value of 1 if present in the word and 0 otherwise. We performed ridge regression on these features with a constant term. We searched for the optimal $l2$ regularization constant among the values [0.1, 1, 10, 100] and picked the optimal one based on training set performance. The train:test split was in the proportion 85%:15% and was randomly chosen for each trial. Across 10k randomized trials, we obtain average training and test set correlations of $91.1\pm 0.6\%$ and $44.6\pm 28.9\%$ respectively.Our final model achieves an out-of-sample correlation of 57% between predicted and observed illusionabilites. Here, the observed illusionability of the words is calculated as the difference between the accuracy of watchers of the illusory videos and the accuracy of listeners, where “accuracy” is defined as the fraction of respondents who were correct. For each word, the predicted illusionability is calculated from doing inference on that word using the averaged regression coefficients of the regression trials where the word is not in the training set.Our predicted illusionability is also calibrated, in the sense that for the words predicted to have an illusionability <0.1, the mean empirical illusionability is 0.04; for words with predicted illusionability in the interval [0.1, 0.2] the mean empirical illusionability is 0.14; for predicted illusionability between [0.2, 0.3] the mean observed is 0.27; and for predicted illusionability >0.3, the mean observed is 0.50. Figure 2 visually depicts the match between the observed and predicted word illusionabilities.We set up the following experiment on naturally occurring sentences. We randomly sampled 300 sentences of lengths 4-8 words inclusive from the novel Little Women BIBREF24 from the Project Gutenberg corpus. From this reduced sample, we selected and perturbed 32 sentences that we expected to be illusionable (listed in Table 4 ). With the speaker, we prepared two formats of each sentence: original video (with original audio), and illusory video (with original audio).We then evaluated the perception of these on 1306 naive test subjects on Amazon Mechanical Turk. The Turkers were shown videos for six randomly selected sentences, three illusory and three original, and were given the prompt: "Press any key to begin video [index #] of 6. Watch the whole video, and then you will be prompted to write down what the speaker said." Only one viewing of any clip was allowed, to simulate the natural setting of observing a live audio/video stream. Each Turker was limited to six videos to reduce respondent fatigue. Turkers were also asked to report their level of confidence in what they heard on a scale from no uncertainty (0%) to complete uncertainty (100%). One hundred and twelve Turkers (8.6%) did not adhere to the prompt, writing unrelated responses, and their results were omitted from analysis. We found that watching the illusory videos led to an average miscomprehension rate of 32.8%, a relative 145% increase from the baseline of listening to the audio alone (Table 5 ). The illusory videos made people less confident about their correct answers. Turkers who correctly identified the audio message in an illusory video self-reported an average uncertainty of 42.9%, which is a relative 123% higher than the average reported by the Turkers who correctly understood the original videos. Examples of mistakes made by listeners of the illusory videos are shown in Table 6 . Overall we found that for 11.5% of the 200 sampled sentences (23 out of the 30 videos we created), the illusory videos increased the error rates by more than 10% above the audio-only baseline.We obtained a sentence-level illusionability prediction model with an out-of-sample correlation of 33% between predicted and observed illusionabilities. Here, the observed illusionability of the sentences was calculated as the difference between the accuracy of watchers of the illusory videos and the accuracy of watchers of original videos, where “accuracy” is defined as the fraction of respondents who were correct. We obtained predicted illusionabilities by simply using the maximum word illusionability prediction amongst the words in each sentence, with word predictions obtained from the word-level model. We attempted to improve our sentence-level predictive model by incorporating how likely the words appear under a natural language distribution, considering three classes of words: words in the sentence for which no illusion was attempted, the words for which an illusion was attempted, and the potentially perceived words for words for which an illusion was attempted. We used log word frequencies obtained from the the top 36.7k most common words from the Project Gutenberg corpus. This approach could not attain better out-of-sample correlations than the naive method. This implies that context is important for sentence-level illusionability, and more complex language models should be used.Finally, comparing word-level and sentence-level McGurk illusionabilities in natural speech, we observe that that the former is significantly higher. A greater McGurk effect at the word level is to be expected–sentences provide context with which the viewer could fill in confusions and misunderstandings. Furthermore, when watching a sentence video compared to a short word video, the viewer's attention is more likely to stray, both from the visual component of the video, which evidently reduces the McGurk effect, as well as the from the audio component, which likely prompts the viewer to rely even more heavily on context. Nevertheless, there remains a significant amount of illusionability at the sentence-level.This work is an initial step towards exploring the density of illusionable phenomena for humans. There are many natural directions for future work. In the vein of further understanding McGurk-style illusions, it seems worth building more accurate predictive models for sentence-level effects, and further investigating the security risks posed by McGurk illusions. For example, one concrete next step in understanding McGurk-style illusions would be to actually implement a system which takes an audio input, and outputs a video dub resulting in significant misunderstanding. Such a system would need to combine a high-quality speech-to-video-synthesis system BIBREF25 , BIBREF26 , with a fleshed-out language model and McGurk prediction model. There is also the question of how to guard against “attacks” on human perception. For example, in the case of the McGurk effect, how can one rephrase a passage of text in such a way that the meaning is unchanged, but the rephrased text is significantly more robust to McGurk style manipulations? The central question in this direction is what fraction of natural language can be made robust without significantly changing the semantics.A better understanding of when and why certain human perception systems are nonrobust can also be applied to make ML systems more robust. In particular, neural networks have been found to be susceptible to adversarial examples in automatic speech recognition BIBREF27 , BIBREF28 and to the McGurk effect BIBREF29 , and a rudimentary approach to making language robust to the latter problem would be to use a reduced vocabulary that avoids words that score highly in our word-level illusionability prediction model. Relatedly, at the interface of cognitive science and adversarial examples, there has been work suggesting that humans can anticipate when or how machines will misclassify, including for adversarial examples BIBREF30 , BIBREF31 , BIBREF32 .More broadly, as the tools for probing the weaknesses of ML systems develop further, it seems like a natural time to reexamine the supposed robustness of human perception. We anticipate unexpected findings. To provide one example, we summarize some preliminary results on audio-only illusions.An audio clip of the word “Laurel" gained widespread attention in 2018, with coverage by notable news outlets such as The New York Times and Time. Roughly half of listeners perceive “Laurel” and the other half perceive “Yanny” or similar-sounding words, with high confidence on both sides BIBREF1 . One of the reasons the public was intrigued is because examples of such phenomena are viewed as rare, isolated instances. In a preliminary attempt to investigate the density of such phenomena, we identified five additional distinct examples (Table 7 ). The supplementary files include 10 versions of one of these examples, where listeners tend to perceive either “worlds” or “yikes.” Across the different audio clips, one should be able to hear both interpretations. The threshold for switching from one interpretation to another differs from person to person.These examples were generated by examining 5000 words, and selecting the 50 whose spectrograms contain a balance of high and low frequency components that most closely matched those for the word “Laurel". Each audio file corresponded to the Google Cloud Text-to-Speech API synthesis of a word, after low frequencies were damped and the audio was slowed $1.3$ - $1.9$ x. After listening to these top 50 candidates, we evaluated the most promising five on a set of 15 individuals (3 female, 12 male, age range 22-33). We found multiple distributional modes of perceptions for all five audio clips. For example, a clip of “worlds” with the high frequencies damped and slowed down 1.5x was perceived by five listeners as “worlds”, four as “yikes/yites” and six as “nights/lights”. While these experiments do not demonstrate a density of such examples with respect to the set of all words—and it is unlikely that illusory audio tracks in this style can be created for the majority of words—they illustrate that even the surprising Yanny or Laurel phenomenon is not an isolated occurrence. It remains to be seen how dense such phenomena can be, given the right sort of subtle audio manipulation. `Our work suggests that for a significant fraction of natural speech, human perception can be altered by using subtle, learnable perturbations. This is an initial step towards exploring the density of illusionable phenomenon for humans, and examining the extent to which human perception may be vulnerable to security risks like those that adversarial examples present for ML systems.We hope our work inspires future investigations into the discovery, generation, and quantification of multimodal and unimodal audiovisual and auditory illusions for humans. There exist many open research questions on when and why humans are susceptible to various types of illusions, how to model the illusionability of natural language, and how natural language can be made more robust to illusory perturbations. Additionally, we hope such investigations inform our interpretations of the strengths and weaknesses of current ML systems. Finally, there is the possibility that some vulnerability to carefully crafted adversarial examples may be inherent to all complex learning systems that interact with high-dimensional inputs in an environment with limited data; any thorough investigation of this question must also probe the human cognitive system.This research was supported in part by NSF award AF:1813049, an ONR Young Investigator award (N00014-18-1-2295), and a grant from Stanford's Institute for Human-Centered AI. The authors would like to thank Jean Betterton, Shivam Garg, Noah Goodman, Kelvin Guu, Michelle Lee, Percy Liang, Aleksander Makelov, Jacob Plachta, Jacob Steinhardt, Jim Terry, and Alexander Whatley for useful feedback on the work. The research was done under Stanford IRB Protocol 46430.

Q: What is the McGurk effect?
A: a perceptual illusion, where listening to a speech sound while watching a mouth pronounce a different sound changes how the audio is heard
When the perception of what we hear is influenced by what we see.

Q: Are humans and machine learning systems fooled by the same kinds of illusions?