“....technologies to understand the human condition and support and enhance human capabilities and experiences”

Sounds of the human vocal instrument
Shrikanth (Shri) Narayanan

University of Southern California

Signal Analysis and Interpretation Laboratory (SAIL)
http://sail.usc.edu

ICA-Aachen 2019

VTSounds-2019-Sept-ICA - September 10, 2019

Broad theme of this talkMultimodal methods in
Human Speech Communication Science
understanding the structure and functionof the human vocal instrument

https://gabbygomez.weebly.com/woodwind-istruments.html!2Wells and Colson, Practical Phonetics (1971)
VTSounds-2019-Sept-ICA - September 10, 2019

Broad theme
Multimodal methods in
Human Speech Communication Science“how do we play the vocal instrument?”
understanding the structure and functionof the human vocal instrument
VOCAL AIRWAY

https://gabbygomez.weebly.com/woodwind-istruments.html!3Wells and Colson, Practical Phonetics (1971)
VTSounds-2019-Sept-ICA - September 10, 2019

Sound out

Co

ns

tri
cti
o

ns

ar
ef

or
me
d

Vocal folds
vibrateAir in/a/

VTSounds-2019-Sept-ICA - September 10, 2019

Vocal vowels synthesized: shaping sounds with specific geometries
https://annex.exploratorium.edu/exhibits/vocal_vowels/vocal_vowels.html

PHARYNX
MOUTH

VTSounds-2019-Sept-ICA - September 10, 2019

Human Vocal Communication•Represents rich information encoding and signaling• information at multiple levels: socio-behavioral, neuro-cognitive, motoric, sensory,.
• carries crucial linguistic and paralinguistic information• intent, desires, emotions, identity, health state,..

•Inherently multimodal• complex orchestration of mental, physiological, physical, social processes• choreographed articulation of tongue, jaw, face, head, hands, body,..
• sophisticated production-perception chain: can learn and adapt•Complex spatio-temporal structure and characteristics

multimodal approaches essential to unpack and understandhuman communication expressions!6
VTSounds-2019-Sept-ICA - September 10, 2019

Talk objective: Observe the vocal instrument in action•Describe multimodal techniques/tools• for vocal production data acquisition, analysis, applications,• to illuminate linguistic and paralinguistic information encoding• with a focus on “data science” approaches•Describe specific imaging, signal processing, and machinelearning possibilities using case studies• dynamics of vocal tract shaping during speech production• some modeling case studies•Highlight applications•health domain scase studies
SPAN: Speech Production and
Articulation kNowledge Group
http://sail.usc.edu/span!7
VTSounds-2019-Sept-ICA - September 10, 2019

Speech Production Studies: Data Are Integral••

Observe, measure, visualize vocal production
Rich history of speech instrumentation & imaging•

Number of techniques: both strengths and limitations– Spatial and temporal resolution– User safety– Flexibility, ease of use, portability– Data interpretability– Specific research and application needs

8
VTSounds-2019-Sept-ICA - September 10, 2019

Acquiring data for probing vocal production

X-ray (Stevens, 1962)

Ultrasound (Stone, 1980)

http://psyc.queensu.ca/~munhallk/05_database.htm
upper lipteeth
lower lip

http://www.speech.umaryland.edu

velum

tongueElectropalatography(courtesy: UCLA Phonetics Lab)

Electromagnetometry
VTSounds-2019-Sept-ICA - September 10, 2019

Using the data
For supporting…..
scientific research and discovery• tools for illuminating mechanisms e.g., individual differences inarticulatory strategies from production data; modeling entrainment

technology development• biologically-informed speech synthesis, biometrics•human-centered conversational agentsclinical applications• objective tools for screening, diagnostics, intervention support
including delivery and behavior change modeling!10
VTSounds-2019-Sept-ICA - September 10, 2019

Newer Possibilities:
Structural magnetic resonance imaging (MRI)Capable of 3D imaging of the hydrogen concentration in human body

Number of advantages:
–
–
–
–
–

Non-invasive, no ionizing radiation
Arbitrary scan plane: Information on complete vocal tract geometry

Excellent, flexible structural differentiation: Good soft tissue contrast, SNR
Amenable to computerized 3D modeling: reconstruction and visualization
Quantitative information: area function and acoustic relations– Variability analyses

Limitations/Challenges––––

Slow: Spatial & Temporal resolution tradeoffs, optimizing to a given application

Noisy images: Susceptibility, blurring artifacts
Imaging teeth
Interaction with other physiological activities: respiration, swallowing, ..
– Clean, Synchronized audio (and other modalities, as needed)– Ease of experimentation, including cost and portability
11
VTSounds-2019-Sept-ICA - September 10, 2019

Early use of MRI: Static Vocal tract Information
Information on 3D vocal tract structure/shape, inter-speaker variations
Accurate vocal tract measurements: area functions, length
Detailed studies on vowels and a number of continuant speech sounds
Facilitated new acoustic modeling studies– Vowels, Nasals, Fricatives, Liquids: English and other languages

Midsagittal vowel images from Haskins (from Goldstein)

3D airway reconstruction forvowel /a/, Univ. Iowa (Story)
http:everest.radiology.uiowa.e
du/nlm/app/vocal/vocal.html
Narayanan, S., Byrd, D., and Kaun, A. (1999), “Geometry,kinematics, and acoustics of Tamil liquid consonants,”
J. Acoust. Soc. Am., pp. 1993—2007.

12

VTSounds-2019-Sept-ICA - September 10, 2019

Support modeling studies: Eg. Fricatives

3D Vocal tract andtongue shapes for /sh/MRI-derived area functions for fricatives

Narayanan, S., Alwan, A., and Haker, K. (1995), “An articulatory study of fricative consonants using magnetic
resonance imaging,” J. Acoust. Soc. Am., vol. 98, pp. 1325—1347.
13
VTSounds-2019-Sept-ICA - September 10, 2019

s

Strident fricative spectra derived
from hybrid source model inputs
using the parametric dipole
spectra: dashed (model); solid(natural speech)
Narayanan, S., and Alwan, A. (2000) “Noisesource models for fricative consonants,”
IEEE Trans. Speech and Audio Processing, vol.
8, no. 3, pp. 328—344.

14
VTSounds-2019-Sept-ICA - September 10, 2019

MRI: Toward real time acquisition for speech (circa 2003)
Improving MRI temporal resolution– A non 2D-FFT acquisition strategy (spiral k-space trajectory) on a GE Signa 1.5T
CV/i scanner with a low-flip angle spiral gradient echo, 9-10 images/second– Adapted pulse sequence originally developed for cardiac imaging– Effective reconstruction rates of 24-35 frames/second
VELUM• sliding window reconstruction techniqueFirst to use real-time MRI andsynchronous noise-cancelled audio
to understand vocal tract movements
during natural speech production.
TONGUE

S. Narayanan, K. Nayak, S. Lee, A. Sethy, and D. Byrd. An approach to real-time magnetic resonanceimaging for speech production. J. Acoust. Soc. Am., 115:1771-1776, 2004.

15

VTSounds-2019-Sept-ICA - September 10, 2019

Spatial vs. Time resolution: speech MRI•

Our newer system (circa 2015) enables visualization of all speech tasks

4

3
2.5
2
1.5
1
0.5
0

Closures of alveolar
trillsconsonantconstrictions

Spatial resolution:(mm2)3.5Cartesian (R=2.4, 1 slice)
Spiral (R=6.5, 1 slice)sustainedvelicsoundsmovements
velotongue
pharyngeal
movements
closure(vowel toconsonanttransitions)coarticulation
events

50

100

150
Time resolution (msec)

200

250

•

Single slice
12 ms/frame(83 fps)300

S. Lingala, Y. Zhu, Y-C. Kim, A. Toutios, S. Narayanan, K. Nayak. A fast and flexible MRI system for the
study of dynamic vocal tract shaping. Magnetic Resonance in Medicine. 77(1): 112-125, 2017

VTSounds-2019-Sept-ICA - September 10, 2019

Approach•Highly accelerated RT-MRI of speech is achieved bysynergistic combination of•Novel custom upper-airway coil design•

Fast spiral readouts•

Constrained reconstruction/TT-GRAPPA

S. Lingala, Y. Zhu, Y-C. Kim, A. Toutios, S. Narayanan, K. Nayak. A fast and flexible MRI system for the
study of dynamic vocal tract shaping. Magnetic Resonance in Medicine. 77(1): 112-125, 2017
VTSounds-2019-Sept-ICA - September 10, 2019

Real-time MRI at 83 fps, 2.4 mm/pixel

18
VTSounds-2019-Sept-ICA - September 10, 2019

3D Real-time MRI Acquisition of Full Vocal Tract• A stack-of-spiral acquisition strategy and constrained image
reconstruction (circa 2018)• Full vocal tract spatial coverage at• high temporal resolution during• natural speech• 16 fps, 200x200x70 mm3 spatial coverage with 2.4x2.4x5.8mm3 / voxel
3D dynamics at three different views

Vocal tract area function/loo/-/lee/-/la/-/za/-/na/-/za/

Mid-sagittal

Axial views

Coronal views

Lim, Y., Zhu, Y., Lingala. S., Byrd D., Narayanan S., Nayak, S. 3D Dynamic MRI of the Vocal Tract During Natural Speech.
Magnetic Resonance in Medicine. 2018
Kim, Y. C., Narayanan, S. S., & Nayak, K. S. (2009). Accelerated three‐dimensional upper airway MRI using compressed sensing.
Magnetic Resonance in Medicine, 61(6), 1434-1440.
VTSounds-2019-Sept-ICA - September 10, 2019llustrating the sounds of the world’s languages
International Phonetic Alphabet (IPA) database
http://sail.usc.edu/span/rtmri_ipa

A. Toutios, S. Lingala, C. Vaz, J. Kim, J. Esling, P. Keating, M. Gordon, D. Byrd, L. Goldstein, K. Nayak, and
S. Narayanan, “Illustrating the Production of the International Phonetic Alphabet Sounds using Fast RealTime Magnetic Resonance Imaging,” in Proc. Interspeech, 2016.
VTSounds-2019-Sept-ICA - September 10, 2019

Real-time MRI at 83 fps, 2.4 mm/pixel

Voiceless fricatives

VTSounds-2019-Sept-ICA - September 10, 2019

Real-time MRI at 83 fps, 2.4 mm/pixel

A child speaker

VTSounds-2019-Sept-ICA - September 10, 2019

USC-TIMIT: A MULTIMODAL ARTICULATORY DATA CORPUS• 10 American English talkers (5M, 5F).
• Real time MRI (5 speakers also withEMA) and synchronized audio.

• 460 sentences each (>20 minutes)• Freely available for speech research.

WEB-LINK (with download info):
http://sail.usc.edu/span/usc-timit/
SAIL homepage: http://sail.usc.edu

Narayanan et al. (2011). A Multimodal Real-Time MRI Articulatory Corpus for Speech Research. InterSpeech.
Narayanan et al. (2014). Real-time magnetic resonance imaging and electromagnetic articulography database for speech
production research. J. Acoust. Soc. Am.
VTSounds-2019-Sept-ICA - September 10, 2019

USC-EMO-MRI corpus
https://sail.usc.edu/span/usc-emo-mri/index.html

A multimodal dataset for emotional speech production• MRI video (23.180 fps) + speech audio (20kHz)• The “Grandfather passage” and 6-7 sentences• 4 acted emotions (neutral, angry, happy and sad)
• Emotion quality evaluation (at least 11 evaluators)
Jangwon Kim et al., “USC-EMO-MRI corpus: An emotional speech production database recorded by real-timemagnetic resonance imaging”, in 10-th ISSP, 2014

24
VTSounds-2019-Sept-ICA - September 10, 2019In the pipeline for upcoming release

USC-Multiview-MRI Speaker Variability corpus
A multimodal dataset from 75 subjects:
1. Real-time MRI at 83 fps (scripted and spontaneous speech)
2. Accelerated volumetric MRI (continuant English phonemes)
3. T2-weighted volumes

S. Lingala, A. Toutios, J. Töger, Y. Lim, Y. Zhu, Y. Kim, Colin Vaz, S. Narayanan, K. Nayak. State-of-the-art MRI
Protocol for Comprehensive Assessment of Vocal Tract Structure and Function. Proc. Interspeech 2016.
VTSounds-2019-Sept-ICA - September 10, 2019

Analysis of data
• Image analysis• Deriving• morphological (structural) details, and• linguistically meaningful articulatory features• Some case studies• Vocal tract morphology• Example linguistic and paralinguistic analyses• Relation between articulatory & acoustic representations• ASR and Speaker Verification• Articulatory strategies

VTSounds-2019-Sept-ICA - September 10, 2019

Measurement of Structural Details

J. TÖGER, T. SORENSEN, K. SOMANDEPALLI, A. TOUTIOS, S. LINGALA, S. NARAYANAN, K. NAYAK. TEST-RETEST REPEATABILITY OF HUMAN
SPEECH BIOMARKERS FROM STATIC AND REAL-TIME DYNAMIC MAGNETIC RESONANCE IMAGING. J. ACOUST. SOC. AM. 141(5): 3323-3336,27
2017
VTSounds-2019-Sept-ICA - September 10, 2019

Vocal Tract Contours
Model-Based Image Segmentation In The Fourier DomainFirst define a contour model segmentation
manually: each articulator in a different color
Now hierarchically optimize the model fit to theimage in the Fourier domain using gradientdescent!
Erik Bresch and Shrikanth Narayanan. Region segmentation in the frequency domain applied to upper airway real-timemagnetic resonance images. IEEE Transactions on Medical Imaging. 28(3): 323--338, March 2009.
28
VTSounds-2019-Sept-ICA - September 10, 2019

Articulator Tracking

Erik Bresch and Shrikanth Narayanan. Region segmentation in the frequency domain applied to upper airway real-time29
magnetic resonance images. IEEE Transactions on Medical Imaging. 28(3): 323--338, March 2009.
VTSounds-2019-Sept-ICA - September 10, 2019

Articulatory Posture & Constriction Task Variables

These feature sets are useful for modeling speech production dynamics
Adam Lammert, Louis Goldstein, Shrikanth Narayanan and Khalil Iskarous. Statistical Methods for Estimationof Direct and Differential Kinematics of the Vocal Tract. Speech Communication. 55: 147–161, 2013.
Vikram Ramanarayanan, Adam Lammert, Louis Goldstein, Shrikanth Narayanan, "Are Articulatory Settings
Mechanically Advantageous for Speech Motor Control?", PLoS ONE, Public Library of Science, vol. 9, no. 8,30
pp. e104168, 2014.
VTSounds-2019-Sept-ICA - September 10, 2019

Tracking Constriction Variables

Vikram Ramanarayanan, Louis Goldstein, Dani Byrd and Shrikanth S. Narayanan, An investigation of articulatory setting using realtime magnetic resonance imaging (2013), in: J. Acoust. Soc. Am., 134:1(510-519)
Sorensen, T., Toutios, A., Goldstein, L., & Narayanan, S. (2016), Characterizing vocal tract dynamics with real-time MRI,
Conference on Laboratory Phonology, Ithaca, NY.

31

VTSounds-2019-Sept-ICA - September 10, 2019

real-time magnetic resonanceimaging

Lingala et al. Magn Reson Med, 2017

tissue-air boundary
segmentation

Bresch & Narayanan IEEE Trans Med Imaging, 2009

constriction degree
measurement

Sorensen et al. Proc Interspeech, 2016

VTSounds-2019-Sept-ICA - September 10, 2019

Speaker-Specific Articulatory Models

Toutios, A., & Narayanan, S. S. (2015). Factor analysis of vocal-tract outlines derived from real-time magnetic resonance imaging
data. International Congress of Phonetic Sciences, Glasgow, UK.
33
VTSounds-2019-Sept-ICA - September 10, 2019

Analysis of data
Image analysis• Deriving• morphological (structural) details, and• linguistically meaningful articulatory features• Some case studies•- Vocal tract morphology- Example linguistic and paralinguistic analyses- Relation between articulatory & acoustic representations- ASR and Speaker Verification- Articulatory strategies

VTSounds-2019-Sept-ICA - September 10, 2019

Different individuals….
nose
tonguevelum..each with a uniquely shaped vocal instrument
35
VTSounds-2019-Sept-ICA - September 10, 2019And with differing articulatory strategies during speech …

Fifteen different individuals producing vowel /i/
VTSounds-2019-Sept-ICA - September 10, 2019

Interspeaker Variability:
Vocal Tract Morphology (Structure)• Confined articulatory environment• Variability across speakers• Highly articulated, layered system• Reflected in acoustical properties

VTSounds-2019-Sept-ICA - September 10, 2019

38
VTSounds-2019-Sept-ICA - September 10, 2019

VOCAL TRACT MORPHOLOGY
Why is morphological structure relevant?
Palate Shape – Principal Components

46% of Variance

30% of Variance

10% of Varianceposterior+ 1.5 std. dev.
- 1.5 std. dev.

anterior
concavity

anteriority

sharpness

Adam Lammert, Michael Proctor and Shrikanth Narayanan. Morphological Variation in the Adult Hard Palate
and Posterior Pharyngeal Wall. Journal of Speech, Language, and Hearing Research. 2013a39

VTSounds-2019-Sept-ICA - September 10, 2019

Anteriority

Sharpnessdistance from teeth

distance from teeth

distance from teeth

δF2

Concavity

δF1

Distance (cm)

THEORETICAL IMPACT OF PALATE SHAPE

Concavity: impact on F1 and F2; Anteriority: impact on F2 only;
Sharpness: marginal
Adam Lammert, Michael Proctor and Shrikanth Narayanan. Interspeaker Variability in Hard Palate Morphology
40
and Vowel Production. Journal of Speech, Language, and Hearing Research. 2013b
VTSounds-2019-Sept-ICA - September 10, 2019

THEORETICAL IMPACT: ACOUSTIC MODELING
Palate Shapes

Formant Sensitivity to
Palatal Concavity

A
B
C
D

F1

300-

Synthesized Vowels

400500600700-

2400 2000 1600 1200 800 400
F2-F1

41

Adam Lammert, Michael Proctor and Shrikanth Narayanan. Interspeaker Variability in Hard Palate Morphology
and Vowel Production. Journal of Speech, Language, and Hearing Research. 2013b
VTSounds-2019-Sept-ICA - September 10, 2019

Experiments on estimating some ofthese vocal tract shape details fromacoustics...

VTSounds-2019-Sept-ICA - September 10, 2019

Inversion: Palatal Concavity

Motor
Controller

Binary Classification
concave or flat
palate?

SpeechArticulation

Vocal
Tract
Shape

Inversion Accuracy: 63% - 71%

Speaker ID
Features• MFCC• Open-smile• GMM UWPP
• Inv. artic.
features43
Li, Lammert, et al. (2013). Automatic Classification of Palatal and Pharyngeal Wall Shape Categories from Speech Acoustics andInverted Articulatory Signals. ISCA Workshop on Speech Production in Automatic Speech Recognition.
VTSounds-2019-Sept-ICA - September 10, 2019

Frequency (Hz)

Vocal Tract Length (cm)Variation in Vocal Tract Length, Acoustics

Vorperian (2009)

Age in Years

Vocal Tract Length (cm)44

VTSounds-2019-Sept-ICA - September 10, 2019

Estimating VT Length From Acoustics! 5 SPAN-TIMIT Subjects: real time MRI data
! Median estimated value (30 sec. read speech)r = 0.98

Could be used to account for speaker differences: new strategies
for Vocal Tract Length Normalization
Lammert, A. & Narayanan, S. On Short-Time Estimation of Vocal Tract Length from Formant Frequencies.
PLOSOne. 2015

45

VTSounds-2019-Sept-ICA - September 10, 2019going beyond linguistic details….
emotional speech production• Sungbok Lee, Shrikanth Narayanan. Vocal Tract Contour Analysis Of Emotional Speech By The Functional Data Curve
•
•
•
•
•

Representation. In Proceedings Of Interspeech, 2010.
Jangwon Kim, Sungbok Lee, Shrikanth Narayanan. A Study Of Interplay Between Articulatory Movement And Prosodic
Characteristics In Emotional Speech Production. In Proceedings Of Interspeech, 2010.
Jangwon Kim, Sungbok Lee and Shrikanth Narayanan. An Exploratory Study Of Manifolds Of Emotional Speech.
Proceedings of IEEE ICASSP. 2010.
Sungbok Lee, Tsuneo Kato and Shrikanth Narayanan. Relation between Geometry and Kinematics of Articulatory Trajectory
Associated with Emotional Speech Production. Proceedings of Interspeech 2008.
Sungbok Lee, Erik Bresch, Jason Adams, Abe Kazemzadeh and Shrikanth Narayanan. A Study of Emotional Speech
Articulation using a Fast Magnetic Resonance Imaging technique. In Proc. InterSpeech 2006.
Sungbok Lee, Serdar Yildirim, Abe Kazemzadeh and Shrikanth Narayanan. An Articulatory Study Of Emotional Speech
Production. In Proc. Interspeech, 2005.
46
VTSounds-2019-Sept-ICA - September 10, 2019

Emotion effects at the Segment level:
Vowel Triangle Example• First and second formant
frequencies for the three vowels,/IY/, /AE/, /UW/ for various
emotions• Distinct constellation fordiﬀerent emotions: Emotions
have diﬀerent eﬀects on
diﬀerent phonemes• Notice that the low vowels /AE/
and /UW/ more aﬀected byemotions than high vowel /IY/
C. M. Lee, S. Yildirim, M. Bulut, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan. Emotion recognition basedon phoneme classes. In Proceedings of ICSLP, 2004.
47
VTSounds-2019-Sept-ICA - September 10, 2019

Example MRI Movies of Emotional Speech Production
neutral

angry

sad

happy

First two principal components of tongue shape contours

word “doctor” (subject AB)

Tongue tip positioning differences across emotions
48
VTSounds-2019-Sept-ICA - September 10, 2019

Emotional Speech Production
Emotion, for a given linguistic context, modulates•kinematics (i.e., movement range and velocity)•constriction place (e.g., tongue-tip positioning)•Speakers seem to exploit possible articulatory freedom for emotional contrast•within allowed variability in a given linguistic context•Emotion expression does not aﬀect the non-linear timing pattern variations much•localized variation in articulatory timing after duration normalization
Implications for information encoding model for speech communication
Sungbok Lee and Shrikanth Narayanan, Effects of Emotion on the Lower Lip Movements at Phrase Boundaries. in:Proceedings of Speech Prosody, 2012.
Jangwon Kim, Sungbok Lee and Shrikanth Narayanan. An Exploratory Study of the Relations between Perceived Emotion
Strength and Articulatory Kinematics. In Proceedings of InterSpeech, 2011.
Sungbok Lee, Shrikanth Narayanan. Vocal tract contour analysis of emotional speech by the functional data curve
representation. In Proceedings of InterSpeech, 2010.
49
VTSounds-2019-Sept-ICA - September 10, 2019

Beyond speech…

SopranoBeatboxing

Erik Bresch and Shrikanth Narayanan. Real-time MRI investigation of resonance tuning in soprano singing. J. Acoust. Soc. Am.,
128(5): EL335-341, 2010.
Michael Proctor, Erik Bresch, Dani Byrd, Krishna Nayak, and Shrikanth Narayanan. Paralinguistic Mechanisms of Production in Human'Beatboxing:' a Real-time Magnetic Resonance Imaging Study. J. Acoust. Soc. Am. 133(2): 1043-1054, 2013.
Reed Blaylock, Nimisha Patil, Timothy Greer, and Shrikanth Narayanan. "Sounds of the human vocal tract." Interspeech, 2017.
Nimisha Patil, Timothy Greer, Reed Blaylock, and Shrikanth Narayanan. "Comparison of Basic Beatboxing Articulations Between
Expert and Novice Artists using Real-Time Magnetic Resonance Imaging." Interspeech, 2017.

50

VTSounds-2019-Sept-ICA - September 10, 2019

“The Soprano Challenge”: Sung vowels• Resonance tuning and vocal tract shaping in soprano singing?
• How do opera singers tune the vocal tract resonances to match thepitch while singing? (Remember they have to be heard!)• Focus on vowel sounds: “LA”, “LE”, “LI”, …
• Derive vocal tract resonance frequencies from images using acoustic
tube models (instead of audio, which would be diﬃcult at high F0)

51
VTSounds-2019-Sept-ICA - September 10, 2019

Image analysis examples /a/• spoken• F0 = 233Hz • F0 = 349Hz • F0 = 622Hz • F0 = 932Hz• Note 11• Note 15• Note 1• Note 5

52
VTSounds-2019-Sept-ICA - September 10, 2019

Vocal tract shapes across vowels in comparison
• /e/

• /i/

• /o/

• /u/

F0 = 932Hz

F0 = 233Hz

spoken• /a/53
VTSounds-2019-Sept-ICA - September 10, 2019

Sung vowels: Resonance tuning

Resonance tuning canbe shown for vowels
with low F1F1F0

Erik Bresch and Shrikanth Narayanan. Real-time MRI investigation of resonance tuning in sopranosinging. J. Acoust. Soc. Am., 128(5): EL335-341, 2010

54

VTSounds-2019-Sept-ICA - September 10, 2019

How Beatboxers Produce Sounds• What do beatboxers use to produce percussion sounds?
• Are these beatboxing sounds similar to those they aremeant to mimic?
• How do they compare to the documented sounds of thelanguages?
• What variation is in the way different beatboxers produce
sounds?
• Unique study:–Few previous visualizations or analyses of beatboxers’ vocal tracts!55
VTSounds-2019-Sept-ICA - September 10, 2019

Observation• Beatboxers have an ability to mimic percussion
sounds with accuracy and consistency• Beatboxers can string together these percussion
sounds to make a beat
• These sounds involve production techniques thatare found in language production!56
Michael Proctor, Erik Bresch, Dani Byrd, Krishna Nayak, and Shrikanth Narayanan. Paralinguistic Mechanisms
of Production in Human 'Beatboxing:' a Real-time Magnetic Resonance Imaging Study. J. Acoust. Soc. Am.
133(2): 1043-1054, 2013.
VTSounds-2019-Sept-ICA - September 10, 2019

Methods• 5 subjects, all native English speakers
Table 1: Scanned Subjects
Skill Level

GenderExperience

AdvancedMale

5 years

Advanced

Female9 years

Intermediate

Female3 years

NoviceMale

6 years

NoviceFemale3 years!57

VTSounds-2019-Sept-ICA - September 10, 2019

Kick Drum

Spectrograms!58
VTSounds-2019-Sept-ICA - September 10, 2019

PF Snare!59

Spectrograms
VTSounds-2019-Sept-ICA - September 10, 2019

What sounds can beatboxers make?
• In addition to sounds attested in language, we also
found sounds unattested in language• Unattested articulations––––

Lateral bilabial trill
Retroflex trill
Inward clickroll
Inward clickuoll with liproll• Unattested airstreams:– Pulmonic ingressive– Lingual egressive!60

Reed Blaylock, Nimisha Patil, Timothy Greer, and Shrikanth Narayanan. "Sounds of the human vocal tract."
Interspeech, 2017.
VTSounds-2019-Sept-ICA - September 10, 2019

Clickroll (lingual egressive)!61

VTSounds-2019-Sept-ICA - September 10, 2019

Inward Clickroll with Liproll!62

VTSounds-2019-Sept-ICA - September 10, 2019

Ongoing Work
• Do beatboxers use the same ready postures when
beatboxing as when speaking?
• How do advanced beatboxers move their vocal tracts
compared to novice beatboxers?
• Are beatboxing and speech sounds represented
separately or together?
!63

sail.usc.edu/span/beatboxingproject/
VTSounds-2019-Sept-ICA - September 10, 2019

Modeling production mechanisms

64
VTSounds-2019-Sept-ICA - September 10, 2019

Articulatory synergies
how talkers use their articulatory organs• Vocal tract is a redundant system• Articulators have overlapping functions• e.g., both jaw and lips contribute to bilabial constrictions• Speakers have several ways to change airway
shape to make a constriction• We call these articulatory synergies

Tanner Sorensen, Asterios Toutios, Louis Goldstein, and Shrikanth Narayanan. Task-dependenceof articulator synergies. 145(3): 1504-1520, J. Acoust. Soc. Am. 2019
VTSounds-2019-Sept-ICA - September 10, 2019

Quantifying Individual Articulatory Synergy
Predicting how diﬀerent speakers navigate this flexible system
Estimate the forward kinematic map, which relates vocal
tract shape to constriction degree• how a speaker uses their forward kinematic map reflects
their articulatory synergy

Analysis-by-synthesis approach• estimate forward kinematic map using real time MRI data
• run dynamical system simulations• simulations display predicted articulatory strategies

modeling goal
quantify how much the jaw, tongue, and lips
contribute to a constriction of the vocal tract
VTSounds-2019-Sept-ICA - September 10, 2019

Approach

67
VTSounds-2019-Sept-ICA - September 10, 2019

Articulatory strategies across speakers
more tongue/lips

ratio of lips/tongue use
to jaw use
by speaker (dots) andconstriction type (x-axis)

more jaw

insights using data from 18 speakers (9F, 9M) in task dynamic simulation
Tanner Sorensen, Asterios Toutios, Louis Goldstein, Shrikanth Narayanan. Characterizing vocal tract dynamics across speakers
using real-time MRI. Proc. Interspeech, 2016 [BEST STUDENT PAPER]
VTSounds-2019-Sept-ICA - September 10, 2019

Alveolar closuresmall jaw movement, speaker M3

prediction

observation
69
VTSounds-2019-Sept-ICA - September 10, 2019

Alveolar closure
large jaw movement, speaker F9prediction

observation
70
VTSounds-2019-Sept-ICA - September 10, 2019

Summary• real time MRI of the vocal tract can be used toestimate forward kinematic map• forward kinematic map differs by speakeraccording to vocal tract geometry• articulatory strategies predicted on the basis of
vocal tract geometry can be compared against
observed strategies→ tool for relating vocal tract structure and function

Tanner Sorensen, Asterios Toutios, Louis Goldstein, and Shrikanth Narayanan. Task-dependenceof articulator synergies. 145(3): 1504-1520, J. Acoust. Soc. Am. 201971
VTSounds-2019-Sept-ICA - September 10, 2019

Insights into someClinical Applications

72
VTSounds-2019-Sept-ICA - September 10, 2019

Clinical Examples

Apraxia

Cancer/Oral tongue

Cancer/Base tongue

Post Glossectomy Scans

Yihe Zu, Shrikanth Narayanan, Yoon-Chul Kim, Krishna Nayak, Christina Bronson-Lowe, Brenda Villegas,
Melody Ouyoung, Uttam Sinha. Evaluation of swallow function post tongue cancer treatment using real-time
MRI: A pilot study. JAMA Otolaryngology - Head & Neck Surgery. 2013
Christina Hagedorn, Michael Proctor, Louis Goldstein, Maria Luisa Gorno Tempini, Shrikanth Narayanan.
Characterizing Covert Articulation in Apraxic Speech Using real-time MRI. In Proceedings of InterSpeech,
2012.
73
VTSounds-2019-Sept-ICA - September 10, 2019

Characterizing Articulation in Apraxic Speech
rtMRI reveals covert (unphonated) articulation of entire words“I can TYPE KNOW know...”

LAB
COR
DOR

Clinical Takeaway: Apraxia of speech affects ability to select appropriate
vocal tract movements for a target word/phrase and coordinate them in time,
suppressing other movements.
Errors may not always be auditorily perceptible
Christina Hagedorn, Michael Proctor, Louis Goldstein, Stephen Wilson, Bruce Miller, Maria Luisa Gorno-Tempini, and Shrikanth S. Narayanan. “Characterizing
Articulation in Apraxic Speech Using Real-time Magnetic Resonance Imaging.” Journal of Speech, Language and Hearing Research (2017)!74
VTSounds-2019-Sept-ICA - September 10, 2019

Head and Neck Cancer
Head and neck cancer impairs speech and swallowing• Cancer-associated cachexia, Peripheral nerve damage• Radiation-induced fibrosis• Surgical treatment effects

Yihe Zu, Shrikanth Narayanan, Yoon-Chul Kim, Krishna Nayak, Christina Bronson-Lowe, Brenda Villegas, Melody
Ouyoung, Uttam Sinha. Evaluation of swallow function post tongue cancer treatment using real-time MRI: A pilot
study. JAMA Otolaryngology - Head & Neck Surgery. 2013!75

VTSounds-2019-Sept-ICA - September 10, 2019

Characterizing Post-Glossectomy Speech
What aspects of consonant and vowel production are
maintained or lost following a partial glossectomy?
How do glossectomy patients compensate for reduced
lingual mass and mobility, post-operatively?
Consonants• Constriction degree (amount of occlusion) is maintained• Constriction location is not maintained• Patients use articulators other than those conventionally used
• e.g., target coronal constriction —> labiodental constriction
Vowels• Constriction degree is not maintained• Exceedingly wide constrictions result in poor vowel intelligibility• Constriction location is maintained

Clinical Takeaway: Patients would likely benefit from pre- and post-operative
speech therapy focused on using jaw height modulation and lip protrusion tocompensate for reduced lingual mobility during vowel production
Christina Hagedorn, Adam Lammert, Mary Bassily, Yihe Zu, Uttam Sinha, Louis Goldstein and Shrikanth S. Narayanan. Characterizing Post-Glossectomy Speech
Using Real-time MRI, In Proceedings of the 10th International Seminar on Speech Production. In Proceedings of the 10th International Seminar on Speech!76
Production (ISSP). 2014.
VTSounds-2019-Sept-ICA - September 10, 2019

Results: Patient M1: Anterior and Posterior Area Measures• Patient M1’s absolute area values greater than non-patient (“typicals”)• not surprising, given resection
77
VTSounds-2019-Sept-ICA - September 10, 2019

Results: Patient M1: Acoustic Measures: Formant frequencies
F2== “front cavity” resonanceF1== “back cavity” resonance

Acoustic Analysis(Hillenbrand et al. 1995)• Vowel space (red) compressed rightward• F1 values more often ‘atypical’ than F2 values• F1 values low: vowels are produced low in the mouth• Almost all atypical F2 values occur for front vowels

78
VTSounds-2019-Sept-ICA - September 10, 2019

USING THE SOUNDS OF THE HUMAN VOCAL INSTRUMENT
A whole range of speech/language technology possibilities• to uncover carries crucial linguistic and paralinguistic information•about intent, desires, emotions, identity, health state,..

Topics of active engineering research and development•Can benefit from the scientific knowledge of speech
Who spoke when, for how long, and about what?
kHz

SpeechSpectrogram
Voice Activity
DetectionS

Speaker
Diarization

1

S

S

S

S

3

2

Come
on!

Ok!

Detected Speech Regions
2

11

3

Speakers in the conversation
I am good.

Hello!

Transcription

1S

How are you?
We should go.

79

VTSounds-2019-Sept-ICA - September 10, 2019

Automatic
SpeechRecognition

So ‘n’ your chest pains have been going on just for two days is that right

What more can we infer beyond words?
Words:

So ‘n’ your chest pains have been going on just for two days is that rightSpeaker:

spkr1 (Doctor)

Gender:

Male

Age:

Adult

Prominent words: So ‘n’ your chest pains have been going on just for two days is that right
Prosodic phrasing: [So ‘n’ your chest pains] [have been going on just for two days] [is that right]
Speech act:

Yes-No Question

Aﬀect:

NeutralAttitude:

PoliteRich information beyond words
VTSounds-2019-Sept-ICA - September 10, 2019

USING THE SOUNDS OF THE HUMAN VOCAL INSTRUMENT
A whole range of speech/language technology possibilities• to uncover carries crucial linguistic and paralinguistic information•about intent, desires, emotions, identity, health state,..

Topics of active engineering research and development•Can benefit from the scientific knowledge of speech
Who spoke when, for how long, and about what?
kHz

SpeechSpectrogram
Voice Activity
DetectionS

Speaker
Diarization

1

S

S

S

S

3

2

Come
on!

Ok!

Detected Speech Regions
2

11

3

Speakers in the conversation
I am good.

Hello!

Transcription

1S

How are you?
We should go.

81

VTSounds-2019-Sept-ICA - September 10, 2019

Concluding Remarks:
Computational Human Speech Science• Multimodal data integral: to advance science and technology• Requires concurrent progress in sensing, processing and modeling• Need to gather and integrate multiple, disparate sources of information
toward getting a more complete picture of human communication• The problem is highly challenging: demands novel approaches• Technological, computational as well as conceptual/theoretical challenges• Open challenges for forward & inverse modeling of speech acoustics• Potential for high impact scientific, technological, and clinical applications• Highly interdisciplinary•

Acquiring, interpreting and utilizing human communication information is an
ongoing interdisciplinary scientific endeavor
82
VTSounds-2019-Sept-ICA - September 10, 2019

“....technologies to understand the human condition and support and enhance human capabilities and experiences”

Work reported represents eﬀorts ofnumerouscolleagues and collaborators
Too many to name, but grateful to all
Supported by NSF, NIH, DoD

SPAN
speech production and articulation knowledge group
SAIL — University of Southern California
investigating language production: from its cognitive conception, toits biomechanical execution, to its signal properties

Papers, Videos, Teaching resources

http://sail.usc.edu/span
VTSounds-2019-Sept-ICA - September 10, 2019

ACKNOWLEDGEMENTS
•
•
•
•
•

NIH Grants DC007124 & DC03172
NSF, ONR, DoJ
USC Imaging Sciences Center
LAC-USC Hospital
USC Center for High Performance Computing

Papers, Videos, Teaching resources

http://sail.usc.edu/span

84
VTSounds-2019-Sept-ICA - September 10, 2019