1Representing ‘how you say’ with ‘what you say’:
English corpus of focused speech and text
reflecting corresponding implications
Naoaki Suzuki, Satoshi Nakamura
Nara Institute of Science and Technology, Japan
INTERSPEECH 2022

2Introduction

Linguistic and Paralinguistic information• Speech communication• Implications can be different• Even with the same linguistic information3

Speech Translation and its Limitation• Speech translation• Translates speech in one language into text/speech in another• Limitation• Unable to consider paralinguistic info
• If the linguistic info is the same, so are the translations

4

An Approach to Paralinguistic Translation• Acoustic to acoustic mapping• Mapping acoustic cues in the source language (SL) to the counterparts in thetarget language (TL) [Aguero+ 2006, Kano+ 2013, Do+ 2018]• What if prosodic counterparts do not exist in the TL?

5

6

Proposed Method

Acoustic – Linguistic Mapping• Three devices for focus [Cruttenden 1997]• Acoustic – linguistic mapping by paraphrasing

7

Research Questions• Fundamentals for the achievement of acoustic – linguisticfocus transformation
1. Corpus construction• Speech having different items in focus• Text reflecting the relevant implications

2. Relationships between focused speech and focused text
• What kind of methods are used for paraphrasing?

89Corpus Construction
How to build the corpus

10

Text Design• Flickr8k [Rashtchian+ 2010]• 8000 images which depict actions relating to people or animals• Five text descriptions are given for each image
A beagle and a golden retriever wrestling in the grass
Two dogs are wrestling in the grass.
Two puppies are playing in the green grass.
two puppies playing around in the grass× 8000
Two puppies play in the grass• Selected 196 short sentences as the source (words length: max six words)• Focus placement• every word to be the target of focus

Place for the Data Collection

11• Amazon Mechanical Turk (MTurk)• a crowd sourcing platform• allows researchers to create tasks called HITs and anonymous users (Workers)
to complete them for a small monetary fee

12Speech Collection• Underlined text

• Instructions• Subjects• Results• 10 British native English speakers• 3 speakers/caption• focused: 2800
• normal (without focus): 600

13Paraphrase Collection• Focused speech•a• Instructions• biker• Subjects• 16 native English speakers• 2 paraphrasers / focused speech• Results• 2100 paraphrases• e.g. ‘It is a biker enjoying a coffee’Paraphrase Evaluation for Ensuring Quality• Pairs of• Instructions• focused speech• paraphrase• Subjects• 16 native English speakers• 3 participants/paraphrase• Results• 1700 paraphrases• Five scales (1 – 5) for both14

Data Example

15

16Analysis
What kind of transformation methods were used forparaphrasing?

Broad categorization by transformation methods• How focus in speech was mapped into focus in text?
• We manually examined the original text – paraphrase pairs• Lexical

17

Broad categorization by transformation methods• How focus in speech was mapped into focus in text?
• We manually examined the original text – paraphrase pairs• Syntactical

18

Occurrences of each transformation per part-of-speech• A certain part-of-speech was more likely to use a certaintransformation method• randomly sampled 50 paraphrases for each part-of-speech• counted each transformation method for each paraphrase

Mean occurrences of each transformation method per part-of-speech (N: Noun, V: Verb, Adj:Adjective, Num: Numeral, Aux: Auxiliary, P: Preposition, Det: Determiner)

19

20Discussion and Conclusion

Discussion
1. Corpus construction• Speech having different items in focus• Text reflecting the relevant implications

2. Relationships between focused speech and focused text
• What kind of methods are used for paraphrasing?
• Broad categorization of transformation methods• Tendency dependent on part-of-speech• Limitation• Lack of context21Conclusion

22• A new direction for paralinguistic translation• Demonstrated the possibility of mapping paralinguistic info to the linguistic
domain with lexical and syntactic devices• The corpus and insights from our analysis will lead us to construct a speechtranslation model which can preserve paralinguistic information

References

23

Aguero, P. D., Adell, J., & Bonafonte, A. (2006, May). Prosody generation for speech-to-speech translation. In 2006 IEEE International
Conference on Acoustics Speech and Signal Processing Proceedings (Vol. 1, pp. I-I). IEEE.
Cruttenden, A. (1997). Intonation. Cambridge University Press.
Derwing, T. M., Thomson, R. I., Foote, J. A., & Munro, M. J. (2012). A longitudinal study of listening perception in adult learners of English:Implications for teachers. Canadian modern language review, 68(3), 247-266.
Do, Q. T., Sakti, S., & Nakamura, S. (2018). Sequence-to-sequence models for emphasis speech translation. IEEE/ACM Transactions on Audio,Speech, and Language Processing, 26(10), 1873-1883.
Downing, L. J. (2006). The prosody and syntax of focus in Chitumbuka. ZAS papers in linguistics, 43, 55-79.
Kano, T., Takamichi, S., Sakti, S., Neubig, G., Toda, T., & Nakamura, S. (2013, August). Generalizing continuous-space translation of paralinguistic
information. In INTERSPEECH (Vol. 445, pp. 25-29).
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010, June). Collecting image annotations using amazon’s mechanical turk.
In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk (pp. 139-147).