Speech-Recognition Interfaces forMusic Information Retrieval:“Speech Completion” and “Speech Spotter”
Masataka Goto
Katunobu Itou
Koji Kitayama
Tetsunori Kobayashi

AIST (National Inst. of Advanced Industrial Sci. & Tech.)
Nagoya University
Waseda University
Waseda University
2004/10/13 ISMIR 2004

No. 1

MIR + Speech Recognition = ?
R Speech-Recognition Interface is Well-Suited to MIR• MIR-based jukebox system with speech-recognition interface
Retrieve a musical piecejust by saying the name of musical piece or artist
Change background musicat homein a carFind musical pieces
at music-listening stations in music storeson karaoke machines

No. 2

Previous MIR Interface
R Text Query• Enter bibliographic information by keyboard or mouse
R Example-based Query• Find a similar musical piece by (a) musical piece(s)R Melody-based Query• Enter a melodic contour by symbols, MIDI, or audio• Query By Humming (QBH) is promising
Require only a microphone
Can easily be used by a novice
R Speech Query• Not exploited how speech recognition can be used for
retrieving music information
No. 3

Our Contribution
R Two Original Speech-Recognition Interfaces• Speech Completion• Speech Spotter

R Two MIR-Based Jukebox Systems• Music retrieval system with the Speech-Completion interface• Music playback system with the Speech-Spotter interface
Find and play back a musical pieceby saying its title or the artist’s nameNo. 4

Speech Completion

Speech Completion

No. 5

Guess What's Happening?
R I’ll Explain Later!

No. 6

Music Retrieval with Speech Completion
R The System Provides “Completion” Assistance• You can retrieve a musical pieceeven if you cannot remember a part of the name
R Learning from Human-Human Conversation

You cannot remember a phrase “Michael Jackson” and hesitate
A listener can help you to recall it“Michael—” (Michael, uh…)

Filling in the rest of a fragment＝Completion
No. 7“Michael Jackson?”Music Retrieval with Speech Completion
R The System Provides “Completion” Assistance• You can retrieve a musical piece
noieven if you cannot remember a part of thennamectuf”
noR Learning from Human-Human Conversationitel!
pemcoaYou cannot remember a phrase“MichaelJackson”
and hesitate
fcr“etanidehyou to recall itsAolistener canehelpcpeoprspr
yof“Michael—” (Michael, uh…)
dobo
N Filling in the rest of a fragment＝Completion
No. 8“Michael Jackson?”Speech Completion
R What is Speech Completion?

• Help a user enter an uncertain piece/artist nameby completing the missing partof a partially uttered fragment“Michael—” (Michael, uh…)

“Michael Jackson?”No. 9

Prevent Completion Becoming Annoying
R How to Invoke Speech Completion Function?

• Should be invoked only when a user needs an assistance
People hesitates when having trouble recalling
R Invoke by Using Filled Pause (Lengthened Vowel)“Michael—” (Michael, uh…)

“Michael Jackson?”• Filled pause is a very natural trigger for human• Can invoke the function intentionally and effortlessly
Frequently used in the same way in Japanese conversation
No. 10Complete in Forward or Backward Direction
R Forward Speech Completion?

• Do not remember the last part of a name
“Michael—” (Michael, uh…)

“Michael Jackson”

R Backward Speech Completion• Do not remember the first part of a name“Nantoka— (Something—) Jackson”
Wildcard keyword
No. 11?
“Michael Jackson”“Janet Jackson”

System Output (Video Demonstration)
R Demonstration of Speech Completion• Enter the Japanese names of musicians and songs“Michael Jackson”“MAIKERU JAKUSON”(in Japanese)“Michael—”“MAIKERU—”

No. 12

Advantages of Speech Completion
R Provide Three Benefits

1. A user can more easily recall poorly remembered names
2. Less labor is needed to input a long name“Supercalifragilisticexpialidocious” (a song from “Mary Poppins”)

3. The user is not forced to utter the entire namecarefully and precisely
R Can Be Used for Various MIR Situations• Playing back a SMF or karaoke track• Making a play list• Editing and downloading music files
No. 13

Speech Spotter

Speech Spotter

No. 14

Guess What's Happening?
R I’ll Explain Later!

No. 15

Music Playback with Speech Spotter
R On-Demand Music Playback Assistancein Human-Human Conversation• You can listen to background music by saying
its title or the artist’s name while talking to another person“Shall we listen to the song `Black or While’ ?”“Yeah! Uhm…, Black or White.”No. 16

How To Achieve On-Demand Assistance
R Problem To Be Solved• Monitor human-human conversations
without disturbing them
• Provide music playback only when asked for it
R Difficult To Achieve By Using Only Microphone• Word-spotting technology
Poor at judging whether the detected keywords are
Command utterances for a computer system or
Conversational utterances for a conversational partner• Using button or camera
Require input devices other than a microphone
Cannot be used in a telephone conversation
No. 17

Speech Spotter
R What is Speech Spotter?

• Regard a user utterance as a command utterance
only when it is intentionally utteredwith a high pitch just after a filled pause (e.g., “er…”)“Shall we listen to the song `Black or While’ ?”“Yeah! Uhm…, Black or White.”No. 18

Speech Spotter
R What is Speech Spotter?

• Regard a user utterance as a command utterance
only when it is intentionally utteredwith a high pitch just after a filled pause (e.g., “er…”)
This combinationis quite
unnatural“Shall we listento the song
`Black or While’ ?”The system accepts
this specially-designed unnatural utterance only
“Yeah! Uhm…, Black or White.”This does not appear in natural conversation
The system can easily spot it!
No. 19System Output (Video Demonstration)
R Demonstration of Speech Spotter• Enter voice commands for music-playback control

No. 20System Implementation
R Implementation of Two Interfaces• Several difficulties should be overcome
R New Technology• Real-time filled-pause detection
Detect a lengthened vowel in any word

• Completion-candidate generation
Generate candidates by tracing on a vocabulary tree

• Endpoint detection
Detect the beginning and end of an utterance• Utterance-pitch classification
Distinguish between normal and high-pitch utterances
No. 21

Conclusion
R Summary• MIR system with Speech Completion
Listen to a piece even if part of its name cannot be recalled• MIR system with Speech Spotter
Share music playback on the telephone
R Practical Speech-Recognition Interfaces for MIR• Cannot be achieved by simply applying
the current ASR (automatic speech recognition) to MIR
R First Step Toward Building Ultimate MIR Interface• Important to explore various speech-recognition interfaces
for MIR
No. 22

Future Directions
R QBH + ASR• Build a unified system using a microphone input
query-by-humming systems
speech-recognition interfaces• Leverage the potential affinity between them+No. 23+