PodCastle

Recent Advances of A Spoken Document Retrieval Service
Improved by Anonymous User Contributions
Masataka Goto and Jun Ogata (AIST, Japan)

What is PodCastle?
 Full-text retrieval of speech data
• Podcasts (audio blogs)• Individual audio files• Video clips(YouTube, Ustream.tv, and Nico Nico Douga)

podcastSearchIn this paper, we describe a public web service,"PodCastle", that provides full-text searching ofJapanese podcasts on the basis of automatic
speech recognition. This is an instance of ourresearch approach, "Speech Recognition
Research 2.0", which is aimed at providing users
with a web service based on Web 2.0 so that theycan experience state-of-the-art speech per-Three Functions

 Speech retrieval web service
based on ASR and crowdsourcing

 Automatic learning from the web• Automatically collect new words/phrases,
their pronunciation, and usage examples• View the transcribed text of speech data
• Each word is colored according to
the degree of ASR reliability• Full text can be indexed and accessedby external search engines (e.g., Google)

News articles (Yahoo! news) and web dictionaries• Difficult to achieve high accuracy• Diversity of topics, vocabularies,and speaking styles• Improve retrieval performancesby correct indices• Improve recognition performancesby automatic learning (adaptation/training)

The subprime loan crisis
is far from over and ...

 Reading function• Add new words to ASR dictionary (0.24M words)

 Users can find and correct ASR errors

ASR result• Full-text search of ASR results• List of speech data containing a query isdisplayed together with text excerpts• Excerpts can be played back individually• Collect and amplify voluntary contributions
by anonymous users

 ASR (automatic speech recognition)
for text transcription

Speech data

 Searching function• Original efficient error correction interface[Ogata & Goto, Interspeech 2005]

 Annotating function (error correction)• Add "annotations" to correct ASR errors• Select the correct candidate from the list
The list is generated by using a confusion networkthat condenses a huge internal word graph• Type in the correct text
• Corrected errors can be used for improving
retrieval and recognition performances Difficulties and Problems• Difficult to support new words/phrases(proper names and buzzwords)
Podcasts often include out-of-vocabulary words• Difficult to launch a spoken document
retrieval service with high accuracy
Users might be disappointed by ASR results

Implementation

PodcastSpeech corpus cannot be prepared in advance

Podcast(MP3+ RSS)
Podcast(MP3+ RSS)
Podcast(MP3+ RSS)
(MP3 + RSS)• Cannot avoid making recognition errors
for various types of speech data

JavaScriptChasenRuby on Rails QuickTime/FlashMySQL/Senna MTASC/Flex

Database managerSpeech recognition manager
Speechrecognizer
recognizer
SpeechSpeechrecognizer
Speechrecognizer
Speechrecognizer

Search engine

PodCastle

Web crawler

GoalUser interface
User interface
User interface

Recent Advances
HistoryExperiences http://podcastle.jp since 2006
• 2006/01• 2006/12

 How widely used? (as of May 31st, 2011)

Started the project
Released to the public• 765 Japanese speech programs
Podcasts and YouTube channels

The world's first speech retrieval projectusing crowdsourcing• 2007/08• Consist of 112,476 audio files in total• 2,593 audio files were partially corrected• 521,938 corrected words (errors)

Interspeech 2007 papers

Speech Recognition Research 2.0• 2008/06

52.8% were corrected by the candidate selection
47.2% were corrected by the text typing

Press release

Reported in TV/web news, newspapers, etc.

• 2009/08• 2009/09• 2011/01• 2011/03• 2011/??

Supported video podcasts
Interspeech 2009 paper
Supported YouTube/Ustream.tv
Supported Nico Nico Douga
Launch the English version

 Recently supported functions• There are users who voluntarily cooperate
in the correction
Speech data recorded by famous artists and TV
personalities tend to receive many corrections
Some podcasts were corrected
almost everyday or every week

 ASR performance improvements• Collaborative training of speech recognizer• Podcast-dependent acoustic model trainedusing transcripts corrected by users• Support video sharing services• Annotate speaker names and paragraphs• Mark (change the color of) correct words
that do not need any correction• Show the percentage of correction[Ogata & Goto, Interspeech 2007, 2009, SSCS 2009]• Confirmed that ASR performance forpodcasts receiving many corrections wasactually improved by this AM training

100% when all words are corrected or marked• Support simultaneous correction by users

Relative error reduction of 21-33%[Ogata & Goto, Interspeech 2009]

Corrections can be automatically shared(synchronized) and shown on their screens
Useful for serious and rapid transcription# episodes (MP3s)
112476# podcasts

1200
1000
8006002008/6 : Press release
Reported in TV news,
newspapers

765

2000
2006/11 2007/05 2007/11 2008/05 2008/11 2009/05 2009/11 2010/05 2010/11 2011/05# episodes (x100)# searches (x100)# corrected words
5219386000
5000
4000# searches
71250# podcasts• Confirmed that ASR performance was also
improved by language model training

3000

400Motivations

2000# correctedepisodes (MP3s)2593

02006/11 2007/05 2007/11 2008/05 2008/11 2009/05 2009/11 2010/05 2010/11 2011/05# corrected episodes• Correction itself is enjoyable and interesting• Users want to contribute• Users want their speech data
to be correctly searched• Users like the content and cannot tolerate
the presence of recognition errors in itSummary Technical contribution• Investigate how far the ASR performancecan be improved
through the cooperative efforts of many end users• PodCastle: Social correction framework
Users gain a real sense of contributing to theconvenience of themselves and other users• Other game-based approaches often
depend on the feeling of fun
Human Computation or GWAPs (games with a purpose)
Lack the feeling that the improved performanceleads to a better user experience ASR contribution• Demonstrate how ASR can be put to use
in situations where a corpus is difficult to prepare Beyond Web 2.0 and Human Computation• Framework for amplifying user contributions
Improvements are automatically spread toother items not contributed by users

2008/6 : Press release
Reported in TV news,
newspapers

1000

 Why did users correct errors?

# corrected words(x100)

Video clip of PodCastle:
http://staff.aist.go.jp/m.goto/PodCastle/
2011/08/28 Interspeech 2011 poster