PodCastle
Goal

PodCastle

 Full-text retrieval of speech data
• Podcasts (audio blogs)• Individual audio files• Video clips (YouTube, Ustream.tv, and Nico Nico Douga)

podcastSearchIn this paper, we describe a public web service,"PodCastle", that provides full-text searching ofJapanese podcasts on the basis of automatic
speech recognition. This is an instance of ourresearch approach, "Speech Recognition
Research 2.0", which is aimed at providing users
with a web service based on Web 2.0 so that theycan experience state-of-the-art speech per- ASR (automatic speech recognition)
for text transcription• Difficult to achieve high accuracy• Diversity of topics, vocabularies,and speaking styles
Speech data

ASR result
The subprime loan crisis
is far from over and ...

http://podcastle.jp

 Speech retrieval web service
based on ASR and crowdsourcing• Collect and amplify voluntary contributions
by anonymous users

 Automatic learning from the web• Automatically collect new words/phrases,
their pronunciation, and usage examples
News articles (Yahoo! news) and web dictionaries• Add new words to ASR dictionary (0.24M words)

 Users can find and correct ASR errors• Original efficient error correction interface[Ogata & Goto, Interspeech 2005]• Improve retrieval performancesby correct indices• Improve recognition performancesby automatic learning (adaptation/training)

Collaborative Training of ASR

Usage Analysis of PodCastle

 Incorporate error corrections(wisdom of crowds) into ASR training

 How widely used? (as of Jan 1st, 2012)• Collaborative training of acoustic models (AMs)
Podcast-dependent acoustic model trainedusing transcripts corrected by users[Ogata & Goto, Interspeech 2007, 2009, SSCS 2009]

Relative error reduction of 21-33%• 877 Japanese speech programs
Podcasts and YouTube channels• Consist of 147,280 audio files in total• 3,279 audio files were partially corrected• 580,765 corrected words (errors)
177.1 corrected words per corrected file on average
51.8% were corrected by the candidate selection
48.2% were corrected by the text typing[Ogata & Goto, Interspeech 2009]• Collaborative training of language models (LMs)[Ogata & Goto, Interspeech 2012: this paper] Generate content-specific LM training data

• There are users who voluntarily cooperate
in the correction
Speech data recorded by famous artists and TV
personalities tend to receive many corrections• Overcome difficulties in preparingtask-specific language models in advanceA common approach of building a task-specific corpus
in advance is impractical because it will be too
costly and time consuming to cover the diversity• Use the benefits of our web service PodCastle

2000
1800
1600
1400
1200
1000
800
600
400
2000

2008/6, 2011/10: Press release
Reported in TV/web news,
newspapers, etc.

# episodes (MP3/video clip)188538# searches
134319# podcasts
914

2006/11 2007/05 2007/11 2008/05 2008/11 2009/05 2009/11 2010/05 2010/11 2011/05 2011/11 2012/05# podcasts# episodes (x100)# searches (x100)

6000

tSelecTypein5000
4000
3000

2008/6, 2011/10:
Press release
Reported in TV/web
news, newspapers

2000

Candidate list is generated by using a confusion network• Transcripts voluntarily corrected by end users
are used for language-model training# corrected words
590579# corrected episodes(MP3/video clip)
3598

1000
0

2006/11 2007/05 2007/11 2008/05 2008/11 2009/05 2009/11 2010/05 2010/11 2011/05 2011/11 2012/05# corrected episodes# corrected words(x100)

Collaborative Training of Language Models
on the Basis of Wisdom of Crowds

Jun Ogata and Masataka Goto (AIST, Japan)

Dynamic Language Modeling System Dynamically mix two kinds of LMs
for each input episode (audio file)• Use corrected ASR transcripts on PodCastle• Iterative adaptation system for LVCSRTopic LM 1 ... Topic LM 25

ChannelMdependent LM

Episodede 1
de 2
Episode...

Episode n-1

ASR

Episode nInitial transcript• Use web news text from news aggregation site
Text articles (Yahoo! news) published during 40 months• Each covers a different topic
Train 25 topic LMs (6 main topics with 25 sub-topics) Channel-dependent LM• Use ASR transcripts corrected by end users• Generate LM on the fly by using all the otherepisodes within the same channel/programLM mixture with initial weights
Initial LM

 Topic LMs (background LMs)

Channel(podcast, YouTube channel,and Ustream channel)

Assume that episodes belonging to each channel
have a linguistic similarity in terms oftopics, vocabularies, and frequent phrases

 Dynamic mixture of topic LMs• Model-level mixture scheme
Combine n-gram probabilities of 26 component LMs(25 topic LMs with channel-dependent LM)

pmix ( w | h)   i pi ( w | h)LM mixture with optimized weights

iMixture weights i are computed by minimizing theperplexity on initial ASR transcript via EM algorithm

Adapted LMExperiments

History Speech data (partially corrected on PodCastle)• Test data: 45 episodes from 8 Japanese podcasts• Training data for channel-dependent LMIDABCDEFGHDomainnews
news
lecturelecturequiz show
lecturechitchatchitchat

Main topic
multi
multi
politics
business
general knowledge
health & caremusicshow business

Test data#episodes (#words)
4 (11,170)
4 (4,937)
20 (13,876)
5 (10,763)
2 (1,910)
2 (3,292)
4 (25,239)
4 (9,251)

Training data
#episodes (#words)
94 (259,840)
23 (24,908)
79 (65,984)
35 (125,432)
30 (39,098)
71 (193,569)
56 (283,414)
60 (127,840)

 ASR system [Ogata, Goto & Eto, Interspeech 2007]

 http://podcastle.jp since 2006• 2006/01• 2006/12

Started the project
Released to the public

The world's first speech retrieval using crowdsourcing• 2008/06• 2011/10• 2011/10

Press release (media coverage)
Press release (media coverage)
Launch the English version

Powered by ASR of CSTR, Univ. of Edinburgh
in collaboration with CSTR and AISTThe Centre for Speech Technology Research
The University of Edinburgh• Multi-pass decoding with unsupervised MLLR
GMM-based audio segmentation (speech/music/others)
Confusion network generation• Baseline acoustic model:
Tied-state triphone (4513 states, 16 Gaussians/state)
Trained with 600 h of presentation speech data (CSJ)• Language model: 3-gram (vocabulary size: 286,345)

WER(%)

 WER results (average)
35

35.3

30
25

33.5

w/ocorrections
w/32.7
corrections
28.9

baseline w/o channel LM
w/ channel LM
Proposed dynamic language modeling system

Summary Dynamic language modelingfor spoken content on the web• Use of topic LMs based on web news
• Use of channel-dependent LM generated byuser transcripts (wisdom of crowds)• Model-level mixture with optimal weights• Plan to deploy the proposed system on thePodCastle web service to improve ASR results

Corrected transcripts are effective in language modeling• Comparison of weight optimization (WER%)
w/o channel LM
w/ channel LM

unsupervised
33.5
28.9

supervised
33.4
28.6

unsupervised: w/ recognition hypothesis
supervised: w/ manual transcriptions

Video clip of PodCastle:
http://staff.aist.go.jp/m.goto/PodCastle/
2012/09/13 Interspeech 2012 poster