Automatic Transcription for a Web 2.0 Serviceto Search Podcasts
Jun Ogata, Masataka Goto, and Kouichirou Eto
National Institute of Advanced Industrial Science and Technology (AIST)

Motivation¾ The amount of speech data isincreasing rapidly on the web¾ Two main difficulties in transcribing podcasts• The contents and recording environments
have a wide variety• Tend to include words and phrases
related to very recent topics• Podcasts have become popular and widespread¾ A novel Web 2.0 application based on
ASR (automatic speech recognition)

Improving techniqueson the basis of Web 2.0

Automatic transcription of podcasts

PodCastle
What
What areare podcasts
podcasts ??

Overview
Overview¾ Podcast search service based on ASR¾ Audio programs distributed on the web[Goto, Ogata, and Eto: Interspeech2007]• Podcast = RSS syndication feed + MP3 files• Updated episodes are automatically downloaded

RSS syndication feed

• Users can search, read, and annotate podcasts• Growing need for full-text speech retrieval service• Existing podcast retrieval services( Podscope and EveryZing (PodZinger) )9 Hide full-text ASR results
9 Users have no means of correcting ASR errors• PodCastle
9 Allow full-text ASR results to be accessed byboth users and external search services
9 Allow users to cooperate with each otherto improve ASR performance• First instance of Speech Recognition Research 2.0

URL: http://podcastle.jp
Video clip: http://staff.aist.go.jp/m.goto/PodCastle/

Metadata
Title: CNN News Update
Description: The latest news happening
in the U.S. and around the world.
Episode 1
Title: CNN News Update (8-21-2007 7 AM EDT)
MP3: http://rss.cnn.com/...08-21-07-7AM.mp3
Episode 2
Title: CNN News Update (8-21-2007 6 AM EDT)
MP3: http://rss.cnn.com/...08-21-07-6AM.mp3
Episode 3
Title: CNN News Update (8-21-2007 5 AM EDT)
MP3: http://rss.cnn.com/...08-21-07-5AM.mp3
Episode ...
(New episodes can be added at arbitrary intervals)

ThreeThree functions
functions¾ Searching function• Full-text search of ASR results• List of episodes containing a search term is displayed together with text excerpts¾ Reading function• View the full-text ASR result to understand the contents• Each word is colored according to the degree of ASR reliability (confidence measure)• Full-text can be accessed by external search engines (e.g., Google)¾ Annotating function• Add “annotations” to correct ASR errors- Select the correct candidate from the candidate list
- Type in the correct text
• Candidate list is generated by using a confusion network

Baseline Transcription System
Acoustic
Acoustic model
model

Decoding
Decoding¾ Various types of speech• Noisy speech, narrow-band speech,
speech with music, etc.
The ETSI advanced front-end

MP3 audio fileGMMs

Audio Segmentation¾Acoustic features

Speech, music,and otherbackground sounds• 12 MFCCs, energy, and delta and delta-delta¾Training data

Speech Segments• 600 hours of presentation speechin the Corpus of Spontaneous Japanese (CSJ)¾ Triphone HMM (4513 tied states, 16 Gaussians/state)

Phone Recognition

Language
Language model
model

Initial Hypothesis¾ Various tasks and wide domains
MLLR• Very difficult to specify the topic and vocabulary¾ Out-of-vocabulary issue• The latest topics and words appear frequently
in newly added episodes on a daily basis

Word Decoding¾ Up-to-date language model

Iterationx2•Lexical tree search with 2-gram•Word graph rescoring with 3-gram• Use of daily updated text from web-news sites
Static corpora

MLLR
Newspaper LM

CSJ LMWeb news LM
Consensus Decoding

Linear interpolation
Confusion Network

Up-to-date LM (trigram, 152163 words)

Adopt to the error correction interface

Experiments
Experiments

281.0

250

Description of test set

48.4

146.6

DE
Confusion network error rate

BC

32.3
24.8

DEA mechanism to keep a speech recognizer always up-to-date is needed

9.1

BC

8.2

A

9.8

13.4

A

12.5
11.1

5

13.1

20

DE

29.1

35C

45.1

45.6

50

99.6

86.8

112.2

104.1

62.1

AB
Word error rate

35.6

• A,B: Commentary focused on the latest economics• C,D: Daily news distributedby a Japanese broadcasting company
D includes background music• E: Chitchat show by a Japanese popular teen idol

50

86.5
72.6

24.1

spontaneous
spontaneous
read
read
spontaneousLM
Up-to-date LM

18.3

192.1
128.2
1159.6
247.3
486.6

150

27.5

Read or
Spontaneous

25.3

A commentary
B commentaryCnews
Dnews
Echitchat

Length(sec.)

28.9

Category

27.5

ID

266.2

Perplexities

Improving Methods on the Basis of Web 2.0¾ Use of Web 2.0 knowledge sources and contents
A huge amount of multimedia contents and annotations aregenerated and accumulated by a number of (anonymous) users

These resources are steadily increasing
and updated on a daily basis

Language
Language model
model adaptation
adaptation using
using RSS
RSS

29.1

25.3

42.4

45.1
35.6
27.5

27.2

Up-to-date LM+ topic dependent LM

34.5¾Use of the text data in RSS for topic adaptation 3525Confusion network error rate

Word error rate45• publishing date, titles,and accompanying text descriptions

27.4¾RSS provides useful data:

Acquiring
Acquiring pronunciations
pronunciations fromfrom thethe webweb¾ Many technical terms and coined words

13.1

12.9

8.2

8.2

9.8

9.0

11.2

11.1

18.3
18.0

22.9

1. Parse RSS title and description
using a Japanese morphological analyzer
15
2. Retrieve text documents related to the topic ofthe episode from the web using a search engine*
5ABCDEABCDE(Only nouns are used as search terms)
3. Build a topic-dependent LM
from the retrieved documents¾ B, D, and E: improved
4. Interpolate the topic-dependent LM¾ A,C: only slight improvementswith the baseline up-to-date LM
No useful information for the topic was given in the RSS* Yahoo! web search API is used to search terms

Acoustic
Acoustic model
model training
training
based
based onon errorerror corrections
corrections byby usersusers¾ Error corrections• Some users may thoroughly correct all errors• Most users will correct only errors
related to the part according to their interests• e.g., PodCastle, Wikipedia …

¾ How to obtain their pronunciations ?

¾ The use of error correction results
to improve ASR is a considerable challenge• Publishes a list of new keywords
including explanations and pronunciations ¾ Investigation in acoustic model training¾ A Web 2.0 service ``Hatena diary keyword’’• Over 200000 keywords are registered at present(about 300 words are added daily)• The amount of speech data was limited• The corrected transcriptions were not perfect(about 50% of errors were corrected by users)
Performance of MLLR-MAP training

ABCDA

FutureFuture workwork¾ Larger-scale experiments¾ Training techniques so as to make best use of user corrections

12.9

8.2

C

7.9

9.0

B

9.1

27.4

E

8.5

11.2

5

DE

25.1

27.8
18.0
16.1

22.9

15

The correct pronunciations can be obtained for
22.9% (3997/17438) of those words

19.3

27.2

25.1

25

34.5

35

The pronunciations were not obtained for
11.5% (17438/152163) of words in our LM

Before training
After training

9.4¾ Investigation

4542.440.1(A web service which provides full-text searching of podcasts)

Confusion network error rate

Word error rate