Proceedings of the 8th Annual Conference of the International Speech Communication Association(Interspeech 2007), pp.2397-2400, August 2007.

PodCastle: A Web 2.0 Approach to Speech Recognition Research
Masataka Goto, Jun Ogata, and Kouichirou Eto
National Institute of Advanced Industrial Science and Technology (AIST)
1-1-1 Umezono, Tsukuba, Ibaraki 305-8568, JAPAN
Abstract

to deal with speech contents that cannot be properly recognizedis to create a speech corpus including such contents and prepare correct transcriptions to train speech recognizers. This approach, however, is impractical for PodCastle because advance
preparation of a corpus covering all podcast contents will be too
costly and time consuming.
The second aim of this study is to dispense with the idea ofusing a pre-prepared corpus to address this problem, and instead
employ the efforts of a large number of users to improve speech
recognition and full-text search performance. Even if a state-ofthe-art speech recognizer is used to recognize podcasts on theweb, a number of errors will naturally occur. PodCastle therefore encourages users to cooperate by correcting these errors
so that those podcasts can be searched more reliably. Furthermore, using the resulting corrections to train the speech recognizer, it implements a mechanism whereby the speech recognition performance is gradually improved. This approach can be
described as “collaborative training for speech recognition”.
In this paper, we introduce the term “Speech Recognition
Research 2.0” to refer to the research approach where the current state of speech recognition technology is intentionally disclosed to users so that speech recognition performance can be
improved through cooperative participation by users. This term
was chosen to reflect the concept of Web 2.0 [1], since this approach brings the benefits of Web 2.0 to speech recognition research. In Section 2 of this paper, we discuss the research approach that Speech Recognition Research 2.0 represents, and in
Section 3 we describe the PodCastle web service as an instanceof this approach. In Section 4, we summarize the contributionsof this research.

In this paper, we describe a public web service, “PodCastle”,
that provides full-text searching of Japanese podcasts on the basis of automatic speech recognition. This is an instance of ourresearch approach, “Speech Recognition Research 2.0”, which
is aimed at providing users with a web service based on Web 2.0
so that they can experience state-of-the-art speech recognition
performance, and at promoting speech recognition technologies
in cooperation with anonymous users. PodCastle enables users
to find podcasts that include a search term, read full texts of
their recognition results, and easily correct recognition errors.
The results of the error correction can then be used to improve
the performance of both full-text search and speech recognition.
Although we know of no state-of-the-art speech recognizer thatcan successfully transcribe all of the various kinds of podcasts,
the mechanism we propose will gradually increase the usefulness and applicability of PodCastle.
Index Terms: information retrieval, speech recognition, error
correction, wisdom of crowds, Web 2.0

1. Introduction
Speech recognition researchers understand what sort of speechis easily recognized by speech recognizers and realize thatspeech recognizers perform best when dealing with clean
speech. On the other hand, most end users of speech recognizers
judge the effectiveness of speech recognition from their limited
experiences and do not necessarily understand how useful stateof-the-art recognizers can be. Users sometimes do not adequately comprehend what sort of voices or recording conditions
make recognition difficult. If they have previously had difficulty being understood by speech recognizers, they often doubt
the usefulness of speech recognition and may stop using it.
The first aim of this study is to address this problem bypromoting the popularization and use of speech recognition byraising end user awareness of state-of-the-art speech recognition performance. For this purpose, we launched a web service
called “PodCastle” that allows a number of anonymous users
to search and read podcasts, and to share the full text of speech
recognition results for podcasts. Podcasts are audio programs
distributed on the web, like radio shows or audio blogs. They
are becoming increasingly popular because updated podcasts(MP3 audio files) can be easily and frequently downloaded byusing RSS syndication feeds. Since various contents have already been published as podcasts, users can grasp the current
state of speech recognition technology just by seeing the resultsof speech recognition applied to published podcasts. This is important because when some users experience recognition errors
while speaking into a microphone, they may become uncomfortable or frustrated and lose their motivation. Such problems
do not occur for PodCastle because users do not have to provide
their own speech input at all.
However, even state-of-the-art speech recognizers cannot
correctly transcribe all podcasts, because their contents andrecording environments vary very widely. A typical approach

2. Speech Recognition Research 2.0“Speech Recognition Research 2.0” is a new research approachto speech recognition which aims at improving speech recognition performance and the usage rate while benefiting from thecooperation of a number of anonymous end users. To achievethis, we propose setting into motion a positive spiral as explained in Figure 1. In the past, this spiral has not necessarily taken hold because of inhibiting factors in each of the threesteps important for popularizing speech recognition. The problems affecting each of these steps are as follows:• With regard to (i) understanding speech recognition performance, users have tended to see the results of speech recognition applied only to their own voices. Once users experience recognition problems with their voices, they tend toincorrectly assume that other people’s voices will also not
be well recognized. On the other hand, speech recognition
researchers have a better understanding of recognition capabilities because they have more opportunity to see the results of speech recognition applied not only to their own
voices but to a large speech corpus.
• As for (ii) contributing to improved speech recognition
performance, users of speech dictation systems can makespeech recognizers adapt to their voices by reading out pre-2397

August 27-31, Antwerp, BelgiumTable 1: A comparison of the conventional approach to speech
recognition research (Speech Recognition Research 1.0) and the
proposed approach (Speech Recognition Research 2.0).

(i) Allowing users to experience speech recognition lets
them better understand its performance.
(ii) Users contribute to improved speech recognition performance.

Speech Recognition Research 1.0 Speech Recognition Research 2.0

Stand-alone application
Dictation
CorpusLimited topics
Transcription
Out-of-vocabulary words
Specialist participation
Individual correction
Personal wisdom
Completed version(iii) Improved performance leads to a better user experience.
Figure 1: A positive spiral leading towards greater use of speech
recognition (through repetition of steps (i), (ii), and (iii)).
defined sentences or add out-of-vocabulary words.1 However, such in-house performance improvements made byeach end user are not made available for re-use by otherusers. Only speech recognition researchers have been able
to improve the performance of speech recognition as awhole. Consequently, it has been difficult to motivate anumber of users to contribute to collaborative efforts thatwill improve performance.
• With regard to (iii) a better user experience, users have had
little opportunity to experience the better performance that
results from ongoing improvements made by researchers.
For example, even if speech recognizers are made available
as open source software (e.g., [2]), these are mainly aimed
at developers; end users have little opportunity to use them
directly. Also, users of most speech recognition products
have only experienced performance improvements throughinfrequent software updates.
By addressing these problems, Speech Recognition Research 2.0 aims to change the usage of speech recognition by
setting the positive spiral of Figure 1 into motion. Table 1 compares this approach with the conventional approach to speech
recognition research, which we will call “Speech Recognition
Research 1.0”. We are not suggesting, though, that Speech
Recognition Research 1.0 is inferior or obsolete, and there is
no doubt that continued research using the Speech Recognition Research 1.0 approach is needed. We ourselves have continued our work on Speech Recognition Research 1.0 as thefoundation for 2.0. It should also be stressed that we are discussing research approaches, and not speech recognition techniques or algorithms themselves, which is why we use the term
“Speech Recognition Research 2.0” instead of “Speech Recognition 2.0”.
In the following, we discuss how the beneficial spiral of
Figure 1 can be put into effect while also explaining the points
in Table 1.
• Instead of developing stand-alone applications such as dictation or spoken dialogue by preparing a corpus for speech
recognizers, Speech Recognition Research 2.0 provides a
web service that allows users to search and browse opento-the-public web-based speech data such as podcasts. In
this way, it promotes understanding of speech recognition
performance (step (i) in Figure 1).
• When recognizing web-based speech data, however, wecannot limit the range of topics and prepare in advance a
suitable corpus with its transcription. This causes many
recognition errors, so Speech Recognition Research 2.0 gets
users to correct the errors, thereby enabling recognition ofa wide variety of speech data on an unlimited range of topics. That is, users cooperate in the preparation of full-texttranscriptions as a form of annotation that can be used when

Web service
Searching/browsing
Web-based data
Unlimited topics
Annotation
Not-yet-annotated words
User participation
Social correction
Wisdom of crowds
Perpetual beta

The above table is influenced by the comparison of Web 1.0 and Web
2.0 by O’Reilly [1]. Research projects that feature more points on theright side of the table are more worthy of the name “Speech Recognition Research 2.0”. However, as with Web 2.0, this does not meanthat all these points have to be featured in any one project.

searching for speech data. It is important that user corrections are also used for training speech recognizers so that
not-yet-corrected errors in other parts or other speech data
can be reduced.2 In Speech Recognition Research 2.0, outof-vocabulary words are regarded as being nothing morethan not-yet-annotated words which will be annotated (corrected) by users and then automatically added to the system
vocabulary. In this way, users can contribute to improved
performance (step (ii) in Figure 1).
• Furthermore, instead of confining this to individual corrections, we propose extending this user participation framework to provide a social correction framework, where anumber of anonymous users can improve the performanceby sharing their correction results over a web service. In
this social framework, users gain a real sense of contributing
to the convenience of other users, and can be motivated tocontribute by seeing the correction activities made by otherusers. In this way, we can use the wisdom of crowds toachieve a better user experience (step (iii) in Figure 1).
In other words, Speech Recognition Research 2.0 can be described as an approach whereby a web service based on speech
recognition that is permanently in beta version (perpetual beta)
is launched and then improved by inviting users to use it on theweb, thereby advancing the research.
As the first instance of Speech Recognition Research 2.0,
we initiated the PodCastle project in January 2006. In thisproject, our goal is to set the positive spiral of Figure 1 into motion by providing the PodCastle web service which is based on
the concepts of both Web 2.0 and Speech Recognition Research
2.0.

3. PodCastle: A podcast search service
based on speech recognition
PodCastle is a social annotation web service where users cansearch, read, and annotate podcasts in text form. Each podcast
consists of a series of episodes of audio data (MP3 files) andtheir metadata (RSS syndication feed) that promotes its circulation. The creator of a podcast (the podcaster) can add new
2 This is an original benefit of Speech Recognition Research 2.0 thatis not provided by Web 2.0. For example, in other services such as
Wikipedia [3] based on the wisdom of crowds, the users’ contributions
are limited to the articles they edit. There is no automatic improvementof other articles.

1 Some systems can adapt automatically to a speaker’s voice during

use without the user’s awareness, and some systems can automaticallyacquire out-of-vocabulary words. However, in neither of these cases it
is possible to share this information between users.

2398

regarding podcasts (step (i) in Figure 1), and the annotating (error correction) function allows them to contribute to improved
performance (step (ii)). This improved performance can thenlead to a better user experience of searching and reading podcasts (step (iii)).
3.1.1. Searching function
This is a function that allows a full-text search of the speech
recognition results (and the results corrected by users). Whenthe user types in a search term, as with an ordinary text-based
search engine, a list of episodes containing this term is displayed together with text excerpts around the term. These excerpts can be played back individually. By selecting one ofthese search results, the user is then able to access its full text
by switching over to the reading function.

Figure 2: PodCastle screen snapshot of an interface for correcting speech recognition errors (competitive candidates are presented underneath the normal recognition results). Five errors
in this excerpt were corrected by selecting from the candidates.
The corrected Japanese sentence means “. . . well, actually theratio of this price range and . . . ”.

3.1.2. Reading function
With this function, as well as listening to a podcast the user canalso view the text of the podcast. This allows users to understand the contents of a podcast even when audio playback is notpossible, and allows them to quickly decide whether they are interested in the podcast’s contents without having to listen to it.
To make errors easy to discover, each word is colored according
to the degree of reliability estimated during speech recognition.
Furthermore, a cursor moves across the text in synchronization
with the audio playback.
Because the full-text result of speech recognition being applied to each episode becomes available to external full-text
search engines, such results can be discovered together with ordinary web pages by these engines. This increases the value of
podcasts by bringing more users into contact with them. Since
this benefits the podcasters, it will motivate them, together withother volunteer users, to use the annotating function.

episodes at arbitrary intervals (daily, weekly, etc.). With RSS,
updated episodes are automatically downloaded from the web
and can be stored in any type of player. Podcasts are often referred to as audio blogs and their popularity has grown because
anybody can publish and download audio data with ease. Just
as full-text search services are essential for accessing text web
pages, there is a growing need for full-text speech retrieval services such as PodCastle.
Although there have been previous research projects forspeech retrieval [4, 5, 6, 7, 8], most do not provide public web
services for podcasts. There are two major exceptions, Podscope [9] and PodZinger [10], which started web services forspeech retrieval targeting English-language podcasts in 2005.
These services use speech recognition to turn podcasts into text,
and can display a list of podcasts that include a search term.
In Podscope, users are shown none of the speech recognition
results and only the title list is provided, although speech dataaround the found search term can be played back. In PodZinger,
users are shown text excerpts (recognition results) surrounding the search term, allowing users to grasp the context of the
podcast more easily. In contrast, PodCastle is the first service
to provide full-text searching of Japanese-language podcasts.
Even if the other services can also support Japanese podcasts
in the future, PodCastle differs significantly in three ways:
1. Although speech recognition has been used in earlier services, they have only displayed parts of the resulting text,
making it impossible to visually ascertain the detailed contents of the podcast without actually listening to it.
2. The full-text results of speech recognition have been hidden
inside, so it has not been possible to search them using otherexisting text-based search engines.
3. Even if users find that search results are degraded by unavoidable speech recognition errors, the users have had nomeans of correcting these errors.
In contrast, PodCastle allows full-text results of speech recognition to be accessed by both users and external search services,
and allows a number of users to cooperate with each other toimprove the speech recognition performance.

3.1.3. Annotating function
This function allows users to add “annotations” to correct any
recognition errors they may come across while searching or
reading. Here, annotation means transcribing the podcast contents, either by selecting the correct candidate from the list ofcompetitive candidates, or by typing in the correct text. For
this purpose, we provide an efficient error correction interface
we earlier proposed [11]. In this interface, shown in Figure 2,
a recognition result excerpt is shown around the cursor andscrolled in synchronization with the audio playback. Each wordin the excerpt is accompanied by other word candidates, which
are generated beforehand by using a confusion network 3 [12]that can condense a huge internal word graph of a large vocabulary continuous speech recognition (LVCSR) system. Note thatusers are not expected to correct all the errors, but can be expected to correct some errors according to their interests.
3.2. Implementation and public release of PodCastle

3.1. Three functions of PodCastle

The implementation overview of PodCastle is shown in
Figure 3. The web crawler collects podcasts, which can be
added by users, and records them in the database manager.
Those podcasts are then recognized by multiple speech recognizers one after another. When a request from a speech recognizer is received by the speech recognition manager, the next

PodCastle supports three functions — searching, reading, andannotating — to satisfy all the points of Speech Recognition Research 2.0 listed in Table 1 and set the positive spiral of Figure 1
into motion. Specifically, the searching and reading functions
let users better understand the speech recognition performance

3 The confusion network was originally introduced in the context ofword error minimization, which minimizes the word error rate of recognition results rather than the sentence error rate [12]. Our original idea
is to apply such an efficient intermediate recognition result to generatecompetitive candidates for efficient error correction [11].

2399Speech recognition manager
Speechrecognizer
recognizer
SpeechSpeechrecognizer
Speechrecognizer
Speechrecognizer

Search engine

Web crawler

PodcastPodcast(MP3+ RSS)
Podcast(MP3+ RSS)
Podcast(MP3+ RSS)(MP3 + RSS)

Database manager

User interface

vides a search engine for podcasts on the basis of the wisdomof crowds. The technical contribution of this study is to investigate how far the performance of speech recognition and fulltext search can be improved by getting speech recognition errors
corrected through the cooperative efforts of many end users. At
the same time, it makes a social contribution in that it helps
web users by providing the world’s first public web service forfull-text search of Japanese-language podcasts.
Another contribution of this study is that it demonstrates
how speech recognition can be put to use in situations where aspeech corpus is almost impossible to prepare in advance. Although speech recognition usually requires a sufficient corpus
to provide useful results, such corpora tend to be costly andlabor-intensive, thus limiting applications. On the other hand,
this study has aimed at “collaborative training for speech recognition” where full-text transcriptions containing recognition errors are first disclosed and then corrected by anonymous users.
Since there are many errors, we run the risk of attracting criticism, but we believe that sharing these results with users will
promote further popularization and use of speech recognition.
We hope that this study will prove the importance and potential of incorporating user contributions into speech recognition,and that various other projects that follow the Speech Recognition Research 2.0 approach will be done, thus adding a new
dimension to this field of research.
Acknowledgments: We thank Ryutaro Kamitsu (Brazil Inc.),
Shunichi Arai (Mellowtone Inc.), and Youhei Sawada for their
web service implementation. We also thank the anonymous
users of PodCastle for correcting speech recognition errors.

User interface
User interface

Figure 3: Implementation overview of PodCastle.
available episode to be recognized is handed over. After the recognizer finishes processing its episode, the recognition result ispassed to the database manager via the speech recognition manager. The database manager controls the processing state of the
podcasts and indexes their speech recognition results together
with the corrections provided by users. Finally, the search engine works as a website that provides the PodCastle user interface with the three functions.
The web server of PodCastle was implemented by using a
web application framework Ruby on Rails 1.1.4, a programming
language Ruby 1.8.4, a web server WEBrick 1.3.1, a database
MySQL 4.1.21, and a morphological parser for the Japanese
language ChaSen 2.3.3. The client interface was implemented
by using a scripting language JavaScript 1.5 and its library
MochiKit 1.3.1, multimedia frameworks QuickTime 7 and Flash7, and an ActionScript 2 compiler MTASC 1.12.
To recognize podcasts, audio data is first segmented into
three categories — speech, music without speech, and otherbackground sounds — by applying GMMs. Speech segments
are then recognized by using our in-house LVCSR system basedon an efficient N-best search algorithm [13] to generate theconfusion networks. This system uses cross-word tied-state
triphone HMMs trained for 39-dimensional MFCC-based features, and a 159k-word trigram language model trained by using both large standard speech corpora and daily-updated web
news. The automatic performance improvements through correction by users can be achieved through various techniques,
such as those for training an acoustic model by using correctedtranscriptions, for making a language model adapt to different topics by using RSS metadata and corrected transcriptions,and for registering out-of-vocabulary words by using a phonetic
typewriter to estimate their pronunciation. The details of ourspeech recognizer are described in another paper [14].
PodCastle was released to the public at http://podcastle.jp
on December 1st, 2006, and has received 7493 search requests
in the three and half months since, even though we have notpromoted it to the public (although we plan to soon). 124 podcasts have been registered, consisting of 5398 episodes in total.
Of these, 239 episodes (4.4%) have been at least partially corrected. We found that there are users who voluntarily cooperate
in the correction as there are in other Web 2.0 services, andthat podcasts recorded by famous artists and TV personalities
tend to receive many corrections. One Web 2.0 principle is totrust users and we also trust users with respect to the qualityof correction: in practice, the correction results obtained so far
have been of high quality. Even if some users deliberately makeinappropriate corrections (vandalism), though, we can develop
countermeasures to acoustically evaluate the reliability of corrections. For example, we can use the likelihood of HMMs forforced alignment with the corrections.

5. References[1] T. O’Reilly, “What is Web 2.0 — design patterns and business models for the next generation of software,” http://
www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-isweb-20.html.
[2] A. Lee, T. Kawahara, and K. Shikano, “Julius — an open source
real-time large vocabulary recognition engine,” in Proc. of Eurospeech 2001, 2001, pp. 1691–1694.
[3] Wikipedia, http://www.wikipedia.org/.
[4] S. Whittaker, et al., “SCAN: Designing and evaluating user interfaces to support retrieval from speech archives,” in Proc. of ACM
SIGIR Conf. on Research and Development in Information Retrieval, 1999, pp. 26–33.
[5] J.-M. V. Thong, et al., “Speechbot: An experimental speech-based
search engine for multimedia content on the web,” IEEE Transactions on Multimedia, vol. 4, no. 1, pp. 88–96, 2002.
[6] L. shan Lee and B. Chen, “Spoken document understanding andorganization,” IEEE Signal Processing Magazine, vol. 22, no. 5,
pp. 42–60, 2005.
[7] Cambridge Multimedia Document Retrieval Project,
http://mi.eng.cam.ac.uk/research/projects/mdr/.
[8] CMU Informedia Digital Video Library Project,
http://www.informedia.cs.cmu.edu/.
[9] Podscope, http://www.podscope.com/.
[10] PodZinger, http://www.podzinger.com/.
[11] J. Ogata and M. Goto, “Speech Repair: Quick error correction justby using selection operation for speech input interfaces,” in Proc.
of Eurospeech 2005, 2005, pp. 133–136.
[12] L. Mangu, E. Brill, and A. Stolcke, “Finding consensus in speech
recognition: Word error minimization and other applications ofconfusion networks,” Computer Speech and Language, vol. 14,
no. 4, pp. 373–400, 2000.
[13] J. Ogata and Y. Ariki, “An efficient lexical tree search for large vocabulary continuous speech recognition,” in Proc. of ICSLP 2000,
2000, pp. 967–970.
[14] J. Ogata, M. Goto, and K. Eto, “Automatic transcription for a
Web 2.0 service to search podcasts,” in Proc. of Interspeech 2007,
2007.

4. Conclusion
We have proposed “Speech Recognition Research 2.0” as a new
approach to speech recognition research that complements existing approaches. As the first instance of this approach’s application, we have launched the PodCastle web service which pro-

2400