ReazonSpeech: A Free and Massive Corpus for Japanese ASR
Yue Yin1 Daijiro Mori1 Seiji Fujimoto2
1 Reazon Holdings, Inc. 2 Clear Code, Inc.
{yue yin, daijiro mori}@reazon.jp fujimoto@clear-code.com
crest of the wave and bring Japanese E2E ASR to a new

Abstractstage by open-sourcing the project.

ReazonSpeech is a 15,000-hour and continuously growing corpus collected from Japanese TV shows free for com-2

Related Workmercial usage. The automatic speech recognition (ASR)

There has been a line of research on creating Japanesemodel trained on ReazonSpeech achieves state-of-the-art

ASR corpora recently. High-quality corpora that require

results with 8.23% character error rate (CER) on JSUT

human eﬀorts such as CSJ[5], and Common Voice[2] havebasic5000[1] and 9.93% on Common Voice[2] v8.0 test

been explored extensively for Japanese ASR for the lastset, on par with the recently released Whisper[3] largev2 model. We released the dataset creation toolkit under

decades. While highly supervised corpora are fundamental for hybrid models and remain essential to E2E

Apache License 2.0 and made both the corpus and the

ASR as benchmark datasets for evaluation, the expen-

pretrained ASR model freely available1）.

sive human labor cost casts a limit on the scalability of

1such highly supervised corpora in terms of E2E train-

Introduction

ing. These days, the popularity of bootstrapping audio-

End-to-end (E2E) automatic speech recognition has re-transcription pairs for ASR training from media like audio-

cently drawn attention for its state-of-the-art results onbooks and subtitled videos using existing ASR resourcesbenchmark datasets in terms of accuracy and its promise

has taken a rise. Corpora like The People’s Speech [6] andfor not needing in-depth expertise compared to hybrid

GigaSpeech[7] have shown the competency of automati-

models[4]. With the release of large-scale corpora andcally generated large-scale corpora for ASR training. For

models for English, the development of English E2E ASR

Japanese, LaboroTVSpeech[8] and JTubeSpeech[9] haveis ﬂourishing more than ever.

Compared to English,

been introduced in recent years and built up a solid foun-Japanese is still resource-scarce and the development of

dation for Japanese E2E ASR. More recently, with the

E2E ASR is largely hindered by this scarcity. This paper
introduces our attempt to construct a freely available large-

release of Whisper[3], sacriﬁcing the supervision for scaleis shown to be promising for E2E ASR. In this work, we ap-

scale high-quality Japanese ASR corpus from TV record-

plied the automatic construction techniques to TV record-

ings via bootstrapping labeling seeding from a small-scaleings and further seeks the potential of constructing large-

corpus. The release of the corpus is bundled with thescale datasets without depending on existing resources.

creation pipeline free-and-open-source for continual evo-

3

lution. The initial release of the corpus consists of 15,735

Corpus Construction

hours of audio data and is by far the largest freely available

A lot of Japanese TV shows are aired with subtitles

Japanese ASR corpus of our knowledge and the construc-that can be leveraged for ASR corpus creation. However,tion is intended to be continuous. With the Amendment tothe subtitles cannot be directly used to construct an ASR

the Copyright Act2）encouraging the use of digital contents

corpus mainly due to the following reasons:in AI going on in Japan since 2019, we hope to ride the1） Sample data and relevant resources can be found at the project
page https://research.reazon.jp/projects/ReazonSpeech2） Detailed information can be found at https://www.bunka.go.
jp/seisaku/chosakuken/hokaisei/h30 hokaisei/

1. timestamps are inaccurate, especially for news andlive TV programs
2. transcriptions are missing for commercials and speechFigure 1

Corpus construction workﬂow

that has its transcription embedded in the video
3. transcriptions are inaccurate, especially when the
speech is informal (eg. deletion of ﬁller phrases)

rors and introduces overheads in computation.
CTC segmentation based

Finding out that using

ASR results overcomplicate the task and suﬀers from missing transcriptions drove us to rethink the possibility to

Of the above limitations, we aim to address 1 and 2 byleverage the inaccurate timestamps that have been ignored.

reﬁning the alignment between text transcription and au-

JTubeSpeech[9] has suggested the viability of anchoringdio. We leave limitation 3 out of account because of its

text in a longer audio segment for corpus creation oncomplexity and the amount of human eﬀort required. The
workﬂow of the construction is illustrated in Figure 1.

Japanese subtitled YouTube videos. The success of thiswork hinted at the possibility to reframe the task to anchor
the text in a longer audio segment which is a superset of

3.1

Pretrained ASR Models for Alignment

the actual utterance by utilizing the fuzzy timestamps in

As suggested in [7, 8, 10], one can au-

the subtitles. As suggested in this work of German corpus

tomatically generate a corpus using ASR-based align-

construction[17], there are several established methods forment and cleansing by leveraging the divide-and-conquer.

alignment:

ASR based

LaboroTVSpeech[8] and GigaSpeech[7] use the strat-• Acoustics based: Montreal Forced Aligner[18], Kaldi

egy proposed in [11] which consists of a pretrained[12]based scripts, etc.

acoustic model and a biased language model through• Text-to-speech based: Aeneas[19]Kaldi[12] interface. Inspired by this approach, we tried• Connectionist Temporal Classiﬁcation outputs based:

ASR-based alignments for the corpus. We broke down

CTC segmentation[17]the audio into utterances using voice activity detection

Amongst the above methods, we chose to experiment withthrough pyannotate[13] and resegmented subtitles into

CTC through ESPnet2[20] interface as we have access tocomplete sentences using GiNZA[14] through SpaCy
framework[15]. We then transcribed the audio segmentsa noise-tolerant pretrained model under the same-domain
corpus LaboroTVSpeech[16]. Moreover, this method isusing an ASR model pretrained on LaboroTVSpeech[16]independent of in-depth language-speciﬁc knowledge andand matched the audio and transcription chunks based oncompatible with our eventual E2E training goal. To test

the ASR results using a dynamic programming-based al-

the compatibility of the CTC segmentation approach with

gorithm as illustrated in Figure 5 in the appendix. By

our task, we used a model from [16] and added 25 seconds

listening to the extracted utterances, we noticed a lot of in-

before the starting point of the subtitled timestamp as thesertion/deletion at the beginning/end, suggesting that thisinaccuracy usually happens at the beginning. The issues weapproach is inadequate to resolve the missing transcription

noticed in ASR based approach are considerably relieved.

issue 2. Besides, certain words occur repeatedly through-The positive results conﬁrmed the viability of this method

out the entire TV show, making the matching error-prone.

on our task.

In our experience with this method, the quality of the constructed corpus and extraction eﬃciency largely depends
on the accuracy and agreement of the intermediate processes, and this dependency unnecessarily aggregates er-

3.2

Bootstrapping Labeling for Alignment

Approach

The success in using CTC segmentation[17] shows the promise of the approach leveraging robust
pretrained ASR models for corpus creation. However, de-pending on resources that have non-permissive licenses

200 utterances of variable lengths created by CTC

casts restrictions on the constructed corpus, not to mention

segmentation using the ASR model pretrained onthat such language resource is a luxury that some languages

LaboroTVSpeech [16], and 79 utterances are deemed ap-

might not have. Therefore we further seek the possibility

propriate. Instead of thresholding CTC scores which is aof loosening the assumption of having an ASR model by

byproduct of the alignment as suggested in JTubeSpeechleveraging public-domain small-scale data. Bootstrapping[9], we use CER(ASR(audio), transcription) ≤ 0.33 as thelabeling methods are shown to be eﬀective in various tasks

threshold to select quality data as it has a higher level ofwhere there is little data available. The process of boot-agreement with human judgment: F1 score of 0.87 while

strapping labeling usually follows this procedure: 1. obtainusing CTC score > −1 only has an F1 score of 0.42.
Extraction eﬃciency evaluationa small amount of labeled data and train a model based on
it 2. use the model to label more data 3. select high-quality
labeled data to add to the training data 4. retrain the model

extraction rate =5. repeat 2-4 until satisfactory results are obtained.
Although this technique has been applied to diﬀerent
tasks including ASR [21, 22, 23], its eﬃcacy on alignment
has not been well-studied. We applied this approach to
our task by starting with training an initial model based on
Common Voice[2] dataset and grew the aligner iteratively
following the procedure described in Figure 2:

To further quan-tify the extraction eﬃciency, we deﬁne the metric aslength of text in constructed corpus
length of text in subtitles

Note that the extraction rate is an approximate metric as
the potential of TV recordings as ASR corpus largely depends on the TV program genre. However, by calculating
this metric using the same TV recordings, we are able tocompare the extraction eﬃciency of diﬀerent models.

1. train 0th-gen ASR model on Mozilla Common
Voice[2] v8.0 (47.7 hrs)
2. use ASR model CTC alignment[17]3. select high-quality aligned utterances
4. retrain ASR model
5. repeat 2-4 until extraction eﬃciency reaches a satisfactory level

Figure 3

Extraction rate for every generation on one-day
amount of unseen recordings

Figure 4

4
Figure 2

CER on JSUT basic5000 for every generation

Experimental Results

bootstrapping labeling ﬂow

Corpus creation results

We used the recordings in

To speed up the development cycle, we orchestrated a

2022.12.27 to compare the extraction rate for every gen-pipeline where ASR training and aligning can be carriederation of models. As shown in Figure 3, the extraction

out in parallel. The paragraphs below describe how high-rate surpassed the LaboroTVSpeech pretrained model[16]quality alignments are selected and how the extraction eﬃciency is measured to monitor the improvements. Figure 3

starting from the 7th generation and reached 70.12% for
the released model. The consistently improving extraction

and 4 show the improvements in extraction eﬃciency andrate showed the eﬀectiveness of bootstrapping labeling in

ASR performance for the models throughout the process.

our task, eventually allowing us to release the model andAlignment quality measurement

We listened tocorpus freely available. It also revealed the possibility ofconstructing large-scale corpora for more languages withfew labeled resources.
ASR results

We trained an E2E ASR model using

a recipe provided by ESPnet2 [24] on ReazonSpeech andevaluated the system on the following benchmark datasets:• JSUT[1] basic 5000: a high-quality dataset covers
common words originally recorded for text-to-speech.
• Common Voice[2] v8.0 test set：4483 utterances withutterances upon request from the authors of the content.
Table 2Comparison of Japanese ASR Corpora
Duration(hrs) Commercial Creation
JSUT[1]
10
Free
RecordedCommon Voice v8.0[2]48Free
Recorded
CSJ[5]
600
ChargedRecorded
JTubeSpeech[9]
1300
Free
Automatic
2050
Negotiable AutomaticLaboroTVSpeech[8]ReazonSpeech
15735
Free
Automaticslightly noisier environments.

Audio processing

To enrich the variability in lead-

We used CER after removing punctuation marks and con-ing blanks and noises and avoid the abrupt starting ofverting numbers to words using num2words[25] to mea-

speech, we allowed more span in the beginning during

sure the ASR performance. The model trained on Reazon-

audio segmentation to reduce the possibility of introduc-

Speech achieves state-of-the-art performance in terms ofing systematic errors brought by CTC segmentation. To
ensure no confounding speech gets prepended to the audio,

CER, on par with the Whisper[3] model trained on 680,000
hours of multilingual data and 7043 hours of Japanese. The

the midpoint between the end of the previous segment andcomparison on model sizes perspective is attached in thethe start of the current segment is set to be the adjusted

appendix. The competitive results we have on the bench-

starting point, and the prepending duration is capped at 3

mark datasets showed the usefulness of our corpus. Also by

seconds.
Text processing

observing the improvements in ASR performance through-We uniﬁed half-width and full-out the bootstrapping process, we conﬁrmed our pipeline’s

width alphanumerics and removed special characters fromcapability of fostering continuous evolution.

the transcriptions. Further processing is left to the judg-

Table 1Comparison of Model CER (%)
JSUT[1] Common Voice[2]Whisper small [3]
14.35
15.18
9.89
11.42
Whisper medium [3]Whisper large-v2[3]
8.17
9.70
ESPnet LaboroTVSpeech[16]
11.72
12.56
ESPnet ReazonSpeech(15k hrs)
8.23
9.93

5

Released Corpus information

ment of researchers who use the corpus. The normalized
transcriptions cover 2599 unique characters and 359,608
unique surface forms under tokenization by MeCab[26]with mecab-ipadic-NEologd[27] as the dictionary.

6

Conclusion
This paper introduces our eﬀorts on the automatic con-struction of a freely available large-scale Japanese ASR

We used 1seg TV recordings from 2021.05.27 tocorpus from TV recording fully from scratch without de-

2022.12.25 to construct the initial release of ReazonSpeechwhich consists of 10, 390, 151 utterances (15, 735 hours

pendency on pretrained models. By conﬁrming the strongof speech and 227.6M characters in the transcriptions).

posed method and corpus. By open-sourcing the project,

The alignment of the corpus is done by the 10th-gen

we hope to lower the barrier of entry for E2E ASR training,

ASR created as the procedure described in 3.2(trained onallowing more individuals and organizations to participate.

10,000 hrs, extraction rate 66.84%, with a threshold at

7

CER(transcription, ASR(audio)) ≤ 0.33). The duration of

ASR performance, we showed the eﬀectiveness of our pro-

Future Workutterance was capped at 14 seconds and the dictionary size

• We plan to maintain the project and release corpus

at 2600 for training eﬃciency. The full corpus without

updates periodically.

duration or dictionary constraints contains 19k hours of• We currently use CER based on ASR results for ﬁl-audio. Details about the corpus and TV program genre
can be found on our project page. To comply with thetering. There might be more appropriate and computationally eﬃcient metrics.

Copyright Act, we shuﬄed the data at the utterance level• Bootstrapping from pretrained multilingual models

to prevent the reconstruction of the TV shows for purposes
except for ASR researches. We will delete the relevant

could make the learning curve less steep as they havea robust acoustic model.

Acknowledgement
We would like to express our gratitude to Taichi
Kakinuma for providing legal guidance on the project.
We are also grateful to the datasets and softwares
we extensively used including but not limited to
ESPnet[20] toolkit,

recipes and models,Common

Voice[2], LaboroTVSpeech[8].

References[1] Ryosuke Sonobe, Shinnosuke Takamichi, and Hiroshi
Saruwatari. JSUT corpus: free large-scale japanese speech
corpus for end-to-end speech synthesis. arXiv preprint
arXiv:1711.00354, 2017.
[2] Rosana Ardila, Megan Branson, Kelly Davis, Michael
Kohler, Josh Meyer, Michael Henretty, Reuben Morais,
Lindsay Saunders, Francis Tyers, and Gregor Weber.
Common voice: A massively-multilingual speech corpus.
In Proceedings of the Twelfth Language Resources
and Evaluation Conference, 2020.
[3] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust
speech recognition via large-scale weak supervision. arXiv
preprint arXiv:2212.04356, 2022.
[4] Jinyu Li. Recent advances in end-to-end automatic speech
recognition. arXiv preprint arXiv:2111.01690, 2021.
[5] Kikuo Maekawa. Corpus of spontaneous japanese: Its
design and evaluation. In Proceedings of The ISCA &IEEE Workshop on Spontaneous Speech Processing
and Recognition (SSPR2003), 2003.

[6] Daniel Galvez, Greg Diamos, and Juan et al. Ciro. The
people’s speech: A large-scale diverse english speech
recognition dataset for commercial usage. arXiv preprint
arXiv:2111.09344, 2021.
[7] Guoguo Chen, Shuzhou Chai, and Guan-Bo Wang et al.
GigaSpeech: An Evolving, Multi-Domain ASR Corpus
with 10,000 Hours of Transcribed Audio. In Proceedingsof Interspeech 2021, pp. 3670–3674, 2021.
[8] Shintaro Ando and Hiromasa Fujihara. Construction of alarge-scale japanese asr corpus on tv recordings. In IEEE
International Conference on Acoustics, Speech andSignal Processing (ICASSP2021), 2021.

[9] Shinnosuke Takamichi, Ludwig Kurzinger, Takaaki Saeki,
Sayaka Shiota, and Shinji Watanabe. JTubeSpeech: 音声
認識と話者照合のために YouTube から構築される日
本語音声コーパス. 言語処理学会第 28 回年次大会,
2022.
[10] Jeong-Uk Bang, Mu-Yeol Choi, Sang-Hun Kim, and OhWook Kwon. Automatic construction of a large-scalespeech recognition database using multi-genre broadcast
data with inaccurate subtitle timestamps. IEICE Transactions on Information and Systems, 2020.
[11] Vimal Manohar, Daniel Povey, and Sanjeev Khudanpur.
Jhu kaldi system for arabic mgb-3 asr challenge using diarization, audio-transcript alignment and transfer learning.
In IEEE 2017 Workshop on Automatic Speech Recog-

nition and Understanding (ASRU2017), 2017.
[12] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, and Burget et al. The kaldi speech recognition toolkit. In IEEE
2011 Workshop on Automatic Speech Recognition
and Understanding (ASRU2011), December 2011.

[13] Hervé Bredin, Ruiqing Yin, Juan Manuel Coria, andGelly et al. pyannote.audio: neural building blocks
for speaker diarization. In IEEE International Conference on Acoustics, Speech, and Signal Processing(ICASSP2020), 2020.

[14] Mai Hiroshi and Masayuki. 短単位品詞の用法曖昧性
解決と依存関係ラベリングの同時学習. 言語処理学会第 25 回年次大会, 2019.
[15] Matthew Honnibal, Ines Montani, Soﬁe Van Landeghem,and Adriane Boyd. spaCy: Industrial-strength Natural
Language Processing in Python. 2020.
[16] Shinji Watanabe. ESPnet2 pretrained model, Shinji Watanabe/laborotv asr train asr conformer2 latest33 raw
char sp valid.acc.ave, fs=16k, lang=jp, 2020. https://zenodo.org/record/4304245.
[17] Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. Ctc-segmentation oflarge corpora for german end-to-end speech recognition.
In Speech and Computer, 2020.
[18] Michael McAuliﬀe, Michaela Socolof, Sarah Mihuc,
Michael Wagner, and Morgan Sonderegger. Montreal
forced aligner: Trainable text-speech alignment using
kaldi. In Proceedings of Interspeech 2017, 2017.
[19] Alberto Pettarin et al. Aeneas, 2017. https://www.
readbeyond.it/aeneas/.
[20] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki
Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta
Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen,
Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: Endto-end speech processing toolkit. In Proceedings of Interspeech 2018, 2018.
[21] Daniel Waegel. A survey of bootstrapping techniques innatural language processing. 2013.
[22] Paul Albert, Diego Ortego, Eric Arazo, Noel E. O’Connor,
and Kevin McGuinness. Relab: Reliable label bootstrapping for semi-supervised learning. arXiv preprint
arXiv:2007.11866, 2020.
[23] Manuel Giollo, Deniz Gunceler, Yulan Liu, and DanielWillett. Bootstrap an End-to-End ASR System by Multilingual Training, Transfer Learning, Text-to-Text Mappingand Synthetic Audio. In Proceedings of Interspeech2021, 2021.
[24] Espnet/egs2/laborotv/asr1, 2021. https://github.com/
espnet/espnet/tree/master/egs2/laborotv/asr1.
[25] Virgil Dupras et al. num2words library: Convert numbers
to words in multiple languages, 2021. https://github.
com/savoirfairelinux/num2words.
[26] Taku Kudo. Mecab : Yet another part-of-speech and morphological analyzer, 2006.
[27] Toshinori Sato, Taiichi Hashimoto, and Manabu Okumura.
単語分かち書き辞書 mecab-ipadic-neologd の実装と情報検索における効果的な使用方法の検討. 言語処
理学会第 23 回年次大会, 2017.

A

Appendix

Figure 5

Example of Aligning using Dynamic Programming

Figure 6

CER vs. Number of Model Parameters