情報処理学会第 82 回全国大会

4Q-02

૒ํ޲ Transformer ‫ޠݴ‬ϞσϧʹΑΔԻ੠ೝࣝԾઆͷϦείΞϦϯάೋ‫ ݟ‬ᰜ 1্೫ ੟ 211.

ࡾଜ ਖ਼ਓ 22‫౎ژ‬େֶ ޻ֶ෦৘ใֶՊ͸͡ΊʹԻ੠ೝࣝ͸ैདྷɼԻ‫ڹ‬৴߸͔ΒԻӆ৘ใΛநग़͢ΔԻ‫ڹ‬ϞσϧɼൃԻࣙॻɼ୯‫ྻܥޠ‬ͷ໬౓Λ‫͢ࢉܭ‬Δ‫ޠݴ‬Ϟσϧʹ෼͚ɼͦΕͧΕֶशͯ͠౷߹͢Δ DNN-HMM Ի੠ೝࣝϞσϧͰ࣮‫͞ݱ‬Ε͖ͯͨɽۙ೥Ͱ͸ͦΕΒΛҰͭͷϞσϧͰ࣮‫͢ݱ‬Δ End-to-End Ի੠ೝࣝ΋ఏҊ͞Ε͍ͯΔɽ྆ϞσϧͰ͸Ұ౓Ի੠ೝࣝԾઆΛग़ྗͨ͠‫ʹޙ‬ɼςΩετσʔλͷΈͰֶश͞Εͨ֎෦‫ޠݴ‬ϞσϧΛ༻͍ͯɼԾઆͷείΞΛ‫͠ࢉܭ‬௚͢ϦείΞϦϯά͕ޮՌΛ͍ࣔͯ͠Δ [1]ɽຊ‫Ͱڀݚ‬͸ɼ‫ޠݴ‬Ϟσϧͱͯ͠૒ํ޲ Transformer [2]Λ༻͍Δ͜ͱΛߟ͑Δɽ૒ํ޲ Transformer ͸ɼॱํ޲ͷਪ࿦ͱ‫޲ํٯ‬ͷਪ࿦ͷग़ྗΛ୯ʹ݁߹ͨ͠૒ํ޲ͷ
RNN ͱ͸ҟͳΓɼࣗ‫ݾ‬஫ҙ‫ʹߏػ‬ΑΓ૒ํ޲ͷ௕͍จ຺Λ௚઀ଊ͑Δ͜ͱ͕Ͱ͖Δɽ Ϟσϧߏ଄ͱͯ͠૒ํ
޲ Transformer Λ࠾༻ͨ͠ BERT [3] ͸ϚεΫ͞Εͨ୯‫ޠ‬Λલ‫ޙ‬ͷจ຺͔Β༧ଌ͢Δࣄલֶश (Masked LanguageModel) Λߦ͏͜ͱͰɼࣗવ‫ॲޠݴ‬ཧͷ༷ʑͳλεΫͰޮՌΛ͖ࣔͯͨ͠ɽ͞ΒʹɼԻ੠ೝࣝԾઆͷϦείΞϦϯάʹ͓͍ͯ΋૒ํ޲ Transformer ‫ޠݴ‬Ϟσϧ͕‫ݕ‬౼͞Ε͍ͯΔ [4]ɽ

2.

ϦείΞϦϯά

W = w1 w2 ... Λ୯‫ྻޠ‬ɼX ΛԻ‫ڹ‬ಛ௃ྔͱ͢ΔɽϦείΞϦϯάͰ͸ɼԻ੠ೝࣝϞσϧͷର਺໬౓ScoreASR := log p(W |X)/|W | ͱ‫ޠݴ‬ϞσϧͷείΞ
ScoreLM Λ૊Έ߹Θͤͯ
Score := (1 − λ)ScoreASR + λScoreLM(1)͕࠷େͱͳΔԾઆΛ N-best Ծઆͷத͔Βબ୒͢Δɽैདྷ༻͍ΒΕΔ‫ޠݴ‬Ϟσϧͱͯ͠ɼN άϥϜɼ୯ํ޲
RNN ‫ޠݴ‬Ϟσϧ౳͕͋Γɼ͜ΕΒ୯ํ޲‫ޠݴ‬ϞσϧʹΑΔείΞ͸ಉ࣌֬཰ScoreLM :=|W |!
t=1log p(wt |W<t ) = log p(w1 , ..., w|W | )(2)ʹΑͬͯ‫͞ࢉܭ‬ΕΔɽҰํɼ૒ํ޲ Transformer ‫ޠݴ‬ϞσϧͷείΞ͸

ScoreLM :=|W |!
t=1log p(wt |W\t )ࡔҪ ৴ี 2Տ‫ ݪ‬ୡ໵ 2‫౎ژ‬େֶ େֶӃ৘ใֶ‫ڀݚ‬Պ
log $ “갈㡮” +log $(“钠陎”) +log $(“䪮遭”)= *+,-./0⿽倯ぢ5SBOTGPSNFSY㾴˘ִ٦ ֿ‫< ך‬."4,> 钠陎 䪮遭 ‫˘׷״ ח‬˘ִ٦ ֿ‫ ך‬갈㡮 <."4,> 䪮遭 ‫˘׷״ ח‬˘ִ٦ ֿ‫ ך‬갈㡮 钠陎 <."4,>‫˘׷״ ח‬˘ִ٦ ֿ‫ך‬湫⵸‫ך‬涪鑧갈㡮 钠陎 䪮遭ٔ‫؝أ‬،ٔٝ‫ؚ‬㼎韋‫˘׷״ ח‬湫䖓‫ך‬涪鑧ਤ 1: ૒ํ޲ Transformer ‫ޠݴ‬ϞσϧʹΑΔείΞ‫ࢉܭ‬ɽϦείΞϦϯάର৅ʮԻ੠ೝٕࣝज़ʯͷ୯‫ޠ‬Λ 1 ‫ͭͣޠ‬[MASK] ʹஔ͖‫ͨ͑׵‬ೖྗ͔Β‫ݩ‬ͷ୯‫ޠ‬Λ༧ଌ͢Δ͜ͱͰ‫ޠݴ‬ϞσϧͷείΞΛಘΔɽ

Transformer ‫ޠݴ‬Ϟσϧͷֶश͸ɼ[MASK] τʔΫϯ͔Β‫ݩ‬ͷ୯‫ޠ‬Λ༧ଌͰ͖ΔΑ͏ʹɼ1 ൃ࿩ͷ 15 % ΛϥϯμϜʹબͼ [MASK] τʔΫϯʹஔ͖‫͑׵‬Δ Masked Language
Model ʹΑ࣮ͬͯ‫͞ݱ‬ΕΔ [4]ɽ3. ఏҊख๏
3.1 ෳ਺ൃ࿩ͷ݁߹
Transformer ͸ࣗ‫ݾ‬஫ҙ‫ʹߏػ‬ΑΓɼ௕͍จ຺Λଊ͑
Δ͜ͱ͕Ͱ͖Δ [2]ɽલ‫ޙ‬ͷൃ࿩ͷԻ੠ೝࣝԾઆΛ݁߹͠είΞ‫ࢉܭ‬Λߦ͏͜ͱͰɼ‫ൃݱ‬࿩ͷԻ੠ೝࣝԾઆ಺Ͱ
ଊ͑Δ͜ͱ͕Ͱ͖ͳ͍จ຺ΛߟྀͰ͖Δͱߟ͑ΒΕΔɽຊ‫Ͱڀݚ‬͸ɼֶशɾϦείΞϦϯά࣌ͱ΋ʹൃ࿩ͷલ‫ޙ‬
1 ൃ࿩Λ݁߹͠ɼ‫ ܭ‬3 ൃ࿩෼Λೖྗͱ͢ΔɽϦείΞϦϯά࣌ʹ͸ɼ݁߹͢Δલ‫ޙ‬ͷൃ࿩͸Ի੠ೝࣝͷର਺໬౓͕࠷େͷ΋ͷΛ࢖༻͢Δɽਤ 1 ʹʮԻ੠ೝٕࣝज़ʯͱ͍͏ൃ࿩΁ͷείΞ‫ࢉܭ‬Λྫࣔ͢Δɽ
3.2 εϜʔδϯά(3)ʹΑͬͯ‫͞ࢉܭ‬ΕΔ [4]ɽW\t ͸ W ͷ t ൪໨ͷ୯‫ޠ‬Λ [MASK] τʔΫϯʹஔ͖‫ͨ͑׵‬΋ͷͰ͋Δɽ૒ํ޲
Rescoring Hypotheses of Automatic Speech Recognition with Bidirectional Transformer Language Model: Hayato Futami, Sei Ueno, Masato
Mimura, Shinsuke Sakai, and Tatsuya Kawahara (Kyoto Univ.)ϚεΫ͞Εͨ୯‫ޠ‬Λલ‫ޙ‬ͷจ຺͔Β༧ଌ͢Δ૒ํ޲ϞσϧͰ͸ɼ ୯ํ޲Ϟσϧʹൺ΂ɼΑΓֶशσʔλͷςΩετͷӨ‫ڹ‬Λड͚΍͘͢ɼύʔϓϨΩγςΟ͕Լ͕Γ΍͍͢ͱߟ͑ΒΕΔɽ͔͠͠ɼύʔϓϨΩςΟ͕௿͗͢ΔϞσϧͰ͸ɼϑΟϥʔ΍Ի੠ೝࣝ‫ޡ‬Γ͕‫·ؚ‬ΕΔϦείΞϦϯά࣌ͱֶश࣌ͷσʔλͷෆҰகʹΑΓɼਖ਼͘͠ϦείΞϦϯά͕Ͱ͖ͳ͍ͱߟ͑ΒΕΔɽͦͷͨΊɼຊ‫ݚ‬

2-175

Copyright 2020 Information Processing Society of Japan.
All Rights Reserved.

情報処理学会第 82 回全国大会‫Ͱڀ‬͸ [5] Λ‫ʹج‬ɼҎԼͷࣜ (4) ʹࣔ͢ιϑτϚοΫε૚ͰͷεϜʔδϯάΛద༻͢Δɽ
exp (αywt )p(wt |W\t ) = "(4)j exp (αyj )ද 1: ఏҊख๏ʹΑΔϦείΞϦϯάͷޮՌԻ੠ೝࣝͷΈbiTLM [4]biTLM (+લ‫ൃޙ‬࿩)yj ΛιϑτϚοΫε૚΁ͷೖྗɼα(< 1) ΛεϜʔδϯάͷύϥϝʔλͱ͢Δ.

4.

biTLM (+લ‫ൃޙ‬࿩ɼεϜʔδϯά)
50-best oracleධՁ࣮‫ݧ‬ຊ‫Ͱڀݚ‬͸σʔληοτͱͯ͠ʰ೔ຊ‫ޠ‬࿩͠‫ݴ‬༿ίʔύεʱ(CSJ) Λ༻͍ΔɽCSJ ͸ CSJ-APS ͱ CSJ-SPS ͷ2 ͭͷαϒίʔύεͰߏ੒͞Ε͍ͯΔɽ͜ͷ͏ͪɼCSJAPS ͷԻ੠ͱςΩετͷϖΞΛԻ੠ೝࣝϞσϧͷֶशɼ
CSJ-APS ͱ CSJ-SPS ͷςΩετΛ‫ޠݴ‬Ϟσϧͷֶशʹ༻͍ͨɽఏ‫͞ڙ‬Ε͍ͯΔ CSJ-APS ༻ͷςετηοτͷ
͏ͪɼςετηοτ 2 Λ։ൃσʔλɼςετηοτ 1 ΛධՁσʔλͱͨ͠ɽ(3) ͷ‫ޠݴ‬ϞσϧͷείΞͷॏΈ λ͸ (0.0025, 0.005, ..., 0.05) ͔ΒධՁσʔλͷ WER Λ࠷খʹ͢Δ΋ͷΛࣔ͢ɽࣜ (4) ͷεϜʔδϯάͷύϥϝʔλ α ͸։ൃσʔλʹΑΓ α = 0.5 ͱܾఆ͢Δɽ‫ޠ‬ኮ͸Ի੠ೝࣝϞσϧɼ‫ޠݴ‬Ϟσϧͱ΋ʹ Byte Pair Encoding [6]Λ༻͍ͯ‫ޠ‬ኮαΠζ 12414 ͷαϒϫʔυ୯Ґͱ͢ΔɽԻ੠ೝࣝϞσϧ Ի‫ڹ‬ಛ௃ྔ͔Βαϒϫʔυ‫ྻܥ‬Λ Endto-End Ͱਪఆ͢ΔϞσϧͱͯ͠ɼ஫ҙ‫ߏػ‬Ϟσϧ [7] Λ༻͍Δɽ஫ҙ‫ߏػ‬Ϟσϧ͸ 5 ૚ͷ૒ํ޲ LSTM Τϯίʔμͱ 1 ૚ͷ୯ํ޲ LSTM σίʔμͰߏ੒͞ΕΔɽਖ਼ଇ
Խख๏ͱͯ͠ϥϕϧεϜʔδϯάɼσʔλ֦ுख๏ͱ͠
ͯ SpecAugment [8] Λ༻͍Δɽ
૒ํ޲ Transformer ‫ޠݴ‬Ϟσϧ ૒ํ޲ Transformer ‫ݴ‬‫ޠ‬Ϟσϧ͸ɼBERTBASE [3] ͱಉ༷ӅΕঢ়ଶ 768 ࣍‫ݩ‬ɼ12 ‫ݸ‬ͷ஫ҙ‫ߏػ‬ϔουΛ࣋ͭ 12 ૚ͷ૒ํ޲ Transformer
ͱ͢Δɽൺֱख๏ͷ N άϥϜ͸ 4 άϥϜ (4-gram)ɼRNN ‫ޠݴ‬Ϟσϧ (RNNLM) ͸ɼ512 ࣍‫ݩ‬ͷग़ྗΛ࣋ͭຒΊࠐΈ૚͓ΑͼӅΕঢ়ଶ 512 ࣍‫ݩ‬Λ࣋ͭ୯ํ޲ LSTM 3 ૚ʹΑΓߏ੒͞ΕΔɽධՁσʔλͷԻ੠ೝࣝ 50-best ީิʹର͢ΔϦείΞϦϯά‫ޙ‬ͷ୯‫ޡޠ‬Γ཰ (WER) ʹΑͬͯධՁΛߦͬͨɽද 1 ʹઌߦ‫[ ڀݚ‬4] (biTLM) ͷ૒ํ޲ Transformer ‫ޠݴ‬ϞσϧͱఏҊͨ͠લ‫ ޙ‬1 ൃ࿩Λ௥Ճͨ͠΋ͷ (biTLM(+લ‫ൃޙ‬࿩)) ͱ͞ΒʹεϜʔδϯάΛద༻ͨ͠΋ͷ (biTLM(+લ‫ൃޙ‬࿩ɺ εϜʔδϯά)) ͱͷൺֱ݁ՌΛࣔ͢ɽͱ΋ʹޮՌ͕֬ೝͰ͖Δɽ
ද 2 ʹఏҊͨ͠૒ํ޲ Transformer ‫ޠݴ‬Ϟσϧͱ N άϥϜɼRNN ‫ޠݴ‬Ϟσϧͱͷൺֱ݁ՌΛࣔ͢ɽલ‫ൃޙ‬࿩Λ௥Ճͨ͠૒ํ޲ Transformer ‫ޠݴ‬Ϟσϧͱͷެਖ਼ͳൺֱͷͨΊɼN άϥϜ‫ޠݴ‬Ϟσϧͱ RNN ‫ޠݴ‬Ϟσϧʹ͓͍ͯ΋લ 2 ൃ࿩Λ௥Ճͯ͠ϦείΞϦϯά݁Ռͷ WERΛධՁͨ͠ɽද 2 Ͱ͸ධՁσʔλ্ͷύʔϓϨΩγςΟ (PPL) ʹ͍ͭͯ΋ൺֱ͍ͯ͠Δɽ૒ํ޲‫ޠݴ‬ϞσϧͷύʔϓϨΩγςΟʹ͍ͭͯ͸ɼࣜ (5) Ͱఆٛ͞ΕΔٖࣅύʔϓϨΩγ

WER(%)
10.57
9.96
9.67
9.52
5.41

oracle ͱ͸ԾઆͷதͰશͯ WER Λ࠷খʹ͢Δ΋ͷΛબ΂ͨ৔߹ͷ WER Ͱ͋ΓɼϦείΞϦϯάʹΑΔ WER վળͷԼ‫͋Ͱݶ‬Δɽද 2: ୯ํ޲‫ޠݴ‬Ϟσϧͱͷൺֱ (3 ൃ࿩)Ի੠ೝࣝͷΈ4-gram
RNNLM
biTLM (+લ‫ൃޙ‬࿩)biTLM(+લ‫ൃޙ‬࿩ɼεϜʔδϯά)

WER(%)
10.57
10.14
9.92
9.67

PPL—
83.43
69.36
9.51

9.52

19.06

ςΟ [5] ͱ͢ΔɽP P L(W ) :=|W |1 !
− log p(wt |W\t )|W | t=1(5)εϜʔδϯάͷࠜ‫ͨ͠ͱڌ‬૒ํ޲‫ޠݴ‬Ϟσϧͷ༧ଌ୯‫ޠ‬ͷεύʔεͳ֬཰෼෍͸௿͍ύʔϓϨΩγςΟͱͯ͠‫ݱ‬Ε͍ͯΔɽ·ͨ୯ํ޲‫ޠݴ‬Ϟσϧʹରͯ͠΋ಉ༷ͷεϜʔδϯάΛߦ͕ͬͨޮՌ͸‫ݟ‬ΒΕͳ͔ͬͨɽ

5. ͓ΘΓʹຊߘͰ͸ɼධՁ࣮‫ʹݧ‬Αͬͯɼ૒ํ޲ Transformer ‫ݴ‬‫ޠ‬Ϟσϧ͕Ի੠ೝࣝԾઆͷϦείΞϦϯάʹ༗ޮͰ͋Δ͜ͱΛࣔ͠ɼ͞ΒʹఏҊͨ͠ෳ਺ൃ࿩ͷ݁߹ͱεϜʔδϯάͷ༗ޮੑΛࣔͨ͠ɽࠓ‫ޙ‬͸ɼΑΓେ‫ن‬໛ͳςΩετσʔλͰͷֶशɼ͞Βʹ͸ɼσίʔυஈ֊Ͱ૒ํ޲ͷจ຺ΛߟྀͰ͖ΔΑ͏ʹ͢Δ͜ͱͰ N-best Ծઆͱͯ͠‫ݱ‬Εͳ͍ਖ਼ղΛσίʔυ͢Δ͜ͱΛ໨ࢦ͢ɽࢀߟจ‫ݙ‬

2-176[1] W. Chan et al.: “Listen, attend and spell: A neural network
for large vocabulary conversational speech recognition” ICASSP,4960–4964, 2016.
[2] A. Vaswani et al.: “Attention Is All You Need” NIPS, 5998–6008,
2017.
[3] J. Devlin et al.: “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” NAACL, 4171–4186, 2019.
[4] J. Shin et al.: “Effective Sentence Scoring Method Using BERT forSpeech Recognition” ACML, 1081–1093, 2019.
[5] X. Chen et al.: “Investigating Bidirectional Recurrent Neural Network Language Models for Speech Recognition” INTERSPEECH,
269–273, 2017.
[6] R. Sennrich et al.: “Neural Machine Translation of Rare Words with
Subword Units” ACL, 1715–1725, 2016.
[7] J. Chorowski et al.: “Attention-Based Models for Speech Recognition” NIPS, 577–585, 2015.
[8] DS. Park et al.: “SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition” INTERSPEECH,
2613–2617, 2019.

Copyright 2020 Information Processing Society of Japan.
All Rights Reserved.