The 19th Annual Conference of the Japanese Society for Artificial Intelligence, 2005

   "!#%$&'( 3F2-04)+*-,/. 0214365 798;:<9=?>6@A>6@ABDCFEHGI@KJMLNG

Speech Recognition Using Incremental Environment Clustering based on Reinforcement Learning

Masafumi Nishida

Yasuo Horiuchi

Akira Ichikawa

Graduate School of Science and Technology, Chiba University

This paper describes a novel approach based on online unsupervised adaptation and clustering using reinforcement
learning. Reinforcement learning technique is a computational approach to learning whereby an agent tries tomaximize the total amount of reward it receives when interacting with a complex, uncertain environment. The
adaptation progresses based on rewards that represents correctness of outputs. The adapted models graduallyaccumulate and cluster with the environmental conditions and can immediately adapt by selecting the optimalmodel from the clusters. We conducted speech recognition experiments by a connected digit recognition in noisy
environments including the variation of speakers and noises. The results verify that the proposed method has ahigher recognition performance than the conventional adaptation method.

TcVuZUXvZODWZwtY\xXPRyz[FQATV]_S^a{|`ZYVbF}Fce~adVVfVRgicehXWXjkt[epTXqUatlV\mF\niz^ZqoZapRRqiXrtXs v
V¤VZ«kV¥ ¬ R[¦[i`X] b w|uxatuytvy|TzZTX{X{|XYX§TX}k¨~vnzXzkXp¦ni­XgZq®tYXfk}¯zq\c© °ZX ±X{tpRv²`X¡Xqebz¢||{|³Zªe´£~µ ~\ªÂz¶tÄ·e¸\Åz¹RÆ ºeuVv»iyt¼VTZ½\{VZYZ¾V}k¿ ~e XzÇkqni_­XÀ¦®tF¯zÁÃc ÂR°Zq| p¦q ÈZÉÒØ |vVÈ|Ù¦gXÉFv|uXÊZ·evXzÚ¦ÓtËXÔ¦ Ìk ceni`|`ttbFb\f|~a´ rV ÍzR¹eÎtni{\[R°V³tÏt´ `VpÖµ b|qt[°Xª\]it ~ZÉkVÕVn ÛXmÖÐVÜ neÑ§ zX§r ÝX_rt× FrZveÞnæXâ ÔzvZ¬ ~ertßz°Xsàktv~Xãt³XÊV´äkµ cË|°XcÌ¦çt±c è|« fVraqe³XgZÈtËX´ É µÌ nnuaåzâ ­t ¬ ®\rz ¯ksZcv[X`|éaÞ|b\êRÝákrVc`X~aab°Ä
kcnâ°Xc§|¬ pRÈXðÉgeqV°XnªztÐV³XÑVV´p¦ëiµì qeneÒñaò±ÈXÝaÉrV¨veec ÙkíF«Vªó v|·ÚFv¶|ÒZ·e îz¸z ï¹¶VÞ·e¸tuiÁÏôV`X¹¦b°\ºi[a~Z³»´ ªtµ nõV ö÷tq\gaÃÊZÀF|kËXÁÏÌkÂ¦nqeâÄX¬ rXªzvk øÏÇzv éa¯\ê¦c°Xc z`Zb¦³Z´c°Xµ |nep¦ùVúqñaòtpqtª\VÛXÜ÷ [\qûHüHýHþ?S/ÿ   ÒtfkXî\qï Ò  ÈXuZÉvV~ÊXzËVzÌk n ~  tÈVcÉ | p|z q uZ"v !$#\É ÞÖÁ²â n
Çz~'$ák),~ +% &z^a(¯ â.'*-_)RÂtcereßFâ|ÝXÇkgZciâzWÇRnagZ^Vv0âz/§pÖ_qZf" !$#\~1(uX2¦v n 31q?i¨@vítí|~5~ "~ 34$A,1$6z> 4("  67$89u0v -_> Ât"c rkBk©eF0c ~ XÁ¨«ÂRp ;ó q:  Þ|ÝrVÞ¦] ¬ Þtn <,=apcIªVD±Jª K Á¨Â\v rEF\u vâtvÇ RSUc u5T|«!,Ë ó# T|!Ë # V÷~51Xø ÞÖW2Y\Á G5ËHRînï ±0c Y\CFv gVÝZZfFrqi LNR(MPSO"[(Q \^]_"`*a*b v

1.

Agent
stReinforcement
for atr t+1 + γ V(st+1 ) - V(st ) TD error
st[1]actor-critic
actor
critic(1)δtrt+1:8522u.jp

atV (st )1: actor-criticactor

TDcriticMAP

HMM[3]N (µ0 , σ0 )µ̂[2]λµ0 + T x̄λ+T(2)Tλm ªV/(ª , HRv nue±u°Xv zgV´ fF q¹Ï v ciduet°Xtq0m´ / ¹XH ~*^~ UR H\n0'$v )u ÒZîtï
rt[Rsv $ga÷ËX ÌX0p¦k q|u ªzH x,(cFy ÷ rX[XaÄXÝagXrXfv I(qe iHÊZt(ËZÌk~ lVcem¦d|nâ q ¬kH,~c§÷ (-z| tY(|gz uXv^~e^ô(  [ H, ciF ôn k(~ § ¨vtª~"Â÷!,Á#¦~Hn\ ©
CF ~,r$¢\R~"!,ne#çU1÷2 z rtâzÇ tgznauX^Xvâ H*gk!,~# ,$F~"c¡£÷e t_vv¥E$e F Âk uH,~R2$n5FB~$a$rVkaªÔ(~5|!Y¦#(1c±2\¤ ¦Eg(¦Ffic Â «µ̂ =actorx̄

TD(Temporal Difference)stActionÉ~5,senj$vhgkkapR~5u d(qeet^ÄX?â±rX@v _rec\âVÇ~5ud$~iíZetõVÂö(vc ^a%,zl â±&f"n5!$re+*# â|it~ Çq|1$~eªtF2 õVin5öc m%*«\V/ &ó pR'(n"qe)f 2.2
T#¦c5{V\w pnN³|´q0o.|µ pr}$~aq^e,°|sr~Rt^¦uZuN~v v $$Rx¶y  µceç\ceW§ètfzgg â ¬ ~"Zz,Ó|!Ô~$¦(R (c¶*~  u®> ~ ceßc ¬ X- ÂRy qpÖ qv(}e*~R ~«¦¬ xUyµMAPactor

δt = rt+1 + γV (st+1 ) − V (st )RewardEnvironment

Actor-Critic1rtCritic V(st)Observation

2.

2.1at

Actor
Policy π(1)λHMMst+1statV (st+1 )2631-33 nishida@faculty.chiba-

1.0

1rt+10.5

st+1YXï }[kqyzæ T¦ë c n5 « Â,Â$q;íµí9î -_-¨×k×\r¦ra|©TVY ~ UV ´ v  |× ¹VYF¹ nin  tVf| nrVcz ~(fz3(£§rt1t_|´ íZ[X Â,dz¹X½Zv uaÂ v c ~$£±raô Ù ¨ô ð~ ñ~kuòRðÏyzTFnetftr|z ÒkÓNció9WXô·pR¯qõNlVö m 4$6Fn  c Vp
Ékc z FceWVpq T|UXlV,m 46

The 19th Annual Conference of the Japanese Society for Artificial Intelligence, 2005
15, 20 dB
45

Speech dataA

Adapted models

SNTD error
δ t < θTD

DTimetDModelselection

EF.
.
.

TD errorδ t < θTD1

÷5ø"ù5ú¦û(üMAP

90.5
66.0
42.6

97.1
91.3
68.0

1:

Distance
calculation

SNDist(D, F) > θdist=> Save model F

20 dB
15 dB
10 dB

. actor-critic(%)

FFÉ ciçzè9§³X´ µ õ|ö§¶t·¸\¹¦ºeV»÷ó r5H?,(@ \"n"$¨÷sZgVr,v_> ©|u0ªc «$«kzó('fV)kf~Vj$kkgnm"â /(¬ H un ª,¬9«-$§t"ge¥\°XfÖ¦tnâ gV(v¬ amÄX/$rX,v0H H,(5n \Tk^ $§zg[X°VÞ|RÝan r,â _¬ © 

33.2

BC

316TD error
δ t < θTD

B430

ý;þ5ÿ 0ý;úhþ5ùÿßü
+97.2
90.9
69.4

97.6
90.7
70.1

« q ³Zce´ dVµ f|°Zg Fuv0v  ¸\· ceÚkdVZf|c g $¤tg ~ilVm(cid|fV~gj(kt vmD÷6t${tÁÃÂt\`|rVb¦~ZÄX|rXÇRvc$ß|G5àz~rV[NF§°|r|¦°X|~FÝaÂ gXc $vi(ßt¤t à|g ÈX É \ qZciua~\d|v lVcf|mW±gil4 gZª\viÊZZt¾VËZ¿ Ìk c z«qtó ª\`Xb ~ØVZÞzÇFÝXcr|§ ðge°Xy ~"Â  n tp¦q
ÉRßtcà«~ q³t´ µ °|õzRö c5Z¤\¶\g·Z¸k¹Öºav t»aÈtÉ¦uVv
ct
dt
faguNv.prq^Éksrciçztè9­r§®°³V¯U´ ±9µ ²Uõ|³µö§´·¶t¶*·¸U¸z¹ ¹RºeV»eÈVÉ~ Ékc « q ³V´ µ °V§Ã ¤zg l|m \ r|eßtàÂ\*avz`Xnb aZc0§Vptr¦ Ò  ÈXÉ[Rua_gv °Xz?³Z@ ´ º~e2 õXöF«\nó ¬9â -$i§ ³[ ~ °|z{\ÊVzË|Ì¦c « q °|È|É  uXvVæVÔk~a`|b ´³|µ´ ~iµ õVöíVuaÂUv0½|$Â,V¾´ ¿R n"¹kÀ,ÃÁ±o»kv(~¾³X¿k´ Zµ ô*µ[¦¬§ÁXi¼\[Fc|q ¶V°|·e¸z¬ ³|¹a´~ °Z÷c ±W÷r5__d$gXrZeV³ZvÐV´oÑ|»µ ÈX~eniÉ`X­VbF®|u¯\ci`XécibêRËX~ÌcX°XkVgZpRNfc§qV§rzªzðgisãtv `Zäz b~ea°X[ZtT\fV³Xe´ía§¨µÂµ niõ|ö±~z,ÊX!z# ËVÌ¦~n|*â }e,¬ ~(Rz~(´ $ ¹F¶³V ´ µ µ ÞiÁÃ~ÂU¾$Ä¿FuX¶ v ÞR~aXÁÏôXÇ¦°n [\«k©eó ~|h|njRõVciö÷°V|gapÊZqt|ª\ËXÌ n â Z¬ rªz_ ÀF FvÁ¨YXÂR}Vq{X y|Tn¨FsZvz´ ¹¦ ~z!$#kµí\~~$Â*¢zÄvZRo»¦¶~a µ °| ~³|Å ´  ¶¦aº(¶zÆr·aÇÈ¸¾$¹V¿¦~Zne°|tft³|r|´  i~¾ÖS¿k·i¸znÀ(¹XÁ÷ceñ¨òµv $ô -¨¬$Â¦¾qe¿\ ³Z$´º2 µ ©X«~"ó ¾T\¿k$u Âv io»~5~az(³Zµ !$´ µ#~5u|Â ¶ `VbtÒZî\°V±ï  ua¶tva·ÊZ¸\t¹RËXºeÌV"~ »V|ceÈXçzÉ è9 §zT|qUXl|m\ÈVÉFniÐ|ÉkÑ±c « r|q ÄÅ(¶¦¶º( µ Ærn"ÇÈ¾$ÉU¿¦Êzcirçzè¶a µ v(µ > c ÀÁ§rV ©t~ Ò|ËXÈ|ÌkÉFnuXâ vZ`|¬ ªtb¦ ~X |v RéaciêR c ð`ZgZbtôzáX°F°Z[a|°|pRqV³|ªz´ µ  ne õttö qiaÊÄ«kó
ò|3rZ1zpviÙF^ qtÚkVª\·nÚFâ Ýa 5r z4$øÏqÇz6t ¯zv ÐVc Ñ|`ZbkÈXÉ~Zu YXFne}|cz±{Zf|ytð_rZTzrT|°Z~UXzXl|k³XmR´c±µc ni«ð_ñgqÇt$g ¯\fVfaci³Xl|´m µn¦õVn ö§D²qt°XªzVpR qV ªzZ r|  vß|àz~e°ZzÈXÉc (¤Vª ª  v u ~"Ë$H  zq
2: actor-critic

TD

δt

20dB

15dB

(3)

λ27% 10dB
MAP7%

39%10dB1.4%

λλ ← λ ± βδt(3)actor-critic
10dB

2.3

actor-critic
2MAP

2.1% actor-critic

0.7%

TD

HMMsP
M
µ̄s = 1/M i=1 µsi

MAP

HMM

4.

actor-criticHMMµ̄h = (µ̄h1 , µ̄h2 , · · · , µ̄hS )(4)

Dist(A, B) =HX

AURORA-2J

B 2||µ̄Ah − µ̄h ||

(4)

h=1H

HMM

 IPSJ SIG-SLP "!#$#%&'() ø"* ü,+ø-. AURORA-2J/ ø ¦ ÷ø"ù021"345687:9;A BDCE FGH021"34I ü ú¦û(üKJ[1] <=?>@LNMPOQRSUT2VWYX $Z%& ,”“ý;[ þ5 ÿ\5ÿ] 2004 ^_`  ab ] , 2-4-10, pp.105-106, 2004.

ÌÎÍPÏÑÐ
3.1 ÒÓ9ÔÖÒFîï  ukÕ vanayÖ\TFf\g|`¦vZbµ°t+ IÖne âNH9Ø kØ*lRÙ m97 ³|´ µ×  ¸ ~ lzÖmFq v
âFßz¹XàtuZÝXv|È|r|íXÉtÂ*ÄV X½rV\Âv q ÜÝ ^$x*kÚ °|y c k{ «kvZóßÐtÞ Ñ\È|É¦r|c  ÉF« qicT|d^UVzltq m× ^·(¦ÚÛ nT~ VUV®z´v  ®U¹Zã~"äàá~À¥ cXua®v å\~ ® × y ·Û  ¹|v ×Nnizâ ft r|~aaíZT(Â,wzZ½ Â³µ´ º ±µ uXar|[av T| ÙUF!,z7 c \# ³X«kv ´ ó µËV}Ìkuae*v n ~â¦R ÝXæ ~r|~ YX}Rc c « «kq ó zHU¦t Yn³|~\´¶3k1 cuev çè ~yVTêéë n çè ~ ìé
3.

AURORA-2J

MAP
MAP

actor-critic

12

1

2
16

MFCC

[2] R. S. Sutton and A. G. Barto, “Reinforcement Learning: AnIntroduction,” MIT Press, 1998.

3920HMM110[3] J. L. Gauvain and C. H. Lee, “Maximum a posteriori estimation for multivariate Gaussian mixture observations ofMarkov chains,” IEEE Trans. Speech and Audio Processing,vol. 2, no. 2, pp. 291-298, 1994.

8440

8
Subway, Babble, Car, Exhibition,
3
SN10,
Restaurant, Street, Airport, Station2