PhD F: Do you wanna go , Barry ?
Grad A: Yeah , OK . Um , so for the past , uh , week an or two , I 've been just writing my , uh , formal thesis proposal . Um , so I 'm taking this qualifier exam that 's coming up in two weeks . And I I finish writing a proposal and submit it to the committee . Um . And uh , should I should I explain , uh , more about what what I 'm proposing to do , and s and stuff ?
Professor D: Yes , briefly .
PhD F: Yeah briefly .
Grad A: OK . Um , so briefly , I 'm proposing to do a n a new p approach to speech recognition using um , a combination of , uh , multi - band ideas and ideas , um ,  {comment} about the uh , acoustic phonec phonetic approach to speech recognition . Um , so I will be using these graphical models that um , that implement the multi - band approach to recognize a set of intermediate categories that might involve , uh , things like phonetic features or other other f feature things that are more closely related to the acoustic signal itself . Um , and the hope in all of this is that by going multi - band and by going into these , um intermediate classifications , that we can get a system that 's more robust to to unseen noises , and situations like that . Um , and so , some of the research issues involved in this are , um , {comment} one , what kind of intermediate categories do we need to classify ? Um , another one is um , what what other types of structures in these multi - band graphical models should we consider in order to um , combine evidence from the sub - bands ? And , uh , the third one is how do we how do we merge all the , uh , information from the individual uh , multi - band classifiers to come up with word word recognition or or phone recognition things . Um , so basically that 's that 's what I 've been doing . And ,
PhD F: So you 've got two weeks , huh ?
Grad A: I got two weeks to brush up on d um , presentation stuff and , um ,
Professor D: Oh , I thought you were finishing your thesis in two weeks .
Grad A: But . Oh , that too .
Professor D: Yeah .
Grad A: Yeah .
PhD F: Are you gonna do any dry runs for your thing ,
Grad A: Yes .
PhD F: or are you just gonna Grad A: Yes . I , um I 'm I 'm gonna do some . Would you be interested ? To help out ?
PhD F: Sure .
Grad A: OK .
PhD F: Sure .
Grad A: Thanks . Yeah .
PhD F: Is that it ?
Grad A: That 's it .
PhD F: Hhh . OK . Uh . Hhh . Let 's see . So we 've got forty minutes left , and it seems like there 's a lot of material . An - any suggestions about where we where we should go next ?
PhD B: Mmm , @ @ .
PhD F: Uh . Do you wanna go , Sunil ? Maybe we 'll just start with you .
PhD B: Yeah . But I actually stuck most of this in our m last meeting with Guenter . Um , but I 'll just Um , so the last week , uh , I showed some results with only SpeechDat - Car which was like some fifty - six percent . And , uh , I didn't h I mean , I I found that the results I mean , I wasn't getting that r results on the TI - digit . So I was like looking into " why , what is wrong with the TI - digits ? " . Why why I was not getting it . And I found that , the noise estimation is a reason for the TI - digits to perform worse than the baseline . So , uh , I actually , picked th I mean , the first thing I did was I just scaled the noise estimate by a factor which is less than one to see if that because I found there are a lot of zeros in the spectrogram for the TI - digits when I used this approach . So the first thing I did was I just scaled the noise estimate . And I found So the the results that I 've shown here are the complete results using the new Well , the n the new technique is nothing but the noise estimate scaled by a factor of point five . So it 's just an ad - hoc I mean , some intermediate result , because it 's not optimized for anything . So the results The trend the only trend I could see from those results was like the the p the current noise estimation or the , uh , noise composition scheme is working good for like the car noise type of thing . Because I 've the only only p very good result in the TI - digits is the noise car noise condition for their test - A , which is like the best I could see that uh , for any non - stationary noise like " Babble " or " Subway " or any " Street " , some " Restaurant " noise , it 's like it 's not performing w very well . So , the  So that that 's the first thing I c uh , I could make out from this stuff . And Professor G: Yeah , I think what is important to see is that there is a big difference between the training modes .
PhD B: Yeah .
Professor G: Uh - huh . If you have clean training , you get also a fifty percent improvement .
PhD B: Yeah .
Professor G: But if you have muddy condition training you get only twenty percent .
PhD B: Yeah . Yeah .
PhD E: Mm - hmm .
PhD B: Uh , and in that twenty percent @ @ it 's very inconsistent across different noise conditions .
Professor G: Mm - hmm . Mmm .
PhD B: So I have like a forty - five percent for " Car noise " and then there 's a minus five percent for the " Babble " ,
Professor G: Mmm .
PhD B: and there 's this thirty - three for the " Station " . And so it 's it 's not it 's not actually very consistent across . So . The only correlation between the SpeechDat - Car and this performance is the c stationarity of the noise that is there in these conditions and the SpeechDat - Car .
Professor G: Mm - hmm .
PhD B: And , uh so so the overall result is like in the last page , which is like forty - seven , which is still very imbalanced because there are like fifty - six percent on the SpeechDat - Car and thirty - five percent on the TI - digits . And uh , ps the fifty - six percent is like comparable to what the French Telecom gets , but the thirty - five percent is way off .
Professor D: I 'm sort of confused but this I 'm looking on the second page ,
PhD B: Oh , yep .
Professor D: and it says " fifty percent " looking in the lower right - hand corner , " fifty percent relative performance " .
Professor G: For the clean training .
Professor D: Is that Professor G: u And if you if you look Professor D: is that fifty percent improvement ?
PhD B: Yeah . For that 's for the clean training and the noisy testing for the TI - digits .
Professor G: Yeah .
Professor D: So it 's improvement over the baseline mel cepstrum ?
PhD B: Yeah . Yeah .
Professor D: But the baseline mel cepstrum under those training doesn't do as well I I 'm I 'm trying to understand why it 's it 's eighty percent That 's an accuracy number , I guess ,
PhD B: Yeah , yeah , yeah .
Professor D: right ? So that 's not as good as the one up above .
PhD B: No .
Professor D: But the fifty is better than the one up above ,
PhD B: Yeah .
Professor D: so I 'm confused .
PhD B: Uh , actually the noise compensation whatever , uh , we are put in it works very well for the high mismatch condition . I mean , it 's consistent in the SpeechDat - Car and in the clean training also it gives it But this fifty percent is is that the the high mismatch performance equivalent to the high mismatch performance in the speech .
PhD F: So n s So since the high mismatch performance is much worse to begin with , it 's easier to get a better relative improvement .
PhD B: Yeah . Yeah . I do . Yeah , yeah . So by putting this noise PhD E: Yeah . Yeah , if we look at the figures on the right , we see that the reference system is very bad .
Professor D: Oh .
PhD B: Yeah . The reference drops like a very fast Professor D: Oh , oh , oh , oh , oh , oh .
PhD E: Like for clean clean training condition .
Professor D: I see .
PhD B: Yeah .
Professor D: I see .
PhD E: Nnn .
Professor D: This is this is TI digits {comment} we 're looking at ?
PhD B: Yeah . Yeah . Oh Professor D: This whole page is TI - digits
PhD B: Oh . Yeah .
Professor D: or this is ?
PhD B: It 's not written anywhere . Yeah , it 's TI - digits . The first r spreadsheet is TI - digits .
Professor D: Mmm . How does clean training do for the , uh , " Car "
Professor G: Hmm .
PhD B: The " Car " ?
Professor D: stuff ?
PhD B: Oh . Still it still , uh that that 's still consistent . I mean , I get the best performance in the case of " Car " , which is the third column in the A condition .
Professor D: No . I mean , this is added noise . I mean , this is TI - digits . I 'm sorry . I meant in in the in the , uh , multi - language , uh , uh , Finnish and PhD B: Uh Professor G: This is next next page .
PhD B: That 's the next next spreadsheet , is Professor G: Hmm .
PhD B: So that is the performance for Italian , Finnish and Spanish .
Professor D: " Training condition " Oh , right . So " clean " corresponds to " high mismatch " .
PhD B: Yeah .
Professor D: And " increase " , That 's increase e
Professor G: Improvement .
PhD B: Improvement . That 's " Percentage increase " is the percentage improvement over the baseline .
Professor G: Yeah . It 's it 's a PhD B: So that 's Professor D: Which means decrease in word error rate ?
PhD B: Yeah .
Professor D: OK , so " percentage increase " means decrease ?
PhD B: Yeah , yeah .
Professor D: OK .
Professor G: Yeah . The the w there was a very long discussion about this on on the on the , uh , Amsterdam meeting .
Professor D: Yeah .
Professor G: How to how to calculate it then .
PhD B: Yeah . There 's there 's a Professor G: I I I guess you are using finally this the scheme which they PhD B: Which is there in the spreadsheet .
Professor G: OK .
PhD B: I 'm not changing anything in there .
Professor G: Mmm .
Professor D: Alright .
PhD B: So . Uh , yeah . So all the hi H M numbers are w very good , in the sense , they are better than what the French Telecom gets . So . But the the only number that 's still I mean , which Stephane also got in his result was that medium mismatch of the Finnish , which is very  which is a very strange situation where we used the we changed the proto for initializing the HMM I mean , this this is basically because it gets stuck in some local minimum in the training . That seventy - five point seven nine in the Finnish mismatch which is that the eleven point nine six what we see .
Professor D: Uh - huh .
Professor G: Mmm .
PhD B: Yeah .
Professor D: So we have to jiggle it somehow ?
PhD B: Yeah so we start with that different proto and it becomes eighty - eight , which is like some fifty percent improvement .
Professor D: S Wait a minute . Start with a different what ?
PhD B: Different prototype , which is like a different initialization for the , uh , s transition probabilities . It 's just that right now , the initialization is to stay more in the current state , which is point four point six , right ? Yeah .
PhD E: Yeah .
PhD B: And if it changes to point five point five , which is equal @ @ for transition and self loop where it becomes eighty - eight percent .
PhD F: Well , but that involves mucking with the back - end ,
PhD B: Yeah . We can't do it .
PhD F: which is not allowed .
PhD B: Yeah .
PhD E: Mmm .
PhD F: Yeah .
PhD B: So .
Professor G: I mean , it uh , like , i i i It is well known , this this medium match condition of the Finnish data has some strange effects .
PhD B: Very s
PhD E: Yeah .
PhD B: It has a very few at uh , actually , c uh , tran I mean , words also .
Professor G: I mean , that is Yeah ,
PhD B: It 's a very , very small set , actually .
Professor G: that too . Yeah . Uh - huh .
PhD B: So there is Professor G: There is a l a There is a lot of Uh , there are a lot of utterances with music in with music in the background .
PhD B: Yeah . Yeah , yeah , yeah . Yeah .
Professor G: Mmm .
Professor D: Uh - huh .
PhD B: Yeah . It has some music also . I mean , very horrible music like like I know .
Professor D: So maybe for that one you need a much smarter VAD ? Mmm ,
PhD B: Uh Professor D: if it 's music .
PhD B: So , that that 's the that 's about the results . And , uh , the summary is like OK . So there are the other thing what I tried was , which I explained in the last meeting , is using the channel zero for , uh , for both dropping and estimating the noise . And that 's like just to f n get a feel of how good it is . I guess the fifty - six percent improvement in the SpeechDat - Car becomes like sixty - seven percent . Like ten percent better . But that 's that 's not a that 's a cheating experiment . So . That 's just So , m w
Professor G: But the but the , uh , forty - seven point nine percent which you have now , that 's already a remarkable improvement in comparison to the first proposal .
PhD B: Yeah . So we had forty - four percent in the first proposal .
Professor G: OK .
PhD B: Yeah .
Professor G: Mm - hmm .
PhD B: We have f a big im So the major improvement that we got was in all the high mismatch cases , because all those numbers were in sixties and seventies because we never had any noise compensations .
Professor G: Mmm .
PhD B: So that 's where the biggest improvement came up . Not much in the well match and the medium match and TI - digits also right now . So this is still at three or four percent improvement over the first proposal .
Professor G: Mmm . Mmm .
Professor D: Yeah , so that 's good .
PhD B: Yeah . So .
Professor D: Then if we can improve the noise estimation , then it should get better .
Professor G: Yeah , I I started thinking about also I mean yeah , uh , I discovered the same problem when I started working on uh , on this Aurora task almost two years ago , that you have the problem with this mulit a at the beginning we had only this multi condition training of the TI - digits .
PhD B: Yeah .
Professor G: And , uh , I I found the same problem . Just taking um , what we were used to u use , I mean , uh , some type of spectral subtraction , {comment} y you get even worse results than the basis
PhD B: Yeah . Yeah ,
Professor G: and uh PhD B: yeah .
Professor G: I I tried to find an explanation for it ,
Professor D: Mmm .
Professor G: so PhD B: So . Yes . Stephane also has the same experience of using the spectral subtraction right ?
Professor G: Mmm .
PhD E: Mm - hmm .
PhD B: Yeah . So here here I mean , I found that it 's if I changed the noise estimate I could get an improvement .
PhD E: Yeah .
PhD B: So that 's so it 's something which I can actually pursue , is the noise estimate .
Professor G: Mm - hmm .
PhD B: And Professor G: Yeah , I think what you do is in when when you have the the this multi - condition training mode , um then you have then you can train models for the speech , for the words , as well as for the pauses where you really have all information about the noise available .
PhD B: Yeah .
Professor G: And it was surprising At the beginning it was not surprising to me that you get really the best results on doing it this way , I mean , in comparison to any type of training on clean data and any type of processing . But it was So , u u it it seems to be the best what wh wh what what we can do in this moment is multi - condition training . And every when we now start introducing some some noise reduction technique we we introduce also somehow artificial distortions .
PhD B: Yeah .
Professor G: And these artificial distortions uh , I have the feeling that they are the reason why why we have the problems in this multi - condition training . That means the H M Ms we trained , they are they are based on Gaussians ,
PhD B: Yeah .
Professor G: and on modeling Gaussians . And if you Can I move a little bit with this ? Yeah . And if we introduce now this this u spectral subtraction , or Wiener filtering stuff So , usually what you have is maybe , um I 'm I 'm showing now an envelope um maybe you 'll f for this time . So usually you have maybe in clean condition you have something which looks like this . And if it is noisy it is somewhere here . And then you try to subtract it or Wiener filter or whatever . And what you get is you have always these problems , that you have this these these these zeros in there .
PhD B: Yeah .
Professor G: And you have to do something if you get these negative values . I mean , this is your noise estimate and you somehow subtract it or do whatever . Uh , and then you have And then I think what you do is you introduce some some artificial distribution in this uh in in the models . I mean , i you you train it also this way but , i somehow there is u u there is no longer a a Gaussian distribution . It is somehow a strange distribution which we introduce with these artificial distortions . And and I was thinking that that might be the reason why you get these problems in the especially in the multi - condition training mode .
PhD B: Yeah , yeah .
Professor D: Mm - hmm .