PhD F: Uh , yeah . Yeah , at first I had a remark why I am wondering why the PDA is always so far . I mean we are always meeting at the beginning of the table and the PDA 's there .
Professor A: Uh . I guess cuz we haven't wanted to move it . We we could  we could move us ,
PhD F: Yeah ?
Professor A: and .
PhD F: OK .
Grad E: That 's right .
PhD F: Well , anyway . Um . Yeah , so . Uh . Since the last meeting we 've we 've tried to put together um the clean low - pass um downsampling , upsampling , I mean , Uh the new filter that 's replacing the LDA filters , and also the um delay issue so that We considered th the the delay issue on the for the on - line normalization . Mmm . So we 've put together all this and then we have results that are not um  very impressive . Well , there is no real improvement .
Professor A: But it 's not wer worse and it 's better better latency ,
PhD F: It 's not Professor A: right ?
PhD F: Yeah . Yeah . Well . Actually it 's better . It seems better when we look at the mismatched case but I think we are like like cheated here by the th this problem that uh in some cases when you modify slight slightly modify the initial condition you end up completely somewhere air somewhere else in the in the space , the parameters .
Professor A: Yeah .
PhD F: So . Well . The other system are for instance . For Italian is at seventy - eight percent recognition rate on the mismatch , and this new system has eighty - nine . But I don't think it indicates something , really . I don't I don't think it means that the new system is more robust
Professor A: Uh - huh .
PhD F: or It 's simply the fact that Well .
Professor A: Well , the test would be if you then tried it on one of the other test sets , if if it was PhD F: Y
Professor A: Right . So this was Italian , right ?
PhD F: Yeah . Yeah .
Professor A: So then if you take your changes
PhD F: It 's similar for other test sets
Professor A: and then PhD F: but I mean from this se seventy - eight um percent recognition rate system , I could change the transition probabilities for the the first HMM and it will end up to eighty - nine also .
Professor A: Uh - huh .
PhD F: By using point five instead of point six , point four as in the the HTK script .
Professor A: Uh - huh . Yeah .
PhD F: So . Well . That 's PhD B: Yeah . Yeah I looked at um  looked at the results when Stephane did that
PhD F: Well . Eh uh PhD B: and it 's it 's really wo really happens .
PhD F: This really happens .
PhD B: I mean th the only difference is you change the self - loop transition probability by a tenth of a percent
PhD F: Yeah .
Professor A: Yeah .
PhD B: and it causes ten percent difference in the word error rate .
Professor A: A tenth of a per cent .
PhD B: Yeah . From point PhD F: Even tenth of a percent ?
PhD B: I I 'm sorry
PhD F: Well , we tried we tried point one ,
PhD B: f for point from You change at point one
PhD F: yeah .
Professor A: Oh !
PhD B: and n not tenth of a percent , one tenth ,
PhD F: Hmm .
Professor A: Yeah .
PhD B: alright ? Um so from point five so from point six to point five and you get ten percent better .
Professor A: Mm - hmm .
PhD B: And it 's  I think it 's what you basically hypothesized in the last meeting about uh it just being very PhD F: Mm - hmm .
PhD B: and I think you mentioned this in your email too it 's just very um PhD F: Mmm , yeah .
PhD B: you know get stuck in some local minimum and this thing throws you out of it I guess .
PhD F: Mm - hmm .
Professor A: Well , what 's what are according to the rules what what are we supposed to do about the transition probabilities ? Are they supposed to be point five or point six ?
PhD B: I think you 're not allowed to Yeah . That 's supposed to be point six , for the self - loop .
PhD F: Yeah .
Professor A: Point It 's supposed to be point six .
PhD B: Yeah . But changing it to point five I think is which gives you much better results , but that 's not allowed .
Professor A: But not allowed ? Yeah . OK .
PhD B: Yeah .
PhD F: Yeah , but even if you use point five , I 'm not sure it will always give you the better results
PhD B: Yeah .
PhD F: on other test set or it
PhD B: Right . We only tested it on the the medium mismatch ,
PhD F: on the other training set , I mean .
PhD B: right ? You said on the other cases you didn't notice PhD F: Yeah . But . I think , yeah . I think the reason is , yeah , I not I it was in my mail I think also , is the fact that the mismatch is trained only on the far microphone . Well , in for the mismatched case everything is um using the far microphone training and testing , whereas for the highly mismatched , training is done on the close microphone so it 's it 's clean speech basically so you don't have this problem of local minima probably and for the well - match , it 's a mix of close microphone and distant microphone and Well .
PhD B: I did notice uh something PhD F: So th I think the mismatch is the more difficult for the training part .
PhD B: Somebody , I think it was Morgan , suggested at the last meeting that I actually count to see how many parameters and how many frames .
Professor A: Mm - hmm .
PhD F: Mm - hmm .
PhD B: And there are uh almost one point eight million frames of training data and less than forty thousand parameters in the baseline system .
Professor A: Hmm .
PhD F: Yeah .
PhD B: So it 's very , very few parameters compared to how much training data .
Professor A: Well . Yes .
PhD D: Mm - hmm .
Professor A: So . And that that says that we could have lots more parameters actually .
PhD B: Yeah . Yeah .
PhD F: Mm - hmm .
PhD B: I did one quick experiment just to make sure I had everything worked out and I just  uh f for most of the um For for all of the digit models , they end up at three mixtures per state . And so I just did a quick experiment , where I changed it so it went to four and um it it it didn't have a r any significant effect at the uh medium mismatch and high mismatch cases and it had  it was just barely significant for the well - matched better . Uh so I 'm r gonna run that again but um with many more uh mixtures per state .
Professor A: Yeah . Cuz at forty thou I mean you could you could have uh Yeah , easily four times as many parameters .
PhD B: Mm - hmm . And I think also just seeing what we saw uh in terms of the expected duration of the silence model ? when we did this tweaking of the self - loop ? The silence model expected duration was really different .
PhD F: Yeah .
PhD B: And so in the case where um it had a better score , the silence model expected duration was much longer .
PhD F: Yeah .
PhD B: So it was like  it was a better match . I think you know if we make a better silence model I think that will help a lot too um for a lot of these cases so but one one thing I I wanted to check out before I increased the um number of mixtures per state was uh in their default training script they do an initial set of three re - estimations and then they built the silence model and then they do seven iterations then the add mixtures and they do another seven then they add mixtures then they do a final set of seven and they quit . Seven seems like a lot to me and it also makes the experiments go take a really long time I mean to do one turn - around of the well matched case takes like a day .
Professor A: Mm - hmm . Mm - hmm .
PhD B: And so you know in trying to run these experiments I notice , you know , it 's difficult to find machines , you know , compute the run on . And so one of the things I did was I compiled HTK for the Linux machines
Professor A: Mm - hmm .
PhD B: cuz we have this one from IBM that 's got like five processors in it ?
Professor A: Right .
PhD B: and so now I 'm you can run stuff on that and that really helps a lot because now we 've got you know , extra machines that we can use for compute . And if I 'm do running an experiment right now where I 'm changing the number of iterations ? from seven to three ?
PhD D: Mm - hmm .
Professor A: Yeah .
PhD B: just to see how it affects the baseline system . And so if we can get away with just doing three , we can do many more experiments more quickly . And if it 's not a a huge difference from running with seven iterations , um , you know , we should be able to get a lot more experiments done .
PhD F: Hmm .
PhD B: And so . I 'll let you know what what happens with that . But if we can you know , run all of these back - ends f with many fewer iterations and on Linux boxes we should be able to get a lot more experimenting done .
Professor A: Mm - hmm .
PhD B: So . So I wanted to experiment with cutting down the number of iterations before I increased the number of Gaussians .
Professor A: Right . Sorry . So um , how 's it going on the PhD F: Um .
Professor A: So . You you did some things . They didn't improve things in a way that convinced you you 'd substantially improved anything .
PhD F: Yeah .
Professor A: But they 're not making things worse and we have reduced latency , right ?
PhD F: Yeah . But actually um actually it seems to do a little bit worse for the well - matched case and we just noticed that Yeah , actually the way the final score is computed is quite funny . It 's not a mean of word error rate . It 's not a weighted mean of word error rate , it 's a weighted mean of improvements .
Professor A: Uh - huh .
PhD F: So . Which means that actually the weight on the well - matched is Well I well what what What happened is that if you have a small improvement or a small if on the well - matched case it will have uh huge influence on the improvement compared to the reference because the reference system is is is quite good for for the well - ma well - matched case also .
PhD B: So it it weights the improvement on the well - matched case really heavily compared to the improvement on the other cases ?
PhD F: No , but it 's the weighting of the of the improvement not of the error rate .
PhD B: Yeah . Yeah , and it 's hard to improve on the on the best case , cuz it 's already so good , right ?
PhD F: Yeah but what I mean is that you can have a huge improvement on the H HMK 's , uh like five percent uh absolute , and this will not affect the final score almost Uh this will almost not affect the final score because this improvement because the improvement uh relative to the the baseline is small Professor A: So they do improvement in terms of uh accuracy ? rather than word error rate ?
PhD F: Uh . Uh improvement ?
Professor A: So PhD F: No , it 's compared to the word er it 's improvement on the word error rate ,
Professor A: OK .
PhD F: yeah . Sorry .
Professor A: So if you have uh ten percent error and you get five percent absolute uh improvement then that 's fifty percent .
PhD F: Mm - hmm .
Professor A: OK . So what you 're saying then is that if it 's something that has a small word error rate , then uh a even a relatively small improvement on it , in absolute terms , will show up as quite quite large in this .
PhD F: Mm - hmm .
Professor A: Is that what you 're saying ?
PhD F: Yeah .
Professor A: Yes .
PhD F: Yeah .
Professor A: OK . But yeah that 's that 's it 's the notion of relative improvement . Word error rate .
PhD F: Yeah . Sure , but when we think about the weighting , which is point five , point three , point two , it 's on absolute on on relative figures ,
Professor A: Yeah .
PhD F: not Professor A: Yeah .
PhD F: So when we look at this error rate
Professor A: No . That 's why I 've been saying we should be looking at word error rate uh and and not not at at accuracies .
PhD F: uh Mmm , yeah . Mmm , yeah .
Professor A: It 's PhD F: Mm - hmm .
Professor A: I mean uh we probably should have standardized on that all the way through . It 's just PhD B: Well .
PhD F: Mm - hmm .
PhD B: I mean , it 's not it 's not that different , right ? I mean , just subtract the accuracy .
Professor A: Yeah but you 're but when you look at the numbers , your sense of the relative size of things is quite different .
PhD B: I mean Oh . Oh , I see . Yeah .
Professor A: If you had ninety percent uh correct and five percent , five over ninety doesn't look like it 's a big difference , but five over ten is is big .
PhD B: Mm - hmm .
PhD F: Mm - hmm .
Professor A: So just when we were looking at a lot of numbers and getting sense of what was important .
PhD B: I see . I see . Yeah . That makes sense .
Professor A: Um .
PhD F: Mmm .
Professor A: Um .
PhD F: Well anyway uh . So . Yeah . So it hurts a little bit on the well - match and yeah .
Professor A: What 's a little bit ? Like PhD F: Like , it 's difficult to say because again um  I 'm not sure I have the um PhD B: Hey Morgan ? Do you remember that Signif program that we used to use for testing signi ? Is that still valid ? I I 've been using that .
Professor A: Yeah . Yeah , it was actually updated .
PhD B: OK .
Professor A: Uh . Jeff updated it some years ago
PhD B: Oh , it was . Oh , I shoul
Professor A: and and uh cleaned it up made some things better in it . So .
PhD B: OK . I should find that new one . I just use my old one from ninety - two or whatever
Professor A: Yeah , I 'm sure it 's not that different but but he  he uh he was a little more rigorous , as I recall .
PhD B: OK .
PhD F: Right . So it 's around , like , point five . No , point six {comment} uh percent absolute on Italian Professor A: Worse .
PhD F: Worse , yep .
Professor A: Out of what ? I mean . s
PhD F: Uh well we start from ninety - four point sixty - four , and we go to ninety - four point O four .
Professor A: Uh - huh . So that 's six six point th
PhD F: Uh .
PhD B: Ninety - three point six four , right ? is the baseline .
PhD F: Oh , no , I 've ninety - four . Oh , the baseline , you mean .
PhD B: Yeah .
PhD F: Well I don't I 'm not talking about the baseline here .
PhD B: Oh . Oh . I 'm sorry .
PhD F: I uh My baseline is the submitted system .
PhD B: Ah ! OK . Ah , ah .
PhD F: Hmm .
Professor A: Yeah .
PhD B: Sorry .
PhD F: Oh yeah . For Finnish , we start to ninety - three point eight - four and we go to ninety - three point seventy - four . And for Spanish we are we were at ninety - five point O five and we go to ninety - three - s point sixty one .
Professor A: OK , so we are getting hurt somewhat .
PhD F: So .
Professor A: And is that wh what do you know what piece you 've done several changes here . Uh , do you know what pie
PhD F: Yeah . I guess I guess it 's it 's the filter . Because nnn , well uh we don't have complete result , but the filter So the filter with the shorter delay hurts on Italian well - matched , which And , yeah . And the other things , like um downsampling , upsampling , don't seem to hurt and the new on - line normalization , neither .
PhD B: I 'm PhD F: So .
PhD B: I 'm really confused about something . If we saw that making a small change like , you know , a tenth , to the self - loop had a huge effect , can we really make any conclusions about differences in this stuff ?
PhD F: Mm - hmm . Yeah that 's th Yeah .
PhD B: I mean , especially when they 're this small . I mean .
PhD F: I think we can be completely fooled by this thing , but I don't know .
Professor A: Well , yeah .
PhD F: So . There is first this thing , and then the yeah , I computed the um  like , the confidence level on the different test sets . And for the well - matched they are around um point six uh percent . For the mismatched they are around like let 's say one point five percent . And for the well - m uh HM they are also around one point five .
Professor A: But OK , so you these these degradations you were talking about were on the well - matched case
PhD F: So .
Professor A: Uh . Do the does the new filter make things uh better or worse for the other cases ?
PhD F: Yeah . But . Uh . About the same . It doesn't hurt . Yeah .
Professor A: Doesn't hurt , but doesn't get a little better , or something .
PhD F: No .
Professor A: No . OK , so um I guess the argument one might make is that , " Yeah , if you looked at one of these cases and you jiggle something and it changes then uh you 're not quite sure what to make of it . But when you look across a bunch of these and there 's some some pattern , um I mean , so eh h here 's all the if if in all these different cases it never gets better , and there 's significant number of cases where it gets worse , then you 're probably hurting things , I would say . So um I mean at the very least that would be a reasonably prediction of what would happen with with a different test set , that you 're not jiggling things with . So I guess the question is if you can do better than this . If you can if we can approximate the old numbers while still keeping the latency down .
PhD F: Mmm . Yeah .
Professor A: Uh , so . Um . What I was asking , though , is uh are what 's what 's the level of communication with uh the O G I gang now , about this and PhD F: Well , we are exchanging mail as soon as we  we have significant results .
Professor A: Yeah .
PhD F: Um . Yeah . For the moment , they are working on integrating the um spectral subtraction apparently from Ericsson .
Professor A: Mm - hmm .
PhD F: Um . Yeah . And so . Yeah . We are working on our side on other things like uh also trying a sup spectral subtraction but of of our own , I mean , another spectral substraction .
Professor A: Mm - hmm .
PhD F: Um . Yeah . So I think it 's it 's OK . It 's going Professor A: Is there any further discussion about this this idea of of having some sort of source code control ?
PhD F: Yeah . Well . For the moment they 're uh everybody 's quite um There is this Eurospeech deadline , so .
Professor A: I see .
PhD F: Um . And . Yeah . But yeah . As soon as we have something that 's significant and that 's better than than what was submitted , we will fix fix the system and But we 've not discussed it it it this yet , yeah .
Professor A: Yeah . Sounds like a great idea but but I think that that um he 's saying people are sort of scrambling for a Eurospeech deadline .
PhD F: Mmm .
Professor A: But that 'll be uh , uh done in a week . So , maybe after this next one .
PhD F: Yeah .
PhD B: Wow ! Already a week ! Man !
Professor A: Yeah .
PhD B: You 're right . That 's amazing .