A: But it 's not wer worse
F: It 's not A: and it 's better better latency ,
A: right ?
F: Yeah .
F: Yeah .
F: Well . Actually it 's better .
F: It seems better when we look at the mismatched case
F: but I think we are like like cheated here by the th this problem
F: that uh in some cases when you modify slight slightly modify the initial condition you end up completely somewhere air somewhere else in the in the space , the parameters .
A: Yeah .
F: So .
F: Well . The other system are for instance .
F: For Italian is at seventy - eight percent recognition rate on the mismatch ,
F: and this new system has eighty - nine .
F: But I don't think it indicates something , really .
F: I don't I don't think it means that the new system is more robust
A: Uh - huh .
F: or F: It 's simply the fact that F: Well .
A: Well , the test would be if you then tried it on one of the other test sets , if if it was F: Y
A: Right . So this was Italian ,
A: right ?
F: Yeah . Yeah .
A: So then if you take your changes
F: It 's similar for other test sets
A: and then F: but I mean from this se seventy - eight um percent recognition rate system , I could change the transition probabilities for the the first HMM and it will end up to eighty - nine also .
A: Uh - huh .
F: By using point five instead of point six , point four as in the the HTK script .
A: Uh - huh .
A: Yeah .
F: So . Well . That 's B: Yeah .
B: Yeah I looked at um  looked at the results when Stephane did that
F: Well .
F: Eh uh B: and it 's it 's really wo really happens .
F: This really happens .
B: I mean th the only difference is you change the self - loop transition probability by a tenth of a percent
F: Yeah .
A: Yeah .
B: and it causes ten percent difference in the word error rate .
A: A tenth of a per cent .
F: Even tenth of a percent ?
B: Yeah .
B: From point F: Well , we tried we tried point one ,
B: I I 'm sorry
B: f for point from You change at point one
F: yeah .
F: Hmm .
A: Oh !
B: and n not tenth of a percent , one tenth ,
B: alright ?
A: Yeah .
B: Um so from point five B: so from point six to point five and you get ten percent better .
A: Mm - hmm .
F: Mm - hmm .
B: And it 's  I think it 's what you basically hypothesized in the last meeting about uh it just being very B: and I think you mentioned this in your email too F: Mmm , yeah .
B: it 's just very um B: you know get stuck in some local minimum
B: and this thing throws you out of it I guess .
F: Mm - hmm .
A: Well , what 's what are according to the rules what what are we supposed to do about the transition probabilities ?
A: Are they supposed to be point five or point six ?
B: I think you 're not allowed to B: Yeah .
B: That 's supposed to be point six , for the self - loop .
F: Yeah .
A: Point It 's supposed to be point six .
B: Yeah .
B: But changing it to point five I think is B: which gives you much better results ,
B: but that 's not allowed .
A: But not allowed ?
A: Yeah . OK .
B: Yeah .
F: Yeah , but even if you use point five , I 'm not sure it will always give you the better results
F: on other test set or it
B: Yeah .
B: Right . We only tested it on the the medium mismatch ,
F: on the other training set , I mean .
B: right ?
F: Yeah .
B: You said on the other cases you didn't notice F: But . I think ,
F: yeah .
F: I think the reason is ,
F: yeah , I not I F: it was in my mail I think also , is the fact that the mismatch is trained only on the far microphone .
F: Well , in for the mismatched case everything is um using the far microphone training and testing ,
F: whereas for the highly mismatched , training is done on the close microphone
F: so it 's it 's clean speech basically
F: so you don't have this problem of local minima probably
F: and for the well - match , it 's a mix of close microphone and distant microphone
F: and F: Well .
B: I did notice uh something F: So th I think the mismatch is the more difficult for the training part .
B: Somebody , I think it was Morgan , suggested at the last meeting that I actually count to see how many parameters and how many frames .
F: Mm - hmm .
A: Mm - hmm .
B: And there are uh almost one point eight million frames of training data
B: and less than forty thousand parameters in the baseline system .
A: Hmm .
F: Yeah .
B: So it 's very , very few parameters compared to how much training data .
D: Mm - hmm .
A: Well . Yes .
A: So .
A: And that that says that we could have lots more parameters actually .
B: Yeah .
F: Mm - hmm .
B: Yeah .
B: I did one quick experiment just to make sure I had everything worked out
B: and I just  uh f for most of the um B: For for all of the digit models , they end up at three mixtures per state .
B: And so I just did a quick experiment , where I changed it so it went to four
B: and um it it it didn't have a r any significant effect at the uh medium mismatch and high mismatch cases
B: and it had  it was just barely significant for the well - matched
B: better .
B: Uh so I 'm r gonna run that again but um with many more uh mixtures per state .
A: Yeah . Cuz at forty thou I mean you could you could have
A: uh A: Yeah , easily four times as many parameters .
B: Mm - hmm .
B: And I think also just seeing what we saw uh in terms of the expected duration of the silence model ?
B: when we did this tweaking of the self - loop ?
F: Yeah .
B: The silence model expected duration was really different .
B: And so in the case where um it had a better score , the silence model expected duration was much longer .
F: Yeah .
B: So it was like  it was a better match .
B: I think you know if we make a better silence model I think that will help a lot too
B: um for a lot of these cases
B: so
B: but one one thing I I wanted to check out before I increased the um number of mixtures per state was uh in their default training script they do an initial set of three re - estimations
B: and then they built the silence model
B: and then they do seven iterations then the add mixtures
B: and they do another seven then they add mixtures
B: then they do a final set of seven and they quit .
B: Seven seems like a lot to me
B: and it also makes the experiments go take a really long time
B: I mean to do one turn - around of the well matched case takes like a day .
A: Mm - hmm .
A: Mm - hmm .
B: And so you know in trying to run these experiments I notice , you know , it 's difficult to find machines , you know , compute the run on .
B: And so one of the things I did was I compiled HTK for the Linux machines
A: Mm - hmm .
B: cuz we have this one from IBM that 's got like five processors in it ?
A: Right .
B: and so now I 'm you can run stuff on that and that really helps a lot because now we 've got you know , extra machines that we can use for compute .
D: Mm - hmm .
B: And if I 'm do running an experiment right now where I 'm changing the number of iterations ? from seven to three ?
A: Yeah .
B: just to see how it affects the baseline system .
B: And so if we can get away with just doing three , we can do many more experiments more quickly .
B: And if it 's not a a huge difference from running with seven iterations , um , you know , we should be able to get a lot more experiments done .
F: Hmm .
B: And so . I 'll let you know what what happens with that .
B: But if we can you know , run all of these back - ends f with many fewer iterations and on Linux boxes we should be able to get a lot more experimenting done .
A: Mm - hmm .
B: So .
B: So I wanted to experiment with cutting down the number of iterations before I increased the number of Gaussians .
A: Right .
A: Sorry .
A: So um , how 's it going on the F: Um .
A: So . You you did some things .
A: They didn't improve things in a way that convinced you you 'd substantially improved anything .
F: Yeah .
A: But they 're not making things worse and we have reduced latency ,
A: right ?
F: Yeah .
F: But actually um actually it seems to do a little bit worse for the well - matched case
F: and we just noticed that F: Yeah , actually the way the final score is computed is quite funny .
F: It 's not a mean of word error rate .
F: It 's not a weighted mean of word error rate ,
F: it 's a weighted mean of improvements .
A: Uh - huh .
F: So .
F: Which means that actually the weight on the well - matched is F: Well I well what what What happened is that if you have a small improvement or a small if on the well - matched case it will have uh huge influence on the improvement compared to the reference
F: because the reference system is is is quite good for for the well - ma well - matched case also .
B: So it it weights the improvement on the well - matched case really heavily compared to the improvement on the other cases ?
F: No , but it 's the weighting of the of the improvement
F: not of the error rate .
B: Yeah .
B: Yeah , and it 's hard to improve on the on the best case ,
B: cuz it 's already so good ,
B: right ?
F: Yeah but what I mean is that you can have a huge improvement on the H HMK 's ,
F: uh like five percent uh absolute ,
F: and this will not affect the final score almost F: Uh this will almost not affect the final score because this improvement because the improvement uh relative to the the baseline is small F: Uh .
A: So they do improvement in terms of uh accuracy ? rather than word error rate ?
F: Uh improvement ?
F: No , it 's compared to the word er it 's improvement on the word error rate ,
A: So F: yeah .
F: Sorry .
A: OK .
A: So if you have uh ten percent error and you get five percent absolute uh improvement then that 's fifty percent .
F: Mm - hmm .
A: OK . So what you 're saying then is that if it 's something that has a small word error rate , then uh a even a relatively small improvement on it , in absolute terms , will show up as quite quite large in this .
F: Mm - hmm .
A: Is that what you 're saying ?
F: Yeah .
A: Yes .
F: Yeah .
A: OK .
A: But yeah that 's that 's it 's the notion of relative improvement .
F: Yeah .
A: Word error rate .
F: Sure , but when we think about the weighting , which is point five , point three , point two , it 's on absolute on on relative figures ,
F: not A: Yeah .
F: So when we look at this error rate
A: Yeah .
A: No .
F: uh A: That 's why I 've been saying we should be looking at word error rate uh and and not not at at accuracies .
F: Mmm , yeah .
F: Mmm , yeah .
A: It 's F: Mm - hmm .
A: I mean uh we probably should have standardized on that all the way through .
A: It 's just B: Well .
F: Mm - hmm .
B: I mean , it 's not it 's not that different ,
B: right ?
B: I mean , just subtract the accuracy .
A: Yeah
B: I mean A: but you 're but when you look at the numbers , your sense of the relative size of things is quite different .
B: Oh . Oh , I see .
B: Yeah .
A: If you had ninety percent uh correct and five percent ,
A: five over ninety doesn't look like it 's a big difference ,
A: but five over ten is is big .
B: Mm - hmm .
F: Mm - hmm .
A: So just when we were looking at a lot of numbers and getting sense of what was important .
B: I see . I see .
B: Yeah .
B: That makes sense .
A: Um .
F: Mmm .
A: Um .
F: Well anyway
F: uh .
F: So .
F: Yeah . So it hurts a little bit on the well - match
F: and
F: yeah .
A: What 's a little bit ?
A: Like F: Like , it 's difficult to say
F: because again
F: um  I 'm not sure I have the
F: um B: Hey Morgan ?
B: Do you remember that Signif program that we used to use for testing signi ?
B: Is that still valid ?
B: I I 've been using that .
A: Yeah .
A: Yeah , it was actually updated .
B: OK .
A: Uh . Jeff updated it some years ago
B: Oh , it was .
B: Oh , I shoul
A: and and uh cleaned it up made some things better in it .
B: OK .
A: So .
B: I should find that new one .
B: I just use my old one from ninety - two or whatever
A: Yeah , I 'm sure it 's not that different
A: but but he  he uh he was a little more rigorous , as I recall .
B: OK .
F: Right .
F: So it 's around , like , point five .
F: No , point six <comment> uh percent absolute on Italian A: Worse .
F: Worse , yep .
A: Out of what ?
A: I mean . s
F: Uh well we start from ninety - four point sixty - four ,
F: and we go to ninety - four point O four .
A: Uh - huh .
A: So that 's six six point th
F: Uh .
B: Ninety - three point six four ,
B: right ?
B: is the baseline .
F: Oh , no , I 've ninety - four .
F: Oh , the baseline , you mean .
B: Yeah .
F: Well I don't I 'm not talking about the baseline here .
B: Oh . Oh .
F: I
B: I 'm sorry .
F: uh F: My baseline is the submitted system .
B: Ah ! OK . Ah , ah .
F: Hmm .
B: Sorry .
A: Yeah .
F: Oh yeah .
F: For Finnish , we start to ninety - three point eight - four
F: and we go to ninety - three point seventy - four .
F: And for Spanish we are we were at ninety - five point O five
F: and we go to ninety - three - s point sixty one .
A: OK , so we are getting hurt somewhat .
F: So .
A: And is that
A: wh what A: do you know what piece A: you 've done several changes here .
A: Uh ,
A: do you know what pie
F: Yeah .
F: I guess I guess it 's it 's the filter .
F: Because nnn , well uh we don't have complete result ,
F: but the filter So the filter with the shorter delay hurts on Italian well - matched ,
F: which F: And ,
F: yeah .
F: And the other things , like um downsampling , upsampling , don't seem to hurt
F: and the new on - line normalization , neither .
B: I 'm F: So .
B: I 'm really confused about something .
B: If we saw that making a small change like , you know , a tenth , to the self - loop had a huge effect , can we really make any conclusions about differences in this stuff ?
F: Mm - hmm .
F: Yeah that 's th
F: Yeah .
B: I mean , especially when they 're this small .
B: I mean .
F: I think we can be completely fooled by this thing ,
F: but F: I don't know .
A: Well , yeah .
F: So . There is first this thing ,
F: and then the F: yeah , I computed the um  like , the confidence level on the different test sets .
F: And for the well - matched they are around um point six uh percent .
F: For the mismatched they are around like let 's say one point five percent .
F: And for the well - m uh HM they are also around one point five .
A: But OK , so you these these degradations you were talking about were on the well - matched case
F: So .
F: Yeah .
A: Uh . Do the does the new filter make things uh better or worse for the other cases ?
F: But .
F: Uh . About the same .
F: It doesn't hurt .
F: Yeah .
A: Doesn't hurt , but doesn't get a little better , or something .
F: No .
A: No .
A: OK , so um I guess the argument one might make is that , " Yeah , if you looked at one of these cases and you jiggle something and it changes then uh you 're not quite sure what to make of it .
A: But when you look across a bunch of these and there 's some some pattern ,
A: um A: I mean , so eh h here 's all the if if in all these different cases it never gets better , and there 's significant number of cases where it gets worse , then you 're probably hurting things , I would say .
A: So um I mean at the very least that would be a reasonably prediction of what would happen with with a different test set , that you 're not jiggling things with .
A: So I guess the question is if you can do better than this .
A: If you can if we can approximate the old numbers while still keeping the latency down .
F: Mmm .
F: Yeah .
A: Uh , so .
A: Um . What I was asking , though , is uh are what 's what 's the level of communication with uh the O G I gang now , about this
A: and F: Well , we are exchanging mail as soon as we  we have significant results .
A: Yeah .
F: Um .
F: Yeah .
F: For the moment , they are working on integrating the um spectral subtraction apparently from Ericsson .
A: Mm - hmm .
F: Um .
F: Yeah .
F: And so .
F: Yeah .
F: We are working on our side on other things like uh also trying a sup spectral subtraction
F: but of of our own , I mean , another spectral substraction .
A: Mm - hmm .
F: Um .
F: Yeah .
F: So I think it 's it 's OK .
F: It 's going A: Is there any further discussion about this this idea of of having some sort of source code control ?
F: Yeah .
F: Well . For the moment they 're F: uh everybody 's quite
F: um F: There is this Eurospeech deadline ,
F: so .
A: I see .
F: Um . And .
F: Yeah .
F: But yeah .
F: As soon as we have something that 's significant and that 's better than than what was submitted ,
F: we will fix fix the system
F: and F: But we 've not discussed it it it this yet ,
F: yeah .
A: Yeah .
A: Sounds like a great idea
A: but but I think that that um he 's saying people are sort of scrambling for a Eurospeech deadline .
F: Mmm .
A: But that 'll be uh , uh done in a week .
A: So , maybe after this next one .
F: Yeah .
B: Wow !
B: Already a week !
B: Man !
A: Yeah .
B: You 're right .
B: That 's amazing .
A: Yeah .
A: Anybo - anybody in the in this group do doing anything for Eurospeech ?
F: S
A: Or , is that what is that F: Yeah we are  We are trying to to do something with the Meeting Recorder digits ,
A: Right .
F: and F: But yeah .
F: Yeah .
F: And the good thing is that there is this first deadline ,
A: Yeah .
F: and , well , some people from OGI are working on a paper for this ,
F: but there is also the um special session about th Aurora
F: which is  uh which has an extended deadline .
F: So .
F: The deadline is in May .
A: For uh  Oh , for Eurospeech ?
F: For th
F: Yeah .
F: So f only for the experiments on Aurora .
A: Oh !
F: So it it 's good ,
F: yeah .
A: Oh , a special dispensation .
A: That 's great .
B: Mm - hmm .
B: Where is Eurospeech this year ?
F: It 's in Denmark .
A: Aalborg A: Aalborg
A: uh
B: Oh .
A: So the deadline A: When 's the deadline ?
F: Hmm ?
A: When 's the deadline ?
F: I think it 's the thirteenth of May .
A: That 's great !
A: It 's great .
A: So we should definitely get something in for that .
F: Yeah .
A: But on meeting digits , maybe there 's A: Maybe .
F: Yeah .
F: So it would be for the first deadline .
A: Maybe .
A: Yeah .
F: Nnn .
A: Yeah . So , I mean , I I think that you could certainly start looking at at the issue