A: and other people think that it 's more important to improve a lot on the mismatch
A: and C: It sounds like they don't really have a good idea about what the final application is gonna be .
A: So , bu
A: l de fff !
A: Mmm .
B: Well , you know , the the thing is that if you look at the numbers on the on the more difficult cases , um , if you really believe that was gonna be the predominant use , none of this would be good enough .
A: Yeah .
A: Mmm .
C: Mm - hmm .
A: Yeah .
B: Nothing anybody 's B: whereas you sort of with some reasonable error recovery could imagine in the better cases that these these systems working .
B: So ,
B: um ,
B: I think the hope would be that it would  uh , it would work well for the good cases
B: and , uh , it would have reasonable reas soft degradation as you got to worse and worse conditions .
B: Um .
C: Yeah .
C: I I guess what I 'm C: I mean , I I was thinking about it in terms of , if I were building the final product
C: and I was gonna test to see which front - end I 'd  I wanted to use ,
C: I would try to weight things depending on the exact environment that I was gonna be using the system in .
C: If I B: But but No .
B: Well , no B: well , no .
B: I mean , it isn't the operating theater .
B: I mean , they don they they don't they don't really know , I think .
C: Yeah .
B: I mean , I th
C: So if if they don't know , doesn't that suggest the way for them to go ?
C: Uh ,
C: you assume everything 's equal .
C: I mean , y y I mean , you B: Well , I mean , I I think one thing to do is to just not rely on a single number B: to maybe have two or three numbers ,
C: Yeah .
C: Right .
B: you know ,
B: and and and say here 's how much you , uh you improve the , uh the the relatively clean case
B: and here 's B: or or well - matched case ,
B: and here 's how here 's how much you ,
C: Mm - hmm .
B: uh C: So not C: So not try to combine them .
B: So .
B: Yeah .
B: Uh , actually it 's true .
C: Yeah .
B: Uh , I had forgotten this ,
B: uh , but , uh , well - matched is not actually clean .
B: What it is is just that ,
C: The training and testing .
B: u uh , the training and testing are similar .
A: Mmm .
B: So ,
B: I guess what you would do in practice is you 'd try to get as many , uh , examples of similar sort of stuff as you could ,
C: Yeah .
B: and then ,
B: uh B: So the argument for that being the the the more important thing , is that you 're gonna try and do that , but you wanna see how badly it deviates from that when when when the , uh it 's a little different .
C: So B: Um ,
C: so you should weight those other conditions v very you know , really small .
B: But B: No .
B: That 's a that 's a that 's an arg
C: I mean , that 's more of an information kind of thing .
B: that 's an ar
B: Well , that 's an argument for it ,
B: but let me give you the opposite argument .
C: Uh - huh .
B: The opposite argument is you 're never really gonna have a good sample of all these different things .
B: I mean , are you gonna have w uh , uh , examples with the windows open ,
B: half open ,
B: full open ?
B: Going seventy , sixty , fifty , forty miles an hour ?
B: On what kind of roads ?
C: Mm - hmm .
B: With what passing you ?
B: With B: uh , I mean ,
C: Mm - hmm .
B: I I I think that you could make the opposite argument that the well - matched case is a fantasy .
C: Mm - hmm .
E: Uh - huh .
B: You know ,
B: so ,
B: I think the thing is is that if you look at the well - matched case versus the po you know , the the medium and the and the fo and then the mismatched case , um , we 're seeing really , really big differences in performance .
B: Right ?
B: And and y you wouldn't like that to be the case .
B: You wouldn't like that as soon as you step outside B: You know , a lot of the the cases it 's is C: Well , that 'll teach them to roll their window up .
B: I mean , in these cases , if you go from the the , uh B: I mean , I don't remember the numbers right off ,
B: but if you if you go from the well - matched case to the medium , it 's not an enormous difference in the in the the training - testing situation ,
B: and and and it 's a really big performance drop .
C: Mm - hmm .
B: You know ,
B: so ,
B: um B: Yeah , I mean the reference one , for instance this is back old on , uh on Italian uh , was like six percent error for the well - matched
B: and eighteen for the medium - matched
B: and sixty for the  for highly - mismatched .
B: Uh ,
B: and , you know , with these other systems we we helped it out quite a bit ,
B: but still there 's there 's something like a factor of two or something between well - matched and medium - matched .
B: And so I think that if what you 're  if the goal of this is to come up with robust features , it does mean B: So you could argue , in fact , that the well - matched is something you shouldn't be looking at at all ,
B: that that the goal is to come up with features that will still give you reasonable performance ,
B: you know , with again gentle degregra degradation ,
B: um , even though the the testing condition is not the same as the training .
C: Hmm .
B: So , you know , I I could argue strongly that something like the medium mismatch , which is you know not compl pathological but B: I mean , what was the the medium - mismatch condition again ?
A: Um , it 's A: Yeah .
A: Medium mismatch is everything with the far microphone ,
A: but trained on , like , low noisy condition ,
A: like low speed
A: and or stopped car
A: and tested on high - speed conditions , I think ,
A: like on a highway
A: and B: Right .
B: So it 's still the same same microphone in both cases ,
A: So A: Same microphone
A: but A: Yeah .
B: but , uh , it 's there 's a mismatch between the car conditions .
B: And that 's B: uh , you could argue that 's a pretty realistic situation
C: Yeah .
A: Mm - hmm .
B: and , uh , I 'd almost argue for weighting that highest .
B: But the way they have it now , it 's I guess it 's it 's B: They they compute the relative improvement first
B: and then average that with a weighting ?
A: Yeah .
B: And so then the that that makes the highly - matched the really big thing .
A: Mm - hmm .
B: Um ,
B: so , u i since they have these three categories , it seems like the reasonable thing to do is to go across the languages and to come up with an improvement for each of those .
A: Mm - hmm .
B: Just say " OK , in the in the highly - matched case this is what happens ,
B: in the  m the , uh this other m medium if this happens ,
B: in the highly - mismatched that happens " .
A: Mm - hmm .
B: And , uh ,
B: you should see , uh , a gentle degradation through that .
A: Mmm .
B: Um .
B: But B: I don't know .
A: Yeah .
B: I think that that B: I I B: I gather that in these meetings it 's it 's really tricky to make anything ac make any <comment> policy change
B: because  everybody has has , uh , their own opinion
B: and A: Mm - hmm .
B: I don't know .
A: Yeah .
B: Yeah .
A: Uh , so A: Yeah .
A: Yeah ,
A: but there is probably a a big change that will be made
A: is that the the baseline th they want to have a new baseline , perhaps ,
A: which is , um , MFCC
A: but with a voice activity detector .
A: And apparently , uh , some people are pushing to still keep this fifty percent number .
A: So they want to have at least fifty percent improvement on the baseline ,
B: Mm - hmm .
A: but w which would be a much better baseline .
B: Mm - hmm .
A: And if we look at the result that Sunil sent , just putting the VAD in the baseline improved , like , more than twenty percent ,
B: Mm - hmm .
A: which would mean then then mean that fifty percent on this new baseline is like , well , more than sixty percent improvement on B: So nobody would be there , probably . Right ?
A: on o e e uh A: Right now , nobody would be there ,
A: but B: Good .
A: Yeah .
B: Work to do .
A: Uh - huh .
B: So whose VAD
B: is Is is this a ?
A: Uh , they didn't decide yet .
A: I guess i this was one point of the conference call also ,
A: but A: mmm ,
A: so I don't know .
A: Um ,
A: but A: Yeah .
E: Oh .
B: Oh , I I think th that would be good .
B: I mean , it 's not that the design of the VAD isn't important ,
B: but it 's just that it it it does seem to be i uh , a lot of work to do a good job on on that
B: and as well as being a lot of work to do a good job on the feature design ,
A: Yeah .
B: so
A: Yeah .
B: if we can cut down on that maybe we can make some progress .
A: M
A: Yeah .
E: Hmm .
A: But I guess perhaps A: I don't know
A: w Yeah .
A: Uh , yeah .
A: Per - e s s someone told that perhaps it 's not fair to do that because the , um to make a good VAD you don't have enough to with the the features that are the baseline features .
A: So A: mmm ,
A: you need more features .
A: So you really need to put more more in the in in the front - end .
B: Yeah .
A: So i
B: Um ,
A: S
B: sure .
C: Wait a minute .
B: But i
B: bu
C: I I 'm confused .
A: Yeah .
C: Wha - what do you mean ?
A: Yeah ,
B: So y so you
A: if i
B: m s Yeah ,
B: but B: Well , let 's say for ins
B: see , MFCC for instance doesn't have anything in it , uh , related to the pitch .
B: So just just for example .
B: So suppose you 've that what you really wanna do is put a good pitch detector on there
B: and if it gets an unambiguous C: Oh ,
C: oh . I see .
A: Mm - hmm .
B: if it gets an unambiguous result then you 're definitely in a in a in a voice in a , uh , s region with speech .
C: So there 's this assumption that the v the voice activity detector can only use the MFCC ?
B: Uh .
A: That 's not clear ,
A: but this  e
B: Well , for the baseline .
C: Yeah .
B: So so if you use other features then y
B: But it 's just a question of what is your baseline .
B: Right ?
C: I g
B: What is it that you 're supposed to do better than ?
C: Yeah .
B: And so
C: I don't s
B: having the baseline be the MFCC 's means that people could choose to pour their ener their effort into trying to do a really good VAD
C: But they seem like two separate issues .
B: or tryi
C: Right ?
C: I mean B: They 're sort of separate .
B: Unfortunately there 's coupling between them ,
B: which is part of what I think Stephane is getting to , is that you can choose your features in such a way as to improve the VAD .
A: Yeah .
B: And you also can choose your features in such a way as to prove improve recognition .
C: But it seems like you should do both .
B: They may not be the same thing .
C: Right ?
B: You should do both
B: and and I I think that this still makes I still think this makes sense as a baseline .
B: It 's just saying , as a baseline , we know A: Mmm .
B: you know , we had the MFCC 's before ,
B: lots of people have done voice activity detectors ,
A: Mm - hmm .
B: you might as well pick some voice activity detector and make that the baseline ,
B: just like you picked some version of HTK and made that the baseline .
A: Yeah .
A: Right .
B: And then let 's try and make everything better .
B: Um ,
B: and if one of the ways you make it better is by having your features be better features for the VAD then that 's so be it .
B: But ,
A: Mm - hmm .
B: uh , uh , uh , at least you have a starting point that 's B: um ,
B: cuz i i some of the some of the people didn't have a VAD at all , I guess .
B: Right ?
B: And and
A: Yeah .
B: then they they looked pretty bad
B: and and in fact what they were doing wasn't so bad at all .
A: Mm - hmm .
B: But ,
A: Mm - hmm .
C: Yeah . It seems like you should try to make your baseline as good as possible .
B: um .
C: And if it turns out that you can't improve on that , well , I mean , then , you know , nobody wins and you just use MFCC .
C: Right ?
B: Yeah .
B: I mean , it seems like ,
B: uh ,
B: it should include sort of the current state of the art that you want are trying to improve ,
B: and MFCC 's , you know , or PLP or something it seems like reasonable baseline for the features ,
B: and anybody doing this task , uh , is gonna have some sort of voice activity detection at some level , in some way .
B: They might use the whole recognizer to do it but rather than a separate thing ,
B: but  but they 'll have it on some level .
B: So ,
B: um .
C: It seems like whatever they choose they shouldn't , you know , purposefully brain - damage a part of the system to make a worse baseline ,
C: or C: You know ?
B: Well , I think people just had
B: it wasn't that they purposely brain - damaged it .
B: I think people hadn't really thought through about the , uh the VAD issue .
C: Mmm .
A: Mm - hmm .
B: And and then when the the the proposals actually came in and half of them had V A Ds and half of them didn't ,
B: and the half that did did well
B: and the half that didn't did poorly .
C: Mm - hmm .
B: So it 's A: Mm - hmm .
A: Um .
B: Uh .
A: Yeah .
A: So
A: we 'll see what happen with this .
A: And A: Yeah .
A: So what happened since , um , last week is A: well , from OGI , these experiments on putting VAD on the baseline .
A: And these experiments also are using , uh , some kind of noise compensation ,
A: so spectral subtraction ,
A: and putting on - line normalization , um , just after this .
A: So I think spectral subtraction , LDA filtering , and on - line normalization ,
A: so which is similar to the pro proposal - one , but with spectral subtraction in addition ,
A: and it seems that on - line normalization doesn't help further when you have spectral subtraction .
C: Is this related to the issue that you brought up a couple of meetings ago with the the musical tones
A: I C: and ?
A: I have no idea ,
A: because the issue I brought up was with a very simple spectral subtraction approach ,
C: Mmm .
A: and the one that they use at OGI is one from from the proposed the the the Aurora prop uh , proposals ,
A: which might be much better .
A: So , yeah .
A: I asked Sunil for more information about that ,
A: but , uh ,
A: I don't know yet .
A: Um .
A: And what 's happened here is that we A: so we have this kind of new , um , reference system which use a nice a a clean downsampling - upsampling ,
A: which use a new filter that 's much shorter
A: and which also cuts the frequency below sixty - four hertz ,
B: Right .
A: which was not done on our first proposal .
B: When you say " we have that " , does Sunil have it now , too ,
A: I
B: or ?
A: No .
A: No .
B: OK .
A: Because we 're still testing .
A: So we have the result for , uh , just the features
B: OK .
A: and we are currently testing with putting the neural network in the KLT .
A: Um , it seems to improve on the well - matched case ,
A: um , but it 's a little bit worse on the mismatch and highly - mismatched A: I mean when we put the neural network .
A: And with the current weighting I think it 's sh it will be better
A: because the well - matched case is better .
A: Mmm .
B: But how much worse since the weighting might change B: how how much worse is it on the other conditions ,
B: when you say it 's a little worse ?
A: It 's like , uh , fff , fff <comment>  um , <comment>   ten percent relative .
A: Yeah .
B: OK .
B: Um .
A: Mm - hmm .
B: But it has the , uh B: the latencies are much shorter .
B: That 's A: Uh - y w when I say it 's worse , it 's not it 's when I I uh , compare proposal - two to proposal - one ,
A: so ,
A: r
A: uh , y putting neural network compared to n not having any neural network .
B: Uh - huh .
A: I mean , this new system is is is better ,
A: because it has um , this sixty - four hertz cut - off ,
A: uh , clean downsampling ,
A: and ,
A: um what else ?
A: Uh , yeah , a good VAD .
A: We put the good VAD .
A: So .
A: Yeah , I don't know . I I j uh , uh pr
B: But the latencies B: but you 've got the latency shorter now .
A: Latency is short F: Isn't it
A: is A: Yeah .
B: Yeah .
A: And so
B: So it 's better than the system that we had before .
A: Yeah .
A: Mainly because  of the sixty - four hertz and the good VAD .
B: OK .
A: And then I took this system and , mmm , w uh , I p we put the old filters also .
A: So we have this good system , with good VAD ,
A: with the short filter and with the long filter ,
A: and ,
A: um ,
A: with the short filter it 's not worse .
A: So A: well , is it B: OK .
B: So that 's that 's all fine .
A: it 's in B: But what you 're saying is that when you do these A: Yes . Uh B: So let me try to understand .
B: When when you do these same improvements to proposal - one ,
A: Mm - hmm .
B: that , uh , on the i
B: things are somewhat better , uh , in proposal - two for the well - matched case
B: and somewhat worse for the other two cases .
A: Yeah .
B: So does , uh B: when you say , uh B: So B: The th now that these other things are in there , is it the case maybe that the additions of proposal - two over proposal - one are less im important ?
A: Yeah .
A: Probably , yeah .