Professor B: Well , that 's what he 's seeing as one of the things they could do .
PhD A: Yeah .
Professor B: It 's just when you when you get all done , I think that they pro I m I I wasn't there but I think they started off this process with the notion that you should be significantly better than the previous standard .
PhD C: Mm - hmm .
Professor B: And , um , so they said " how much is significantly better ? what do you ? " And and so they said " well , you know , you should have half the errors , " or something , " that you had before " .
PhD A: Mm - hmm . Hmm .
PhD C: Mm - hmm .
PhD A: Yeah .
Professor B: So it 's , uh , But it does seem like
PhD C: Hmm .
Professor B: i i it does seem like it 's more logical to combine them first and then do the PhD A: Combine error rates and then Professor B: Yeah .
PhD A: Yeah . Well Professor B: Yeah .
PhD A: But there is this this is this still this problem of weights . When when you combine error rate it tends to give more importance to the difficult cases , and some people think that Professor B: Oh , yeah ?
PhD A: well , they have different , um , opinions about this . Some people think that it 's more important to look at  to have ten percent imp relative improvement on well - matched case than to have fifty percent on the m mismatched , and other people think that it 's more important to improve a lot on the mismatch and So , bu
PhD C: It sounds like they don't really have a good idea about what the final application is gonna be .
PhD A: l de fff ! Mmm .
Professor B: Well , you know , the the thing is that if you look at the numbers on the on the more difficult cases , um , if you really believe that was gonna be the predominant use , none of this would be good enough .
PhD A: Yeah . Mmm . Yeah .
Professor B: Nothing anybody 's PhD C: Mm - hmm .
Professor B: whereas you sort of with some reasonable error recovery could imagine in the better cases that these these systems working . So , um , I think the hope would be that it would  uh , it would work well for the good cases and , uh , it would have reasonable reas soft degradation as you got to worse and worse conditions . Um .
PhD C: Yeah . I I guess what I 'm I mean , I I was thinking about it in terms of , if I were building the final product and I was gonna test to see which front - end I 'd  I wanted to use , I would try to weight things depending on the exact environment that I was gonna be using the system in .
Professor B: But but No .
PhD C: If I Professor B: Well , no well , no . I mean , it isn't the operating theater . I mean , they don they they don't they don't really know , I think .
PhD C: Yeah .
Professor B: I mean , I th
PhD C: So if if they don't know , doesn't that suggest the way for them to go ? Uh , you assume everything 's equal . I mean , y y I mean , you Professor B: Well , I mean , I I think one thing to do is to just not rely on a single number to maybe have two or three numbers ,
PhD C: Yeah .
Professor B: you know ,
PhD C: Right .
Professor B: and and and say here 's how much you , uh you improve the , uh the the relatively clean case and here 's or or well - matched case , and here 's how here 's how much you ,
PhD C: Mm - hmm .
Professor B: uh PhD C: So not Professor B: So .
PhD C: So not try to combine them .
Professor B: Yeah . Uh , actually it 's true .
PhD C: Yeah .
Professor B: Uh , I had forgotten this , uh , but , uh , well - matched is not actually clean . What it is is just that , u uh , the training and testing are similar .
PhD C: The training and testing .
PhD A: Mmm .
Professor B: So , I guess what you would do in practice is you 'd try to get as many , uh , examples of similar sort of stuff as you could , and then ,
PhD C: Yeah .
Professor B: uh So the argument for that being the the the more important thing , is that you 're gonna try and do that , but you wanna see how badly it deviates from that when when when the , uh it 's a little different .
PhD C: So Professor B: Um ,
PhD C: so you should weight those other conditions v very you know , really small .
Professor B: But No . That 's a that 's a that 's an arg
PhD C: I mean , that 's more of an information kind of thing .
Professor B: that 's an ar Well , that 's an argument for it , but let me give you the opposite argument . The opposite argument is you 're never really gonna have a good sample of all these different things .
PhD C: Uh - huh .
Professor B: I mean , are you gonna have w uh , uh , examples with the windows open , half open , full open ? Going seventy , sixty , fifty , forty miles an hour ? On what kind of roads ?
PhD C: Mm - hmm .
Professor B: With what passing you ? With uh , I mean ,
PhD C: Mm - hmm .
Professor B: I I I think that you could make the opposite argument that the well - matched case is a fantasy .
PhD C: Mm - hmm .
Professor B: You know , so ,
Grad E: Uh - huh .
Professor B: I think the thing is is that if you look at the well - matched case versus the po you know , the the medium and the and the fo and then the mismatched case , um , we 're seeing really , really big differences in performance . Right ? And and y you wouldn't like that to be the case . You wouldn't like that as soon as you step outside You know , a lot of the the cases it 's is PhD C: Well , that 'll teach them to roll their window up .
Professor B: I mean , in these cases , if you go from the the , uh I mean , I don't remember the numbers right off , but if you if you go from the well - matched case to the medium , it 's not an enormous difference in the in the the training - testing situation , and and and it 's a really big performance drop .
PhD C: Mm - hmm .
Professor B: You know , so , um Yeah , I mean the reference one , for instance this is back old on , uh on Italian uh , was like six percent error for the well - matched and eighteen for the medium - matched and sixty for the  for highly - mismatched . Uh , and , you know , with these other systems we we helped it out quite a bit , but still there 's there 's something like a factor of two or something between well - matched and medium - matched . And so I think that if what you 're  if the goal of this is to come up with robust features , it does mean So you could argue , in fact , that the well - matched is something you shouldn't be looking at at all , that that the goal is to come up with features that will still give you reasonable performance , you know , with again gentle degregra degradation , um , even though the the testing condition is not the same as the training .
PhD C: Hmm .
Professor B: So , you know , I I could argue strongly that something like the medium mismatch , which is you know not compl pathological but I mean , what was the the medium - mismatch condition again ?
PhD A: Um , it 's Yeah . Medium mismatch is everything with the far microphone , but trained on , like , low noisy condition , like low speed and or stopped car and tested on high - speed conditions , I think , like on a highway and Professor B: Right .
PhD A: So Professor B: So it 's still the same same microphone in both cases ,
PhD A: Same microphone but Yeah .
Professor B: but , uh , it 's there 's a mismatch between the car conditions . And that 's uh , you could argue that 's a pretty realistic situation
PhD C: Yeah .
PhD A: Mm - hmm .
Professor B: and , uh , I 'd almost argue for weighting that highest . But the way they have it now , it 's I guess it 's it 's They they compute the relative improvement first and then average that with a weighting ?
PhD A: Yeah .
Professor B: And so then the that that makes the highly - matched the really big thing .
PhD A: Mm - hmm .
Professor B: Um , so , u i since they have these three categories , it seems like the reasonable thing to do is to go across the languages and to come up with an improvement for each of those .
PhD A: Mm - hmm .
Professor B: Just say " OK , in the in the highly - matched case this is what happens , in the  m the , uh this other m medium if this happens , in the highly - mismatched that happens " .
PhD A: Mm - hmm .
Professor B: And , uh , you should see , uh , a gentle degradation through that .
PhD A: Mmm .
Professor B: Um . But I don't know .
PhD A: Yeah .
Professor B: I think that that I I I gather that in these meetings it 's it 's really tricky to make anything ac make any {comment} policy change because  everybody has has , uh , their own opinion
PhD A: Mm - hmm .
Professor B: and I don't know .
PhD A: Yeah .
Professor B: Yeah .
PhD A: Uh , so Yeah . Yeah , but there is probably a a big change that will be made is that the the baseline th they want to have a new baseline , perhaps , which is , um , MFCC but with a voice activity detector . And apparently , uh , some people are pushing to still keep this fifty percent number . So they want to have at least fifty percent improvement on the baseline , but w which would be a much better baseline .
Professor B: Mm - hmm . Mm - hmm .
PhD A: And if we look at the result that Sunil sent , just putting the VAD in the baseline improved , like , more than twenty percent ,
Professor B: Mm - hmm .
PhD A: which would mean then then mean that fifty percent on this new baseline is like , well , more than sixty percent improvement on on o e e uh Professor B: So nobody would be there , probably . Right ?
PhD A: Right now , nobody would be there , but Yeah .
Professor B: Good . Work to do .
PhD A: Uh - huh .