Professor D: OK . So , uh You can fill those out , uh after , actually , so So , I got , uh these results from , uh , Stephane . Also , um , I think that , uh um we might hear later today , about other results . I think s that , uh , there were some other very good results that we 're gonna wanna compare to . But , r our results from other other places , yeah .
PhD A: I I 'm sorry ? I didn't
Professor D: Um , I got this from you
PhD A: Yeah .
Professor D: and then I sent a note to Sunil about the cuz he has been running some other systems
PhD A: Mm - hmm .
Professor D: other than the the ICSI OGI one .
PhD A: Oh yeah .
Professor D: So um , I wan wanna wanna see what that is . But , uh , you know , so we 'll see what it is comparatively later . But it looks like , um
PhD A: M yeah .
Professor D: You know most of the time , even I mean even though it 's true that the overall number for Danish we didn't improve it If you look at it individually , what it really says is that there 's , um , uh Looks like out of the six cases , between the different kinds of , uh , matching conditions out of the six cases , there 's basically , um , a couple where it stays about the same , uh , three where it gets better , and one where it gets worse .
PhD A: Yeah .
Professor D: Uh , go ahead .
PhD A: Y Actually , uh , um , for the Danish , there 's still some kind of mystery because , um , um , when we use the straight features , we are not able to get these nice number with the ICSI OGI one , I mean . We don't have this ninety - three seventy - eight , we have eight
PhD E: Eighty - nine forty - four .
PhD A: yeah . Uh , so , uh , that 's probably something wrong with the features that we get from OGI . Uh , and Sunil is working on on trying to to check everything .
Professor D: Oh , and and we have a little time on that and actually so
PhD A: Hmm ?
Professor D: We have a little bit of time on that , actually .
PhD A: Yeah .
Professor D: We have a day or so , so When when when do you folks leave ?
PhD A: Uh , Sunday .
Professor D: Sunday ? So So , uh Yeah , until Saturday midnight , or something , we have W we we have time , yeah . Well , that would be good . That 'd be good .
PhD A: Yeah .
Professor D: Yeah . Uh , and , you know , i u when whenever anybody figures it out they should also , for sure , email Hynek because Hynek will be over there telling people what we did , so he should know .
PhD A: Mmm . Yeah .
Professor D: Good , OK . So , um So , we 'll we 'll hold off on that a little bit . I mean , even with these results as they are , it 's it 's it 's really not that bad . But but , uh , um And it looks like the overall result as they are now , even without , you know , any any bugs being fixed is that , uh , on the the other tasks , we had this average of , uh , forty uh nine percent , or so , improvement . And here we have somewhat better than that than the Danish , and somewhat worse than that on the German , but I mean , it sounds like , uh , one way or another , the methods that we 're doing can reduce the error rate from from mel ceptrum down by , you know a fourth of them to , uh , a half of them . Somewhere in there , depending on the exact case . So So that 's good . I mean , I think that , uh , one of the things that Hynek was talking about was understanding what was in the other really good proposals and and trying to see if what should ultimately be proposed is some , uh , combination of things . Um , if , uh Cuz there 's things that they are doing there that we certainly are not doing . And there 's things that we 're doing that they 're not doing . And and they all seem like good things .
PhD A: Yeah .
Professor D: So
PhD E: Mmm , yeah .
PhD C: How much how much better was the best system than ours ?
Professor D: So Well , we don't know yet .
PhD C: Mmm .
Professor D: Uh , I mean , first place , there 's still this thing to to work out , and second place second thing is that the only results that we have so far from before were really development set results .
PhD C: Oh , OK .
Professor D: So , I think in this community that 's of interest . It 's not like everything is being pinned on the evaluation set . But , um , for the development set , our best result was a little bit short of fifty percent . And the best result of any system was about fifty - four , where these numbers are the , uh , relative , uh , reduction in , uh , word error rate .
PhD C: Oh , OK .
Professor D: And , um , the other systems were , uh , somewhat lower than that . There was actually there was much less of a huge range than there was in Aurora one . In Aurora one there were there were systems that ba basically didn't improve things .
PhD C: Hmm .
Professor D: And here the the worst system still reduced the error rate by thirty - three percent , or something , in development set .
PhD C: Oh , wow .
Professor D: So so , you know , sort of everybody is doing things between , well , roughly a third of the errors , and half the errors being eliminated , uh , and varying on different test sets and so forth .
PhD C: Mm - hmm .
Professor D: So I think Um It 's probably a good time to look at what 's really going on and seeing if there 's a there 's a way to combine the best ideas while at the same time not blowing up the amount of , uh , resources used , cuz that 's that 's critical for this this test .
PhD C: Do we know anything about who who 's was it that had the lowest on the dev set ?
Professor D: Um , uh , the , uh , the there were two systems that were put forth by a combination of of , uh , French Telecom and Alcatel . And , um they they differed in some respects , but they e em one was called the French Telecom Alcatel System the other was called the Alcatel French Telecom System , uh , which is the biggest difference , I think . But but there 're there 're there 're some other differences , too . Uh , and and , uh , they both did very well ,
PhD C: Uh - huh .
Professor D: you know ? So , um , my impression is they also did very well on on the the , uh , evaluation set , but , um , I I we haven't seen you 've - you haven't seen any final results for that
PhD C: And they used the main thing that that they used was spectral subtraction ?
Professor D: yeah .
PhD C: Or
Professor D: There is a couple pieces to it . There 's a spectral subtraction style piece it was basically , you know , Wiener filtering . And then then there was some p some modification of the cepstral parameters , where they PhD A: Yeah , actually , something that 's close to cepstral mean subtraction . But , uh , the way the mean is adapted um , it 's signal dependent . I 'm I 'm , uh So , basically , the mean is adapted during speech and not during silence .
Professor D: Yeah .
PhD A: But it 's very close to to cepstral mean subtraction .
Professor D: But some people have done  exactly that sort of thing , of of and the I mean it 's not To to look in speech only , to try to m to measure these things during speech ,
PhD A: Yeah , yeah .
Professor D: that 's p that 's not that uncommon . But i it it so it looks like they did some some , uh , reasonable things , uh , and they 're not things that we did , precisely . We did unreasonable things , which because we like to try strange things , and and , uh , and our things worked too .
PhD C: Hmm .
Professor D: And so , um , uh , it 's possible that some combination of these different things that were done would be the best thing to do . But the only caveat to that is that everybody 's being real conscious of how much memory and how much CPU they 're using
PhD C: Mm - hmm .
Professor D: because these ,   uh , standards are supposed to go on cell phones with m moderate resources in both respects .
PhD C: Did anybody , uh , do anything with the models as a an experiment ? Or
Professor D: Uh , they didn't report it , if they did .
PhD C: N nobody reported it ?
Professor D: Yeah . I think everybody was focused elsewhere . Um , now , one of the things that 's nice about what we did is , we do have a a , uh a filtering , which leads to a a , uh a reduction in the bandwidth in the modulation spectrum , which allows us to downsample . So , uh , as a result of that we have a reduced , um , transmission rate for the bits .
PhD C: Mm - hmm .
Professor D: That was misreported the first time out . It it said the same amount because for convenience sake in the particular way that this is being tested , uh , they were repeating the packets . So it was they were s they they had twenty - four hundred bits per second , but they were literally creating forty - eight hundred bits per second , um , even though y it was just repeated .
PhD C: Oh . Mm - hmm . Right .
Professor D: So , uh , in practice
PhD C: So you could 've had a repeat count in there or something .
Professor D: Well , n I mean , this was just a ph phoney thing just to to fit into the the software that was testing the errors channel errors and so on .
PhD C: Oh . Oh .
Professor D: So so in reality , if you put this this system in into , uh , the field , it would be twenty - four hundred bits per second , not forty - eight hundred . So , um , so that 's a nice feature of what what we did . Um , but , um , well , we still have to see how it all comes out .
PhD C: Hmm .
Professor D: Um , and then there 's the whole standards process , which is another thing altogether .
PhD C: When is the development set I mean , the , uh , uh , test set results due ? Like the day before you leave or something ?

Summary: The deployed model generally decreased error rates, but the extent to which it did so varied based on the language as well as where it was deployed. They learned that similar projects experienced the same challenges. Still, the model was not efficient and likely used too much CPU and memory for the benefit it provided.