D: OK .
D: So , uh
D: You can fill those out , uh after , actually ,
D: so
D: So , I got , uh these results from , uh , Stephane .
D: Also , um , I think that , uh um we might hear later today , about other results .
D: I think s that , uh , there were some other very good results that we 're gonna wanna compare to .
D: But , r our results from other other places ,
D: yeah .
A: I I 'm sorry ?
A: I didn't
D: Um , I got this from you
A: Yeah .
D: and then I sent a note to Sunil about the cuz he has been running some other systems
D: other than the the ICSI OGI one .
A: Mm - hmm .
A: Oh yeah .
D: So um , I wan wanna wanna see what that is .
D: But , uh , you know , so we 'll see what it is comparatively later .
D: But it looks like , um
A: M yeah .
D: You know most of the time , even I mean even though it 's true that the overall number for Danish we didn't improve it
D: If you look at it individually , what it really says is that there 's , um , uh
D: Looks like out of the six cases , between the different kinds of , uh , matching conditions out of the six cases , there 's basically , um , a couple where it stays about the same ,
D: uh , three where it gets better , and one where it gets worse .
A: Yeah .
D: Uh , go ahead .
A: Y Actually , uh , um , for the Danish , there 's still some kind of mystery
A: because , um , um , when we use the straight features , we are not able to get these nice number
A: with the ICSI OGI one , I mean .
A: We don't have this ninety - three seventy - eight , we have
A: eight
E: Eighty - nine forty - four .
A: yeah .
A: Uh , so , uh , that 's probably something wrong with the features that we get from OGI .
A: Uh , and Sunil is working on on trying to to check everything .
D: Oh ,
D: and and we have a little time on that and actually
D: so
A: Hmm ?
D: We have a little bit of time on that , actually .
A: Yeah .
D: We have a day or so ,
D: so
D: When when when do you folks leave ?
A: Uh , Sunday .
D: Sunday ?
D: So
D: So , uh
D: Yeah ,
D: until Saturday midnight , or something , we have
D: W we we have time ,
D: yeah .
D: Well , that would be good .
D: That 'd be good .
A: Yeah .
D: Yeah .
D: Uh , and , you know , i u when whenever anybody figures it out they should also , for sure , email Hynek
D: because Hynek will be over there telling people what we did ,
D: so he should know .
A: Mmm .
D: Good ,
A: Yeah .
D: OK .
D: So , um
D: So , we 'll we 'll hold off on that a little bit .
D: I mean , even with these results as they are , it 's it 's it 's really not that bad .
D: But but , uh , um
D: And it looks like the overall result as they are now , even without , you know , any any bugs being fixed is that , uh , on the the other tasks , we had this average of , uh , forty uh nine percent , or so , improvement .
D: And here we have somewhat better than that than the Danish , and somewhat worse than that on the German ,
D: but I mean , it sounds like , uh , one way or another , the methods that we 're doing can reduce the error rate from from mel ceptrum down by , you know a fourth of them to , uh , a half of them .
D: Somewhere in there , depending on the exact case .
D: So
D: So that 's good .
D: I mean , I think that , uh , one of the things that Hynek was talking about was understanding what was in the other really good proposals
D: and D: and trying to see if what should ultimately be proposed is some , uh , combination of things .
D: Um , if , uh D: Cuz there 's things that they are doing there that we certainly are not doing .
D: And there 's things that we 're doing that they 're not doing .
D: And and they all seem like good things .
A: Yeah .
E: Mmm , yeah .
D: So
D: So
C: How much how much better was the best system than ours ?
D: Well , we don't know yet .
C: Mmm .
D: Uh , I mean , first place , there 's still this thing to to work out ,
D: and second place second thing is that the only results that we have so far from before were really development set results .
C: Oh , OK .
D: So , I think in this community that 's of interest .
D: It 's not like everything is being pinned on the evaluation set .
D: But , um , for the development set , our best result was a little bit short of fifty percent .
D: And the best result of any system was about fifty - four ,
D: where these numbers are the , uh , relative , uh , reduction in , uh , word error rate .
C: Oh ,
C: OK .
D: And , um , the other systems were , uh , somewhat lower than that .
D: There was actually there was much less of a huge range than there was in Aurora one .
D: In Aurora one there were there were systems that ba basically didn't improve things .
C: Hmm .
D: And here the the worst system still reduced the error rate by thirty - three percent , or something , in development set .
C: Oh , wow .
D: So so , you know , sort of everybody is doing things between , well , roughly a third of the errors , and half the errors being eliminated , uh , and varying on different test sets and so forth .
C: Mm - hmm .
D: So I think
D: Um It 's probably a good time to look at what 's really going on and seeing if there 's a there 's a way to combine the best ideas
D: while at the same time not blowing up the amount of , uh , resources used ,
D: cuz that 's that 's critical for this this test .
C: Do we know anything about C: who who 's was it that had the lowest on the dev set ?
D: Um , uh , the , uh , the there were two systems that were put forth by a combination of of , uh , French Telecom and Alcatel .
D: And , um they they differed in some respects ,
D: but they e em one was called the French Telecom Alcatel System the other was called the Alcatel French Telecom System , uh , which is the biggest difference , I think .
D: But but there 're there 're there 're some other differences , too .
D: Uh , and and , uh , they both did very well ,
C: Uh - huh .
D: you know ?
D: So , um , my impression is they also did very well on on the the , uh , evaluation set ,
D: but , um , I I we haven't seen D: you 've - you haven't seen any final results for that
D: yeah .
C: And they used the main thing that that they used was spectral subtraction ?
C: Or
D: There is a couple pieces to it .
D: There 's a spectral subtraction style piece it was basically , you know , Wiener filtering .
D: And then then there was some p some modification of the cepstral parameters , where they A: Yeah ,
A: actually , something that 's close to cepstral mean subtraction .
A: But , uh , the way the mean is adapted um , it 's signal dependent .
A: I 'm I 'm , uh
A: So , basically , the mean is adapted during speech and not during silence .
D: Yeah .
A: But it 's very close to to cepstral mean subtraction .
D: But some people have done  exactly that sort of thing ,
D: of of and the I mean it 's not To to look in speech only , to try to m to measure these things during speech ,
A: Yeah ,
A: yeah .
D: that 's p that 's not that uncommon .
D: But i it it so it looks like they did some some , uh , reasonable things ,
D: uh , and they 're not things that we did , precisely .
D: We did unreasonable things , which because we like to try strange things ,
D: and and , uh , and our things worked too .
C: Hmm .
D: And so , um ,
D: uh , it 's possible that some combination of these different things that were done would be the best thing to do .
D: But the only caveat to that is that everybody 's being real conscious of how much memory and how much CPU they 're using
D: because these ,   uh , standards are supposed to go on cell phones with m moderate resources in both respects .
C: Mm - hmm .
C: Did anybody , uh , do anything with the models as a an experiment ?
C: Or
D: Uh , they didn't report it , if they did .
C: N nobody reported it ?
D: Yeah .
D: I think everybody was focused elsewhere .
D: Um , now , one of the things that 's nice about what we did is , we do have a a , uh a filtering , which leads to a a , uh a reduction in the bandwidth in the modulation spectrum , which allows us to downsample .
D: So , uh , as a result of that we have a reduced , um , transmission rate for the bits .
C: Mm - hmm .
D: That was misreported the first time out .
D: It it said the same amount because for convenience sake in the particular way that this is being tested , uh , they were repeating the packets .
D: So it was they were s they they had twenty - four hundred bits per second , but they were literally creating forty - eight hundred bits per second , um , even though y it was just repeated .
C: Oh .
C: Mm - hmm .
C: Right .
D: So , uh , in practice
C: So you could 've had a repeat count in there or something .
D: Well , n I mean , this was just a ph phoney thing just to to fit into the the software that was testing the errors channel errors and so on .
C: Oh .
D: So C: Oh .
D: so in reality , if you put this this system in into , uh , the field , it would be twenty - four hundred bits per second , not forty - eight hundred .
D: So ,
D: um , so that 's a nice feature of what what we did .
D: Um ,
D: but , um , well , we still have to see how it all comes out .
C: Hmm .
D: Um , and then there 's the whole standards process , which is another thing altogether .
C: When is the development set I mean , the , uh , uh , test set results due ?
C: Like the day before you leave or something ?
D: Uh , probably the day after they leave ,
D: but we 'll have to  we 'll have to stop it the day before <comment> we leave .
A: Yeah , yeah .
A: So
C: Huh .
D: I think tha I think the the meeting is on the thirteenth or something .
A: Yeah ,
A: this Tuesday ,
A: yeah .
D: And , uh ,
D: they , uh
D: Right .
D: And the the , uh , results are due like the day before the meeting or something .
D: So
A: Yeah ,
A: probably ,
A: well
D: I th I think I I think they are ,
A: Yeah , well
D: yeah .
D: So  um , since we have a bit farther to travel than some of the others , uh , we 'll have to get done a little quicker .
D: But , um , I mean , it 's just tracing down these bugs .
D: I mean , just exactly this sort of thing of , you know , why why these features seem to be behaving differently , uh , in California than in Oregon .
C: Hmm .
D: Might have something to do with electricity shortage .
D: Uh , we didn't we didn't have enough electrons here
D: and
D: Uh , but , um
D: Uh , I think , you know , the main reason for having D: I mean , it only takes w to run the the two test sets in just in computer time is just a day or so ,
D: right ?
D: So
A: Yeah ,
A: it 's very short interval .
D: yeah .
D: So , I think the who the whole reason for having as long as we have , which was like a week and a half , is is because of bugs like that .
D: So
D: Huh
D: So , we 're gonna end up with these same kind of sheets that have the the percentages and so on just for the A: Yeah , so there are two more columns in the sheets ,
D: Oh , I guess it 's the same sheets ,
D: yeah ,
A: two . Yeah ,
A: it 's the same sheets ,
D: yeah D: just with the missing columns filled in .
A: yeah .
A: Yeah .
D: Yeah .
D: Well , that 'll be good .
D: So , I 'll dis I 'll disregard these numbers .
D: That 's that 's that 's good .
A: So , Hynek will try to push for trying to combine , uh , different things ?
A: Or
D: Uh , well that 's um
A: Hmm ?
D: yeah
D: I mean , I think the question is " Is there is there some advantage ? "
D: I mean , you could just take the best system and say that 's the standard .
D: But the thing is that if different systems are getting at good things , um , a again within the constraint of the resources , if there 's something simple that you can do
D: Now for instance , uh , it 's , I think , very reasonable to have a standard for the terminal 's side and then for the server 's side say , " Here 's a number of things that could be done . "
D: So , um , everything that we did could probably just be added on to what Alcatel did ,
D: and i it 'd probably work pretty well with them , too .
D: So ,
D: um , uh , that 's one one aspect of it .
D: And then on the terminal 's side , I don't know how much , um , memory and and CPU it takes , but it seems like the filtering Uh ,
D: I mean , the VAD stuff they both had ,
D: right ?
D: And , um , so and they both had some kind of on - line normalization ,
D: right ?
A: Uh , yeah .
D: Of sorts ,
D: yeah ?
D: So D: so , it seems like the main different there is the is the , uh , filtering .
D: And the filtering I think if you can D: shouldn't take a lot of memory to do that Uh , and I also wouldn't think the CPU , uh , would be much either for that part .
D: So , if you can if you can add those in um then , uh , you can cut the data rate in half .
A: Yeah .
D: So it seems like the right thing to do is to on the on the terminal 's side , take what they did , if it if it does seem to generalize well to German and Danish ,
D: uh , take what they did add in a filter , and add in some stuff on the server 's side
D: and and and that 's probably a reasonable standard .
D: Um Uh
A: They are working on this already ?
A: Because yeah , Su - Sunil told me that he was trying already to put some kind of , uh , filtering in the  France Telecom .
D: Yeah , so that 's that 's that 's what
D: That would be ideal would be is that they could , you know , they could actually show that , in fact , a combination of some sort , uh , would work even better than what what any of the systems had .
D: And , um , then it would it would , uh be something to to discuss in the meeting .
D: But , uh , not clear what will go on .
D: Um , I mean , on the one hand , um , sometimes people are just anxious to get a standard out there .
D: I mean , you can always have another standard after that ,
D: but this process has gone on for a while on already
D: and and people might just wanna pick something and say , " OK , this is it . "
D: And then , that 's a standard .
D: Uh , standards are always optional .
D: It 's just that , uh , if you disobey them , then you risk not being able to sell your product ,
D: or  Uh um
D: And people often work on new standards while an old standard is in place and so on .
D: So it 's not final even if they declared a standard .
D: The other hand , they might just say they just don't know enough yet to to declare a standard .
D: So
D: you you you will be you will become experts on this and know more far more than me about the tha this particular standards process once you you go to this meeting .
D: So ,
D: be interested in hearing .
D: So , uh , I 'd be , uh , interested in hearing , uh , your thoughts now
D: I mean you 're almost done .
D: I mean , you 're done in the sense that , um , you may be able to get some new features from Sunil , and we 'll re - run it .
D: Uh , but other than that , you 're you 're basically done ,
D: right ?
D: So , uh , I 'm interested in hearing hearing your thoughts about where you think we should go from this .
A: Yeah .
D: I mean , we tried a lot of things in a hurry ,
D: and , uh , if we can back off from this now and sort of take our time with something , and not have doing things quickly be quite so much the constraint , what what you think would be the best thing to do .
A: Uh , well
A: Hmm
A: Well , first , uh , to really have a look at at the speech  from these databases
A: because , well , we tried several thing ,
A: but we did not really look at what what 's happening , and where is the noise , and
D: OK .
A: Eh
D: It 's a novel idea .
D: Look at the data .
D: OK .
A: Yeah .
D: Or more generally , I guess , what what is causing the degradation .
A: Yeah ,
A: yeah .
A: Actually , there is one thing that well Um , generally we we think that most of the errors are within phoneme classes ,
A: and
A: so I think it could be interesting to to see if it I don't think it 's still true when we add noise ,
A: and so we have I I guess the confusion ma the confusion matrices are very different when when we have noise , and when it 's clean speech .
A: And probably , there is much more between classes errors for noisy speech .
D: Mm - hmm .
A: And so , um
A: Yeah , so perhaps we could have a a large gain , eh , just by looking at improving the , uh , recognition , not of phonemes , but of phoneme classes , simply .
D: Mm - hmm .
A: And which is a s a s a simpler problem , perhaps , but which is perhaps important for noisy speech .
D: The other thing that strikes me , just looking at these numbers is , just taking the best cases ,
D: I mean , some of these , of course , even with all of our our wonderful processing , still are horrible kinds of numbers .
D: But just take the best case , the well - matched uh , German case after er well - matched Danish after we A: Mm - hmm .
D: the kind of numbers we 're getting are about eight or nine uh p percent error per digit .
A: Mm - hmm .
A: Yeah .
D: This is obviously not usable ,
D: right ?
A: No .
D: I mean , if you have ten digits for a phone number <comment> I mean , every now and then you 'll get it right .
A: Sure .
D: I mean , it 's it 's , uh , um
D: So , I mean , the other thing is that , uh D: And and a and and also , um part of what 's nice about this is that this is , uh , um a realistic almost realistic database .
D: I mean , it 's still not people who are really trying to accomplish something ,
D: but but , uh , within the artificial setup , it isn't noise artificially added , you know , simulated , uh , additive noise .
A: Mm - hmm .
D: It 's real noise condition .
D: And , um , the the training the training , I guess , is always done on the close talking
A: No ,
A: actually actually the well - matched condition is still quite di still quite difficult .
D: No ?
A: I mean , it 's they have all these data from the close mike and from the distant mike , from different driving condition , open window , closed window ,
D: Yeah .
A: and they take all of this
A: and they take seventy percent , I think , for training and thirty percent for testing .
E: Mm - hmm .
A: So , training is done on different conditions and different microphones ,
A: and testing also is done on different microphone and conditions .
A: So , probably if we only take the close microphones , I guess the results should be much much better than this .
D: I see .
A: Mmm .
D: Oh , OK ,
D: that explains it partially .
A: Uh
D: Wha - what about
D: i in so the the A: Yeah ,
A: so there is this , the mismatched is , um the same kind of thing ,
D: go ahead .
A: but the driving conditions , I mean the speed and the kind of road , is different for training and testing ,
A: is that right ?
E: Yeah .
A: And the last condition is close microphone for training and distant for testing .
A: Yeah .
D: Uh , OK ,
D: so
A: So  s so D: I see .
D: So , yeah ,
D: so the high so the right so the highly mismatched case is in some sense a good model for what we 've been , you know , typically talking about when we talk about additive noise in D: And so and i i k it does correspond to a realistic situation in the sense that , um , people might really be trying to , uh , call out telephone numbers or some or something like that , in in their cars