D: Uh - huh .
B: That would say at least for people , having the pitch is really , really important ,
B: which would be interesting in itself .
D: Uh , yeah . But B: Um ,
B: if i on the other hand , if it stayed up near five percent , then I 'd say " boy , LPC n twelve is pretty crummy " .
B: You know ?
D: Uh - huh .
B: So I I I 'm not sure I 'm not sure how we can conclude from this anything about that our system is close to the human performance .
D: Ye
D: Yeah . Well , the point is that eh l ey the point is that , um , what I what I listened to when I re - synthesized the LP - the LPC - twelve spectrum is in a way what the system , uh , is hearing ,
D: cuz @ @ all the all the , um , excitation all the D: well , the excitation is is not taken into account .
D: That 's what we do with our system .
D: And
B: Well , you 're not doing the LPC D: in this case B: I mean , so so what if you did a D: Well , it 's not LPC , sure ,
D: but B: What if you did LPC - twenty ?
D: LPC ?
B: Twenty .
B: Right ?
B: I mean , th the thing is LPC is not a a really great representation of speech .
D: Mm - hmm .
D: Mm - hmm .
B: So , all I 'm saying is that you have in addition to the w the , uh , removal of pitch , you also are doing , uh , a particular parameterization ,
D: Mm - hmm .
B: which ,
B: um , uh D: Mmm .
B: Uh , so , let 's see ,
B: how would you do ? So , fo
D: But that 's that 's what we do with our systems .
D: And B: No . Actually , we d we we don't ,
B: because we do we do , uh , uh , mel filter bank , for instance .
D: Yeah , but is it that is it that different , I mean ?
B: Right ?
B: Um , I don't know what mel , uh , based synthesis would sound like ,
D: I
D: Mm - hmm .
B: but certainly the spectra are quite different .
D: Mm - hmm .
A: Couldn't you t couldn't you , um , test the human performance on just the original audio ?
D: This is the one percent number .
B: Yeah , it 's one percent .
D: Mm - hmm .
B: He 's trying to remove the pitch information
A: Oh , oh . OK ,
D: Mm - hmm .
B: and make it closer to what to what we 're seeing as the feature vectors .
A: I see .
A: OK . So , y uh , your performance was one percent ,
D: Uh - huh .
A: and then when you re - synthesize with LPC - twelve it went to five .
D: Yeah .
A: OK .
B: I mean We were we were j It it it 's a little bit still apples and oranges
B: because we are choosing these features in order to be the best for recognition .
D: Uh - huh .
B: And , um ,
B: i if you listen to them they still might not be very Even if you made something closer to what we 're gonna i it might not sound very good .
D: Yeah .
B: Uh , and i the degradation from that might might actually make it even harder , uh , to understand than the LPC - twelve .
B: So all I 'm saying is that the LPC - twelve puts in synthesis
B: puts in some degradation
D: Uh - huh .
B: that 's not what we 're used to hearing ,
B: and is , um B: It 's not it 's not just a question of how much information is there , as if you will always take maximum advantage of any information that 's presented to you .
D: Mm - hmm .
B: In fact , you hear some things better than others .
B: And so it it isn't A: But B: But , I agree that it says that , uh , the kind of information that we 're feeding it is probably , um , um , a little bit , um , minimal .
B: There 's definitely some things that we 've thrown away .
B: And that 's why I was saying it might be interesting if you  an interesting test of this would be if you if you actually put the pitch back in .
B: So , you just extract it from the actual speech and put it back in ,
B: and see does that is that does that make the difference ?
D: Uh - huh .
B: If that if that takes it down to one percent again , then you 'd say " OK , it 's it 's in fact having , um , not just the spectral envelope but also the also the the pitch that , uh , <comment> @ @ <comment> has the information that people can use , anyway . "
D: Mmm .
A: But from this it 's pretty safe to say that the system is with either two to seven percent away from the performance of a human .
A: Right ?
A: So it 's somewhere in that range .
B: Well , or it 's it 's B: Yeah , so A: Two two to six percent .
B: It 's it 's one point four times , uh , to , uh , seven times the error ,
D: To f seven times , yeah .
B: for Stephane .
D: Um .
B: So , uh B: uh , but i I don't know .
D: But <comment> but B: I do don't wanna take you away from other things .
B: But that 's  that 's what that 's the first thing that I would be curious about , is , you know , i i when you we
D: But the signal itself is like a mix of um , of a a periodic sound and , @ @ <comment> uh , unvoiced sound , and the noise
B: Mm - hmm .
D: which is mostly , uh , noise .
D: I mean not periodic .
D: So , what what do you mean exactly by putting back the pitch in ?
D: Because A: In the LPC synthesis ?
B: Yeah . You did LPC re - synthesis A: I think D: I
D: Uh - huh .
B: L PC re - synthesis .
B: So , uh and you did it with a noise source ,
D: Mm - hmm .
B: rather than with with a s periodic source .
B: Right ?
B: So if you actually did real re - synthesis like you do in an LPC synthesizer , where it 's unvoiced you use noise ,
B: where it 's voiced you use , uh , periodic pulses .
D: Um .
B: Right ?
D: Yeah , but it 's neither purely voiced or purely unvoiced .
D: Esp - especially because there is noise .
B: Well , it might be hard to do it
D: So B: but it but but the thing is that if you  um , if you detect that there 's periodic s strong periodic components , then you can use a voiced voice thing .
D: Oh .
D: Uh - huh .
D: Yeah .
B: Yeah . I mean , it 's probably not worth your time .
B: It 's it 's a side thing
B: and and and there 's a lot to do .
D: Uh - huh , yeah .
B: But I 'm I 'm just saying , at least as a thought experiment , that 's what I would wanna test .
D: Mm - hmm .
B: Uh , I wan would wanna drive it with a a a two - source system rather than a than a one - source system .
D: Mm - hmm .
D: Mm - hmm .
B: And then that would tell you whether in fact it 's B: Cuz we 've talked about , like , this harmonic tunneling or other things that people have done based on pitch ,
B: maybe that 's really a key element .
B: Maybe maybe , uh , uh , without that , it 's it 's not possible to do a whole lot better than we 're doing .
B: That that could be .
D: Yeah .
D: That 's what I was thinking by doing this es experiment ,
D: like B: Yeah .
D: Mmm . Evi
B: But , I mean , other than that , I don't think it 's B: I mean , other than the pitch de information , it 's hard to imagine that there 's a whole lot more in the signal that that , uh that we 're throwing away that 's important .
D: Yeah , but D: Yeah . Mm - hmm . Yeah , right .
B: Right ? I mean , we 're using a fair number of filters in the filter bank
B: and uh D: Mm - hmm .
D: Uh , yeah .
B: Hmm .
B: Yeah .
D: Um .
D: Yeah , that 's it .
B: Yeah . That look
B: Yeah .
B: That 's that 's I mean , one one percent is sort of what I would I would figure .
B: If somebody was paying really close attention , you might get B: I would actually think that if , you looked at people on various times of the day and different amounts of attention , you might actually get up to three or four percent error on digits .
D: Mm - hmm .
B: Uh , uh D: Um .
B: So it 's B: you know , we 're not we 're not incredibly far off .
B: On the other hand , with any of these numbers except maybe the one percent , it 's st it 's not actually usable in a commercial system with a full telephone number or something .
D: Uh - huh .
D: Yeah . At these noise levels .
D: Yeah . Mm - hmm .
B: Yeah .
B: Right .
D: Well , yeah . These numbers , I mean .
D: Mmm .
B: Good .
B: Um , while we 're still on Aurora stuff maybe you can talk a little about the status with the , uh , Wall Street Journal things for it .
A: So I 've , um , downloaded , uh , a couple of things from Mississippi State .
A: Um , one is their software A: their , uh , LVCSR system .
A: Downloaded the latest version of that .
A: Got it compiled and everything .
A: Um , downloaded the scripts .
A: They wrote some scripts that sort of make it easy to run the system on the Wall Street Journal , uh , data .
A: Um , so I haven't run the scripts yet .
A: Uh , I 'm waiting there was one problem with part of it
A: and I wrote a note to Joe asking him about it .
A: So I 'm waiting to hear from him .
A: But , um , I did print something out just to give you an idea about where the system is .
A: Uh , they on their web site they , uh , did this little table of where their system performs relative to other systems that have done this this task .
A: And , um , the Mississippi State system using a bigram grammar , uh , is at about eight point two percent .
A: Other comparable systems from , uh  were getting from , uh , like six point nine , six point eight percent .
A: So they 're B: This is on clean test set ?
A: This is on clean on clean stuff . Yeah .
A: They they 've started a table where they 're showing their results on various different noise conditions
A: but they they don't have a whole lot of it filled in
A: and  and I didn't notice until after I 'd printed it out that , um , they don't say here what these different testing conditions are .
A: You actually have to click on it on the web site to see them .
A: So I I don't know what those numbers really mean .
B: What kind of numbers are they getting on these on the test conditions ?
A: Well , see , I was a little confused
A: because on this table , I 'm the they 're showing word error rate .
A: But on this one ,
A: I I don't know if these are word error rates
A: because they 're really big .
A: So , under condition one here it 's ten percent .
A: Then under three it goes to sixty - four point six percent .
B: Yeah , that 's probably Aurora .
B: I mean A: Yeah .
A: So m I guess maybe they 're error rates
A: but they 're , uh they 're really high .
B: I I I don't find that surpri
A: So B: I mean , we B: W what 's what 's some of the lower error rates on on on uh , some of the higher error rates on , uh , some of these w uh , uh , highly mismatched difficult conditions ?
B: What 's a ?
D: Uh . Yeah , it 's around fifteen to twenty percent .
A: Correct ?
D: And the baseline , eh A: Accuracy ?
D: Uh , error rate .
B: Yeah .
D: Twenty percent error rate ,
B: Yeah . So twenty percent error rate on digits .
D: and B: So if you 're doing so if you 're doing ,
D: and A: Oh , oh , on digits .
A: Yeah .
D: On digits .
D: And this is so so still the baseline .
A: OK .
B: you know ,
B: sixty - thousand D: Right ?
A: Yeah .
B: Yeah ,
B: and if you 're saying sixty - thousand word recognition , getting sixty percent error on some of these noise condition not at all surprising .
A: Yeah .
D: The baseline is sixty percent also on digits ,
A: Oh , is it ?
D: on the m more mismatched conditions .
A: OK .
D: So .
B: Yeah .
A: So , yeah , that 's probably what it is then .
A: Yeah . So they have a lot of different conditions that they 're gonna be filling out .
B: It 's a bad sign when you looking at the numbers , you can't tell whether it 's accuracy or error rate .
A: Yeah .
A: Yeah . It 's it 's gonna be hard .
A: Um ,
A: they 're I I 'm still waiting for them to release the , um , multi - CPU version of their scripts ,
A: cuz right now their script only handles processing on a single CPU ,
A: which will take a really long time to run .
A: So .
B: This is for the training ?
A: But their s
A: Uh I beli
A: Yes , for the training also .
B: OK .
A: And , um , they 're supposed to be coming out with it any time ,
A: the multi - CPU one .
B: OK .
A: So , as soon as they get that , then I 'll I 'll grab those too
A: and so w
B: Yeah . Cuz we have to get started ,
B: cuz it 's cuz , uh ,
A: Yeah .
A: Yeah . I 'll go ahead and try to run it though with just the single CPU one ,
B: if the A: and I they they , um , released like a smaller data set that you can use that only takes like sixteen hours to train and stuff .
A: So I can I can run it on that just to make sure that the  the thing works and everything .
B: Oh ! Good .
B: Yeah .
E: Hmm .
B: Cuz we 'll B: I guess the actual evaluation will be in six weeks or something .
B: So .
B: Is that about right you think ?
D: Uh , we don't know yet , I I think .
B: Really , we don't know ?
D: Uh - huh .
D: Um .
B: Hmm .
A: It wasn't on the conference call this morning ?
D: No .
A: Hmm .
A: Did they say anything on the conference call about , um , how the Wall Street Journal part of the test was going to be run ?
A: Because I I thought I remembered hearing that some sites were saying that they didn't have the compute to be able to run the Wall Street Journal stuff at their place ,
D: No .
D: Mmm .
A: so there was some talk about having Mississippi State run the systems for them .
A: And I A: Did did that come up at all ?
D: Uh , no .
D: Well , this first , this was not the point at all of this the meeting today
A: Oh , OK .
D: and ,
D: uh , frankly , I don't know
B: Some
D: because I d <comment> didn't read also the most recent mails about the large - vocabulary task .
D: But , uh , did you do you still , uh , get the mails ?
D: You 're not on the mailing list or what ?
A: Hmm - mm .
A: The only , um , mail I get is from Mississippi State D: Uh - huh .
A: so D: Oh , yeah . So we should have a look at this .
A: about their system .
A: I I don't get any mail about B: I have to say , there 's uh something funny - sounding about saying that one of these big companies doesn't have enough cup compute power do that ,
B: so they 're having to have it done by Mississippi State .
A: Yeah .
B: It just  just sounds funny .
A: Yeah . It does .
B: But ,
B: anyway .
A: Yeah . I 'm I 'm wondering about that
A: because there 's this whole issue about , you know , simple tuning parameters , like word insertion penalties .
D: Mm - hmm .
A: And whether or not those are going to be tuned or not ,
A: and <comment> So .
D: Mm - hmm .
A: I mean , it makes a big difference .
A: If you change your front - end , you know , the scale is completely can be completely different ,
A: so .
A: It seems reasonable that that at least should be tweaked to match the front - end .
A: But D: You didn't get any answer from Joe ?
A: I did ,
A: but Joe said , you know , " what you 're saying makes sense
D: Uh - huh .
A: and I don't know " .
D: Uh - huh .
A: So he doesn't know what the answer is .
A: I mean , that 's th We had this back and forth a little bit about , you know , are sites gonna are you gonna run this data for different sites ?
A: And , well , if if Mississippi State runs it , then maybe they 'll do a little optimization on that parameter ,
A: and , uh A: But then he wasn't asked to run it for anybody .
A: So i it 's it 's just not clear yet what 's gonna happen .
D: Mm - hmm .
A: Uh , he 's been putting this stuff out on their web site and for people to grab
A: but I haven't heard too much about what 's happening .
B: So it could be I mean , Chuck and I had actually talked about this a couple times , and and over some lunches , I think , that , um , one thing that we might wanna do B: The - there 's this question about , you know , what do you wanna scale ?
B: Suppose y you can't adjust these word insertion penalties and so forth ,
B: so you have to do everything at the level of the features .
B: What could you do ?
B: And , uh , one thing I had suggested at an earlier time was maybe some sort of scaling ,
B: some sort of root or or something of the , um , uh , features .
B: But the problem with that is that isn't quite the same ,
B: it occurred to me later ,
B: because what you really want to do is scale the , uh , @ @ <comment> the range of the likelihoods rather than D: Nnn , the dist
D: Yeah .
B: But , what might get at something similar , it just occurred to me , is kind of an intermediate thing B: is because we do this strange thing that we do with the tandem system , at least in that system what you could do is take the , um , uh , values that come out of the net ,
B: which are something like log probabilities ,
B: and scale those .
B: And then , uh , um  then at least those things would have the right values
B: or the right the right range .
B: And then that goes into the rest of it and then that 's used as observations .
B: So it 's it 's , um , another way to do it .
D: Mm - hmm .
D: Mm - hmm .
D: But , these values are not directly used as probabilities anyway .
B: I know they 're not .
D: So there are there is B: I know they 're not .
B: But but ,
B: you know D: Uh - huh .
B: So because what we 're doing is pretty strange and complicated , we don't really know what the effect is at the other end .
D: Mm - hmm .
B: So , um , my thought was maybe B: I mean , they 're not used as probabilities ,
B: but the log probabilities B: we 're taking advantage of the fact that something like log probabilities has more of a Gaussian shape than Gaus - than probabilities ,
B: and so we can model them better .
B: So , in a way we 're taking advantage of the fact that they 're probabilities ,
B: because they 're this quantity that looks kind of Gaussian when you take it 's log .
B: So , <comment> uh , maybe maybe it would have a a reasonable effect to do that .
D: Mm - hmm .
B: I d I don't know .
B: But , I mean , I guess we still haven't had a  a ruling back on this .
B: And we may end up being in a situation where we just you know really can't change the word insertion penalty .