C: which didn't have any of the F: Right . But all of these effects were j like a couple percent .
C: Uh .
F: Right ?
F: I mean , y the C: Well , but if you add them all up you have , uh , almost five percent difference now .
F: Add all of them .
F: I thought one was one point five percent and one was point eight .
C: Yeah . And now we have another percent with the V T
F: That 's three point three .
C: Um , actually , and it 's ,
C: um ,
C: What 's actually qu interesting is that with C: um , well , you m prob maybe another half percent if you do the VTL in training ,
C: and then interestingly , if you optimize you get more of a win out of rescoring the , um , uh , the N best lists ,
C: uh , and optimizing the weights ,
C: um , uh than D: Than you do with the standard ?
C: Yeah .
F: Yeah . But the part that 's actually adjustment of the front - end per se as opposed to doing putting VTLN in or something is it was a couple percent .
C: So C: Right .
F: Right ? It was it was there was  there was one thing that was one and a half percent and one that was point eight .
F: So and and F: let me see if I remember what they were .
F: One of them was , uh , the change to , uh because it did it all at once , <comment> to uh , from bark scale to mel scale ,
C: Mm - hmm .
F: which I really feel like saying in quotes ,
F: because @ @ <comment> they 're essentially the same scale
G: Yeah . Why did that cha ?
F: but the but but but any i individual particular implementation of those things puts things in a particular place .
C: Mm - hmm .
F: So that 's why I wanted to look F: I still haven't looked at it yet .
F: I I wanna look at exactly where the filters were in the two ,
C: Mm - hmm .
F: and it  it 's probably something like there 's one fewer or one more filter in the sub one kilohertz band
F: and for whatever reason with this particular experiment it was better
C: Mm - hmm .
F: one way or the other .
G: Hmm .
F: Um , it could be there 's something more fundamental
F: but it you know , I I don't know it yet .
F: And the other and the other that was like one and a half or something ,
F: and then there was point eight percent ,
F: which was F: what was the other thing ?
D: Well , that was combined with the triangular .
D: Right ?
F: Yeah . Those those two were together .
D: Yeah . Right .
F: We d weren't able to separate them out
F: cuz it was just done in one thing .
F: But then there was a point eight percent which was something else .
F: Do you remember the ?
D: The low - frequency cut - off .
F: Oh , yeah . So that was that was , uh that one I can claim credit for , uh , i in terms of screwing it up in the first place .
F: So that someone e until someone else fixed it ,
F: which is that , um , I never put F: when I u
F: We had some problems before with offsets .
F: This inf this went back to , uh , I think Wall Street Journal .
F: So we we had , uh C: Hmm .
F: ea everybody else who was doing Wall Street Journal knew that there were big DC offsets in th in these data in those data
F: and and and nobody happened to mention it to us ,
C: Hmm .
F: and we were getting these , like , really terrible results ,
F: like two , three times the error everybody else was getting .
F: And then in casual conversation someone ment mentioned " uh , well , I guess , you know , of course you 're taking care of the offsets . "
F: I said " what offsets ? "
C: Mm - hmm .
F: And at that point , you know , we were pretty new to the data
F: and we 'd never really , like , looked at it on a screen
F: and then when we just put it on the screen <comment> and wroop !
F: there 's this big DC offset .
C: Mm - hmm .
F: So , um , in PLP
G: There was a like a hum or some or when they recorded it ?
F: No . It 's just , it it 's it 's not at all uncommon for for recorded electronics to have different , um , DC offsets .
G: Or just ?
G: Huh .
F: It 's it 's , you know , no big deal .
F: It 's you know , you could have ten , twenty , maybe thirty millivolts , whatever ,
F: and it 's consistently in there .
F: The thing is , most people 's front - ends have pre - emphasis with it , with zero at zero frequency ,
F: so that it 's irrelevant .
F: Uh , but with P L P , we didn't actually have that .
F: We had we had the equivalent of pre - emphasis in a a , uh , Fletcher - Munson style weighting that occurs in the middle of P L
F: but it doesn't actually have a zero at zero frequency ,
G: Hmm .
F: like , eh , uh , typical simple fr pre - emphasis does .
F: We had something more fancy .
F: It was later on
F: it didn't have that .
F: So at that point I reali " oh sh we better have a have a high - pass filter "
F: just , you know  just take care of the problem .
F: So I put in a high - pass filter at , uh , I think ninety ninety hertz or so uh , for a sixteen kilohertz sampling rate .
F: And I never put anything in to adjust it for different different sampling rates .
F: And so well , so , you know , the code doesn't know anything about that
F: and so this is all at eight kilohertz
F: and so it was at forty - five hertz instead of at  instead of at ninety .
C: Hmm .
F: So , um ,
F: I don't know if Dan fixed it or or , uh , what he C: Well , he made it a parameter .
F: He made it a parameter .
F: So . Yeah , I guess if he did it right , he did fix it
F: and then and then it 's taking care of sampling rate ,
F: which is great .
D: What what is the parameter ?
F: He had a D: Is it , uh , just the f lower cut - off that you want ?
C: It 's called , uh , H - HPF .
F: H F: Yeah . Does HPF on on his feat feature .
C: u And but HPF , you know , when you put a number after it , uses that as the hertz value of the cut - off .
D: Mm - hmm .
F: Yeah .
D: Oh , OK .
C: So .
F: I mean , frankly , we never did that with the RASTA filter either ,
F: so the RASTA filter is actually doing a different thing in the modulation spectral domain depending on what sampling rate you 're doing ,
C: Mm - hmm .
F: which is another old old bug of mine .
C: Mm - hmm .
F: But , um F: Um . So that that was the problem there was th we we we had always intended to cut off below a hundred hertz
C: Mm - hmm .
F: and it just wasn't doing it ,
F: so now it is .
F: So , that hep that helped us by , like , eight tenths of a percent .
F: It still wasn't a big deal .
C: OK .
C: Well , but , um Well , uh , again , after completing the current experiments , we 'll  we can add up all the uh differences
F: Oh , yeah .
F: But but , I guess my my point was that that , um , the hybrid system thing that we did was , uh , primitive in many ways .
C: and  and an
C: Y
C: Right .
F: And I think I agree with you that if we fixed lots of different things and they would all add up , we would probably have a a a competitive system .
F: But I think not that much of it is due to the front - end per se .
F: I think maybe a couple percent of it is , as far as I can see from this .
C: Mm - hmm .
F: Uh , unless you call well , if you call VTL the front - en front - end , that 's , uh , a little more .
F: But that 's sort of more both , kind of .
D: One experiment we should we 'll probably need to do though when um , at some point , is , since we 're using that same the net that was trained on PLP without all these things in it , for the tandem system , we may wanna go back and retrain ,
F: Right ?
F: But .
C: Well , that 's what I meant , in fact . Yeah .
D: yeah , yeah ,
D: for the tandem . You know ,
D: so we can see if it what effect it has on the tandem processing .
C: So so , the thing is is do we expect ?
F: Mm - hmm .
C: eh At this point I 'm as I mean , you know e I 'm wondering is it Can we expect , uh , a tandem system to do better than a properly trained you know , a Gaussian system trained directly on the features with , you know , the right ch choice of parameters ?
F: Well , that 's what we 're seeing in other areas . Yes .
F: Right ?
F: So , it 's so ,
F: um , um D: So , we But but we may not . I mean , if it doesn't perform as well , we may not know why .
D: Right ?
D: Cuz we need to do the exact experiment .
C: Right .
F: I mean , the reason to think it should is because you 're putting in the same information and you 're transforming it to be more discriminative .
F: So . Um .
F: Now the thing is , in some databases I wouldn't expect it to necessarily give you much
F: and and part of what I view as the real power of it is that it gives you a transformational capability for taking all sorts of different wild things that we do ,
F: not just th the standard front - end ,
F: but other things , like with multiple streams and so forth ,
C: Mm - hmm .
F: and allows you to feed them to the other system with this through this funnel .
F: Um , so I think I think that 's the real power of it .
F: I wouldn't expect huge in huge improvements .
F: Um , but it should at least be roughly the same and maybe a little better .
C: Mm - hmm .
F: If it 's , you know , like way way worse then , you know C: Right .
D: So , Morgan , an another thing that Andreas and I were talking about was ,
D: so @ @ <comment> in the first experiment that he did we just took the whole fifty - six , uh , outputs
D: and that 's , um , basically compared to a thirty - nine input feature vector from either MFCC or PLP .
F: Mm - hmm .
F: Mm - hmm .
D: But one thing we could do is F: Let let me let me just ask you something .
F: When you say take the fifty - six outputs , these are the pre final nonlinearity outputs
D: Yeah . Through the regular tandem outputs .
F: and they 're and F: through the KLT .
D: Through the KLT .
F: OK .
D: All that kinda stuff .
F: And so so then you u Do you use all fifty - six of the KLT
F: or ?
D: That 's what we did .
D: Right ?
F: OK .
D: So one thing we were wondering is , if we did principal components and , say , took out just thirteen , and then did deltas and double - deltas on that F: Yes .
F: Yes .
D: so we treated the th first thirteen as though they were standard features .
F: Yeah .
D: I mean , did Dan do experiments like that to ?
F: Uh . Talk with Stephane .
F: He did some things like that .
F: It was either him or Carmen .
F: I forget .
D: Mmm .
C: Mm - hmm .
F: I mean these were all different databases and different you know , in HTK and all that ,
D: Yeah .
F: so i it it may not apply .
F: But my recollection of it was that it didn't make it better but it didn't make it worse .
D: Hmm .
F: But , again , given all these differences , maybe it 's more important in your case that you not take a lot of these low - variance , uh , components .
D: Cuz in a sense , the net 's already got quite a bit of context in those features ,
F: Yeah .
D: so if we did deltas and double - deltas on top of those , we 're getting sort of even more .
F: Which could be good or not .
D: Yeah .
F: Yeah .
F: Yeah . Worth trying .
C: But there the main point is that , um , you know , it took us a while but we have the procedure for coupling the two systems debugged now
C: and I mean , there 's still conceivably some bug somewhere in the way we 're feeding the tandem features C: uh , either generating them or feeding them to this to the SRI system ,
D: Mm - hmm .
D: Yeah .
C: but
F: There might be ,
C: it 's F: cuz that 's a pretty big difference .
C: Yeah .
F: But
C: And I 'm wondering how we can how we can debug that .
D: Yeah .
C: I mean how C: Um .
F: Hmm .
C: I 'm actually f quite sure that the feeding the features into the system and training it up ,
F: What if ?
C: that that I think that 's this that 's essentially the same as we use with the ce with the P L P fe features .
C: And that 's obviously working great .
C: So . I um .
D: Yeah . There could be a bug in in the somewhere before that .
C: There we could C: the another degree of freedom is how do you generate the K L T transform ?
D: Mm - hmm .
F: That 's C: Right ?
C: We to
D: Right .
F: well , and another one is the normalization of the inputs to the net .
C: Yeah .
F: These nets are trained with particular normalization
F: and when that gets screwed up it it can really hurt it .
D: I 'm doing what Eric D: E Eric coached me through then that part of it ,
D: so I 'm pretty confident in that .
F: OK .
D: I mean , the only slight difference is that I use normalization values that , um , Andreas calculated
D: from the original <comment> PLP ,
C: Right .
D: which is right .
D: N
D: Yeah .
C: Right .
D: So , I u I do D: Oh , we actually don't do that normalization for the PLP ,
D: do we ?
D: For the st just the straight PLP features ?
C: No . The the SRI system does it .
D: S R I system does that . Right .
C: Yeah .
F: Right .
F: Well , you might e e
D: So that 's that 's another C: So , there 's there is there is room for bugs that we might not have discovered ,
D: Yeah .
F: Yeah .
D: Mm - hmm .
F: Yeah . I I would actually double check with Stephane at this point ,
C: but F: cuz he 's probably the one here F: I mean , he and Dan are the ones who are at this point most experienced with the tandem
D: Mm - hmm .
F: thing and there may there may be some little bit here and there that is not not being handled right .
D: Yeah . It 's hard with features ,
D: cuz you don't know what they should look like .
D: I mean , you can't just , like , print the the values out in ASCII and , you know , look at them , see if they 're F: Not unless you had a lot of time
G: Well F: and G: eh , and also they 're not I mean , as I understand it , you you don't have a way to optimize the features for the final word error .
G: Right ?
C: Right .
G: I mean , these are just discriminative ,
G: but they 're not , um , optimized for the final C: They 're optimized for phone discrimination ,
G: Right . So it there 's always this question of whether you might do better with those features if there was a way to train it for the word error metric that you 're actually that you 're actually C: not for F: That 's right .
F: Well , the other F: Yeah , th the C: Mm - Mmm .
F: Well , you actually are .
F: But but it but in an indirect way .
G: Well , right . It 's indirect , so you don't know F: So wha w what an and you may not be in this case ,
F: come to think of it ,
F: because , uh , you 're just taking something that 's trained up elsewhere .
F: So , what what you what you do in the full procedure is you , um , uh , have an embedded training .
F: So in fact you the the net is trained on , uh , uh , a , uh , Viterbi alignment of the training data that comes from your full system .
F: And so that 's where the feedback comes all around , so that it is actually discriminant .
F: You can prove that it 's it 's a ,
F: uh F: If you believe in the Viterbi assumption that , uh , getting the best path , uh , is almost equivalent to getting the best , uh , total probability , um , then you actually do improve that by , uh by training up on local local , uh local frames .
F: But , um , we aren't actually doing that here ,
F: because we did we did that for a hybrid system ,
F: and now we 're plugging it into another system
F: and so it isn't  i i i it wouldn't quite apply here .
D: So another huge experiment we could do would be to take the tandem features , uh , do SRI forced alignments using those features , and then re - do the net with those .
C: Do y
F: Mm - hmm .
G: Mmm , uh Exactly . Exactly .
F: Yeah .
G: So that you can optimize it for the word error .
F: Yeah . Another thing is since you 're not using the net for recognition per se but just for this transformation , it 's probably bigger than it needs to be .
C: But G: Yeah .
F: So that would save a lot of time .
C: And there 's a mismatch in the phone sets .
D: Mmm .
C: So , you 're using a l a long a larger phone set than what F: Yeah . Actually all those things could could could could , uh could affect it as well .
D: Yeah . Yeah .
F: The other thing , uh , just to mention that Stephane this was an innovation of Stephane 's , which was a pretty neat one ,
F: uh , and might particularly apply here , given all these things we 're mentioning .
F: Um , Stephane 's idea was that , um , discriminant , uh , approaches are great .
F: Even the local ones , given , you know , these potential outer loops which , you know , you can convince yourself turn into the global ones .
F: Um , however , there 's times when it is not good .
F: Uh , when something about the test set is different enough from the training set that that , uh , the discrimination that you 're learning is is is not a good one .
C: Mm - hmm .
F: So , uh , his idea was to take as the input feature vector to the , uh , Gaussian mixture system , uh , a concatenation of the neural net outputs and the regular features .
C: Oh , we already talked about that .
G: Yeah . That F: Yeah .
D: Mm - hmm .
G: Didn't you did you do that already
C: El
C: Yeah .
G: or ?
C: No , but we we when when we when I first started corresponding with Dan about how to go about this , I think that was one of the things that we definitely went there .
G: Oh . That makes a lot of sense .
G: Huh .
F: Yeah .
F: Yeah . I mean , I 'm sure that Stephane wasn't the first to think of it ,
C: Yeah .
F: but actually Stephane did it
F: and  and and it helped a lot .
C: Uh - huh . And i does it help ?
F: Yeah .
C: Oh , OK .
F: So that 's that that 's our current best best system in the , uh  uh , in the Aurora thing .
C: Oh . OK .
G: Yeah . That makes sense .
F: Yeah .
C: And do you do a KLT transform on the con on the combined feature vector ?
G: As you should never do worse .
F: I I , uh , missed what you said .
C: Do you d you do a KLT transform on the combined feature vector ?
F: Yeah .
C: OK .
F: Well , actually , I , uh you should check with him ,
F: because he tried several different combinations .
C: Because you end up with this huge feature vector , so that might be a problem , a unless you do some form of dimensionality reduction .
F: Yeah . I , uh , th
F: what I don't remember is which came out best .
F: So he did one where he put o put e the whole thing into one KLT ,
F: and another one , since the the PLP things are already orthogonalized , he left them alone and and just did a KLT on the on the on the net outputs
C: Mm - hmm .
F: and then concatenated that .
C: Mmm .
F: And I don't remember which was better .
D: Did he did he try to ?
D: So he always ended up with a feature vector that was twice as long as either one of the ?
F: No . I don't know , i I I don't know .