B: Yeah .
D: But he I think he d he didn't he probably didn't use all the training data that was available .
D: And his dictionary probably wasn't as tuned to um conversational speech as as the as ours is .
D: So .
B: That 's That 's certainly one thing , yeah .
D: And the dictionary made a huge difference .
B: Uh . Yeah .
D: Uh . We we made some improvements to the dictionary 's uh to the dictionary about two years ago
D: which resulted in a uh something like a four percent absolute error rate reduction on Switchboard ,
D: which B: Well the other thing is , dipping deep into history and into uh our resource management days , when we were collaborating with SRI before ,
D: Mm - hmm .
D: Mmm .
D: Mm - hmm .
B: uh it was I think , it is was a really key uh starting point for us that we actually got our alignment .
B: When we were working together we got our initial alignments from Decipher , uh at the time .
D: Mm - hmm .
D: Yeah .
B: Uh . And . Later we got away from it because because once we had decent systems going then it was it was typically better to use our own systems
F: Yeah .
D: Mm - hmm .
B: cuz they were self consistent
B: but but certainly to start off when we were trying to recover from our initial hundred and forty percent error uh rate .
B: Uh . But that was a that was a good good good way to start .
D: Yeah .
D: OK .
B: And we 're not quite that bad with our our Switchboard systems but it was they certainly aren't as good as SRI 's ,
D: Yeah .
B: so F: W What is the performance on s the best Switchboard system that we 've done ?
D: Right .
F: Roughly ?
B: Well , the hybrid system we never got better than about fifty percent error .
B: And uh it was B: I think there 's just a whole lot of things that uh no one ever had time for .
B: We never did really fix up the dictionary .
B: Uh we always had a list of a half dozen things that we were gonna do
B: and and a lot of them were pretty simple and we never did .
D: Yeah .
D: Mmm .
D: But that w
B: Uh , we never did an never did any adaptation
D: Even that that number D: Right .
B: uh , we never did any D: And And that number I think was on Switchboard - one data ,
D: right ?
D: Where the error rate now is in the twenties .
B: Yeah .
D: So , um .
B: Yeah .
B: So we were D: That 's yet s
B: Yeah .
B: We were probably at least a factor or two off .
D: Right .
B: Yeah .
D: So it would be So it would be good t to sort of r re uh B: Yeah .
D: just at least to give us an idea of how well the hybrid system would do .
B: Yeah .
B: But I think again it 's B: Yeah .
B: It 's the conver it 's the s conversational speech bit .
B: Because our our Broadcast News system is actually pretty good .
D: Mm - hmm .
B: He knows .
D: Right .
D: And the other thing that that would help us to evaluate is to see how well the M L P is trained up .
D: Right ?
D: Because it 's a pretty good um indicator of that .
B: Mm - hmm .
D: So it 's sort of a sanity check of the M L P outputs before we go ahead and train up the uh you know , use them as a basis for the tandem system .
F: Mm - hmm .
B: Yeah .
B: It 'll still probably be worse .
B: I mean , it 's it 'd be context independent and so on .
D: No .
D: Sure .
F: Should we Should we bother with um using the net before doing uh embedded training ?
D: Not D: But B: But .
F: I mean should should we even use that ?
D: Oh oh that 's a good question .
F: Or should I just go straight to D: Yeah , we we weren't sure whether it 's worth to just use the alignments um from the S R I recognizer
D: or whether to actually go through one or more iterations of embedded training where you realign .
A: Try it .
A: You run it ?
A: Keep keep both versions ?
A: See which one 's better ?
B: Uh , yeah .
B: I mean . I think I agree with Ad
B: I mean basically you would then You proceed with the embedded training .
D: Mm - hmm .
D: Mm - hmm .
B: It 's gonna take you a while to train at this net anyway .
D: Right .
B: And while it 's training you may as well test the one you have and see how it did .
D: OK .
F: Mmm .
D: Alright .
A: I could make arguments either way .
D: But But so I A: You know , it 's D: Well but i
A: Sort of given up guessing .
D: But in your experience I mean uh have you seen big improvements in s on some tasks with embedded training ?
D: Or was it sort of small - ish uh improvements that you got
B: Uh well . It depended on the task .
B: I mean I think in this one I would sort of expect it to be important
D: Right .
B: because we 're coming from uh , alignments that were achieved with an extremely different system .
D: That are from another D: Right .
D: Right .
A: Although , I mean we 've done it with When we were combining with the Cambridge recurrent neural net , embedded training made it worse .
B: Uh .
A: Which I 've never figured out .
B: Right .
B: But I mean i
A: I think it 's a bug .
D: So you you started training with outputs from a with alignments that were generated by the Cambridge uh system ?
A: Yep .
D: And then D: Uh .
A: Yeah .
B: Yeah .
D: Hmm . Well , that might probably just D: Hmm .
D: That was probably because your initial system I mean your system was ba worse than Cambridge 's .
D: And you Um .
B: Was it ?
B: I don't think it was .
A: No
A: they were they were comparable .
D: It wasn't ?
B: No .
D: Really ?
A: They were very close .
B: Yeah .
D: That 's weird .
B: Excuse me ?
D: That 's That 's weird .
D: No I mean it 's weird that it did B: Oh !
A: That 's what I said .
D: I 'm sorry .
D: It 's w
D: It 's weird that it got worse .
F: That 's ambiguous .
B: Um . No .
B: Uh . Tha - u we we 've see I mean and wi with the numbers OGI numbers task we 've seen a number of times people doing embedded trainings and things not getting better .
D: Oh actually it 's not that weird
D: because we have seen D: We have seen cases where acoustic retraining the acoustic models after some other change made matters worse rather than better .
B: Yeah .
B: It just D: Yeah .
B: But I But I would I would suspect that something that that had um a very different Um feature set , for instance B: I mean they were using pretty diff similar feature sets to us .
A: Yep .
B: I I would expect that something that had a different feature set would would uh benefit from D: Mm - hmm .
F: What about uh hidden unit size on this .
B: Oh , wait a minute ,
B: and the other thing uh ,
F: Oh .
B: sorry ,
B: it was the other thing is that what was in common <comment> to the Cambridge system and our system is they both were training posteriors .
A: Right .
B: So I mean , uh , that 's another pretty big difference
A: Ah yeah .
A: That 's another big difference .
B: and uh , one bac at least B: Back at D: You mean with soft targets ?
D: Or ?
D: Sorry , I 'm sor
D: I missed D: What What 's the key issue here ?
B: Oh , that uh both the Cambridge system and our system were were training posteriors .
B: And if we 're we 're coming from alignments coming from the SRI system , it 's a likelihood - based system .
B: So so that 's another difference .
D: Yeah .
B: I mean . You know , there 's diffe different front - end different different uh , um , training criterion B: Uh , I would think that in a that an embedded uh embedded uh training would have at least a good shot of improving it some more .
D: Mm - hmm .
B: But we don't know .
D: OK .
B: You gonna say something ?
F: Yeah .
F: I was wondering uh you know what size net I should F: Anybody have any intuitions or suggestions ?
B: Uh , how much training data ?
F: Well , I was gonna start off with the small train set .
B: And how How many hours is that ?
F: That 's why I was F: I I 'm not sure how much that is .
D: Uh , I think that has about D: Well i you 'd would be gender - dependent training ,
D: right ?
F: Gender - dependent , yeah .
D: So So I think it 's uh that 's about mmm , something like thirty hours .
F: Thirty hours .
D: Thirty hours per gender .
A: I 'm not sure what this 'll mean .
F: In the small training set ?
A: Hello ?
D: I I think so .
D: I 'll A: Excuse me ?
D: It 's definitely less than a hundred D: You know , it 's more like like thirty forty hours
A: Alright .
D: something like that .
A: Wrong number .
F: They called to tell us that ?
D: Yeah .
B: Yeah .
B: Um . So . Uh . after run
F: I mean , I didn't want to do too big ,
B: Right .
F: just B: So
B: At least a couple thousand hidden units .
B: I mean .
D: Mm - hmm .
B: It 's it 's th the thing I 'll I 'll think about it a little more
B: but it it 'd be toss up between two thousand and four thousand .
F: Mmm .
B: You definitely wouldn't want the eight thousand .
B: It 's m It 's more than F: And a thousand is too small ?
B: Oh let me think about it ,
B: but I think that that uh th at some point there 's diminishing returns .
F: Mm - hmm .
B: I mean it doesn't actually get worse , typically ,
A: Mm - hmm .
B: but it but but there is diminishing returns
B: and you 're doubling the amount of time .
D: Remember you 'll have a smaller output layer so there 's gonna be fewer parameters there .
A: But not by a lot .
B: Not by much .
D: And then B: Fifty s Fifty four to forty eight ?
A: Vast majority is from the input unit .
D: OK .
D: Yeah .
B: Yeah .
B: Yeah .
B: It 'll have a very tiny effect .
A: Right , because you used the context windows
A: and so the input to hidden is much , much larger .
B: Yeah .
D: Oh I see ,
D: I see ,
D: yeah , of course .
B: Yeah .
D: Yeah .
D: It 's negligible , OK .
B: Yeah , so it 's it 'd be way , way less than ten percent of the difference .
D: Uh - huh .
B: Uh . There 's uh B: How bi how big
B: Let 's see .
B: What am I trying to think of ?
F: The The net that that we did use already uh was eight thousand hidden units
F: and that 's the one that Eric trained up .
B: Right .
B: And that was trained up on uh like a hundred and forty hours of of speech .
D: Was that gender - dependent or independent ?
F: Gender - dependent .
B: Oh .
B: So that would be like trained on s sixty or seventy hours .
D: Mm - hmm .
B: So , uh ,
B: yeah definitely not the one thousand uh two thousand fr
B: I mean the four thousand will be better
B: and the two thousand will be almost will be faster and almost as good .
A: It 'll be faster .
F: Yeah .
B: So .
F: Maybe I 'll start off with two thousand just to see .
D: Mm - hmm .
B: Yeah .
D: OK .
F: OK .
B: Yeah , thirty hours is like a hundred and ten thousand uh seconds .
B: Uh , so that 's like eleven eleven million frames .
B: And a two thousand hidden unit net is uh I guess about seven , eight hundred thousand parameters .
B: So that 's probably That 's probably fine .
B: I mean a four thousand is well within the range that you could benefit from
B: but the two thousand 'd be faster
B: so D: Right .
D: I actually have to go .
D: So .
B: Alright .
B: Uncle Bernie 's rule is ten to one .
B: Bernie Woodrow 's Rule of B: yeah B: Uncle Bernie B: yeah .
B: Yes sir .
A: We 're just waiting for you to leave .
A: Anything else ?
B: Nah .
B: Since we have nothing to talk about we only talked for an hour .
A: OK .
B: So .
A: If A: yeah that 's right .
C: Yeah .
A: Uh , well , we started late .
F: Transcript
B: de - ba - de . de - ba - de
B: That 's all folks !