Professor D: Um , by the way , there is a another , uh , suggestion that would apply , uh , to the second configuration , um , which , uh , was made , uh , by , uh , Hari . And that was that , um , if you have uh feed two streams into HTK , um , and you , uh , change the , uh variances if you scale the variances associated with , uh these streams um , you can effectively scale the streams . Right ? So , um , you know , without changing the scripts for HTK , which is the rule here , uh , you can still change the variances
PhD A: Mm - hmm .
Professor D: which would effectively change the scale of these these , uh , two streams that come in .
PhD A: Uh , {comment} yeah .
Professor D: And , um , so , um , if you do that , for instance it may be the case that , um , the MLP should not be considered as strongly , for instance .
PhD A: Mmm .
Professor D: And , um , so this is just setting them to be , excuse me , of equal equal weight . Maybe it shouldn't be equal weight .
PhD B: Maybe .
Professor D: Right ? You know , I I 'm sorry to say that gives more experiments if we wanted to look at that , but but , uh , um , you know on the other hand it 's just experiments at the level of the HTK recognition .
PhD A: Mmm .
Professor D: It 's not even the HTK ,
PhD A: Yeah .
Professor D: uh , uh PhD B: Yeah . Yeah .
Professor D: Well , I guess you have to do the HTK training also .
PhD B: so this is what we decided to do .
Professor D: Uh , do you ? Let me think . Maybe you don't . Uh . Yeah , you have to change the No , you can just do it in as once you 've done the training Grad C: And then you can vary it . Yeah .
Professor D: Yeah , the training is just coming up with the variances so I guess you could you could just scale them all .
PhD A: Scale
Professor D: Variances .
PhD A: Yeah . But Is it i th I mean the HTK models are diagonal covariances , so I d Is it Professor D: That 's uh , exactly the point , I think , that if you change um , change what they are PhD A: Hmm . Mm - hmm .
Professor D: It 's diagonal covariance matrices , but you say what those variances are .
PhD A: Mm - hmm .
Professor D: So , that you know , it 's diagonal , but the diagonal means th that then you 're gonna it 's gonna it 's gonna internally multiply it and and uh , uh , i it im uh implicitly exponentiated to get probabilities , and so it 's it 's gonna it 's it 's going to affect the range of things if you change the change the variances of some of the features .
PhD A: Mmm . Mmm .
PhD B: do ?
Professor D: So , i it 's precisely given that model you can very simply affect , uh , the s the strength that you apply the features . That was that was , uh , Hari 's suggestion .
PhD A: Yeah . Yeah .
Professor D: So , um PhD B: Yeah .
Professor D: Yeah . So . So it could just be that h treating them equally , tea treating two streams equally is just just not the right thing to do . Of course it 's potentially opening a can of worms because , you know , maybe it should be a different number for for each kind of test set , or something ,
PhD A: Mm - hmm .
Professor D: but OK .
PhD A: Yeah .
Professor D: So I guess the other thing is to take you know if one were to take , uh , you know , a couple of the most successful of these ,
PhD A: Yeah , and test across everything .
Professor D: and uh Yeah , try all these different tests .
PhD A: Mmm .
PhD B: Yeah .
PhD A: Yeah .
Professor D: Alright . Uh .
PhD A: So , the next point , yeah , we 've had some discussion with Steve and Shawn , um , about their um , uh , articulatory stuff , um . So we 'll perhaps start something next week .
Professor D: Mm - hmm .
PhD A: Um , discussion with Hynek , Sunil and Pratibha for trying to plug in their our our networks with their within their block diagram , uh , where to plug in the the network , uh , after the the feature , before as um a as a plugin or as a anoth another path , discussion about multi - band and TRAPS , um , actually Hynek would like to see , perhaps if you remember the block diagram there is , uh , temporal LDA followed b by a spectral LDA for each uh critical band . And he would like to replace these by a network which would , uh , make the system look like a TRAP . Well , basically , it would be a TRAP system . Basically , this is a TRAP system kind of TRAP system , I mean , but where the neural network are replaced by LDA . Hmm . Um , yeah , and about multi - band , uh , I started multi - band MLP trainings , um mmh {comment} Actually , I w I w hhh {comment} prefer to do exactly what I did when I was in Belgium . So I take exactly the same configurations , seven bands with nine frames of context , and we just train on TIMIT , and on the large database , so , with SPINE and everything . And , mmm , I 'm starting to train also , networks with larger contexts . So , this would would be something between TRAPS and multi - band because we still have quite large bands , and but with a lot of context also . So Um Yeah , we still have to work on Finnish , um , basically , to make a decision on which MLP can be the best across the different languages . For the moment it 's the TIMIT network , and perhaps the network trained on everything . So . Now we can test these two networks on with with delta and large networks . Well , test them also on Finnish
PhD B: Mmm .
PhD A: and see which one is the the the best . Uh , well , the next part of the document is , well , basically , a kind of summary of what everything that has been done . So . We have seventy - nine M L Ps trained on one , two , three , four , uh , three , four , five , six , seven ten on ten different databases .
Professor D: Mm - hmm .
PhD A: Uh , the number of frames is bad also , so we have one million and a half for some , three million for other , and six million for the last one . Uh , yeah ! {comment} As we mentioned , TIMIT is the only that 's hand - labeled , and perhaps this is what makes the difference . Um . Yeah , the other are just Viterbi - aligned . So these seventy - nine MLP differ on different things . First , um with respect to the on - line normalization , there are that use bad on - line normalization , and other good on - line normalization . Um . With respect to the features , with respect to the use of delta or no , uh with respect to the hidden layer size and to the targets . Uh , but of course we don't have all the combination of these different parameters Um . s What 's this ? We only have two hundred eighty six different tests And no not two thousand .
Professor D: Ugh ! I was impressed boy , two thousand .
PhD A: Yeah .
PhD B: Ah , yes .
Professor D: OK .
PhD B: I say this morning that @ @ thought it was the Professor D: Alright , now I 'm just slightly impressed , OK .
PhD A: Um . Yeah , basically the observation is what we discussed already . The MSG problem , um , the fact that the MLP trained on target task decreased the error rate . but when the M - MLP is trained on the um is not trained on the target task , it increased the error rate compared to using straight features . Except if the features are bad uh , actually except if the features are not correctly on - line normalized . In this case the tandem is still better even if it 's trained on not on the target digits .
Professor D: Yeah . So it sounds like yeah , the net corrects some of the problems with some poor normalization .
PhD A: Yeah .
Professor D: But if you can do good normalization it 's it 's uh OK .
PhD A: Yeah .
PhD B: Yeah .
PhD A: Uh , so the fourth point is , yeah , the TIMIT plus noise seems to be the training set that gives better the best network .
Professor D: So So - Let me bef before you go on to the possible issues .
PhD A: Mm - hmm .
Professor D: So , on the MSG uh problem um , I think that in in the um , in the short time solution um , that is , um , trying to figure out what we can proceed forward with to make the greatest progress ,
PhD A: Mm - hmm .
Professor D: uh , much as I said with JRASTA , even though I really like JRASTA and I really like MSG ,
PhD A: Mm - hmm .
Professor D: I think it 's kind of in category that it 's , it it may be complicated .
PhD A: Yeah .
Professor D: And uh it might be if someone 's interested in it , uh , certainly encourage anybody to look into it in the longer term , once we get out of this particular rush uh for results .
PhD A: Mm - hmm .
Professor D: But in the short term , unless you have some some s strong idea of what 's wrong ,
PhD A: I don't know at all but I 've perhaps I have the feeling that it 's something that 's quite quite simple or just like nnn , no high - pass filter
Professor D: Yeah , probably .
PhD A: or Mmm . Yeah . My But I don't know .
Professor D: There 's supposed to well MSG is supposed to have a an on - line normalization though , right ?
PhD A: It 's There is , yeah , an AGC - kind of AGC . Yeah . Yeah . Yeah .
Professor D: Yeah , but also there 's an on - line norm besides the AGC , there 's an on - line normalization that 's supposed to be uh , yeah ,
PhD A: Mmm .
Professor D: taking out means and variances and so forth . So .
PhD A: Yeah .
Professor D: In fac in fact the on - line normalization that we 're using came from the MSG design ,
PhD A: Um .
Professor D: so it 's PhD A: Yeah , but Yeah . But this was the bad on - line normalization . Actually . Uh . Are your results are still with the bad the bad PhD B: Maybe , may No ? With the better PhD A: With the O - OLN - two ?
PhD B: No ?
PhD A: Ah yeah , you have you have OLN - two ,
PhD B: Oh ! Yeah , yeah , yeah ! With " two " , with " on - line - two " .
PhD A: yeah .
PhD B: Yeah , yeah ,
Professor D: " On - line - two " is good .
PhD A: So it 's , is the good yeah .
PhD B: yeah . Yep , it 's a good .
Professor D: " Two " is good ?
PhD A: And Professor D: No , " two " is bad .
PhD A: Yeah .
PhD B: Well , actually , it 's good with the ch with the good .
Professor D: OK . Yeah . So Yeah , I I agree . It 's probably something simple uh , i if if uh someone , you know , uh , wants to play with it for a little bit . I mean , you 're gonna do what you 're gonna do
PhD A: Mmm .
Professor D: but but my my guess would be that it 's something that is a simple thing that could take a while to find .
PhD A: But Yeah . Mmm . I see , yeah .
Professor D: Yeah .
PhD A: And Professor D: Uh . {comment} And the other the results uh , observations two and three , Um , is
PhD A: Mmm .
Professor D: uh Yeah , that 's pretty much what we 've seen . That 's that what we were concerned about is that if it 's not on the target task If it 's on the target task then it it it helps to have the MLP transforming it .
PhD A: Mmm .
Professor D: If it uh if it 's not on the target task , then , depending on how different it is , uh you can get uh , a reduction in performance .
PhD A: Mmm .
Professor D: And the question is now how to how to get one and not the other ? Or how to how to ameliorate the the problems .
PhD A: Mmm .
Professor D: Um , because it it certainly does is nice to have in there , when it  when there is something like the training data .
PhD A: Mm - hmm . Um . Yeah . So , the the reason Yeah , the reason is that the perhaps the target the the task dependency the language dependency , and the noise dependency Professor D: So that 's what you say th there . I see .
PhD A: Well , the e e But this is still not clear because , um , I I I don't think we have enough result to talk about the the language dependency . Well , the TIMIT network is still the best but there is also an the other difference , the fact that it 's it 's hand - labeled .
Professor D: Hey ! Um , just you can just sit here . Uh , I d I don't think we want to mess with the microphones but it 's uh Just uh , have a seat . Um . s Summary of the first uh , uh forty - five minutes is that some stuff work and works , and some stuff doesn't OK ,
PhD A: We still have uh this One of these perhaps ?
PhD B: Yeah .
PhD A: Mm - hmm .
Professor D: Yeah , I guess we can do a little better than that but  I think if you if you start off with the other one , actually , that sort of has it in words and then th that has it the associated results .
PhD B: Um .
Professor D: OK . So you 're saying that um , um , although from what we see , yes there 's what you would expect in terms of a language dependency and a noise dependency . That is , uh , when the neural net is trained on one of those and tested on something different , we don't do as well as in the target thing . But you 're saying that uh , it is Although that general thing is observable so far , there 's something you 're not completely convinced about . And and what is that ? I mean , you say " not clear yet " . What what do you mean ?
PhD A: Uh , mmm , uh , {comment} I mean , that the the fact that s Well , for for TI - digits the TIMIT net is the best ,  which is the English net .
Professor D: Mm - hmm .
PhD A: But the other are slightly worse . But you have two two effects , the effect of changing language and the effect of training on something that 's Viterbi - aligned instead of hand hand - labeled .
PhD B: Yeah .
PhD A: So . Um . Yeah .
Professor D: Do you think the alignments are bad ? I mean , have you looked at the alignments at all ? What the Viterbi alignment 's doing ?
PhD A: Mmm . I don't I don't know . Did - did you look at the Spanish alignments Carmen ?
PhD B: Mmm , no .
Professor D: Might be interesting to look at it . Because , I mean , that is just looking but um , um It 's not clear to me you necessarily would do so badly from a Viterbi alignment . It depends how good the recognizer is
PhD A: Mm - hmm .
Professor D: that 's that the the engine is that 's doing the alignment .
PhD A: Yeah . But Yeah . But , perhaps it 's not really the the alignment that 's bad but the just the ph phoneme string that 's used for the alignment
Professor D: Aha !
PhD A: Mmm .
PhD B: Yeah .
Professor D: The pronunciation models and so forth
PhD A: I mean for We It 's single pronunciation , uh Professor D: Aha .
PhD A: French French s uh , phoneme strings were corrected manually
Professor D: I see .
PhD A: so we asked people to listen to the um the sentence and we gave the phoneme string and they kind of correct them . But still ,  there there might be errors just in the in in the ph string of phonemes . Mmm . Um . Yeah , so this is not really the Viterbi alignment ,  in fact , yeah . Um , the third The third uh issue is the noise dependency perhaps but , well , this is not clear yet because all our nets are trained on the same noises and Professor D: I thought some of the nets were trained with SPINE and so forth . So it And that has other noise .
PhD A: Yeah . So Yeah . But Yeah . Results are only coming for for this net . Mmm .
Professor D: OK , yeah , just don't just need more more results there with that @ @ .
PhD A: Yeah . Um . So . Uh , from these results we have some questions with answers . What should be the network input ? Um , PLP work as well as MFCC , I mean . Um . But it seems impor important to use the delta . Uh , with respect to the network size , there 's one experiment that 's still running and we should have the result today , comparing network with five hundred and one thousand units . So , nnn , still no answer actually .
Professor D: Hm - hmm .
PhD A: Uh , the training set , well , some kind of answer . We can , we can tell which training set gives the best result , but we don't know exactly why . Uh , so .
Professor D: Uh . Right , I mean the multi - English so far is is the best .
PhD A: Yeah .
Professor D: " Multi - multi - English " just means " TIMIT " ,
PhD A: Yeah .
Professor D: right ?
PhD B: Yeah .
Professor D: So uh That 's Yeah . So . And and when you add other things in to to broaden it , it gets worse uh typically .
PhD A: Mmm . Mm - hmm .
Professor D: Yeah .
PhD A: Then uh some questions without answers .
Professor D: OK .
PhD A: Uh , training set , um ,
Professor D: Uh - huh .
PhD A: uh , training targets Professor D: I like that . The training set is both questions , with answers and without answers .
PhD A: It 's Yeah . Yeah .
Professor D: It 's sort of , yes it 's mul it 's multi - uh - purpose .
PhD A: Yeah .
Professor D: OK .
PhD A: Uh , training s Right . So Yeah , the training targets actually , the two of the main issues perhaps are still the language dependency and the noise dependency . And perhaps to try to reduce the language dependency , we should focus on finding some other kind of training targets .
Professor D: Mm - hmm .
PhD A: And labeling s labeling seems important uh , because of TIMIT results .
Professor D: Mm - hmm .
PhD A: Uh . For moment you use we use phonetic targets but we could also use articulatory targets , soft targets , and perhaps even , um use networks that doesn't do classification but just regression so uh , train to have neural networks that um , um , uh ,
Professor D: Mm - hmm .
PhD A: does a regression and well , basically com com compute features and noit not , nnn , features without noise . I mean uh , transform the fea noisy features in other features that are not noisy . But continuous features . Not uh uh , hard targets .
Professor D: Mm - hmm . Mm - hmm .
PhD A: Uh