PhD C: Yeah . Right . We 've fff {comment} done some strange things like removing C - zero or C - one from the   the vector of parameters , and we noticed that C - one is almost not useful at all . You can remove it from the vector , it doesn't hurt .
PhD E: Really ? ! That has no effect ?
PhD C: Um .
PhD E: Eh Is this in the baseline ? or in uh PhD C: In the No , in the proposal .
PhD E: in uh - huh , uh - huh .
Professor B: So we were just discussing , since you mentioned that , in it w
PhD C: Mm - hmm .
Professor B: driving in the car with Morgan this morning , we were discussing a good experiment for b for beginning graduate student who wants to run a lot of who wants to get a lot of numbers on something
PhD C: Mm - hmm .
Professor B: which is , like , " imagine that you will you will start putting every co any coefficient , which you are using in your vector , in some general power .
PhD E: In some what ?
Professor B: General pow power . Like sort of you take a s power of two , or take a square root , or something .
PhD E: Mm - hmm .
PhD C: Mm - hmm .
Professor B: So suppose that you are working with a s C - zer C - one .
PhD E: Mm - hmm .
Professor B: So if you put it in a s square root , that effectively makes your model half as efficient . Because uh your uh Gaussian mixture model , right ? computes the mean .
PhD E: Mm - hmm .
Professor B: And and uh i i i but it 's the mean is an exponent of the whatever , the the this Gaussian function .
PhD E: You 're compressing the range ,
Professor B: So you 're compressing the range of this coefficient , so it 's becoming less efficient .
PhD E: right ? of that Professor B: Right ?
PhD E: Mm - hmm .
Professor B: So . So . Morgan was @ @ and he was he was saying well this might be the alternative way how to play with a with a fudge factor , you know , uh in the PhD E: Oh .
Professor B: you know , just compress the whole vector .
PhD E: Yeah .
Professor B: And I said " well in that case why don't we just start compressing individual elements , like when when because in old days we were doing when when people still were doing template matching and Euclidean distances , we were doing this liftering of parameters , right ?
PhD E: Uh - huh .
Professor B: because we observed that uh higher parameters were more important than lower for recognition . And basically the the C - ze C - one contributes mainly slope ,
PhD E: Right .
Professor B: and it 's highly affected by uh frequency response of the of the recording equipment and that sort of thing ,
PhD C: Mm - hmm .
PhD E: Mm - hmm .
Professor B: so so we were coming with all these f various lifters .
PhD E: Mm - hmm .
Professor B: uh Bell Labs had he this uh uh r raised cosine lifter which still I think is built into H HTK for reasons n unknown to anybody , but but uh we had exponential lifter , or triangle lifter , basic number of lifters .
PhD E: Hmm .
Professor B: And . But so they may be a way to to fiddle with the f with the f
PhD E: Insertions .
Professor B: Insertions , deletions , or the the giving a relative uh basically modifying relative importance of the various parameters .
PhD C: Mm - hmm .
Professor B: The only of course problem is that there 's an infinite number of combinations and if the if you s if y
PhD E: Oh . Uh - huh . You need like a some kind of a Professor B: Yeah , you need a lot of graduate students , and a lot of computing power .
PhD E: You need to have a genetic algorithm , that basically tries random permutations of these things .
Professor B: I know . Exactly . Oh . If you were at Bell Labs or I d d I shouldn't be saying this in on on a mike , right ? Or I uh IBM , that 's what maybe that 's what somebody would be doing .
PhD E: Yeah .
Grad A: Hmm .
Professor B: Oh , I mean , I mean the places which have a lot of computing power , so because it is really it 's a p it 's a it 's it will be reasonable search
PhD E: Mm - hmm . Yeah .
Professor B: uh but I wonder if there isn't some way of doing this uh search like when we are searching say for best discriminants .
PhD E: You know actually , I don't know that this wouldn't be all that bad . I mean you you compute the features once ,
Professor B: Yeah . Yeah .
PhD E: right ? And then these exponents are just applied to that Professor B: Absolutely . And hev everything is fixed .
PhD E: So .
Professor B: Everything is fixed . Each each PhD E: And is this something that you would adjust for training ? or only recognition ?
Professor B: For both , you would have to do . Yeah .
PhD E: You would do it on both .
Professor B: You have to do bo both .
PhD E: So you 'd actually Professor B: Because essentially you are saying " uh this feature is not important " .
PhD E: Mm - hmm .
Professor B: Or less important , so that 's th that 's a that 's a painful one , yeah .
PhD E: So for each uh set of exponents that you would try , it would require a training and a recognition ?
Professor B: Yeah . But but wait a minute . You may not need to re uh uh retrain the m model . You just may n may need to c uh give uh less weight to to uh a mod uh a component of the model which represents this particular feature . You don't have to retrain it .
PhD E: Oh . So if you Instead of altering the feature vectors themselves , you you modify the the the Gaussians in the models .
Professor B: You just multiply . Yeah . Yep . You modify the Gaussian in the model , but in the in the test data you would have to put it in the power , but in a training what you c in a training uh in trained model , all you would have to do is to multiply a model by appropriate constant .
PhD E: Uh - huh . But why if you 're if you 're multi if you 're altering the model , why w in the test data , why would you have to muck with the uh cepstral coefficients ?
Professor B: Because in uh test in uh test data you ca don't have a model . You have uh only data . But in a in a tr
PhD E: No . But you 're running your data through that same model .
Professor B: That is true , but w I mean , so what you want to do You want to say if uh obs you if you observe something like Stephane observes , that C - one is not important , you can do two things .
PhD E: Mm - hmm . Mm - hmm .
Professor B: If you have a trained trained recognizer , in the model , you know the the the the component which I I mean di dimension wh
PhD E: Mm - hmm . All of the all of the mean and variances that correspond to C - one , you put them to zero .
Professor B: To the s you you know it . But what I 'm proposing now , if it is important but not as important , you multiply it by point one in a model .
PhD E: Yeah .
Professor B: But but but PhD E: But what are you multiplying ? Cuz those are means , right ?
Grad A: You 're multiplying the standard deviation ?
PhD E: I mean you 're Grad A: So it 's Professor B: I think that you multiply the I would I would have to look in the in the math , I mean how how does the model uh PhD E: I think you Professor B: Yeah .
PhD E: Yeah , I think you 'd have to modify the standard deviation or something , so that you make it wider or narrower .
Grad A: Cuz Yeah .
Professor B: Yeah .
PhD C: Yeah .
Professor B: Effectively , that 's that that 's I Exactly . That 's what you do . That 's what you do , you you you modify the standard deviation as it was trained .
Grad A: Yeah .
Professor B: Effectively you , you know y in f in front of the of the model , you put a constant . S yeah effectively what you 're doing is you is you are modifying the the the deviation . Right ?
Grad A: The spread ,
PhD E: Oop .
Grad A: right .
PhD E: Sorry .
Professor B: Yeah , the spread .
Grad A: It 's the same same mean ,
PhD E: So .
Grad A: right ?
Professor B: And and and PhD E: So by making th the standard deviation narrower , {comment} uh your scores get worse for Professor B: Yeah .
PhD E: unless it 's exactly right on the mean .
Professor B: Your als No . By making it narrower ,
PhD E: Right ?
Professor B: uh y your PhD E: I mean there 's you 're you 're allowing for less variance .
Grad A: Mm - hmm .
Professor B: Yes , so you making this particular dimension less important . Because see what you are fitting is the multidimensional Gaussian , right ?
PhD E: Mm - hmm .
Professor B: It 's a it has it has uh thirty - nine dimensions , or thirteen dimensions if you g ignore deltas and double - deltas .
PhD E: Mm - hmm . Mm - hmm .
Professor B: So in order if you in order to make dimension which which Stephane sees uh less important , uh uh I mean not not useful , less important , what you do is that this particular component in the model you can multiply by w you can you can basically de - weight it in the model . But you can't do it in a in a test data because you don't have a model for th I mean uh when the test comes , but what you can do is that you put this particular component in and and you compress it . That becomes uh th gets less variance , subsequently becomes less important .
PhD E: Couldn't you just do that to the test data and not do anything with your training data ?
Professor B: That would be very bad , because uh your t your model was trained uh expecting uh , that wouldn't work . Because your model was trained expecting a certain var variance on C - one .
PhD E: Uh - huh .
Professor B: And because the model thinks C - one is important . After you train the model , you sort of y you could do you could do still what I was proposing initially , that during the training you you compress C - one that becomes then it becomes less important in a training .
PhD E: Mm - hmm .
Professor B: But if you have if you want to run e ex extensive experiment without retraining the model , you don't have to retrain the model . You train it on the original vector . But after , you wh when you are doing this parametric study of importance of C - one you will de - weight the C - one component in the model , and you will put in the you will compress the this component in a in the test data . s by the same amount .
PhD E: Could you also if you wanted to if you wanted to try an experiment uh by leaving out say , C - one , couldn't you , in your test data , uh modify the all of the C - one values to be um way outside of the normal range of the Gaussian for C - one that was trained in the model ? So that effectively , the C - one never really contributes to the score ?
PhD C: Mm - hmm .
Professor B: No , that would be a severe mismatch ,
PhD E: Do you know what I 'm say
Professor B: right ? what you are proposing ? N no you don't want that .
PhD E: Yeah , someth
Professor B: Because that would then your model would be unlikely . Your likelihood would be low , right ? Because you would be providing severe mismatch .
PhD E: Mm - hmm . But what if you set if to the mean of the model , then ? And it was a cons you set all C - ones coming in through your test data , you you change whatever value that was there to the mean that your model had .
Professor B: No that would be very good match , right ?
PhD E: Yeah .
Professor B: That you would PhD C: Which Well , yeah , but we have several means . So .
Professor B: I see what you are sa saying ,
PhD C: Right ?
Grad A: Saying .
Professor B: but uh , no , no I don't think that it would be the same .  I mean , no , the If you set it to a mean , that would No , you can't do that . Y you ca you ca Ch - Chuck , you can't do that .
PhD E: Oh , that 's true , right , yeah , because you you have PhD C: Wait . Which Professor B: Because that would be a really f fiddling with the data ,
PhD E: Yeah .
Professor B: you can't do that .
PhD E: Mm - hmm . Mm - hmm .
Professor B: But what you can do , I 'm confident you ca
PhD E :
Professor B: well , I 'm reasonably confident and I putting it on the record , right ? I mean y people will listen to it for for centuries now , is what you can do , is you train the model uh with the with the original data .
Grad A: Mm - hmm .
Professor B: Then you decide that you want to see how important C C - one is . So what you will do is that a component in the model for C - one , you will divide it by by two . And you will compress your test data by square root .
PhD E: Mm - hmm .
Professor B: Then you will still have a perfect m match . Except that this component of C - one will be half as important in a in a overall score .
PhD E: Mm - hmm . Mm - hmm .
Professor B: Then you divide it by four and you take a square , f fourth root . Then if you think that some component is more is more important then th th th it then then uh uh i it is , based on training , then you uh multiply this particular component in the model by by by PhD E: You 're talking about the standard deviation ?
Professor B: yeah .
PhD E: Yeah .
Professor B: Yeah , multiply this component uh i it by number b larger than one ,
PhD E: Mm - hmm .
Professor B: and you put your data in power higher than one . Then it becomes more important . In the overall score , I believe .
PhD C: Yeah , but , at the PhD E: But don't you have to do something to the mean , also ?
Professor B: No .
PhD C: No .
Grad A: Yeah .
Professor B: No .
PhD C: But I think it 's uh the The variance is on on the denominator in the in the Gaussian equation . So . I think it 's maybe it 's the contrary . If you want to decrease the importance of a c parameter , you have to increase it 's variance .
Professor B: Yes . Right . Yes .
PhD D: Multiply .
Professor B: Exactly . Yeah . So you so you may want to do it other way around ,
PhD C: Hmm . That 's right . OK .
Professor B: yeah .
PhD C: Mm - hmm .
Grad A: Right .
PhD E: But if your If your um original data for C - one had a mean of two .
Professor B: Uh - huh .
PhD E: And now you 're you 're you 're changing that by squaring it . Now your mean of your C - one original data has {comment} is four . But your model still has a mean of two . So even though you 've expended the range , your mean doesn't match anymore .
PhD C: Mm - hmm .
Professor B: Let 's see .
PhD E: Do you see what I mean ?
PhD C: I think What I see What could be done is you don't change your features , which are computed once for all ,
Professor B: Uh - huh .
PhD C: but you just tune the model . So . You have your features . You train your your model on these features .
PhD E: Mm - hmm .
PhD C: And then if you want to decrease the importance of C - one you just take the variance of the C - one component in the in the model and increase it if you want to decrease the importance of C - one or decrease it PhD E: Yeah .
Professor B: Yeah .
PhD E: Right .
Professor B: Yeah . You would have to modify the mean in the model . I you I agree with you . Yeah . Yeah , but I mean , but it 's it 's i it 's do - able ,
PhD C: Well .
PhD E: Yeah , so y
Professor B: right ? I mean , it 's predictable . Uh . Yeah .
PhD E: It 's predictable , yeah .
Professor B: Yeah . Yeah , it 's predictable .
PhD C: Mmm .
PhD E: Yeah . But as a simple thing , you could just just muck with the variance .
PhD C: Just adjust the model , yeah .
PhD E: to get uh this uh this the effect I think that you 're talking about ,
Professor B: Mm - hmm .
PhD E: right ?
Professor B: It might be .
PhD E: Could increase the variance to decrease the importance .
Professor B: Mm - hmm .
PhD C: Mm - hmm .
PhD E: Yeah , because if you had a huge variance , you 're dividing by a large number , {comment} you get a very small contribution .
Grad A: Doesn't matter PhD C: Yeah , it becomes more flat
Grad A: Right .
PhD C: and Professor B: Yeah .
PhD C: Yeah .
PhD E: Yeah .
Grad A: Yeah , the sharper the variance , the more more important to get that one right .
PhD E: Hmm .
Professor B: Mm - hmm .
PhD E: Yeah , you know actually , this reminds me of something that happened uh when I was at BBN . We were playing with putting um pitch into the Mandarin recognizer .
Professor B: Mm - hmm .
PhD E: And this particular pitch algorithm um when it didn't think there was any voicing , was spitting out zeros . So we were getting uh when we did clustering , we were getting groups uh of features
Professor B: p Pretty new outliers , interesting outliers , right ?
PhD E: yeah , with with a mean of zero and basically zero variance .
Professor B: Variance .
PhD E: So , when ener {comment} when anytime any one of those vectors came in that had a zero in it , we got a great score . I mean it was just , {nonvocalsound} you know , incredibly {nonvocalsound} high score , and so that was throwing everything off .
PhD C: Mm - hmm .
PhD E: So if you have very small variance you get really good scores when you get something that matches .
Professor B: Yeah .
PhD C: Mm - hmm .
PhD E: So . So that 's a way , yeah , yeah That 's a way to increase the yeah , n That 's interesting . So in fact , that would be That doesn't require any retraining .
Professor B: Yeah . No . No .
PhD C: No , that 's right . So it 's
PhD E: So that means it 's just
Professor B: Yeah .
PhD C: just tuning the models and testing , actually .
PhD E: recognitions .
Professor B: Yeah .
PhD E: Yeah .
PhD C: It would be quick .
PhD E: You you have a step where you you modify the models , make a d copy of your models with whatever variance modifications you make , and rerun recognition .
Professor B: Yeah .
PhD C: Mm - hmm .
Professor B: Yeah . Yeah . Yeah .
PhD E: And then do a whole bunch of those .
Professor B: Yeah .