A: And pick whichever 's better .
I: We would actually we would run some sort of segmentation .
I: Thilo 's is as good as any , probably .
I: Um , and then we would do an unsupervised clustering of of the segments , to I: and and put the similar ones into bins that would be sort of pseudo - speakers .
C: Mm - hmm .
I: And then we would do our standard processing on these pseudo - speakers .
I: And that turns out to work very well on Broadcast News , SPINE I: those types of tasks , where you don't have the speaker segmentation given to you .
A: S does the clustering ? Do you give it sort of a target number of clusters ? Or is it ? <comment> adapted in some way ?
I: Um , you can either do it by target number or by some measure of dissimilarity that you use as a threshold .
A: Mm - hmm .
A: That 's what I 'm just thinking one of the big differences with Broadcast News and these meetings is we have m many fewer participants .
H: The other thing is that you actually have direction here .
I: Right .
H: So , unlike these corpora that are recorded with other microphones , like The right way to do this I guess , w you know , in the future would be , well , in general , Thilo 's sitting there , and this PZM is gonna You know , he 's roughly in that location .
A: Speaker ID .
A: Yeah .
A: Well , there 're different ways of thinking about it .
A: I mean that that would be true if w you had a meeting situation with multiple mikes .
A: But if you only had your PDA sitting in front of you H: Well , any case where the people are not all sitting at the same place and they 're not moving around too much .
A: And you have more than one mike .
C: Yeah . If you don't have one more than one mike , you don't have a very good handle on location .
H: W well , you have distance , and you have C: That 's it 's not enough .
H: I mean , that , um , Jane 's i n The pickup of Adam on this mike is gonna be different than me , in terms of energy and so forth over the whole meeting .
A: Oh , so just from clustering . You might be able to cluster it better because of that .
H: You might get some clustering from the speaker and some of it from the characteristics of the distance ,
C: Mmm .
A: From mike .
A: Yeah .
C: Yeah .
H: and C: But , say , if if you had a a cardioid mike or something sitting someplace , then  sitting there , then it would Its its response to him would be about the same as the response to him , and so on .
A: And transfer functions .
H: Right . Exactly .
I: Well , you can do  You can you can you can do certain normalizations like , you know , gain control ,
A: Well , I think there 're lots of lots of ways of doing it .
H: They 're both picked up in the clustering , I guess .
C: Mm - hmm .
I: uh , before you do the clustering to rule out those those types of things .
H: Or to just do the clustering , knowing that you 're capturing both .
H: It 's just that the kind of clustering we 've done before hasn't had that , uh , distance factor , or location factor in it in the same way .
C: I see .
C: Yeah .
H: And so we 're not really modeling it directly .
H: If somebody does , we sh maybe we add that
I: Mmm .
A: Yeah . That 's an interesting H: because I think it would be a pretty big difference .
H: When you listen , you can sort of tell where people are ,
A: It 's a big difference .
A: Yeah .
I: OK . That would be fun fun to try .
H: not which s you know , side ,
H: but A: Well , humans are really good at that A: transfer function through the head , and things like that .
H: Right . Even with one microphone .
A: So you know , even if you only have one ear , you can still get get good transfers .
C: Yeah .
C: Yeah .
H: Right . So our our clustering is not gonna be intelligent that way .
A: So
H: It 's just gonna pick up whatever energy difference , or whatever , is A: Yeah .
C: But Anyway , I I 'd it 'd be neat to have that ,
C: because you know we 've been at this for a little while
C: and we don't have have any results yet with with conversational speech at a distance .
I: Mm - hmm .
I: OK .
C: So , um I: Hmm .
A: Yeah .
C: We should at least get a first one .
A: Something .
A: Yeah .
H: OK .
F: Mm - hmm .
C: Um , and the other thing This would kind of be a Hail Mary ,
C: but but , uh  Uh , Dave does have this stuff that is helping on digits ,
C: and , you know , and so with then it 'd be , you know , just throw that in and see .
I: Oh , yeah . Then we should A: Yeah . So it 'd be cool to see if it helped .
I: Well , first you have to filter the whole training set and retrain .
H: Yeah .
E: Mm - hmm .
C: Yeah .
A: That would be ft quick ,
H: Mm - hmm .
A: since I think he did it in Matlab .
C: Uh , well , he can do it in something else .
F: Hmm . Interesting .
A: Yep .
C: But , I mean  you know , it 's A: Can't you export C from Matlab ,
C: Um C: Actually , we 're experimenting with phase stuff now ,
A: or is that Mathematica ?
C: and and , uh , thi this , uh first result he got , uh , was really great .
C: It actually , uh uh  didn't exactly eliminate the reverberation ,
C: but it completely got rid of the speech .
I: Oh .
I: That would solve all of our problems .
F: Wow .
C: Yeah . Well , I was thinking that .
I: Wouldn't it ?
A: Well , so just take the inverse and you 're fine .
C: So , it 's C: That 's C: You think I didn't s tell him that ?
C: No . I got pretty excited ,
C: because it completely got rid of the speech .
C: So , I was thinking " well , so ,
A: So that 's a speech detector .
A: That 's great .
F: That 's interesting .
C: you know , it could be useful for lots of things " .
C: So , we have to F: Did it get rid of other stuff , too , though ?
I: Hmm .
C: What ?
F: Did it get rid of other stuff besides the speech ?
A: Yeah . Just subtract that C: Well , we have to sort of check that out .
A: subtract that from the original signal and you 're set .
C: That 's F: OK .
F: Wow . Interesting .
C: Yeah .
C: Yeah .
I: Right . Then you can do like an you can estimate the the noise estimates .
A: Noise estimate .
A: Signal to noise .
I: Right ? Yeah .
A: That 's great .
H: Mm - hmm .
C: Reminds me of when th when Herve and I were first playing with c uh , context - dependent things for nets ,
C: and and at one point we took out the speech input ,
C: so we only had priors , and our performance went up .
I: Mmm
F: Wow .
A: I guess that 's why Herve always talks about using the priors as one of the mixtures in in his all - ways combos .
I: Hmm .
C: S Yeah .
C: Well , of course it was a bug .
C: But , I mean , it 's it 's a but it was pretty f f It was pretty funny anyway .
A: But still F: Wow . Interesting .
I: Yeah .
F: Hmm .
A: So if you run er , your recognizer with all probabilities equal , what do you get out ?
A: Probably garbage .
B: Whatever the language model says .
A: I bet the pruning D: Yep .
A: The pruning probably prunes everything out .
I: You get out Switchboard .
I: That 's just the lang the language model .
A: Yeah . That 's right .
C: So that 's how it was generated .
I: Ha
I: Yeah . So we have this new speaker adaptation .
I: Um
I: A b Oh , it 's a s sort of feature normalization
I: t
I: uh , like f speaker adaptation ,
I: which , uh , which I wr which I wrote about in the last status report ,
I: which seems to be helping about a percent and a half on Hub - five .
I: So , um . We haven't tried that yet on the meetings ,
I: uh , but hopefully it 'll help there , too .
F: I wanna ask , um So , you know that the data I ' v I have upgraded it considerably .
F: So I 've probably made I probably corrected something like ,
F: well , s
F: It 's really a substantial amount of things that I 've caught , changed um , added to it .
F: Including a lot of , uh , backchannels .
F: So , when you 're d running things , if you run it on the old F: So . If you if you run it on the new version , then the numbers will be um , and you compare it to the to runs on the old version , then you 're gonna end up with more of an improvement than would actually be the case .
I: Hmm .
A: Different .
I: Mm - hmm .
I: Well , we we have a frozen we do all our experiments with a frozen version of the transcripts as of , I don't know
F: As of February ?
I: A a as of I: no , a little I: I don't know .
I: When when did we grab the transcripts ?
F: So a l Like the HLT paper ?
E: The b for the f ? It was m m March , probably .
H: F Sorry .
I: The H: For these meetings ?
I: We 're talking about which which version we 're using for evaluating the recognition .
I: Which version of the transcripts .
H: Right . They 're somewhere in between January and late March , or something like that ?
E: Yeah . Something around there .
F: OK . Well , so long as you have the same baseline , then you 'll be able to tell .
A: But they are channelized ones , though ?
I: No , no .
I: Yeah . O obviously . Yeah .
H: N
H: N
H: They 're meetings that are now channelized , but they were not from F: So long as i so long as it 's the same baseline you 'll be able to tell .
I: The uh , the other thing is I: Th - the the F: But but I 'm just gonn I 'm just saying that if you were to compare that with running that on the new data , that it would be an a more optimistic outcome .
I: Hmm . And I: Well , the and the other thing is , it takes only a a minute to rescore all the old outputs with I: If you had new transcripts , then we j we just re rescore the old A: Right . Cuz you haven't done any training .
I: Sorry ?
A: Right . Cuz we 're not doing it for training .
H: Yeah . We haven't a modified the recognizer at all .
A: Right . So it 's really it would be really easy to re - do it .
H: So , actually , um , at some point we should update and rescore everything with , you know , the corrected transcripts ,
I: W we just We save A: Yeah . It 'd be interesting just to see i how much it changes .
I: Well , the the H: just to h
A: I bet it wouldn't change a lot .
I: Right .
E: What are the nature of most of the changes ?
F: Yeah .
A: We can take we can have a pool .
H: Sometimes the changes are , um , cases where the recognizer would get it wrong anyway ,
H: cuz it was some word that we didn't have in the vocabulary ,
A: That 's just what I was thinking .
I: So , I I The thing is , when H: or H: But it does help to get the backchannels back in , and things like that .
I: Right .
I: So , whenever the Right now , the s the scoring is based on segments .
I: Um , which is not great
I: because , for instance I: So so , the the <nonvocalsound> other way to do the scoring is using a a NIST format called STM . S
I: uh , Segment Time Marked ,
A: Yep . I we know about it .
I: where I: So , um , I have to convert the , uh , transcripts into this format , and then the scoring program actually looks at the times .
I: And , uh , you know , it You can have a different segmentation in your recognizer output than in your references ,
I: and it will still do the right thing .
I: So , that 's what we need to basically , uh , to A: Tran
I: Hmm ?
A: Transcriber will export STM .
I: Oh .
A: In case you care .
I: Well . But then there 's other changes .
I: So . I mean , there 's other A: OK .
I: We we strip away a lot of the mark - up , uh , in the transcripts ,
I: which , you know , isn't relevant to the scoring of the speech recognition output .
A: Right .
I: So
H: Would that ? Doesn't that only change the scoring if a word has moved into a different segment ?
H: I mean , I don't think that 's har I hardly ever see that .
H: I think most of them are pretty good .
I: No . I mean if suppo
I: I I assume you also changed some boundaries .
F: Mmm . I did .
I: Right ?
I: So , if we want to use new transcripts with a different segmentation , then we can't use them in the current way we do scoring .
I: We have to m m switch to this o
H: Oh , you mean you would need to re - run the recognition .
I: No . We have to r
I: There 's a difference H: I mean , if you re - ran the recognition , then you just run it I: Right . Right .
H: Oh , I see .
H: If you wanna use the old H: You can actually never , though , really infer what you would get with a different I: That 's true .
H: You know ? It 's it 's probably more fair to re - run the whole th re - run that .
H: Uh , in other words , it 's not really a scoring script problem .
H: It 's I: No . But if you w just want to see what I: Like , suppose you fixed some typ No , Jane fixed some typos and you wanna see what effect does that have on the word error ,
I: then we can f we c
A: Right . But but If the segments change , that won't work .
H: Y Yeah . Yeah .
H: You It 'll sort of work , but it 's not exactly what you would maybe get from recognition .
A: Well , I mean , what happens if you break one segment into two ?
H: You never know .
A: Suddenly the don they don't match at all and you can't line them up anymore .
I: No , but that 's what the pr
I: Well , that 's what I 'm saying .
H: Well , you c You you s
I: You can line them up based on the times .
H: You can H: Yeah .
I: So the scoring program with if you give it an STM reference file , it will actually compare the words based on their time marks .
A: Right .
H: Yeah .
I: So therefore , you can ,
I: um A: Does STM do it per word , or por per utterance ?
I: It 's per utterance ,
I: but it it allows As long as you hypothesize the word in the right segment in the reference , it gives you credit for that .
A: Th - what I thought .
A: Alright .
I: So it does a  it does a word alignment , like you have to do for scoring ,
H: Mm - hmm .
A: And then it does also some I: but it constrains the words to lie within the t the time bins of the reference .
A: Within the segment .
A: Yep .
A: I see .
I: And fo for you to get a credit for it .
I: So .
F: That sounds great .
I: So so , it 's it should be just a straightforward re - formatting issue of the references .
I: So .
F: Yeah . I mean , t y I was thinking , th the other day , that this is not just conversational speech .
F: The fact that is has so much technological jargon in it . I it makes it u considerably harder to , um me e e e to transcribe and to to double check and all those things .
F: So , I think you 're gonna find a substantial gain in terms of the word ac accuracy .
F: So long as those words are in your vocabulary .
H: Well , it definitely helps with , um , forced alignment , too ,
A: One percent .
H: because , you know , whe when we know the the the true words and we 're adding them to the vocabulary and training a language model and so forth for future meetings , especially the like , the front - end meetings or meetings with a lot of jargon in them , that is not represented in Switchboard or CallHome , then it then it really helps .
F: Well , all of them are . All of our all of our meetings have a lot of jargon in them . But we know what those words are .
H: So
H: Yeah .
F: I mean , I don't know how you get them into the vocabulary ,
F: but it would seem @ @ F: Now , I have to I have to  I really need to , uh , raise a question about the term " cheating " .
F: OK ?
F: And the reason is , um , if I understand that , " cheating " is a term which is used to apply for basically what all of linguistics corpus linguistics does , and what what my transcribers are doing and what I do , which is a methodology whereby you actually physically mark things in the data .
A: All al
F: Like the transcription ,
F: like the words that are actually there .
A: All we mean by that is that we 're giving the recognizer more information than it would have if you were running it raw ,
A: over a meeting that no person has ever listened to or transcribed .
F: OK . I mean , that that part is OK .
F: But I but I do wonder sometimes if it might be possible to use a term that 's a little bit less evaluative , like , um , " hand - marked " .
C: Uh , but cheating is is p
C: Sure . But chea but cheating is is is pretty commonly used to mean this .
I: It 's a it 's a technical term .
H: It I guess it It 's been around for a long time ,
C: It really is A: It is . Yep .
H: and it it 's sort of , um Because it 's so strong a word , people don't take it that seriously ,
C: Yeah .
C: Yeah . I i i i
H: in It 's not negatively viewed .
H: It just really means C: I it 's i
C: It 's even more than that .
C: I think it it really gives a very strong perspective , that you know that what you were doing is not an un - biased experiment .
C: If you don't say that , then people think " oh , they did that and they threw that , but that doesn't represent what would happen in the real world " .
C: But if you say " we did a n cheating experiment " , which is really the standard way you 'd say it , it says you deliberately put in a piece of me information that you would not have in the real world so that you can learn something .
C: Just as part as your process .
C: So it 's I d I don't know ,
C: I F: OK . I guess what I 'm I 'm thinking , is just in if uh , when these are presented in an inte interdisciplinary context , it might be nice to add that explanation ?
C: i
F: Cuz otherwise it sounds like a pejorative statement on an alternative methodology ?
F: Cuz it sounds like it sort of devalues the the , um , all other approach , which is to to put those distinctions in .
C: Maybe . I I 've heard this at ICSLP for years and years and years , though .
C: I mean , it 's it 's pretty pretty common . People say this .
I: If i Well , you i i in a To a broader audience you could call it a diagnostic experiment , rather than a cheating experiment .
H: Actually , Jane is right .
H: Like , in conversation analysis , I 've never heard people use this ,
H: because they 're not using an automatic system .
H: So , it really I: But they don't run experiments .
C: Yeah .
H: Right . I 'm j Well ,
F: They can do experiments .
H: you can do experiments ,
H: but the it 's cheating relative to what we call a system where we can completely control this black box .
H: And it 's not a very smart system .
H: It it only knows what we give it .
H: And if w If it knows more than we would really give it when it runs , we call it cheating .
H: But it 's H: Yeah , it 's f only used in a community that does some type of computational modeling ,
C: Yeah . In pattern recognition , or in H: I think . It 's really not used in any kind of community doing experiments on human perception ,
C: Yeah .
H: or C: Right .
H: You know . So she certainly can define it .
C: Yeah . It 's machine machine experiments .
C: Um , I mean , w when when the , uh , the neural net w uh wave hit in the mid - eighties ,
C: and by the late eighties there were uh , we were reviewing thousands of papers that were coming out in neural nets .
C: It was really hot .
C: Everybody thought it would do everything .
C: Um , and a really common error that people were making was , they were just reporting their , uh uh , classification results on the data that they were training on .