A: and it 's Yeah , it seems like I get into that a certain way and then something else intervenes <comment> and I have to stop .
A: Cleaning up the things like the , uh , uh , places where the transcriber was uncertain ,
A: and and doing spot - checking here and there .
A: So , um , uh , I guess it would make sense to wait until th that 's done ,
A: um , but but F: Well , le let me put in another sort of a milestone kind of as as I did with the , uh , uh the the pipeline .
A: Yeah .
F: Um , we are gonna have this DARPA meeting in the middle of July ,
F: and I think it w it 'd be A: Yes .
F: given that we 've been we 've given a couple public talks about it already , spaced by months and months , I think it 'd be pretty bad if we continued to say none of this is available .
F: Um .
A: It 'll certainly be done by then . Yeah .
F: Right . So we can s we we wanna be able to say " here is a subset that is available right now "
A: Mm - hmm .
A: That 's right .
F: and that 's has been through the legal issues and so forth .
A: That 's right .
F: So .
A: Yeah . That 's right . So that F: OK ?
A: OK .
F: So , by before July .
C: And they don't have to approve , you know , th an edited version , they can just give their approval to whatever version
A: Well , maybe F: Well , in principle , yes . But , I mean , i if if if somebody actually did get into some legal issue with it then we
C: Bu
C: Yeah . But th I mean , the editing will continue . Presumably if if s errors are found , they will be fixed ,
C: but they won't change the the content of the meetings .
D: Content , really .
A: Well , see , this is the this is the issue . Subtleties .
C: So .
G: Well , i if Jane is clarifying question question , then , you know , how can they agree to it before they know her final version ?
A: The other thing , too , is there can be subtleties where a person uses this word instead of that word , which @ @ <comment> could 've been transcribed in the other way .
F: Yeah .
G: Thing A: And no and they wouldn't have been slanderous if it had been this other word .
A: You know ?
F: I it you know , there there is a point at which I agree it becomes ridiculous
F: because , you know , you could do this final thing and then a year from now somebody could say , you know , that should be a period and not a question mark .
F: Right ?
F: And you don't you there 's no way that we 're gonna go back and ask everybody " do you approve this , uh , you know this document now ? "
F: So F: So I think what it is is that the the the the thing that they sign I I haven't looked at it in a while , but it has to be open enough that it sort of says " OK , from now on you know , now that I 've read this , you can use do anything you want with these data . "
A: Mm - hmm .
F: And , uh But , i I think we wanna F: So , assuming that it 's in that kind of wording , which I don't remember , um , I think i we just wanna have enough confidence ourselves that it 's so close to the final form it 's gonna be in , a year from now that they 're A: Mm - hmm .
A: I agree .
A: Mmm . I totally agree .
F: Uh .
A: It 's just , uh , a question of , uh , if if the person is using the transcript as the way of them judging what they said and whether it was slanderous , then it seems like it 's it 's i it needs to be more correct than if we could count on them re - listening to the meeting .
A: Because it becomes , eh , in a way a a f uh , a legal document
A: i if they 've agreed to that .
F: Well , I forget how we end
F: Right . I forget how we ended up on this ,
F: but I remember my taking the position of not making it so so easy for everybody to observe everything
F: and Adam was taking the position of of having it be really straightforward for people to check every aspect of it including the audio .
F: And I don't remember who won , Adam or me ,
F: but A: Well , if it 's only the transcript , though I mean , th this this is my point , that that
F: uh , the F: Uh , that that 's why I 'm bringing this up again , because I can't remember how we ended up .
A: then it becomes F: That it was the transcrip
F: He wanted to do a web interface that would make it A: Well , if it 's just the audio F: that would give you access to the transcript and the audio .
A: Well .
F: That 's what Adam wanted .
A: Mm - hmm .
F: And I don't remember how we ended up .
G: I mean , with the web interface it 's interesting , because you could allow the person who signs to be informed when their transcript changes ,
G: or something like that .
G: And , I mean , I would say " no " . Like , I don't wanna know ,
G: but some people might be really interested
G: and then y In other words , they would be informed if there was some significant change other than typos and things like that .
F: You decided you were whispering Satanic incantations under your breath when you were G: Well , I don't know what happened to the small heads thing ,
G: but I j Um , I 'm just saying that , like , you know , you can sort of say that any things that are deemed F: They disappeared from view .
G: Anyway . I mean , I agree that at some point people probably won't care about typos but they would care about significant meaning changes
G: and then they could be asked for their consent , I guess , if if those change .
G: Cuz assumi assuming we we don't really distribute things that have any significant changes from what they sign anyway .
C: Tha
C: That 's C: How about having them approve the audio and not the transcripts ?
G: Oh , my God .
F: Uh .
A: That would be simpler ,
A: if we could count on them listening .
G: But no one will listen to the hours and hours of D: Talk .
C: Well , that 's O K .
C: We just have to give them a chance to listen to it , and if they don't , that 's their problem .
B: That 's B: hmm , hmm .
G: You you d
G: That 's like A: Unfortunately , uh , in in the sign thing that they signed , it says " transcripts " .
C: No , I 'm serious .
A: " You 'll be you 'll be provided the transcripts when they 're available . "
C: Really ?
F: Yeah .
G: I I I think
B: Mmm .
C: Mmm .
E: Yeah .
G: that 's a lot to ask for people that have been in a lot of meetings .
A: Yeah .
A: Yeah .
F: W
F: anyway , haven't we we 've gone down this path a number of times . I know this can lead to extended conversations and and not really get anywhere ,
F: so let let me just suggest that  uh , off - line that , uh , the people involved figure it out and take care of it before it 's July .
A: Yes .
F: OK . So so that in July we can tell people " yes , we have this and you can use it " .
A: Yes .
A: It 's done , ready , available .
A: Good .
F: Uh .
F: So , let 's see .
F: What else we got ?
F: Uh . Don did did a report about his project in class
F: and , uh an oral and written written version .
G: Well .
F: So that was stuff he was doing with you .
G: I mean , it 's I guess one thing we 're learning is that the amount F: Yeah .
G: We have eight meetings there
G: because we couldn't use the non - native all non - native meetings
G: and it 's , well , probably below threshold on enough data for us for the things we 're looking at
G: because the prosodic features are very noisy and so you you need a lot of data in order to model them .
G: Um , so we 're starting to see some patterns
G: and we 're hoping that maybe with , I don't know , double or triple the data with twenty meetings or so , that we would start to get better results .
G: But we did find that some of the features that , I gue Jane would know about , that are expressing sort of the distance of , um , boundaries from peaks in the utterance and some local , um , range pitch range effects , like how close people are to their floor , are showing up in these classifiers ,
G: which are also being given some word features that are cheating , cuz they 're true words .
G: Um , so these are based on forced alignment .
G: Word features like , um , word frequency
G: and whether or not something 's a backchannel and so forth .
G: So , we 're starting to see , I think , some interesting patterns .
F: So the dominant features , including everything , were those those quasi - cheating things .
F: Right ?
F: Where these are B: Sometimes not .
G: I think it depends what you 're looking at , a actually .
B: Yeah . Sometimes positions in sentences obviously , or in spurts , was helpful .
G: Right .
B: I don't know if that 's cheating , too .
G: Um ,
C: Spurts wouldn't be .
C: Right ?
G: spurts is not cheating except that of course you know the real words ,
G: but roughly speaking , the recognized words are gonna give you a similar type of position .
B: Right .
B: Right . Would they give you the same number of words , though ?
G: It 's either early or late .
F: Right .
G: Not exactly , but i
C: No
B: But ra somewhat ?
G: Y yeah it should be .
F: On the average .
G: Well , we don't know
G: and actually that 's one of the things we 're interested in doing , is a sort of C: Have you tried using just time , as opposed to number of words ?
F: Uh - huh .
G: So .
B: I think ti uh B: Just p time position , like when the word starts ?
C: Yeah .
C: Well , no , I mean t time time position relative to the beginning of the spurt .
G: Eh B: I don't know if that was in the G: You know , uh
B: Start .
G: Yeah ,
G: uh , we didn't try it ,
B: Yeah . There 's all these things to do .
G: but it 's s
B: Like , there 's a lot of different features you could just pull out .
C: Yeah .
C: I mean that wouldn't be cheating because you can detect pause pretty well within the time .
B: Right .
G: Right .
F: How about time position normalized by speak
G: And it depends on speaking rate F: Yeah . Yeah .
G: speaking rate . Yeah .
G: Yeah . That 's actually why I didn't use it at first .
B: Yeah .
F: Yeah .
G: But we one of the interesting things was
C: Mm - hmm .
G: I guess you reported on some te punctuation type G: finding sentence boundaries , finding disfluency boundaries ,
B: Yeah .
G: and then I had done some work on finding from the foreground speech whether or not someone was likely to interrupt ,
G: so where you know , if I 'm talking now and someone and and Andreas is about to interrupt me , is he gonna choose a certain place in my speech , either prosodically or word - based .
G: And there the prosodic features actually showed up
G: and a neat thing G: even though the word features were available .
G: And a neat thing there too is I tried some  putting the speaker G: So , I gave everybody a short version of their name .
G: So the real names are in there ,
G: which we couldn't use .
G: Uh , we should use I Ds or something .
G: And those don't show up .
G: So that means that overall , um , it wasn't just modeling Morgan , or it wasn't just modeling a single person ,
F: Mm - hmm .
G: um ,
G: but was sort of trying to , uh , get a general idea G: the model the tree classifier was trying to find general locations that were applicable to different speakers ,
G: even though there are huge speaker effects .
G: So .
G: The but the main limitation now is I because we 're only looking at things that happen every ten words or every twenty words , we need more more data and more data per speaker .
G: So .
G: It 'd also be interesting to look at the EDU meetings
G: because we did include meeting type as a feature ,
G: so
G: whether you were in a r Meeting Recorder meeting or a Robustness meeting did matter to interrupts
G: because there are just fewer interrupts in the Robustness meetings .
B: Mm - hmm .
G: And so the classifier learns more about Morgan than it does about sort of the average person ,
F: Mm - hmm .
G: which is not bad .
G: It 'd probably do better than  Um , but it wasn't generalizing .
G: So it 's F: Yeah .
G: And I think Don , um Well , we have a long list of things he 's starting to look at now over the summer ,
G: where we can G: And he 'll be able to report on more things in the future .
G: But it was great that we could at least go from the  you know , Jane 's transcripts and the , uh , recognizer output and get it to this point .
G: And I think it 's something Mari can probably use in her preliminary report G: like , " yeah , we 're at the point where we 're training these classifiers
G: and we 're just reporting very preliminary but suggestive results that some features , both word and pro prosodic , work . "
G: The other thing that was interesting to me is that the pitch features are better than in Switchboard .
G: And I think that really is from the close - talking mikes ,
G: cuz the pitch processing that was done has much cleaner behavior than than the Switchboard telephone bandwidth .
C: W wh wh wh
C: Better in what sense ?
G: Um . Well , first of all , the pitch tracks are m have less , um , halvings and doublings than than Switchboard
G: and there 's a lot less dropout ,
G: so if you ask how many regions where you would normally expect some vowels to be occurring are completely devoid of pitch information ,
F: Mm - hmm .
G: in other words the pitch tracker just didn't get a high enough probability of voicing for words G: for for , you know , five word
F: Hmm .
G: there are much fewer than in Switchboard .
G: So the missing  We had a big missing data problem in Switchboard
G: and , so the features weren't as reliable
G: cuz they were often just not available .
G: So that 's actually good .
D: Could it have to do with the the lower frequency cut - off on the Switchboard ?
G: Ma - maybe . I mean , the tele we had telephone bandwidth for Switchboard
G: and we had the an annoying sort of telephone handset movement problem that I think may also affect it .
D: Hmm .
G: So we 're just getting better signals in in this data .
G: Which is nice .
G: So .
F: Yeah .
G: Anyway , Don 's been doing a great job
G: and we hope to continue with , um , Andreas 's help and also some of Thilo 's help on this ,
F: Great .
G: to to try to get a non - cheating version of how all this would work .
E: Y
E: Yeah . Sure .
E: Yeah .
F: Has has , uh ? We just I think , just talked about this the other day , but h has has anybody had a chance to try changing , uh , insertion penalty sort of things with the with the , uh  uh , using the tandem system input for the ?
C: Oh , yeah . I tried that .
C: It didn't , um , help dramatically .
D: Were they out of balance ?
C: The D: I didn't I didn't notice .
C: There were a little C: the relative number of C: I think there were a higher number of deletions , actually .
F: Oh .
G: Deletions ?
C: So , you , uh So , actually it it preferred to have a positive er , negative insertion penalty ,
C: which means that , um F: Uh - huh .
C: But , you know , it didn't change th the C: by adjusting that the , um F: OK .
C: Yeah . The error changed by probably one percent or so .
C: But , you know , given that that word error rate is so high , that 's not a F: OK . So that F: So that 's So that 's not the problem .
C: That 's not the problem .
F: Yeah .
C: No .
C: But , uh , we s just , um ,
C: uh C: you know , Chuck and I talked and the @ @ <comment> next thing to do is probably to tune the um , the size of the Gaussian system , um , @ @ <comment> to to this to this feature vector , which we haven't done at all .
C: We just used the same configuration as we used for the  for the standard system .
F: Hmm .
C: And , for instance , uh , Dan @ @ <comment> Dan just sent me a message saying that CMU used , um , something like ten Gaussians per cluster C: You know , each each mixture has ten Gaussians
D: Mm - hmm .
D: Hmm . We 're using sixty - four ,
C: and and we 're using sixty - four ,
D: right ?
C: so that 's obviously a big difference
D: Yeah .
C: and it might be way off
C: and give very poorly trained , uh , you know , Gaussians that way ,
F: Hmm .
C: uh , an and poorly trained mixture weights .
C: So so , we have C: The turn - around time on the training when we train only the a male system with , uh , you know , our small training set , is less than twenty - four hours ,
C: so we can run lots of  uh , basically just brute force ,
C: try a whole bunch of different um , settings .
F: OK .
C: And , uh , with the new machines it 'll be even better .
C: So .
F: Yeah . We get twelve of those ,
F: huh ?
C: Yeah .
C: But the PLP features work um , uh , you know , continue to improve the ,
F: OK .
C: um C: As I said before , the uh using Dan 's , uh , uh , vocal tract normalization option works very well .
F: Mm - hmm .
C: So , um , @ @ <comment> I ran one experiment where we 're just did the vocal tract le normalization only in the test data ,
C: so I didn't bother to retrain the models at all ,
C: and it improved by one percent ,
C: which is about what we get with  uh , with , you know , just @ @ <comment> actually doing both training and test normalization , um , with , um , the , uh  uh , with the standard system .
C: So , in a few hours we 'll have the numbers for the for retraining everything with vocal tract length normalization
C: and So , that might even improve it further .
F: Great .
C: So , it looks like the P L - fea P features <comment> do very well now with after having figured out all these little tricks to to get it to work .
F: Yeah .
C: So .
G: Wait . So you mean you improve one percent over a system that doesn't have any V T L in it already ?
F: Good .
C: Exactly . Yeah .
G: OK .
F: Yeah .
F: OK . So then then we 'll have our baseline to to compare the currently hideous , uh , uh , new thing with .
C: Right . a
F: But C: Right . And and what that suggests also is of course that the current Switchboard MLP isn't trained on very good features .
F: Yeah .
C: Uh , because it was trained on whatever , you know , was used , uh , last time you did Hub - five stuff ,