PhD E: Yeah . I talked to Helen .
Postdoc A: And and and she was And so , I asked her I mean , They 're very perceptive . I really want to have this meeting of the transcribers . I haven't done it yet , but I wanna do that and she 's out of town , um , for a couple of weeks , but I wanna do that when she returns . Um , cuz she was saying , you know , in a in a span of very short period we asked It seems like the ones that need to be adjusted are these these these things , and she was saying the short utterances , uh , the , um PhD G: Hmm .
PhD E: Mmm . Yeah .
Postdoc A: you know , I mean , you 're You 're aware of this . But but actually i it 's so correct for so much of the time , that it 's an enormous time saver
PhD E: Yeah .
Postdoc A: and it just gets tweaked a little around the boundaries . So .
PhD G: That 's great .
Postdoc A: Um . Yeah . I think it 'd be interesting to combine these .
PhD E: Yeah .
PhD G: Is there actually a record of where they change ? I mean , you can compare , do a diff on the just so that we knew Postdoc A: You could do it . It 's it 's complicated in that um , hhh , i hhh , i
PhD E: Yeah . Actually , when when they create new yeah , new segments or something , it will be , uh , not that easy but hmm . I think one could do that .
PhD G: I mean , if we keep a old copy of the old time marks
PhD E: Yeah .
PhD G: just so that if we run it we know whether we 're which ones were cheating
PhD E: Yeah . Yeah . That would be great , yeah , to know that .
PhD G: and
Postdoc A: There is a there is one problem with that and that is when they start part way through then what I do is I merge what they 've done with the pre - segmented version .
PhD G: which one would be good .
PhD E: Yeah .
Postdoc A: So it 's not a pure it 's not a pure condition . Wha - what you 'd really like is that they started with pre - segmented and were pre - segmented all the way through .
PhD G: Mm - hmm .
Postdoc A: And , um @ @ {comment} I , uh the it wasn't possible for about four of the recent ones . But , it will be possible in the future
PhD E: Yeah .
Postdoc A: because we we 're , um .
PhD E: It would .
PhD G: Mmm , that 's great .
PhD E: Yeah .
PhD G: Yeah . As long as we have a record , I guess , of the original automatic one , we can always find out how well we would do fr from the recognition side by using those boundaries .
PhD E: Yeah . Yeah .
PhD G: Um .
PhD E: Yeah .
PhD G: You know , a completely non - cheating version .
PhD E: Yeah .
PhD G: Also if you need someone to record this meeting , I mean , I 'm happy to for the transcribers I could do it , or Chuck or Adam .
Postdoc A: Thank you .
Professor F: OK . So , uh , u you were saying something about organizing the meeting info ?
PhD D: Yeah . So , um , uh , Jane and Adam and I had a meeting where we talked about the reorganization of the directory structure for all of the meeting Professor F: Did you record it ?
PhD D: No . For all the Meeting Recorder data . We should have . Um . And so we 've got a plan for what we 're gonna do there . And then , Jane also s prepared a um , started getting all of the the meetings organized , so she prepared a  a spreadsheet , which I spent the last couple of days adding to . So I went through all of the data that we have collected so far , and have been putting it into , uh , a spreadsheet with start time , the date , the old meeting name , the new meeting name , the number of speakers , the duration of the meeting , comments , you know , what its transcription status is , all that kind of stuff . And so , the idea is that we can take this and then export it as HTML and put it on the Meeting Recorder web page so we can keep people updated about what 's going on .
PhD G: Oh , great .
PhD D: Um , I 've gotta get some more information from Jane cuz I have some some gaps here that I need to get her to fill in , but so far , um , as of Monday , the fourteenth , um , we 've had a total number of meeting sixty - two hours of meetings that we have collected . And , um Uh , some other interesting things , average number of speakers per meeting is six . Um , and I 'm gonna have on here the total amount that 's been transcribed so far , but I 've got a bunch of uh , that 's what I have to talk to Jane about , figuring out exactly which ones have have been completed and so forth . But , um , this 'll be a nice thing that we can put up on the the web site and people can be informed of the status of various different ones . And it 'll also list , uh , like under the status , if it 's at IBM or if it 's at ICSI , uh , or if it 's completed or which ones we 're excluding and and there 's a place for comments , so we can , um , say why we 're excluding things and so forth . So .
Professor F: Now would the ones that , um , are already transcribed we h we have enough there that c you know , we 've already done some studies and so forth and um , shouldn't we go through and do the business - es u of of having the , um , uh , participants approve it , uh , for approve the transcriptions for distribution and so forth ?
Postdoc A: Um , interesting idea . In principle , I I would say yes , although I still am doing some the final - pass editing , trying to convert it over to the master file as the being the channelized version and it 's Yeah , it seems like I get into that a certain way and then something else intervenes {comment} and I have to stop . Cleaning up the things like the , uh , uh , places where the transcriber was uncertain , and and doing spot - checking here and there . So , um , uh , I guess it would make sense to wait until th that 's done , um , but but Professor F: Well , le let me put in another sort of a milestone kind of as as I did with the , uh , uh the the pipeline .
Postdoc A: Yeah .
Professor F: Um , we are gonna have this DARPA meeting in the middle of July ,
Postdoc A: Yes .
Professor F: and I think it w it 'd be given that we 've been we 've given a couple public talks about it already , spaced by months and months , I think it 'd be pretty bad if we continued to say none of this is available . Um .
Postdoc A: It 'll certainly be done by then . Yeah .
Professor F: Right . So we can s we we wanna be able to say " here is a subset that is available right now "
Postdoc A: Mm - hmm . That 's right .
Professor F: and that 's has been through the legal issues and so forth .
Postdoc A: That 's right .
Professor F: So .
Postdoc A: Yeah . That 's right . So that Professor F: OK ?
Postdoc A: OK .
Professor F: So , by before July .
PhD C: And they don't have to approve , you know , th an edited version , they can just give their approval to whatever version
Postdoc A: Well , maybe Professor F: Well , in principle , yes . But , I mean , i if if if somebody actually did get into some legal issue with it then we
PhD C: Bu Yeah . But th I mean , the editing will continue . Presumably if if s errors are found , they will be fixed , but they won't change the the content of the meetings .
PhD D: Content , really .
Postdoc A: Well , see , this is the this is the issue . Subtleties .
PhD C: So .
PhD G: Well , i if Jane is clarifying question question , then , you know , how can they agree to it before they know her final version ?
Postdoc A: The other thing , too , is there can be subtleties where a person uses this word instead of that word , which @ @ {comment} could 've been transcribed in the other way .
Professor F: Yeah .
PhD G: Thing Postdoc A: And no and they wouldn't have been slanderous if it had been this other word . You know ?
Professor F: I it you know , there there is a point at which I agree it becomes ridiculous because , you know , you could do this final thing and then a year from now somebody could say , you know , that should be a period and not a question mark . Right ? And you don't you there 's no way that we 're gonna go back and ask everybody " do you approve this , uh , you know this document now ? " So So I think what it is is that the the the the thing that they sign I I haven't looked at it in a while , but it has to be open enough that it sort of says " OK , from now on you know , now that I 've read this , you can use do anything you want with these data . "
Postdoc A: Mm - hmm .
Professor F: And , uh But , i I think we wanna So , assuming that it 's in that kind of wording , which I don't remember , um , I think i we just wanna have enough confidence ourselves that it 's so close to the final form it 's gonna be in , a year from now that they 're Postdoc A: Mm - hmm . I agree . Mmm . I totally agree . It 's just , uh , a question of , uh , if if the person is using the transcript as the way of them judging what they said and whether it was slanderous , then it seems like it 's it 's i it needs to be more correct than if we could count on them re - listening to the meeting .
Professor F: Uh .
Postdoc A: Because it becomes , eh , in a way a a f uh , a legal document i if they 've agreed to that .
Professor F: Well , I forget how we end Right . I forget how we ended up on this , but I remember my taking the position of not making it so so easy for everybody to observe everything and Adam was taking the position of of having it be really straightforward for people to check every aspect of it including the audio . And I don't remember who won , Adam or me , but Postdoc A: Well , if it 's only the transcript , though I mean , th this this is my point , that that
Professor F: uh , the Uh , that that 's why I 'm bringing this up again , because I can't remember how we ended up .
Postdoc A: then it becomes Professor F: That it was the transcrip He wanted to do a web interface that would make it Postdoc A: Well , if it 's just the audio Well .
Professor F: that would give you access to the transcript and the audio . That 's what Adam wanted .
Postdoc A: Mm - hmm .
Professor F: And I don't remember how we ended up .
PhD G: I mean , with the web interface it 's interesting , because you could allow the person who signs to be informed when their transcript changes , or something like that . And , I mean , I would say " no " . Like , I don't wanna know , but some people might be really interested and then y In other words , they would be informed if there was some significant change other than typos and things like that .
Professor F: You decided you were whispering Satanic incantations under your breath when you were PhD G: Well , I don't know what happened to the small heads thing , but I j Um , I 'm just saying that , like , you know , you can sort of say that any things that are deemed Professor F: They disappeared from view .
PhD G: Anyway . I mean , I agree that at some point people probably won't care about typos but they would care about significant meaning changes and then they could be asked for their consent , I guess , if if those change . Cuz assumi assuming we we don't really distribute things that have any significant changes from what they sign anyway .
PhD C: Tha That 's How about having them approve the audio and not the transcripts ?
PhD G: Oh , my God .
Postdoc A: That would be simpler ,
Professor F: Uh .
Postdoc A: if we could count on them listening .
PhD G: But no one will listen to the hours and hours of PhD D: Talk .
PhD C: Well , that 's O K .
Grad B: That 's PhD C: We just have to give them a chance to listen to it , and if they don't , that 's their problem .
Grad B: hmm , hmm .
PhD G: You you d That 's like Postdoc A: Unfortunately , uh , in in the sign thing that they signed , it says " transcripts " .
PhD C: No , I 'm serious .
Postdoc A: " You 'll be you 'll be provided the transcripts when they 're available . "
PhD C: Really ?
Grad B: Mmm .
PhD C: Mmm .
Professor F: Yeah .
PhD E: Yeah .
PhD G: I I I think
Postdoc A: Yeah .
PhD G: that 's a lot to ask for people that have been in a lot of meetings .
Postdoc A: Yeah .
Professor F: W anyway , haven't we we 've gone down this path a number of times . I know this can lead to extended conversations and and not really get anywhere , so let let me just suggest that  uh , off - line that , uh , the people involved figure it out and take care of it before it 's July .
Postdoc A: Yes .
Professor F: OK . So so that in July we can tell people " yes , we have this and you can use it " .
Postdoc A: Yes . It 's done , ready , available . Good .
Professor F: Uh . So , let 's see . What else we got ? Uh . Don did did a report about his project in class and , uh an oral and written written version .
PhD G: Well .
Professor F: So that was stuff he was doing with you . Yeah .
PhD G: I mean , it 's I guess one thing we 're learning is that the amount We have eight meetings there because we couldn't use the non - native all non - native meetings and it 's , well , probably below threshold on enough data for us for the things we 're looking at because the prosodic features are very noisy and so you you need a lot of data in order to model them . Um , so we 're starting to see some patterns and we 're hoping that maybe with , I don't know , double or triple the data with twenty meetings or so , that we would start to get better results . But we did find that some of the features that , I gue Jane would know about , that are expressing sort of the distance of , um , boundaries from peaks in the utterance and some local , um , range pitch range effects , like how close people are to their floor , are showing up in these classifiers , which are also being given some word features that are cheating , cuz they 're true words . Um , so these are based on forced alignment . Word features like , um , word frequency and whether or not something 's a backchannel and so forth . So , we 're starting to see , I think , some interesting patterns .
Professor F: So the dominant features , including everything , were those those quasi - cheating things . Right ? Where these are Grad B: Sometimes not .
PhD G: I think it depends what you 're looking at , a actually .
Grad B: Yeah . Sometimes positions in sentences obviously , or in spurts , was helpful . I don't know if that 's cheating , too .
PhD G: Right . Um ,
PhD C: Spurts wouldn't be . Right ?
PhD G: spurts is not cheating except that of course you know the real words ,
Grad B: Right .
PhD G: but roughly speaking , the recognized words are gonna give you a similar type of position .
Grad B: Right . Would they give you the same number of words , though ?
Professor F: Right .
PhD G: It 's either early or late .
PhD C: No
PhD G: Not exactly , but i
Grad B: But ra somewhat ?
Professor F: On the average .
PhD G: Y yeah it should be . Well , we don't know and actually that 's one of the things we 're interested in doing , is a sort of Professor F: Uh - huh .
PhD C: Have you tried using just time , as opposed to number of words ?
PhD G: So .
Grad B: I think ti uh Just p time position , like when the word starts ?
PhD C: Yeah .
Grad B: I don't know if that was in the PhD C: Well , no , I mean t time time position relative to the beginning of the spurt .
PhD G: Eh You know , uh
Grad B: Start .
PhD G: Yeah ,
Grad B: Yeah . There 's all these things to do .
PhD G: uh , we didn't try it , but it 's s
Grad B: Like , there 's a lot of different features you could just pull out .
PhD C: Yeah . I mean that wouldn't be cheating because you can detect pause pretty well within the time .
Grad B: Right .
PhD G: Right .
Professor F: How about time position normalized by speak
PhD G: And it depends on speaking rate Professor F: Yeah . Yeah .
PhD G: speaking rate . Yeah .
Grad B: Yeah .
PhD G: Yeah . That 's actually why I didn't use it at first .
Professor F: Yeah .
PhD C: Mm - hmm .
PhD G: But we one of the interesting things was I guess you reported on some te punctuation type Grad B: Yeah .
PhD G: finding sentence boundaries , finding disfluency boundaries , and then I had done some work on finding from the foreground speech whether or not someone was likely to interrupt , so where you know , if I 'm talking now and someone and and Andreas is about to interrupt me , is he gonna choose a certain place in my speech , either prosodically or word - based . And there the prosodic features actually showed up and a neat thing even though the word features were available . And a neat thing there too is I tried some  putting the speaker So , I gave everybody a short version of their name . So the real names are in there , which we couldn't use . Uh , we should use I Ds or something . And those don't show up . So that means that overall , um , it wasn't just modeling Morgan , or it wasn't just modeling a single person ,
Professor F: Mm - hmm .
PhD G: um , but was sort of trying to , uh , get a general idea the model the tree classifier was trying to find general locations that were applicable to different speakers , even though there are huge speaker effects . So . The but the main limitation now is I because we 're only looking at things that happen every ten words or every twenty words , we need more more data and more data per speaker . So . It 'd also be interesting to look at the EDU meetings because we did include meeting type as a feature , so whether you were in a r Meeting Recorder meeting or a Robustness meeting did matter to interrupts because there are just fewer interrupts in the Robustness meetings .
Grad B: Mm - hmm .
PhD G: And so the classifier learns more about Morgan than it does about sort of the average person ,
Professor F: Mm - hmm .
PhD G: which is not bad . It 'd probably do better than  Um , but it wasn't generalizing .
Professor F: Yeah .
PhD G: So it 's And I think Don , um Well , we have a long list of things he 's starting to look at now over the summer , where we can And he 'll be able to report on more things in the future . But it was great that we could at least go from the  you know , Jane 's transcripts and the , uh , recognizer output and get it to this point . And I think it 's something Mari can probably use in her preliminary report like , " yeah , we 're at the point where we 're training these classifiers and we 're just reporting very preliminary but suggestive results that some features , both word and pro prosodic , work . " The other thing that was interesting to me is that the pitch features are better than in Switchboard . And I think that really is from the close - talking mikes , cuz the pitch processing that was done has much cleaner behavior than than the Switchboard telephone bandwidth .
PhD C: W wh wh wh Better in what sense ?
PhD G: Um . Well , first of all , the pitch tracks are m have less , um , halvings and doublings than than Switchboard and there 's a lot less dropout , so if you ask how many regions where you would normally expect some vowels to be occurring are completely devoid of pitch information ,
Professor F: Mm - hmm .
PhD G: in other words the pitch tracker just didn't get a high enough probability of voicing for words for for , you know , five word
Professor F: Hmm .
PhD G: there are much fewer than in Switchboard . So the missing  We had a big missing data problem in Switchboard and , so the features weren't as reliable cuz they were often just not available .
PhD D: Could it have to do with the the lower frequency cut - off on the Switchboard ?