Professor F: Right .
PhD G: I mean , I both actu as as a submission and  you know , as a paper . Um but Grad A: Well yeah , you sent it in late .
Professor F: Yeah , I guess you first you have to do the first one ,
Grad A: Yeah .
Professor F: and then Yeah .
PhD G: We actually exceeded the delayed deadline by o another day , so .
PhD B: Oops .
Professor F: Oh they they had some extension that they announced or something ?
PhD G: Well yeah . Liz had sent them a note saying " could we please have another " {comment} I don't know , " three days " or something , and they said yes .
PhD D: And then she said " Did I say three ?
Grad A: Oh ,
PhD D: I meant four . "
Grad A: that was the other thing uh ,
PhD G: But u
Grad A: uh , Dave Gelbart sent me email , I think he sent it to you too , {comment} that um , there 's a special topic , section in si in Eurospeech on new , corp corpors corpora . And it 's not due until like May fifteenth .
Professor F: Oh this isn't the Aurora one ?
Grad A: No .
Professor F: It 's another one ?
Grad A: It 's a different one .
PhD B: No it 's Yeah . Yeah .
Grad E: Huh !
Grad A: And uh ,
Professor F: Oh !
PhD B: I got this mail from Grad A: I s forwarded it to Jane as I thought being the most relevant person . Um So , I thought it was highly relevant Postdoc C: Yeah I 'm Professor F: That 's Grad A: have you did you look at the URL ?
Postdoc C: Yeah . I think so too . Um , I haven't gotten over to there yet ,
Grad A: Mm - hmm .
Postdoc C: but what our discussion yesterday , I really I I wanna submit one .
PhD B: Was this SmartKom message ? I think Christoph Draxler sent this ,
Postdoc C: Yeah . And , you offered to to join me , if you want me to .
Grad A: I 'll help ,
PhD B: yeah .
Grad A: but obviously I can't , really do , most of it ,
Postdoc C: Yeah . Yeah , that 's right .
PhD G: I think several people sent this ,
Grad A: so .
PhD B: Yeah .
Postdoc C: Uh - huh .
PhD G: yeah .
Grad A: But any any help you need I can certainly provide .
Professor F: Well ,
PhD G: Yeah .
Professor F: that 's that 's a great idea .
PhD G: Well there there were some interesting results in this paper , though . For instance that Morgan uh , accounted for fifty - six percent of the Robustness meetings in terms of number of words .
Grad A: Wow .
Postdoc C: In in terms of what ? In term
PhD G: Number of words .
Postdoc C: One ? Wow ! OK .
Grad A: That 's just cuz he talks really fast .
Postdoc C: Do you mean ,
Professor F: n No .
Grad A: I know
PhD B: Oh . Short words .
Postdoc C: because is it partly , eh , c correctly identified words ? Or is it or just overall volume ?
PhD G: No . Well , according to the transcripts .
Grad A: But re well regardless . I think it 's he 's he 's in all of them ,
Postdoc C: Oh . OK .
Professor F: Yeah .
PhD G: I mean , we didn't mention Morgan by name
Grad A: and he talks a lot .
PhD G: we just Grad A: One participant .
Professor F: Well we have now , but PhD G: We we we something about Grad A: Did you identify him as a senior member ?
PhD G: No , we as identify him as the person dominating the conversation .
Professor F: Well .
Grad A: Yeah .
Postdoc C: OK .
Professor F: I mean I get these AARP things , but I 'm not se really senior yet , but PhD G: Right
Professor F: Um ,
PhD G: Hmm .
Professor F: but uh , other than that delightful result , what was the rest of the paper about ?
PhD G: Um , well it was about it had three sections
Professor F: You sent it to me but I haven't seen it yet .
PhD G: uh three kinds of uh results , if you will . Uh , the one was that the just the the amount of overlap
Grad A: The good , the bad , and the ugly .
PhD G: um , s in terms of in terms of number of words and also we computed something called a " spurt " , which is essentially a stretch of speech with uh , no pauses exceeding five hundred milliseconds . Um , and we computed how many overlapped i uh spurts there were and how many overlapped words there were . Um , for four different corpora , the Meeting Recorder meetings , the Robustness meetings Switchboard and CallHome , and , found and sort of compared the numbers . Um , and found that the , uh , you know , as you might expect the Meeting Recorder meetings had the most overlap uh , but next were Switchboard and CallHome , which both had roughly the same , almost identical in fact , and the Robustness meetings were had the least , so One sort of unexpected result there is that uh two - party telephone conversations have about the same amount of overlap ,
Grad A: I 'm surprised .
PhD G: sort of in gen you know order of magnitude - wise as , uh as face - to - face meetings with multiple Grad A: I have I had better start changing all my slides !
PhD G: Yeah . Also , I in the Levinson , the pragmatics book , {comment} in you know , uh , textbook , there 's I found this great quote where he says you know you know , how people it talks about how uh how how people are so good at turn taking ,
Postdoc C: Mm - hmm . Yeah . Yeah .
PhD G: and so they 're so good that generally , u the overlapped speech does not is less than five percent .
Postdoc C: Oh , that 's interesting . Yeah .
PhD G: So , this is way more than five percent .
Grad E: Did he mean face like face - to - face ? Or ?
PhD G: Well , in real conversations ,
Grad E: Hmm .
PhD G: everyday conversations .
Postdoc C: Mm - hmm .
PhD G: It 's s what these conversation analysts have been studying for years and years there .
Postdoc C: Mm - hmm .
PhD B: But Postdoc C: Well , of course , no , it doesn't necessarily go against what he said , cuz he said " generally speaking " . In order to to go against that kind of a claim you 'd have to big canvassing .
Grad A: Hmm .
PhD B: And in f
PhD G: Well , he he made a claim Grad A: Well PhD G: Well Grad A :
PhD B: But Professor F: Yeah , we we have pretty limited sample here .
PhD B: Five percent of time or five percent of what ?
Grad A: Yeah , I was gonna ask that too .
Postdoc C: Yeah .
PhD B: Yeah .
Postdoc C: Exactly .
PhD G: Well it 's time .
PhD B: Yeah , so Postdoc C: It 's i it 's not against his conclusion ,
PhD G: So  but still but still u
Postdoc C: it just says that it 's a bi bell curve , and that , you have something that has a nice range , in your sampling .
PhD G: Yeah . So there are slight There are differences in how you measure it , but still it 's  You know , the difference between um between that number and what we have in meetings , which is more like , you know , close to in meetings like these , uh you know , close to twenty percent .
Postdoc C: Mm - hmm . Mm - hmm .
Professor F: But what was it like , say , in the Robustness meeting , for instance ?
PhD G: That Grad A: But PhD G: Robustness meeting ? It was about half of the r So , in terms of number of words , it 's like seventeen or eigh eighteen percent for the Meeting Recorder meetings and about half that for , uh , the Robustness .
Professor F: Maybe ten percent ?
Grad A: But I don't know if that 's really a fair way of comparing between , multi - party , conversations and two - party conversations . Yeah . I I I don't know .
PhD B: Then then then you have to Grad A: I mean that 's just something PhD D: Yeah , I just wonder if you have to normalize by the numbers of speakers or something .
Postdoc C: Yeah .
PhD B: Then Yeah , then normalize by by something like that ,
Postdoc C: Yeah , that 's a good point .
PhD G: Well , we didn't get to look at that ,
PhD B: yeah .
Postdoc C: Yeah .
PhD G: but this obvious thing to see if if there 's a dependence on the number of uh participants .
Postdoc C: Good idea .
Grad A: I mean I bet there 's a weak dependence . I 'm sure it 's it 's not a real strong one .
PhD B: Yeah .
PhD G: Right .
Grad A: Right ? Because you
PhD D: Cuz not everybody talks .
Grad A: Right .
PhD G: Right .
PhD D: Yeah .
Grad A: You have a lot of a lot of two - party , subsets within the meeting .
PhD G: Right .
Postdoc C: Uh - huh .
Grad A: Well regardless it 's an interesting result regardless .
PhD G: So Right .
Postdoc C: Yes , that 's right .
PhD G: And and and then and we also d computed this both with and without backchannels ,
Postdoc C: Mm - hmm .
PhD G: so you might think that backchannels have a special status because they 're essentially just Grad A: Uh - huh . So , did we all said " uh - huh " and nodded at the same time ,
PhD G: R right .
Grad A: so .
PhD G: But , even if you take out all the backchannels so basically you treat backchannels l as nonspeech , as pauses ,
Grad A: Mm - hmm .
Professor F: Mm - hmm .
PhD G: you still have significant overlap . You know , it goes down from maybe For Switchboard it goes down from I don't know f um {comment} I don't know f fourteen percent of the words to maybe uh I don't know , eleven percent or something it 's it 's not a dramatic change ,
Grad A: Mm - hmm .
PhD G: so it 's Anyway , so it 's uh That was that was one set of results , and then the second one was just basically the  the stuff we had in the in the HLT paper on how overlaps effect the recognition performance .
Postdoc C: Hmm .
Grad A: Nope . Right .
Professor F: Mm - hmm .
PhD G: And we rescored things um , a little bit more carefully . We also fixed the transcripts in in numerous ways . Uh , but mostly we added one one number , which was what if you uh , basically score ignoring all So so the the conjecture from the HLT results was that most of the added recognition error is from insertions due to background speech . So , we scored all the recognition results , uh , in such a way that the uh Grad A: Oh by the way , who 's on channel four ? You 're getting a lot of breath .
PhD B: Yeah . I j was just wondering .
Grad E: That 's PhD B: Yeah .
Grad E: That 's me .
PhD G: uh , well Don 's been working hard .
Grad E: That 's right .
PhD G: OK , so  so if you have the foreground speaker speaking here , and then there 's some background speech , may be overlapping it somehow , um , and this is the time bin that we used , then of course you 're gonna get insertion errors here and here .
Grad A: Right .
PhD G: Right ? So we scored everything , and I must say the NIST scoring tools are pretty nice for this , where you just basically ignore everything outside of the , uh , region that was deemed to be foreground speech . And where that was we had to use the t forced alignment , uh , results from s for so That 's somewhat that 's somewhat subject to error , but still we we  Uh , Don did some ha hand - checking and and we think that based on that , we think that the results are you know , valid , although of course , some error is gonna be in there . But basically what we found is after we take out these regions so we only score the regions that were certified as foreground speech , {comment} the recognition error went down to almost uh , the level of the non - overlapped speech . So that means that even if you do have background speech , if you can somehow separate out or find where it is , uh , the recognizer does a good job ,
Grad A: That 's great .
PhD B: Yeah .
PhD G: even though there is this back
Grad A: Yeah , I guess that doesn't surprise me , because , with the close - talking mikes , the the signal will be so much stronger .
PhD G: Right . Right .
Professor F: Mm - hmm .
PhD G: Mm - hmm . Um ,
Grad A: What what sort of normalization do you do ?
PhD G: so Uh , well , we just @ @ {comment} we do u you know , vit
Grad A: I mean in you recognizer , in the SRI recognizer .
PhD G: Well , we do uh , VTL  vocal tract length normalization , w and we uh you know , we we uh , make all the features have zero mean and unit variance .
Grad A: Over an entire utterance ?
Professor F: And Grad A: Or windowed ?
PhD G: Over over the entire c over the entire channel .
PhD B: Don't train PhD G: Over the Grad A: Hmm .
PhD G: but you know . Um , now we didn't re - align the recognizer for this . We just took the old So this is actually a sub - optimal way of doing it ,
Grad A: Right .
Professor F: Right .
PhD G: right ? So we took the old recognition output and we just scored it differently . So the recognizer didn't have the benefit of knowing where the foreground speech a start
Professor F: Were you including the the lapel in this ?
PhD G: Yes .
Professor F: And did the did did the la did the the problems with the lapel go away also ? Or PhD G: Um , it Yeah .
Professor F: fray for for insertions ?
PhD G: It u not per I mean , not completely , but yes ,
Professor F: Less so .
PhD G: dramatically . So we have to um Professor F: I mean , you still PhD G: Well I should bring the should bring the table with results . Maybe we can look at it Monday .
Professor F: I would presume that you still would have somewhat higher error with the lapel for insertions than PhD G: Yes . It 's It 's Professor F: Yeah .
PhD G: Yes . Yeah .
Professor F: Cuz again , looking forward to the non - close miked case , I think that we s still PhD G: Mm - hmm .
Grad A: I 'm not looking forward to it .
Professor F: i it 's the high signal - to - noise ratio
PhD G: Right .
Professor F: here that that helps you .
PhD G: u s Right . So so that was number that was the second set of uh , the second section . And then , the third thing was , we looked at ,  uh , what we call " interrupts " , although that 's that may be a misnomer , but basically we looked at cases where Uh , so we we used the punctuation from the original transcripts and we inferred the beginnings and ends of sentences . So , you know Postdoc C: Di - did you use upper - lower case also , or not ?
PhD G: Um Postdoc C: U upper lower case or no ?
PhD G: Hmm ?
Postdoc C: OK .
PhD G: No , we only used , you know , uh periods , uh , question marks and exclamation . And we know that there 's th that 's not a very g I mean , we miss a lot of them ,
Postdoc C: Yeah . That 's OK but PhD G: but but it 's f i i
Postdoc C: Comma also or not ?
PhD G: No commas . No . And then we looked at locations where , uh , if you have overlapping speech and someone else starts a sentence , you know , where do these where do other people start their turns not turns really , but you know , sentences ,
PhD B: Ah .
PhD G: um So we only looked at cases where there was a foreground speaker and then at the to at the so the the foreground speaker started into their sentence and then someone else started later .
PhD B: Somewhere in between the start and the end ?
PhD G: OK ? And so what PhD B: OK .
PhD G: Sorry ?
PhD B: Somewhere in between the start and the end of the foreground ?
PhD G: Yes . Uh , so that such that there was overlap between the two sentences .
PhD B: Yeah .
PhD G: So , the the question was how can we what can we say about the places where the second or or actually , several second speakers , um start their " interrupts " , as we call them .
PhD D: Three words from the end .
Grad A: At pause boundaries .
PhD G: w And we looked at this in terms of um Grad A: On T - closures , only .
PhD G: So so we had  we had um u to for for the purposes of this analysis , we tagged the word sequences , and and we time - aligned them . Um , and we considered it interrupt if it occurred in the middle of a word , we basically you know , considered that to be a interrupt as if it were at at the beginning of the word . So that , if any part of the word was overlapped , it was considered an interrupted word .
Professor F: Mm - hmm .
PhD G: And then we looked at the the locatio the , um , you know , the features that the tags because we had tagged these word strings , {comment} um , that that occurred right before these these uh , interrupt locations .
PhD B: Tag by uh
PhD G: And the tags we looked at are the spurt tag , which basically says or actually Sorry . End of spurt . So  whether there was a pause essentially here , because spurts are a defined as being you know , five hundred milliseconds or longer pauses , and then we had things like discourse markers , uh , backchannels , uh , disfluencies . um , uh , filled pauses So disfluen the D 's are for , um , the interruption points of a disfluency , so , where you hesitate , or where you start the repair there . Uh , what else do we had . Uh , repeated you know , repeated words is another of that kind of disfluencies and so forth . So we had both the beginnings and ends of these uh so , the end of a filled pause and the end of a discourse marker . And we just eyeballed I mean we didn't really hand - tag all of these things . We just looked at the distribution of words , and so every " so yeah " , and " OK " , uh , and " uh - huh " were were the were deemed to be backchannels and " wow " and " so " and uh " right " , uh were um  Not " right " . " Right " is a backchannel . But so , we sort of just based on the lexical  um , identity of the words , we we tagged them as one of these things . And of course the d the interruption points we got from the original transcripts . So , and then we looked at the disti so we looked at the distribution of these different kinds of tags , overall uh , and and and particularly at the interruption points . And uh , we found that there is a marked difference so that for instance after so at the end after a discourse marker or after backchannel or after filled pause , you 're much more likely to be interrupted than before . OK ? And also of course after spurt ends , which means basically in p inside pauses . So pauses are always an opportunity for So we have this little histogram which shows these distributions and , um ,
PhD D: I wonder PhD G: you know , it 's it 's it 's not No big surprises , but it is sort of interesting from Grad A: It 's nice to actually measure it though .
PhD G: Yeah .
PhD D: I wonder about the cause and effect there . In other words uh if you weren't going to pause you you will because you 're g being interrupted .
PhD G: Well we 're ne
PhD D: Uh PhD G: Right . There 's no statement about cause and effect .
PhD D: Yeah , right . No , no , no .
PhD G: This is just a statistical correlation ,
PhD D: Right , I I see . Yeah .
PhD G: yeah .
Professor F: But he yeah , he 's he 's right , y I mean maybe you weren't intending to pause at all , but  You were intending to stop for fifty - seven milliseconds ,
PhD G: Right .
Professor F: but then Chuck came in
PhD G: Right .
PhD D: Yeah .
Professor F: and so you paused for a second
PhD G: Right . Anyway . {comment} So ,
Professor F: or more .
PhD G: uh , and that was basically it . And and we so we wrote this and then , we found we were at six pages , and then we started cutting furiously
PhD B: Oops .
PhD G: and threw out half of the material again , and uh played with the LaTeX stuff and Grad A: Made the font smaller and the narrows longer .
PhD G: uh , and until it fi
PhD B: Font smaller , yeah .
PhD G: No , no . W well , d you couldn't really make everything smaller
PhD B: Put the abstract end .
PhD G: but we s we put Oh , I I Grad A: Took out white space .
PhD G: you know the the gap between the two columns is like ten millimeters ,
PhD B: Yeah .
PhD G: so I d shrunk it to eight millimeters and that helped some . And stuff like that .
PhD D: Wasn't there wasn't there some result , Andreas Professor F: Yeah PhD D: I I thought maybe Liz presented this at some conference a while ago about uh , backchannels
PhD G: Mm - hmm . Mm - hmm .
PhD D: uh , and that they tend to happen when uh the pitch drops . You know you get a falling pitch . And so that 's when people tend to backchannel .
PhD G: Yeah . Well PhD D: Uh - i i do you rem
PhD G: y We didn't talk about , uh , prosodic , uh , properties at all ,
PhD D: Right . Right . But PhD G: although that 's I I take it that 's something that uh Don will will look at
Grad E: Yeah , we 're gonna be looking at that .
PhD G: now that we have the data and we have the alignment , so . This is purely based on you know the words
PhD D: Mm - hmm .
PhD G: and Postdoc C: I have a reference for that though . Uh - huh .
PhD D: Oh you do .
PhD G: Yeah .
PhD D: So am I recalling correctly ?
PhD G: Anyway , so .
Postdoc C: Well , I didn't know about Liz 's finding on that ,
PhD D: About Postdoc C: but I know of another paper that talks about something
PhD D: Uh - huh .
Postdoc C: that PhD D: Hmm .
Grad E: I 'd like to see that reference too .
Postdoc C: OK .
PhD D: It made me think about a cool little device that could be built to uh to handle those people that call you on the phone and just like to talk and talk and talk . And you just have this little detector that listens for these drops in pitch and gives them the backchannel . And so then you hook that to the phone and go off
Grad A: Yeah . Uh - huh .
PhD D: and do the  do whatever you r wanna do ,
PhD G: Oh yeah . Well PhD D: while that thing keeps them busy .
PhD G: There 's actually uh there 's this a former student of here from Berkeley , Nigel Nigel Ward .
PhD D: Uh - huh . Sure .
PhD G: Do you know him ?
PhD D: Yeah .
PhD G: He did a system uh , in he he lives in Japan now , and he did this backchanneling , automatic backchanneling system .
Professor F: Right .
PhD G: It 's a very PhD D: Oh !
PhD G: So , exactly what you describe ,
PhD D: Huh .
PhD G: but for Japanese . And it 's apparently for Japa - in Japanese it 's really important that you backchannel . It 's really impolite if you don't , and So .
Professor F: Huh . Actually for a lot of these people I think you could just sort of backchannel continuously and it would pretty much be fine .
PhD D: It wouldn't matter ? Yeah .
Grad E: Yeah . That 's w That 's what I do .
PhD D: Random intervals .
Grad A: There was there was of course a Monty Python sketch with that . Where the barber who was afraid of scissors was playing a a tape of clipping sounds , and saying " uh - huh " , " yeah " , " how about them sports teams ? "
PhD G: Anyway . So the paper 's on - line and y I I think I uh I CC ' ed a message to Meeting Recorder with the URL so you can get it .
Grad A: Yep .
Professor F: Yeah .
Grad A: Printed it out , haven't read it yet .
Professor F: Yeah .
PhD G: Um , uh one more thing . So I I 'm actually  about to send Brian Kingbury an email saying where he can find the the s the m the material he wanted for the s for the speech recognition experiment , so but I haven't sent it out yet because actually my desktop locked up , like I can't type anything . Uh b so if there 's any suggestions you have for that I was just gonna send him the PhD D: Is it the same directory that you had suggested ?
PhD G: I made a directory . I called it um Postdoc C: He still has his Unix account here , you know .
PhD G: Well this isn't Postdoc C: Yeah .
PhD G: He does ?
Postdoc C: And he and he 's PhD G: Yeah but but but he has to Postdoc C: I 'd hafta add him to Meeting Recorder , I guess ,
PhD G: he prefe he said he would prefer FTP
Postdoc C: but OK .
PhD G: and also , um , the other person that wants it There is one person at SRI who wants to look at the um , you know , the uh the data we have so far ,
Postdoc C: OK .
PhD G: and so I figured that FTP is the best approach . So what I did is I um   @ @ {comment} I made a n new directory after Chuck said that would c that was gonna be a good thing . Uh , so it 's " FTP  pub
Grad A: Pub real .
PhD G: real " Exactly . MTGC What is it again ? CR Grad A: Ask Dan Ellis .
Professor F: u R D RDR , yeah .
PhD G: Or Yeah . Right ? The same the same as the mailing list ,
Professor F: Yeah ,
PhD G: and Professor F: the  No vowels .
PhD G: Yeah . Um ,
Professor F: Yeah
PhD G: and then under there Um actually Oh and this directory , is not readable . It 's only uh , accessible . So , in other words , to access anything under there , you have to be told what the name is .
Grad A: Right .
PhD G: So that 's sort of a g quick and dirty way of doing access control .
Professor F: Mm - hmm .
PhD G: So uh , and the directory for this I call it I " ASR zero point one " because it 's sort of meant for recognition .
Professor F: So anyone who hears this meeting now knows the Grad A: Beta ?
PhD G: And then then in there I have a file that lists all the other files , so that someone can get that file and then know the file names and therefore download them . If you don't know the file names you can't Professor F: Is that a dash or a dot in there ?
PhD G: I mean you can Grad A: Don't don't don't say .
PhD G: Dash . Anyway . So all I all I was gonna do there was stick the the transcripts after we the way that we munged them for scoring , because that 's what he cares about , and um , and also and then the the waveforms that Don segmented . I mean , just basically tar them all up f I mean w for each meeting I tar them all into one tar file and G - zip them and stick them there .
Grad A: I uh , put digits in my own home directory home FTP directory ,
PhD G: And so .
Grad A: but I 'll probably move them there as well .
PhD G: Oh , OK .
PhD D: So we could point Mari to this also for her March O - one request ?
PhD G: OK . Yeah . March O - one .
PhD D: Or PhD G: Oh !
PhD D: You n Remember she was PhD G: Oh she wanted that also ?
PhD D: Well she was saying that it would be nice if we had they had a Or was she talking Yeah . She was saying it would be nice if they had eh the same set , so that when they did experiments they could compare .
PhD G: Right , but they don't have a recognizer even .
PhD D: Yeah .
Grad E: Um I
PhD G: But yeah , we can send I can CC Mari on this so that she knows PhD D: Yeah . So , for the thing that Postdoc C: That 's good .
PhD D: We need to give Brian the beeps file ,
PhD G: Right .
PhD D: so I was gonna probably put it Grad A: We can put it in the same place . Just put in another directory .
PhD D: Yeah , it I 'll make another directory .