PhD F: um Yeah . And so then it would just be a matter of getting making sure to handle the annotations that are , you know , not at the word - level and , um , t to import the
Grad B: Where are those annotations coming from ?
PhD F: Well , right now , I g Jane would  would Grad C: Mm - hmm .
PhD F: Yeah .
Postdoc E: Are you talking about the overlap a annotations ?
PhD F: Yeah , any kind of annotation that , like , isn't already there . Uh , you know , anything you can envision .
Postdoc E: Yeah . So what I was imagining was um , so Dave says we can have unlimited numbers of green ribbons . And so put , uh , a a green ribbon on for an overlap code . And since we w we I I think it 's important to remain flexible regarding the time bins for now . And so it 's nice to have However , you know , you want to have it , uh , time time uh , located in the discourse . So , um , if we if we tie the overlap code to the first word in the overlap , then you 'll have a time - marking . It won't it 'll be independent of the time bins , however these e evolve , shrink , or whatever , increase , or Also , you could have different time bins for different purposes . And having it tied to the first word in an overlap segment is unique , uh , you know , anchored , clear . And it would just end up on a separate ribbon .
Grad C: Right .
Postdoc E: So the overlap coding is gonna be easy with respect to that . You look puzzled .
PhD D: I I just I don't quite understand what these things are .
Postdoc E: OK .
PhD D: Uh .
Postdoc E: What , the codes themselves ?
PhD D: Well , th overlap codes .
Postdoc E: Or the ?
PhD D: I 'm not sure what that @ @ Grad C: Well , I mean , is that PhD D: It probably doesn't matter .
Postdoc E: Well , we don't have to go into the codes .
Grad C: I mean , it doesn't .
PhD D: No , I d
Postdoc E: We don't have to go into the codes .
Grad C: I mean , that not for the topic of this meeting .
Postdoc E: But let me just No . W the idea is just to have a separate green ribbon , you know , and and and let 's say that this is a time bin . There 's a word here . This is the first word of an overlapping segment of any length , overlapping with any other , uh , word uh , i segment of any length . And , um , then you can indicate that this here was perhaps a ch a backchannel , or you can say that it was , um , a usurping of the turn , or you can you know , any any number of categories . But the fact is , you have it time - tagged in a way that 's independent of the , uh , sp particular time bin that the word ends up in . If it 's a large unit or a small unit , or
PhD A: Mm - hmm .
Postdoc E: we sh change the boundaries of the units , it 's still unique and and , uh , fits with the format ,
PhD F: Right .
Postdoc E: flexible , all that .
PhD A: Um , it would be nice um , eh , gr this is sort of r regarding uh , uh it 's related but not directly germane to the topic of discussion , but , when it comes to annotations , um , you often find yourself in the situation where you have different annotations of the same , say , word sequence . OK ?
Postdoc E: Yeah .
PhD A: And sometimes the word sequences even differ slightly because they were edited s at one place but not the other .
Postdoc E: Yeah .
PhD A: So , once this data gets out there , some people might start annotating this for , I don't know , dialogue acts or , um , you know , topics or what the heck . You know , there 's a zillion things that people might annotate this for . And the only thing that is really sort of common among all the versi the various versions of this data is the word sequence , or approximately .
Postdoc E: Yep .
PhD F: Or the time .
PhD A: Or the times . But , see , if you 'd annotate dialogue acts , you don't necessarily want to or topics you don't really want to be dealing with time - marks .
PhD F: I guess .
PhD A: You 'd it 's much more efficient for them to just see the word sequence , right ?
PhD F: Mm - hmm .
PhD A: I mean , most people aren't as sophisticated as as we are here with , you know , uh , time alignments and stuff . So So the the the point is Grad C: Should should we mention some names on the people who are n ?
PhD A: Right . So , um , the p my point is that you 're gonna end up with , uh , word sequences that are differently annotated . And you want some tool , uh , that is able to sort of merge these different annotations back into a single , uh , version . OK ? Um , and we had this problem very massively , uh , at SRI when we worked , uh , a while back on , uh well , on dialogue acts as well as , uh , you know , um , what was it ? uh ,
PhD F: Well , all the Switchboard in it .
PhD A: utterance types . There 's , uh , automatic , uh , punctuation and stuff like that .
PhD F: Yeah .
PhD A: Because we had one set of annotations that were based on , uh , one version of the transcripts with a particular segmentation , and then we had another version that was based on , uh , a different s slightly edited version of the transcripts with a different segmentation . So , we had these two different versions which were you know , you could tell they were from the same source but they weren't identical . So it was extremely hard to reliably merge these two back together to correlate the information from the different annotations .
Grad C: Yep . I I don't see any way that file formats are gonna help us with that .
PhD A: No .
Grad C: It 's it 's all a question of semantic .
PhD A: No . But once you have a file format , I can imagine writing not personally , but someone writing a tool that is essentially an alignment tool , um , that mediates between various versions ,
PhD F: Mm - hmm .
Grad C: Yeah .
PhD A: and uh , sort of like th uh , you know , you have this thing in UNIX where you have , uh , diff .
Grad C: Diff .
PhD F: W - diff or diff .
PhD A: There 's the , uh , diff that actually tries to reconcile different two diffs f {comment} based on the same original .
PhD F: Yeah .
Postdoc E: Is it S - diff ?
Grad C: Yep .
Postdoc E: Mmm .
PhD A: Something like that , um , but operating on these lattices that are really what 's behind this uh , this annotation format .
Grad C: Yep .
PhD A: So Grad C: There 's actually a diff library you can use to do things like that that so you have different formats .
PhD F: You could definitely do that with the PhD A: So somewhere in the API you would like to have like a merge or some some function that merges two two versions .
Grad C: Yeah , I think it 's gonna be very hard . Any sort of structured anything when you try to merge is really , really hard
PhD A: Right .
Grad C: because you ha i The hard part isn't the file format . The hard part is specifying what you mean by " merge " .
PhD A: Is Exactly .
Grad C: And that 's very difficult .
PhD F: But the one thing that would work here actually for i that is more reliable than the utterances is the the speaker ons and offs . So if you have a good ,
Grad C: But this is exactly what I mean , is that that the problem i
PhD F: um Yeah . You just have to know wha what to tie it to .
Grad C: Yeah , exactly . The problem is saying " what are the semantics ,
PhD F: And Grad C: what do you mean by " merge " ? "
PhD F: Right , right .
PhD A: Right . So so just to let you know what we where we kluged it by , uh , doing uh , by doing Hhh .
Grad C: So .
PhD A: Both were based on words , so , bo we have two versions of the same words intersp you know , sprinkled with with different tags for annotations .
Grad C: And then you did diff .
PhD A: And we did diff . Exactly !
Grad C: Yeah , that 's just what I thought .
PhD A: And that 's how Grad C: That 's just wh how I would have done it .
PhD A: Yeah . But , you know , it had lots of errors and things would end up in the wrong order , and so forth . Uh , so , um , if you had a more Grad C: Yep .
PhD A: Uh , it it was a kluge because it was basically reducing everything to uh , to uh , uh , to textual alignment .
Grad C: A textual PhD A: Um , so PhD F: But , d isn't that something where whoever if  if the people who are making changes , say in the transcripts , cuz this all happened when the transcripts were different ye um , if they tie it to something , like if they tied it to the acoustic segment if they You know what I mean ? Then Or if they tied it to an acoustic segment and we had the time - marks , that would help .
Grad C: Yep .
PhD F: But the problem is exactly as Adam said , that you get , you know , y you don't have that information or it 's lost in the merge somehow ,
Postdoc E: Well , can I ask one question ?
PhD F: so Postdoc E: It it seems to me that , um , we will have o an official version of the corpus , which will be only one one version in terms of the words where the words are concerned . We 'd still have the the merging issue maybe if coding were done independently of the PhD A: And you 're gonna get that
Postdoc E: But but PhD A: because if the data gets out , people will do all kinds of things to it . And , uh , s you know , several years from now you might want to look into , um , the prosody of referring expressions . And someone at the university of who knows where has annotated the referring expressions . So you want to get that annotation and bring it back in line with your data .
Grad C: Right .
PhD A: OK ?
Grad C: But unfortunately they 've also hand - edited it .
Postdoc E: OK , then PhD F: But they 've also Exactly . And so that 's exactly what we should somehow when you distribute the data , say that you know , that have some way of knowing how to merge it back in and asking people to try to do that .
PhD A: Yeah .
Grad C: Yep .
PhD A: Right .
Postdoc E: Well , then the PhD D: What 's what 's wrong with doing times ? I Postdoc E: I agree . That was what I was wondering .
PhD F: Uh , yeah , time is the Grad C: Well ,
Postdoc E: Time is unique . You were saying that you didn't think we should PhD F: Time is passing !
PhD A: Time time times are ephemeral .
Postdoc E: Andreas was saying Yeah .
Grad C: what if they haven't notated with them , times ?
PhD F: Yeah . He he 's a language modeling person , though .
PhD A: Um Grad C: So so imagine I think his his example is a good one . Imagine that this person who developed the corpus of the referring expressions didn't include time .
PhD A: Mm - hmm . Yeah .
Grad C: He included references to words .
Postdoc E: Ach !
PhD A: Yeah .
Grad C: He said that at this word is when when it happened .
Postdoc E: Well , then PhD A: Or she .
Grad C: Or she .
Postdoc E: But then couldn't you just indirectly figure out the time tied to the word ?
PhD F: But still they Exactly .
Grad C: Sure . But what if what if they change the words ?
PhD F: Yeah .
Postdoc E: Not Well , but you 'd have some anchoring point . He couldn't have changed all the words .
PhD D: But can they change the words without changing the time of the word ?
Grad C: Sure . But they could have changed it a little . The the point is , that that they may have annotated it off a word transcript that isn't the same as our word transcript , so how do you merge it back in ? I understand what you 're saying .
PhD A: Mmm . Mm - hmm .
Grad C: And I I guess the answer is , um , it 's gonna be different every time . It 's j it 's just gonna be Postdoc E: Yeah .
PhD F: Yeah .
Grad C: I it 's exactly what I said before ,
PhD F: You only know the boundaries of the Grad C: which is that " what do you mean by " merge " ? " So in this case where you have the words and you don't have the times , well , what do you mean by " merge " ? If you tell me what you mean , I can write a program to do it .
PhD F: Right . Right . You can merge at the level of the representation that the other person preserved and that 's it .
Grad C: Right . And that 's about all you can do .
PhD F: And beyond that , all you know is is relative ordering and sometimes even that is wrong .
Grad C: So so in so in this one you would have to do a best match between the word sequences ,
PhD F: So .
PhD A: Mm - hmm .
Grad C: extract the times f from the best match of theirs to yours , and use that .
PhD F: And then infer that their time - marks are somewhere in between .
Grad C: Right .
PhD F: Yeah , exactly .
Postdoc E: But it could be that they just uh , I mean , it could be that they chunked they they lost certain utterances and all that stuff ,
Grad C: Right , exactly . So it could get very , very ugly .
Postdoc E: or PhD F: Definitely .
Postdoc E: Yeah .
PhD F: Definitely . Alright .
Postdoc E: That 's interesting .
PhD F: Well , I guess , w I I didn't want to keep people too long and Adam wanted t people I 'll read the digits . If anyone else offers to , that 'd be great . And
PhD A: Ah , well .
Grad C: Yeah .
PhD F: if not , I guess PhD A: For th for the {nonvocalsound} for the benefit of science we 'll read the digits .