Grad C: Yeah , the other thing We should look at ATLAS , the NIST thing ,
PhD F: Oh .
PhD A: Mmm .
Grad C: and see if they have anything at that level .
PhD F: Uh Grad C: I mean , I 'm not sure what to do about this with ATLAS , because they chose a different route . I chose something that Th - there are sort of two choices . Your your file format can know about know that you 're talking about language and speech , which is what I chose , and time , or your file format can just be a graph representation . And then the application has to impose the structure on top . So what it looked like ATLAS chose is , they chose the other way , which was their file format is just nodes and links , and you have to interpret what they mean yourself .
PhD F: And why did you not choose that type of approach ?
Grad C: Uh , because I knew that we were doing speech , and I thought it was better if you 're looking at a raw file to be t for the tags to say " it 's an utterance " , as opposed to the tag to say " it 's a link " .
PhD F: OK . OK .
Grad C: So , but PhD F: But other than that , are they compatible ? I mean , you could sort of Grad C: Yeah , they 're reasonably compatible .
PhD F: I mean , you you could PhD D: You could probably translate between them .
Grad C: Yep .
PhD F: Yeah , that 's w So ,
Grad C: So , well , the other thing is if we choose to use ATLAS , which maybe we should just do , we should just throw this out before we invest a lot of time in it .
PhD F: OK . I don't So this is what the meeting 's about ,
Grad C: Yeah .
PhD F: just sort of how to Um , cuz we need to come up with a database like this just to do our work . And I actually don't care , as long as it 's something useful to other people , what we choose .
Grad C: Yeah .
PhD F: So maybe it 's maybe oth you know , if if you have any idea of how to choose , cuz I don't .
Grad C: The only thing Yeah .
PhD A: Do they already have tools ?
Grad C: I mean , I I chose this for a couple reasons . One of them is that it 's easy to parse . You don't need a full XML parser . It 's very easy to just write a Perl script to parse it .
PhD A: As long as uh each tag is on one line .
Grad C: Exactly . Exactly . Which I always do .
PhD F: And you can have as much information in the tag as you want , right ?
Grad C: Well , I have it structured . Right ? So each type tag has only particular items that it can take .
PhD F: Can you But you can add to those structures if you Grad C: Sure . If you have more information . So what What NIST would say is that instead of doing this , you would say something like " link {nonvocalsound} start equals , um , you know , some node ID ,
PhD F: Yeah . So Grad C: end equals some other node ID " , and then " type " would be " utterance " .
PhD A: Hmm .
Grad C: You know , so it 's very similar .
PhD F: So why would it be a a waste to do it this way if it 's similar enough that we can always translate it ?
PhD D: It probably wouldn't be a waste . It would mean that at some point if we wanted to switch , we 'd just have to translate everything .
Grad C: Write a translator . But it se Since they are developing a big PhD F: But it but that sounds PhD D: But that 's I don't think that 's a big deal .
PhD F: As long as it is Grad C: they 're developing a big infrastructure . And so it seems to me that if if we want to use that , we might as well go directly to what they 're doing , rather than PhD A: If we want to Do they already have something that 's that would be useful for us in place ?
PhD D: Yeah . See , that 's the question . I mean , how stable is their Are they ready to go ,
Grad C: The I looked at it PhD D: or ?
Grad C: The last time I looked at it was a while ago , probably a year ago , uh , when we first started talking about this .
PhD D: Hmm .
Grad C: And at that time at least it was still not very complete . And so , specifically they didn't have any external format representation at that time . They just had the sort of conceptual node uh , annotated transcription graph , which I really liked . And that 's exactly what this stuff is based on . Since then , they 've developed their own external file format , which is , uh , you know , this sort of s this sort of thing . Um , and apparently they 've also developed a lot of tools , but I haven't looked at them . Maybe I should .
PhD A: We should we should find out .
PhD F: I mean , would the tools would the tools run on something like this , if you can translate them anyway ?
Grad C: Um , th what would would would what would worry me is that maybe we might miss a little detail
PhD A: It 's a hassle
PhD F: I mean , that I guess it 's a question that PhD A: if PhD F: uh , yeah .
Grad C: that would make it very difficult to translate from one to the other .
PhD F: OK .
PhD A: I I think if it 's conceptually close , and they already have or will have tools that everybody else will be using , I mean , it would be crazy to do something s you know , separate that PhD F: OK .
Grad C: Yeah , we might as well . Yep .
PhD F: Yeah .
Grad C: So I 'll I 'll take a closer look at it .
PhD F: Actually , so it 's that that would really be the question , is just what you would feel is in the long run the best thing .
Grad C: And Right .
PhD F: Cuz once we start , sort of , doing this I don't we don't actually have enough time to probably have to rehash it out again
Grad C: The Yep . The other thing the other way that I sort of established this was as easy translation to and from the Transcriber format .
PhD F: and s Right .
Grad C: Um ,
PhD F: Right .
Grad C: but PhD F: I mean , I like this . This is sort of intuitively easy to actually r read ,
Grad C: Yep .
PhD F: as easy it could as it could be . But , I suppose that as long as they have a type here that specifies " utt " , um ,
Grad C: It 's almost the same .
PhD F: it 's yeah , close enough that Grad C: The the the the point is with this , though , is that you can't really add any supplementary information . Right ? So if you suddenly decide that you want PhD F: You have to make a different type .
Grad C: Yeah . You 'd have to make a different type .
PhD F: So Well , if you look at it and Um , I guess in my mind I don't know enough Jane would know better , {comment} about the types of annotations and and But I imagine that those are things that would well , you guys mentioned this , {comment} that could span any it could be in its own channel , it could span time boundaries of any type ,
Grad C: Right .
PhD F: it could be instantaneous , things like that . Um , and then from the recognition side we have backtraces at the phone - level .
Grad C: Right .
PhD F: If if it can handle that , it could handle states or whatever . And then at the prosody - level we have frame sort of like cepstral feature files ,
Grad C: Yep .
PhD F: uh , like these P - files or anything like that . And that 's sort of the world of things that I And then we have the aligned channels , of course ,
Grad C: Right .
PhD A: It seems to me you want to keep the frame - level stuff separate .
PhD F: and Yeah .
PhD A: And then PhD F: I I definitely agree and I wanted to find actually a f a nicer format or a maybe a more compact format than what we used before .
Grad C: Right .
PhD F: Just cuz you 've got ten channels or whatever and two hours of a meeting . It 's it 's a lot of Grad C: Huge .
PhD A: Now now how would you how would you represent , um , multiple speakers in this framework ? Were You would just represent them as Grad C: Um ,
PhD A: You would have like a speaker tag or something ?
Grad C: there 's a spea speaker tag up at the top which identifies them and then each utt the way I had it is each turn or each utterance , {comment} I don't even remember now , had a speaker ID tag attached to it .
PhD A: Mm - hmm . OK .
Grad C: And in this format you would have a different tag , which which would , uh , be linked to the link . So so somewhere else you would have another thing that would be ,
PhD F: Yeah .
Grad C: um Let 's see , would it be a node or a link ? Um And so so this one would have , um , an ID is link {comment} link seventy - four or something like that .
PhD A: Mm - hmm .
Grad C: And then somewhere up here you would have a link that that , uh , you know , was referencing L - seventy - four and had speaker Adam .
PhD A: Is i ?
Grad C: You know , or something like that .
PhD F: Actually , it 's the channel , I think , that PhD A: Well , channel or speaker or whatever .
PhD F: I mean , w yeah , channel is what the channelized output out
PhD A: It doesn't Grad C: This isn't quite right .
PhD A: Right .
Grad C: I have to look at it again .
PhD F: Yeah , but PhD A: But but so how in the NIST format do we express a hierarchical relationship between , um , say , an utterance and the words within it ? So how do you tell that these are the words that belong to that utterance ?
Grad C: Um , you would have another structure lower down than this that would be saying they 're all belonging to this ID .
PhD A: Mm - hmm .
PhD D: So each thing refers to the utterance that it belongs to .
Grad C: Right . And then each utterance could refer to a turn ,
PhD D: So it 's it 's not hi it 's sort of bottom - up .
Grad C: and each turn could refer to something higher up .
PhD F: And what if you actually have So right now what you have as utterance , um , the closest thing that comes out of the channelized is the stuff between the segment boundaries that the transcribers put in or that Thilo put in , which may or may not actually be , like , a s it 's usually not um , the beginning and end of a sentence , say .
Grad C: Well , that 's why I didn't call it " sentence " .
PhD F: So , right . Um , so it 's like a segment or something .
Grad C: Yeah .
PhD F: So , I mean , I assume this is possible , that if you have someone annotates the punctuation or whatever when they transcribe , you can say , you know , from for from the c beginning of the sentence to the end of the sentence , from the annotations , this is a unit , even though it never actually i It 's only a unit by virtue of the annotations at the word - level .
Grad C: Sure . I mean , so you would you would have yet another tag .
PhD F: And then that would get a tag somehow .
Grad C: You 'd have another tag which says this is of type " sentence " .
PhD F: OK . OK .
Grad C: And , what PhD F: But it 's just not overtly in the PhD A: OK .
PhD F: Um , cuz this is exactly the kind of PhD A: So PhD F: I think that should be possible as long as the But , uh , what I don't understand is where the where in this type of file that would be expressed .
Grad C: Right . You would have another tag somewhere . It 's well , there 're two ways of doing it .
PhD F: S so it would just be floating before the sentence or floating after the sentence without a time - mark .
Grad C: You could have some sort of link type type equals " sentence " , and ID is " S - whatever " . And then lower down you could have an utterance . So the type is " utterance " equals " utt " . And you could either say that No . I don't know PhD A: So here 's the thing .
Grad C: I take that back .
PhD A: Um Grad C: Can you can you say that this is part of this ,
PhD F: See , cuz it 's PhD A: Hhh .
PhD F: it 's PhD D: You would just have a r
PhD F: S
Grad C: or do you say this is part of this ? I think PhD D: You would refer up to the sentence .
PhD F: But they 're PhD A: Well , the thing PhD F: they 're actually overlapping each other , sort of .
Grad C: So PhD A: the thing is that some something may be a part of one thing for one purpose and another thing of another purpose .
Grad C: Right .
PhD A: So f
PhD F: You have to have another type then , I guess .
PhD A: s Um , well , s let 's let 's ta so let 's Grad C: Well , I think I 'm I think w I had better look at it again
PhD F: Yeah .
PhD A: so Grad C: because I I 'm PhD F: OK . OK .
PhD A: y So for instance @ @ {comment} sup
Grad C: There 's one level there 's one more level of indirection that I 'm forgetting .
PhD A: Suppose you have a word sequence and you have two different segmentations of that same word sequence . f Say , one segmentation is in terms of , um , you know , uh , sentences . And another segmentation is in terms of , um , I don't know , {comment} prosodic phrases . And let 's say that they don't nest . So , you know , a prosodic phrase may cross two sentences or something .
Grad C: Right .
PhD A: I don't know if that 's true or not but let 's as
PhD F: Well , it 's definitely true with the segment .
PhD A: Right .
PhD F: That 's what I exactly what I meant by the utterances versus the sentence could be sort of PhD A: Yeah . So , you want to be s you want to say this this word is part of that sentence and this prosodic phrase .
PhD F: Yeah .
PhD A: But the phrase is not part of the sentence
PhD F: Yeah .
PhD A: and neither is the sentence part of the phrase .
PhD F: Right .
Grad C: I I 'm pretty sure that you can do that , but I 'm forgetting the exact level of nesting .
PhD A: So , you would have to have two different pointers from the word up one level up , one to the sent
Grad C: So so what you would end up having is a tag saying " here 's a word , and it starts here and it ends here " .
PhD A: Right .
Grad C: And then lower down you would say " here 's a prosodic boundary and it has these words in it " . And lower down you 'd have " here 's a sentence ,
PhD A: Right .
PhD F: An - Right .
Grad C: and it has these words in it " .
PhD F: So you would be able to go in and say , you know , " give me all the words in the bound in the prosodic phrase
Grad C: Yep .
PhD F: and give me all the words in the " Yeah .
Grad C: So I think that 's that would wor
PhD F: Um , OK .
Grad C: Let me look at it again .
PhD A: Mm - hmm . The the o the other issue that you had was , how do you actually efficiently extract , um find and extract information in a structure of this type ?
PhD F: OK .
Grad C: So .
PhD F: That 's good .
PhD A: So you gave some examples like PhD F: Well , uh , and , I mean , you guys might I don't know if this is premature because I suppose once you get the representation you can do this , but the kinds of things I was worried about is ,
PhD A: No , that 's not clear .
PhD F: uh PhD A: I mean , yeah , you c sure you can do it ,
PhD F: Well , OK . So i if it PhD A: but can you do it sort of l l you know , it PhD F: I I mean , I can't do it , but I can um ,
PhD A: y y you gotta you gotta do this you you 're gonna want to do this very quickly
Grad C: Well PhD A: or else you 'll spend all your time sort of searching through very complex data structures PhD F: Right . You 'd need a p sort of a paradigm for how to do it . But an example would be " find all the cases in which Adam started to talk while Andreas was talking and his pitch was rising , Andreas 's pitch " . That kind of thing .
Grad C: Right . I mean , that 's gonna be Is the rising pitch a feature , or is it gonna be in the same file ?
PhD F: Well , the rising pitch will never be hand - annotated . So the all the prosodic features are going to be automatically Grad C: But the I mean , that 's gonna be hard regardless ,
PhD F: So they 're gonna be in those Grad C: right ? Because you 're gonna have to write a program that goes through your feature file and looks for rising pitches .
PhD A: Yeah .
PhD F: So Right . So normally what we would do is we would say " what do we wanna assign rising pitch to ? " Are we gonna assign it to words ? Are we gonna just assign it to sort of when it 's rising we have a begin - end rise representation ? But suppose we dump out this file and we say , uh , for every word we just classify it as , w you know , rise or fall or neither ?
Grad C: OK . Well , in that case you would add that to this format
PhD F: OK .
Grad C: r
PhD F: So we would basically be sort of , um , taking the format and enriching it with things that we wanna query in relation to the words that are already in the file ,
Grad C: Right .
PhD F: and then querying it .
PhD A: You want sort of a grep that 's that works at the structural on the structural representation .
PhD F: OK .
Grad C: You have that . There 's a standard again in XML , specifically for searching XML documents structured X - XML documents , where you can specify both the content and the structural position .
PhD A: Yeah , but it 's it 's not clear that that 's That 's relative to the structure of the XML document ,
PhD F: If PhD A: not to the structure of what you 're representing in the document .
Grad C: You use it as a tool . You use it as a tool , not an end - user . It 's not an end - user thing .
PhD A: Right .
Grad C: It 's it 's you would use that to build your tool to do that sort of search .
PhD A: Right . Be Because here you 're specifying a lattice .
PhD F: Uh PhD A: So the underlying that 's the underlying data structure . And you want to be able to search in that lattice .
PhD F: But as long as the Grad C: It 's a graph , but PhD A: That 's different from searching through the text .
PhD F: But it seems like as long as the features that Grad C: Well , no , no , no . The whole point is that the text and the lattice are isomorphic . They represent each other completely .
PhD A: Um Grad C: So that I mean th
PhD F: That 's true if the features from your acoustics or whatever that are not explicitly in this are at the level of these types .
PhD A: Hhh .
PhD F: That that if you can do that Grad C: Yeah , but that 's gonna be the trouble no matter what . Right ? No matter what format you choose , you 're gonna have the trou you 're gonna have the difficulty of relating the the frame - level features PhD F: That 's right . That 's true . That 's why I was trying to figure out what 's the best format for this representation .
Grad C: Yep .
PhD F: And it 's still gonna be PhD A: Hmm .
PhD F: it 's still gonna be , uh , not direct .
Grad C: Right .
PhD F: You know , it Or another example was , you know , uh , where in the language where in the word sequence are people interrupting ? So , I guess that one 's actually easier .
PhD D: What about what about , um , the idea of using a relational database to , uh , store the information from the XML ? So you would have XML basically would Uh , you you could use the XML to put the data in , and then when you get data out , you put it back in XML . So use XML as sort of the the transfer format ,
Grad C: Transfer .
PhD D: uh , but then you store the data in the database , which allows you to do all kinds of good search things in there .
Grad C: The , uh One of the things that ATLAS is doing is they 're trying to define an API which is independent of the back store ,
PhD F: Huh .
Grad C: so that , uh , you could define a single API and the the storage could be flat XML files or a database .
PhD D: Mm - hmm .
Grad C: My opinion on that is for the s sort of stuff that we 're doing , {comment} I suspect it 's overkill to do a full relational database , that , um , just a flat file and , uh , search tools I bet will be enough .
PhD A: But Grad C: But that 's the advantage of ATLAS , is that if we actually take decide to go that route completely and we program to their API , then if we wanted to add a database later it would be pretty easy .
PhD D: Mm - hmm . Mm - hmm .
PhD F: It seems like the kind of thing you 'd do if I don't know , if people start adding all kinds of s bells and whistles to the data . And so that might be I mean , it 'd be good for us to know to use a format where we know we can easily , um , input that to some database if other people are using it .