C: Exactly .
C: Exactly .
C: Which I always do .
F: And you can have as much information in the tag as you want ,
F: right ?
C: Well , I have it structured .
C: Right ?
C: So each type tag has only particular items that it can take .
F: Can you But you can add to those structures if you C: Sure .
C: If you have more information .
C: So what What NIST would say is that instead of doing this , you would say something like " link <nonvocalsound> start equals , um , you know , some node ID ,
F: Yeah .
F: So C: end equals some other node ID " ,
C: and then " type " would be " utterance " .
A: Hmm .
C: You know , so it 's very similar .
F: So why would it be a a waste to do it this way if it 's similar enough that we can always translate it ?
D: It probably wouldn't be a waste .
D: It would mean that at some point if we wanted to switch , we 'd just have to translate everything .
C: Write a translator .
C: But it se
F: But it but that sounds C: Since they are developing a big D: But that 's D: I don't think that 's a big deal .
F: As long as it is C: they 're developing a big infrastructure .
C: And so it seems to me that if if we want to use that , we might as well go directly to what they 're doing , rather than A: If we want to A: Do they already have something that 's that would be useful for us in place ?
D: Yeah .
D: See , that 's the question .
D: I mean , how stable is their D: Are they ready to go ,
C: The I looked at it D: or ?
C: The last time I looked at it was a while ago ,
C: probably a year ago ,
D: Hmm .
C: uh , when we first started talking about this .
C: And at that time at least it was still not very complete .
C: And so , specifically they didn't have any external format representation at that time .
C: They just had the sort of conceptual node uh , annotated transcription graph ,
C: which I really liked .
C: And that 's exactly what this stuff is based on .
C: Since then , they 've developed their own external file format ,
C: which is , uh , you know , this sort of s this sort of thing .
C: Um , and apparently they 've also developed a lot of tools ,
C: but I haven't looked at them .
C: Maybe I should .
A: We should we should find out .
F: I mean , would the tools would the tools run on something like this , if you can translate them anyway ?
C: Um , th what would would would what would worry me is that maybe we might miss a little detail
F: I mean , that I guess it 's a question that A: It 's a hassle
F: uh , yeah .
A: if C: that would make it very difficult to translate from one to the other .
F: OK .
A: I I think if it 's conceptually close , and they already have or will have tools that everybody else will be using , I mean , it would be crazy to do something s you know , separate that F: OK .
C: Yeah , we might as well .
C: Yep .
F: Yeah .
C: So I 'll I 'll take a closer look at it .
F: Actually ,
F: so it 's F: that that would really be the question , is just what you would feel is in the long run the best thing .
C: And C: Right .
F: Cuz once we start , sort of , doing this I don't we don't actually have enough time to probably have to rehash it out again
C: The C: Yep .
F: and C: The other thing the other way that I sort of established this was as easy translation to and from the Transcriber format .
F: s
F: Right .
C: Um ,
F: Right .
C: but F: I mean , I like this .
F: This is sort of intuitively easy to actually r read ,
C: Yep .
F: as easy it could as it could be .
F: But ,
F: I suppose that as long as they have a type here that specifies " utt " ,
F: um ,
C: It 's almost the same .
F: it 's yeah , close enough that C: The the the the point is with this , though , is that you can't really add any supplementary information .
C: Right ?
C: So if you suddenly decide that you want F: You have to make a different type .
C: Yeah .
C: You 'd have to make a different type .
F: So F: Well , if you look at it
F: and Um ,
F: I guess in my mind I don't know enough Jane would know better , <comment> about the types of annotations
F: and and But I imagine that those are things that would well , you guys mentioned this , <comment> that could span any F: it could be in its own channel ,
F: it could span time boundaries of any type ,
C: Right .
F: it could be instantaneous ,
F: things like that .
F: Um ,
F: and then from the recognition side we have backtraces at the phone - level .
F: If if it can handle that , it could handle states or whatever .
C: Right .
F: And then at the prosody - level we have frame sort of like cepstral feature files ,
C: Yep .
F: uh , like these P - files or anything like that .
F: And that 's sort of the world of things that I F: And then we have the aligned channels , of course ,
C: Right .
F: and A: It seems to me you want to keep the frame - level stuff separate .
F: Yeah .
F: I I definitely agree
A: And then F: and I wanted to find actually a f a nicer format or a maybe a more compact format than what we used before .
C: Right .
F: Just cuz you 've got ten channels or whatever and two hours of a meeting .
F: It 's it 's a lot of A: Now now how would you how would you represent , um , multiple speakers in this framework ?
C: Huge .
A: Were You would just represent them as C: Um ,
A: You would have like a speaker tag or something ?
C: there 's a spea speaker tag up at the top which identifies them
C: and then each utt
C: the way I had it is each turn or each utterance , <comment> I don't even remember now , had a speaker ID tag attached to it .
A: Mm - hmm .
A: OK .
C: And in this format you would have a different tag ,
C: which which would , uh , be linked to the link .
F: Yeah .
C: So so somewhere else you would have another thing that would be ,
C: um C: Let 's see , would it be a node or a link ?
C: Um C: And so so this one would have , um , an ID is link <comment> link seventy - four or something like that .
C: And then somewhere up here you would have a link that that , uh , you know , was referencing L - seventy - four and had speaker Adam .
A: Mm - hmm .
F: Actually , it 's the channel , I think , that A: Is i ?
C: You know ,
C: or something like that .
F: I mean , w
A: Well , channel or speaker or whatever .
F: yeah , channel is what the channelized output out
A: It doesn't C: This isn't quite right .
C: I have to look at it again .
A: Right .
F: Yeah , but A: But but so how in the NIST format do we express a hierarchical relationship between , um , say , an utterance and the words within it ?
A: So how do you tell that these are the words that belong to that utterance ?
C: Um , you would have another structure lower down than this that would be saying they 're all belonging to this ID .
A: Mm - hmm .
D: So each thing refers to the utterance that it belongs to .
C: Right .
D: So it 's it 's not hi it 's sort of bottom - up .
C: And then each utterance could refer to a turn ,
C: and each turn could refer to something higher up .
F: And what if you actually have F: So right now what you have as utterance , um , the closest thing that comes out of the channelized is the stuff between the segment boundaries that the transcribers put in or that Thilo put in ,
F: which may or may not actually be , like , a s it 's usually not um , the beginning and end of a sentence , say .
C: Well , that 's why I didn't call it " sentence " .
F: So ,
F: right .
F: Um , so it 's like a segment or something .
C: Yeah .
F: So ,
F: I mean , I assume this is possible , that if you have someone annotates the punctuation or whatever when they transcribe , you can say , you know , from for from the c beginning of the sentence to the end of the sentence , from the annotations , this is a unit ,
F: even though it never actually F: i It 's only a unit by virtue of the annotations at the word - level .
C: Sure .
C: I mean , so you would you would have yet another tag .
F: And then that would get a tag somehow .
C: You 'd have another tag which says this is of type " sentence " .
F: OK .
F: OK .
C: And ,
F: But it 's just not overtly in the C: what A: OK .
F: Um , cuz this is exactly the kind of A: So F: I think that should be possible as long as the F: But , uh , what I don't understand is where the where in this type of file that would be expressed .
C: Right .
C: You would have another tag somewhere .
C: It 's well , there 're two ways of doing it .
F: S so it would just be floating before the sentence or floating after the sentence without a time - mark .
C: You could have some sort of link type C: type equals " sentence " ,
C: and ID is " S - whatever " .
C: And then lower down you could have an utterance .
C: So the type is " utterance " equals " utt " .
C: And you could either say that C: No .
C: I don't know C: I take that back .
A: So here 's the thing .
A: Um F: See , cuz it 's C: Can you can you say that this is part of this ,
A: Hhh .
F: it 's D: You would just have a r
F: S
C: or do you say this is part of this ?
C: I think F: But they 're D: You would refer up to the sentence .
A: Well , the thing F: they 're actually overlapping each other , sort of .
C: So A: the thing is that some something may be a part of one thing for one purpose and another thing of another purpose .
C: Right .
F: You have to have another type then , I guess .
A: So f
A: s
A: Um ,
A: well ,
C: Well , I think I 'm I think w I had better look at it again
A: s let 's let 's ta so let 's F: Yeah .
F: OK .
A: so C: because I I 'm F: OK .
A: y So for instance @ @ <comment> sup
C: There 's one level there 's one more level of indirection that I 'm forgetting .
A: Suppose you have a word sequence
A: and you have two different segmentations of that same word sequence .
A: f Say , one segmentation is in terms of , um , you know , uh , sentences .
A: And another segmentation is in terms of , um , I don't know , <comment> prosodic phrases .
A: And let 's say that they don't nest .
A: So , you know , a prosodic phrase may cross two sentences or something .
C: Right .
A: I don't know if that 's true or not
A: but let 's as
F: Well , it 's definitely true with the segment .
A: Right .
F: That 's what I exactly what I meant by the utterances versus the sentence could be sort of A: Yeah .
A: So , you want to be s you want to say this this word is part of that sentence and this prosodic phrase .
F: Yeah .
F: Yeah .
A: But the phrase is not part of the sentence
A: and neither is the sentence part of the phrase .
F: Right .
C: I I 'm pretty sure that you can do that ,
C: but I 'm forgetting the exact level of nesting .
A: So , you would have to have two different pointers from the word up one level up ,
A: one to the sent
C: So so what you would end up having is a tag saying " here 's a word ,
C: and it starts here and it ends here " .
A: Right .
C: And then lower down you would say " here 's a prosodic boundary
C: and it has these words in it " .
C: And lower down you 'd have " here 's a sentence ,
F: An - Right .
A: Right .
C: and it has these words in it " .
F: So you would be able to go in and say , you know , " give me all the words in the bound in the prosodic phrase
F: and give me all the words in the "
C: Yep .
F: Yeah .
C: So I think that 's that would wor
F: Um , OK .
C: Let me look at it again .
A: Mm - hmm .
F: OK .
A: The the o the other issue that you had was , how do you actually efficiently extract , um find and extract information in a structure of this type ?
C: So .
F: That 's good .
A: So you gave some examples like F: Well ,
F: uh , and , I mean , you guys might F: I don't know if this is premature
F: because I suppose once you get the representation you can do this ,
F: but the kinds of things I was worried about is ,
A: No , that 's not clear .
F: uh A: I mean , yeah , you c sure you can do it ,
F: Well , OK .
F: So i if it A: but can you do it sort of l l you know , it F: I I mean , I can't do it ,
F: but I can F: um ,
A: y y you gotta you gotta do this you you 're gonna want to do this very quickly
C: Well A: or else you 'll spend all your time sort of searching through very complex data structures F: Right .
F: You 'd need a p sort of a paradigm for how to do it .
F: But an example would be " find all the cases in which Adam started to talk while Andreas was talking and his pitch was rising ,
F: Andreas 's pitch " .
F: That kind of thing .
C: Right .
C: I mean , that 's gonna be C: Is the rising pitch a feature ,
C: or is it gonna be in the same file ?
F: Well , the rising pitch will never be hand - annotated .
F: So the all the prosodic features are going to be automatically C: But the C: I mean , that 's gonna be hard regardless ,
F: So they 're gonna be in those C: right ?
C: Because you 're gonna have to write a program that goes through your feature file and looks for rising pitches .
A: Yeah .
F: So F: Right .
F: So normally what we would do is we would say " what do we wanna assign rising pitch to ? "
F: Are we gonna assign it to words ?
F: Are we gonna just assign it to sort of F: when it 's rising we have a begin - end rise representation ?
F: But suppose we dump out this file
F: and we say , uh , for every word we just classify it as , w you know , rise or fall or neither ?
C: OK .
C: Well , in that case you would add that to this format
F: OK .
F: So we would basically be sort of , um , taking the format and enriching it with things that we wanna query in relation to the words that are already in the file ,
C: r
C: Right .
F: and then querying it .
F: OK .
A: You want sort of a grep that 's that works at the structural on the structural representation .
C: You have that .
C: There 's a standard again in XML , specifically for searching XML documents structured X - XML documents , where you can specify both the content and the structural position .
A: Yeah , but it 's it 's not clear that that 's F: If A: That 's relative to the structure of the XML document ,
A: not to the structure of what you 're representing in the document .
C: You use it as a tool .
C: You use it as a tool ,
C: not an end - user .
C: It 's not an end - user thing .
A: Right .
C: It 's it 's you would use that to build your tool to do that sort of search .
A: Right .
A: Be
A: Because here you 're specifying a lattice .
F: Uh A: So the underlying that 's the underlying data structure .
F: But as long as the A: And you want to be able to search in that lattice .
C: It 's a graph ,
C: but A: That 's different from searching through the text .
F: But it seems like as long as the features that C: Well ,
C: no , no , no .
C: The whole point is that the text and the lattice are isomorphic .
C: They represent each other completely .
A: Um C: So that C: I mean th
F: That 's true if the features from your acoustics or whatever that are not explicitly in this are at the level of these types .
A: Hhh .
F: That that if you can do that C: Yeah , but that 's gonna be the trouble no matter what .
C: Right ?
C: No matter what format you choose , you 're gonna have the trou you 're gonna have the difficulty of relating the the frame - level features F: That 's right .
F: That 's true .
F: That 's why I was trying to figure out what 's the best format for this representation .
C: Yep .
F: And it 's still gonna be F: it 's still gonna be , uh , not direct .
A: Hmm .
F: You know , it C: Right .
F: Or another example was , you know , uh , where in the language where in the word sequence are people interrupting ?
F: So ,
F: I guess that one 's actually easier .
D: What about D: what about , um , the idea of using a relational database to , uh , store the information from the XML ?
D: So you would have D: XML basically would D: Uh ,
D: you you could use the XML to put the data in ,
D: and then when you get data out , you put it back in XML .
D: So use XML as sort of the the transfer format ,
C: Transfer .
D: uh , but then you store the data in the database ,
D: which allows you to do all kinds of good search things in there .
C: The , uh C: One of the things that ATLAS is doing is they 're trying to define an API which is independent of the back store ,
F: Huh .
C: so that , uh , you could define a single API and the the storage could be flat XML files or a database .
D: Mm - hmm .
C: My opinion on that is for the s sort of stuff that we 're doing , <comment> I suspect it 's overkill to do a full relational database , that , um , just a flat file and , uh , search tools I bet will be enough .
A: But C: But that 's the advantage of ATLAS , is that if we actually take decide to go that route completely and we program to their API , then if we wanted to add a database later it would be pretty easy .
D: Mm - hmm .
D: Mm - hmm .
F: It seems like the kind of thing you 'd do if I don't know , if people start adding all kinds of s bells and whistles to the data .
F: And so that might be F: I mean , it 'd be good for us to know to use a format where we know we can easily , um , input that to some database if other people are using it .
C: Yep .
F: Something like that .
C: I guess I 'm just a little hesitant to try to go whole hog on sort of the the whole framework that that NIST is talking about , with ATLAS and a database and all that sort of stuff ,
F: So C: cuz it 's a big learning curve , just to get going .
D: Hmm .
A: Hmm .
C: Whereas if we just do a flat file format , sure , it may not be as efficient
C: but everyone can program in Perl and and use it .
F: OK .
C: Right ?
C: So , as opposed to A: But this is A: I I 'm still , um , not convinced that you can do much at all on the text on the flat file that that you know , the text representation .
A: e
A: Because the text representation is gonna be , uh , not reflecting the structure of of your words and annotations .
A: It 's just it 's C: Well , if it 's not representing it , then how do you recover it ?
C: Of course it 's representing it .
A: No .
A: You you have to C: That 's the whole point .
A: what you have to do is you have to basically A: Y yeah . You can use Perl to read it in and construct a internal representation that is essentially a lattice .
A: But , the D: Yeah .
C: OK .
A: and then C: Well , that was a different point .
C: Right ?
C: So what I was saying is that A: Right .
A: But that 's what you 'll have to do .
C: For Perl if you want to just do Perl .
A: Bec - be
C: If you wanted to use the structured XML query language , that 's a different thing .
C: And it 's a set of tools that let you specify given the D - DDT DTD of the document , um , what sorts of structural searches you want to do .