Meeting transcript: Bmr018 (icsi)

me001: How about channel

fe008: OK.

me011: We're recording.

me013: Yeah, go ahead.

me001: Alright.

me013: Alright, and no crash.

fe008: Hmm.

me011: I pre-crashed it.

me013: Yeah.

mn017: Pre-crashed!

fe016: It never crashes on me.

me011: I think it's actually - it depends on if the temp files are there or not,

fe016: What is - what is that?

me011: that - at least that's my current working hypothesis,

fe016: Ah.

me011: that I think what happens is it tries to clear the temp files and if they're too big, it crashes.

fe016: Ah.

me018: When the power went out the other day and I restarted it, it crashed the first time.

me011: Oh, that's right.

me018: After the power out-

fe016: So then there would be no temp files.

me013: Yeah.

fe016: OK. Hmm.

me011: Uh, no, it doesn't - it doesn't clear those necessarily, so.

fe016: Oh wait - It - it doesn't clear them, OK.

me013: Hmm, no connection.

me011: It's - i- they're called temp files, but they're not actually in the temp directory they're in the scratch, so.

me011: They're not backed up, but they're not

me011: erased either on power failure.

fe016: But that's usually the meeting that I recorded, and it neve- it doesn't crash on me.

me011: Oh well.

me018: Well this wasn't - Actually, this wasn't a- before your meeting, this was, um, Tuesday afternoon when, um,

fe016: Oh -

me018: uh, Robert just wanted to do a little recording, and the power had gone out earlier in the day.

fe016: Oh, right. OK.

fe016: Huh, OK.

me013: I don't know when would be a good excuse for it, but I just can't wait to be giving a talk t- and - and - and use the example from last week with everybody t- doing the digits at once.

me011: Yeah.

fe008: That was fun.

me013: I'd love to play somebody that.

fe008: That was fun.

fe016: It was quick.

me013: It was. It was really efficient.

me018: Talk about a good noise shield. You know?

me018: You wanted to pe- keep people from listening in, you could like have that playing outside the room. Nobody could listen in.

me013: Yeah.

fe016: Well, I had this idea we could make our whole meeting faster that way.

me013: Yeah.

me013: Everybody give the reports about what they were doing at exactly the same time, yeah.

fe016: And we'll just all leave, and -

me018: And then we'll - we'll go back later and review the individual channels, right? If you wanna know what -

me011: Yep, and then everyone can listen to it later.

me011: Yes. Absolutely.

me013: Actually isn't that what we have been doing?

fe016: Yeah.

me011: It's what it sounds like.

me018: Practically, huh. With all the overlaps.

fe008: Yeah.

me013: What are we doing?

me011: I -

me011: Since I've been gone all week, I didn't send out a reminder for an agenda, so.

me013: Yeah, and I'm just -

me011: Do we have anything to talk about or should we just read digits and go?

me018: I wouldn't mind hearing how the conference was.

me013: What conference?

fe016: Uh,

me011: Yeah, really.

fe016: I had one question about - Aren't the UW folks coming this weekend?

me011: It's all a blur.

me011: Yep.

mn017: No. The next, right?

me011: Next weekend, week from -

fe016: Next weekend?

me013: That is right. The next weekend.

fe016: Sorry, not - not - not the days coming up, but -

mn017: It's like the -

me011: A week from Saturday.

fe016: Yeah, within ten days. So, are we - do we have like an agenda or anything that we should be -

me013: That's when they're coming. That's correct.

me013: No, but that would be a good idea.

fe016: OK.

me013: Why don't we w-

mn017: So - so the deal is that I can,

mn017: um,

mn017: uh, I can be available after,

mn017: uh, like

mn017: ten thirty or something. I don't know how s- how early you wanted to -

me013: They're not even gonna be here until eleven or so.

mn017: Oh, OK.

me011: That's good.

mn017: So -

fe016: Wait, this is on - on Sunday? Or Saturday?

me013: Cuz they're flying up that day.

me013: Saturday.

me013: Saturday.

mn017: Saturday.

fe016: OK.

me011: Well, y-

me013: S- Saturday.

mn017: Mm-hmm.

me013: Yeah.

me011: Eurospeech is due on Friday and then I'm going down to San -

me011: uh, San Jose Friday night, so, if - you know, if we start nice and late Saturday that's a good thing.

me013: No, I mean, they're flying up from - from -

me011: Seattle.

me013: down from

me013: Seattle.

me011: They're flying from somewhere to somewhere,

me013: Yeah,

me011: yeah.

me013: and they'll end up here. So b- and also Brian Kingsbury is actually flying

me013: from, uh, the east coast

me013: on that - that morning.

fe008: Excellent.

mn017: Oh.

me013: So, i- I - I will be - I mean, he's taking a very early flight and we do have the time work difference running the right way,

me013: but I still think that there's no way we could start before eleven. It might end up really being twelve.

me013: So when we get closer we'll find people's plane schedules, and let everybody know.

me013: Uh,

me013: So. That's good.

me011: But, uh, yeah maybe an agenda, or at least some things to talk about would be a good idea.

me013: Well we can start gathering those - those ideas, but then we - we should firm it up by next - next Thursday's meeting.

fe008: Will we have time to, um,

fe008: to prepare something that we - in the format we were planning for the IBM transcribers by then, or -?

me011: Oh yeah.

me011: Absolutely.

fe008: OK.

me011: So have you heard back from Brian about that,

me011: Chuck?

me018: Yes, um, he's - I - I'm sorry, I should have forwarded that along.

me018: Uh, oh I - I think I mentioned at the last meeting, he said that, um,

me018: he talked to them and it was fine - with the beeps they would be - That's easy for them to do.

me011: Great.

me011: OK.

me011: So, uh, oh, though Thi- Thilo isn't here, um, but, uh,

me011: I - I have the program to insert the beeps. What I don't have is something to parse the output

me011: of

me011: the channelized

me011: transcripts to find out where to put the beeps, but that should be really easy to do.

me011: So do we have a meeting that that's been done with,

fe008: He's - he's -

me011: that we've tightened it up to the point where we can actually

me011: give it to IBM and have them try it out?

fe008: He generated, um, a channel-wise presegmented version of a meeting, but it was Robustness rather than EDU so I guess

fe008: depends on whether we're willing to use Robustness?

me011: Mm-hmm.

me018: Well for this experiment I think we can use pre- pretty much anything. This experiment of just -

fe008: OK.

me011: Well we had -

me011: we had talked about doing maybe EDU as a good

me011: choice, though. Well,

me011: whatever we have.

me018: Well we've talked about that as being the next ones we wanted to transcribe.

me011: Right.

me018: But for the purpose of sending him a sample one to - f- I - I don't think it matte-

fe008: OK.

me011: Yeah, maybe it doesn't matter.

fe008: Great. I'll - I'll - I'll, um, get - make that available.

me011: OK, and has it

me011: been corrected?

fe008: Oh, well, wait. Um -

me011: Hand-checked? Cuz that was one of the

me011: processes we were talking about as well.

me018: Right, so we need to run

fe008: That's right.

me018: Thilo's thing on it, and then we go in and adjust the boundaries.

fe008: Yeah that's right. Yeah, we haven't done that. I - I could set someone on that tomorrow.

me011: And time how long it takes.

me018: Right.

me018: OK. And we probably don't have to do necessarily a whole meeting for that if we just wanna send them a sample to try.

fe008: I think they're coming -

fe008: OK.

fe008: What would be a good number of minutes?

me018: I don't know, maybe we can figure out how long it'll take to - to do.

me011: Um,

me011: I don't know, it seems to me w-

me011: we

me011: probably should go ahead and do a whole meeting because we'll have to transcribe the whole meeting anyway sometime.

me013: Yes except that if they had - if there was a choice between

me013: having fifteen minutes that was fully the way you wanted it, and having a whole meeting that didn't get at what you wanted for them -

me013: It's just dependent of how much -

me011: Like I -

me011: I mean I guess if we have to do it again anyway,

me011: but, uh

me013: Yeah.

me018: I guess, the only thing I'm not sure about is, um,

me018: how quickly can the transcribers scan over and fix the boundaries, and -

fe008: Mm-hmm.

me018: I mean, is it pretty easy?

me011: I think it's gonna be one or two times real time at -

me011: Wow,

me011: excuse me, two or more times real time, right?

me011: Cuz they have to at least listen to it.

me013: Can we pipeline it so that say there's, uh, the transcriber gets done with a quarter of the meeting and then we - you run it through this other -

me013: other stuff?

me011: Well the other stuff is IB M.

me013: Uh,

me011: I'm just thinking that

me011: from a data -

me013: OK, so.

me011: keeping-track-of-the- data point of view, it may be best

me011: to send them whole meetings at a time and not try to send them bits and pieces.

me013: Oh, that's right. So the first thing is the automatic thing,

me011: Right.

me013: and then it's - then it's - then it's the transcribers tightening stuff up, and then it's IBM.

fe008: Mm-hmm.

fe008: Mm-hmm, mm-hmm.

me011: Right.

me013: OK, so you might as well ha- run the automatic thing over the entire meeting,

me013: and then -

me013: and then, uh,

me013: you would give IBM whatever was

me013: fixed.

fe008: And have them fix it over the entire meeting too?

me011: Right.

me013: Well, yeah, but start from the beginning and go to the end, right? So if they were only half way through then that's what you'd give IBM.

fe008: OK.

me013: Right?

me018: As of what point?

me018: I mean.

me018: The - I guess the question on my mind is do we wait for the transcribers to adjust the marks for the whole meeting before we give anything to IBM, or do we go ahead and send them a sample?

me013: Why wouldn't we s- w- i- if they were going sequentially through it, why wouldn't

me018: Let their -

me013: we give them -

me013: I mean i- are we trying to get something done by the time Brian comes?

me018: Well I - I - I mean, I don't know.

me011: That was the question.

me011: Though.

me013: So if we - if we were,

me018: Yeah.

me013: then it seems like giving them something, whatever they had gotten up to, would be better than nothing.

me018: Uh. That - I agree. I agree.

me011: Well, I don't think -

me011: I mean, h-

me011: they - they typically work for what, four hours, something like that?

fe008: Hmm, I gue- hmm.

me011: I think the- they should be able to get through a whole meeting in one sitting.

me011: I would think, unless it's a lot harder than we think it is, which it could be,

me011: certainly.

fe008: If it's got like for speakers then

fe008: I guess - I mean if - hhh

me011: Or seven or eight.

me018: We're just doing the individual channels, right?

fe008: Individual channels. Yeah.

me018: So it's gonna be, depending on the number of people in the meeting,

fe008: I guess there is this issue of, you know, if - if the segmenter thought there was no speech on - on a particular stretch, on a particular channel, and there really was,

me018: um,

me011: Well -

fe008: then, if it didn't show up in a mixed signal to verify, then it might be overlooked,

fe008: so, I mean, the question is "should - should a transcriber listen to the entire thing or can it

fe008: g- can it be based on the mixed signal?" And I th- eh so far as I'm concerned it's fine to base it on the mixed signal at this point, and -

me011: That's what it seems to me too, in that

me011: if they need to, just like in the other cases, they can listen to the individual,

fe008: And that cuts down the time.

me011: if they need to.

fe008: Yeah.

me011: But they don't have to for most of it.

fe008: Yeah, that's good. So. Yeah. Good, good, good.

me018: I don't see how that will work, though.

fe008: What - what aspect?

me013: So you're talking about tightening up time boundaries? So how do you -

me018: Yeah.

me011: So, they have the normal channeltrans

me011: interface where they have each individual speaker has their own line,

me018: Yeah.

me011: but you're listening to the mixed signal and you're tightening the boundaries,

me011: correcting the boundaries. You shouldn't have to tighten them too much because Thilo's program does that.

fe008: Should be pretty good, yeah.

fe016: Except for it doesn't do well on short things, remember. It will miss them. It will miss most of the really short things.

me011: Right, so - so you'll have to I -

me011: Uh-huh.

fe016: Like that.

fe008: But those would be - those would be -

fe016: Uh-huh.

me011: Uh-huh!

fe016: It will - it will miss - Yeah, you have to say "uh-huh" more slowly to -

me011: Sorry.

fe016: to get c- No, I'm s- I'm actually serious. So it will miss stuff like that which -

me011: I'll work on that.

me011: Well, so - so that's something that the transcribers will have to - have to do.

me018: I -

fe008: Yeah, but presumably, most of those they

fe008: should be able to hear from the mixed signal unless they're embedded in the heavil- heavy overlap section when - in which case they'd be listening to the channels anyway.

fe016: Right, and that's what I'm not sure about.

me018: That's - that's what I'm - I'm concerned about the part.

fe008: Yeah, I am too. And I think it's an empirical question.

me018: Can't we - uh couldn't we just have, um,

me018: I don't know, maybe this just doesn't fit with the software, but I guess

me018: if I didn't know anything about Transcriber and I was gonna make something to let them adjust boundaries, I would just show them

me018: one channel at a time, with the marks, and let them adju-

fe008: Oh they can -

me011: Well, but then they have to do -

me011: but then they - for this meeting they would have to do seven times real time,

fe008: Yeah, that's it.

me011: and it would probably be more than that.

fe008: Yeah.

me011: Right? Because they'd have to at least listen to each channel

me011: all the way through.

me018: But i- but it's very quick, right? I mean, you scan - I mean, if you have a display of the waveform.

fe008: And if - Uh-huh.

me011: Oh, you're talking about visually. I just don't think -

fe008: Yeah. w- Well,

fe008: the other problem is the breaths cuz you also see the breaths on the waveform. I've - I've looked at the int- uh, s- I've tried to do that with a single channel,

fe008: and - and you do see all sorts of other stuff besides just the voice.

me018: Uh-huh.

me011: Yeah, and I - I think that

me011: they're going much more on acoustics than they are on visuals.

fe008: Well that - that I'm not sure. What you - the digital - what the digital task that you had your interface?

me011: So.

fe008: Um, I know for a fact that one of those - sh- she could really well - she could judge what th- what the number was based on the - on the waveform.

me011: Yeah, that's actually true. Yeah, you're right.

me011: You're absolutely right.

me011: Yeah, I found the same thing that when I was scanning through the wave form

me011: I could see when someone started to read digits just by the shapes.

fe008: Yeah, she could tell which one was seven.

me011: Um, maybe.

fe008: Yeah.

me013: So I don't - I'm - I'm now entirely confused about what they do. So, they're - they're looking at

me011: But -

me013: a mixed signal, or they're looking - what - what are they looking at visually?

fe008: Well, they have a choice. They could choose any signal to look at.

fe008: I've tried lookin- but usually they look at the mixed.

fe008: But I've - I've tried looking at the single signal and - and

fe008: in order to judge when it - when it was speech and when it wasn't, but the problem is then you have breaths which - which show up on the signal.

me011: Oh.

me013: But the procedure that you're imagining, I mean, people vary from this, is that they have the mixed signal wave form in front of them,

fe008: Yes.

fe008: Yes.

me013: and they have multiple, uh, well, let's see, there isn't -

me013: we don't have transcription yet.

me013: So - but there's markers of some sort that have been happening automatically, and those show up on the mixed signal? There's a clicks?

me011: Right.

fe008: Yes.

fe008: Oh, they show up on the separate ribbons. So you have a separate ribbon for each channel,

me011: N-

me011: the t-

me011: Right.

me013: There're separate ribbons.

fe008: and - and i- i- it'll be - because it's being segmented as channel at a time with his - with Thilo's new procedure, then

fe008: you don't have the correspondence of the times across the bins - uh across the ribbons uh you could have -

me013: And is there a line moving across the waveform as it goes? OK, so

me011: Yes.

fe008: Yes.

me013: The way you're imaging is they kind of play it, and they see oh this happened, then this happened, then - and if it's about right, they just sort of let it slide,

me011: Right.

fe008: Yeah.

me011: Right.

me013: and if it - if it - there's a question on something, they stop and maybe look at the individual wave form.

me011: Right.

fe008: Oh, well not - not "look".

me011: Well, they wouldn't look at it

me011: at this point.

me011: They would just listen.

me013: They - they might look at it, right?

me011: Well, the problem is that the -

me011: the interface

me011: doesn't really allow you to switch visuals.

fe008: Not very quickly. You can but it takes time.

me011: The problem is that - that -

me011: the Tcl-TK interface with

me011: the visuals, it's very slow to load waveforms.

fe008: That's it.

me013: Uh-huh.

me011: And so when I tried - that -

me011: that was the first thing I tried when I first started it,

fe008: Oh, oh. Visually.

me011: right?

fe008: You can - you can switch quickly between the audio, but you just can't get the visual display to show quickly.

fe008: So you have to - It takes, I don't know, three, four minutes to - Well, I mean, it takes - it takes long enough -

fe016: Yeah, it's very slow

fe008: It takes long enough cuz it has to reload the I - I don't know exactly what it's doing frankly cuz - but it t- it takes long enough that it's just not a practical alternative.

fe016: to do that.

fe016: That w-

me011: Well it -

me011: it

me011: does some sort of shape pre-computation so that it can then scroll it quickly, yeah. But then you can't

me001: But you can cancel that.

fe016: Yeah.

me011: change the resolution or scroll quickly.

me001: Oh, really?

me011: So.

me001: Huh!

fe008: Now you could set up multiple windows, each one with a different signal showing,

fe008: and then look between the windows. Maybe that's the solution.

me011: I mean, we - we could do different interfaces, right?

me001: What if you preload them all?

me011: I mean, so - so we could use like XWaves instead of Transcriber,

fe008: Yeah.

me011: and it loads faster,

me011: certainly.

me001: What if you were to preload all the channels or -

me001: or initially -

me001: like doesn't -

me011: Well that's what I tried originally. So I - I actually before,

me011: uh,

me011: Dave Gelbart did this, I did an interface which showed each waveform and ea-

me011: a ribbon for each waveform,

me001: Mm-hmm.

me011: but the problem with it is even with just three waveforms it was just painfully slow to scroll.

me001: Oh, OK.

me011: So you just scroll a screen and it would, you know go

me011: "kur-chunk!"

fe008: Mm-hmm.

me011: And so it just was not doable with the current interface.

fe008: You know, I am thinking if we have a meeting with only four speakers and, you know, you could

fe008: fire up a Transcriber interface for, y- you know, in different windows, multiple ones, one for each channel. And it's sort of a - a hack but I mean it would be one way of seeing the visual form.

me011: I think that if we decide that

me011: we need -

me011: that they need to see the visuals, we need to change the interface so that they can do that.

fe008: Yeah. Yeah.

fe016: That's actually what I thought of, loading the chopped up waveforms, I mean,

me011: An-

me013: So -

fe016: you know, that - that would make it faster -

me001: Hmm.

me011: But isn't -

me011: The chopped up waveforms. Isn't that -

fe016: So.

me018: The problem is if - if anything's cut off, you can't expand it from the chopped up -

me001: Right.

fe016: Right, but if you

me011: And wouldn't that be the same as the mixed signal?

fe016: a- at some point -

fe016: No, I mean the individual channels that were chopped up that - it'd be nice to be able to go back and forth between those

fe016: short segments. Cuz you don't really nee- like

me013: Mm-hmm.

fe016: nine tenths of the time you're throwing most of them out, but what you need are tho- that particular channel, or that particular location, and,

me011: Yeah.

fe016: um,

fe008: Yeah.

fe016: might be nice, cuz we save those out already,

fe016: um,

fe016: to be able to do that. But it won't work for IBM of course, it only works here cuz they're not saving out the individual

fe016: channels.

fe008: Well, I - I do think that this - this will be a doable procedure, and have them starting with mixed and, um,

me013: Yeah.

me013: OK.

fe008: then when they get into overlaps, just have them systematically check all the channels to be sure that there isn't something hidden from - from audio view.

me011: Yeah.

me011: Yeah, hopefully, I mean -

me011: The mixed signal, the overlaps are pretty audible

me011: because it is volume equalized. So I think they should be able to hear.

me011: The only problem is - is,

me011: you know, counting how many and if they're really correct or not.

me011: So, I don't know.

fe016: I don't know that you can locate them very well from the mixed signal,

me011: Right but -

me011: but once - once you know that they happen, you can at least listen

fe016: but you would know that they were there, and then you would switch. Right. And then you would switch into the other -

me011: to the close talking, so.

me013: But right now, to do this limitation, the switching is going to be switching of the audio? Is what she's saying.

me011: Right.

fe008: Yeah.

me013: So -

me011: Right, so - so

me011: did Dave -

me013: so they're using their ears to do these markings anyway.

fe008: Yes. Yes.

me011: Did Dave do that change where you can actually just click rather than having to go up to the menu

fe016: Yeah.

fe008: Click -

me011: to listen to the individual channels?

fe008: Um,

me011: I had suggested it before. I just don't know whether he did it or not.

fe008: I'm not sure what - click what - click on the ribbon? Yeah, you can get that - oh, oh, get - you can get the, uh - you can get it to switch audio?

me011: Yeah.

me011: Yeah.

fe008: Uh, not last I tried, but, um, maybe he's changed it again.

me011: We should get him to do that

me011: because, uh, I think that would be much, much faster than going to the menu.

fe008: I disagree.

fe008: There's a reason I disagree, and that is that, uh, you - it's very good to have a dissociation between the visual and the audio. There're times when

fe008: I wanna hear the mixed signal, bu- but I want to transcribe on the single channel. So right now -

me011: Then maybe just buttons down at the bottom

fe008: Maybe, I just don't - I don't see that it's a -

me011: next to it. Just something so that it's not in the menu option so that you can do it much faster.

fe008: Well, I mean, that's the i- I - I think that might be a personal style thing. I find it really convenient the way - the way it's set up right now.

me011: Well

me011: it just seems to me that if you wanna quickly -

me011: "well was that Jane, no, was that Chuck, no, was that Morgan", right now, you have to go up to the menu,

me011: and each time, go up to the menu, select it, listen to that channel then click below, and then go back to the menu, select the next one, and then click below.

fe008: That's fine.

fe008: Yeah, it's true.

me011: So you can definitely streamline that with the i- with the interface.

fe008: Yeah, it could be faster, but, you know, I mean, th- in the ideal world -

fe008: Yeah. No I - I agree that'd be nice. Yeah. OK.

me011: What?

me011: OK.

me013: So, um,

me013: Done with that? Does any - I forget, does anybody, uh,

me013: working on any - any Eurospeech submission related to this?

me011: Hhh! I would like to try to do something on digits but I just don't know if we have time. I mean, it's due next Friday so we have to do the experiments and write the paper.

me011: So, I'm gonna try,

me011: but, uh,

me011: we'll just have to see. So actually I wanna get together with both Andreas and, uh,

me011: uh,

me011: Stephane

me011: with their respective systems.

me013: Yeah.

me013: Yeah there was that - we- that's right, we had that one conversation about, uh,

me013: what - what - what did it mean for, uh, one of those speakers to be pathological, was it a -

me011: Right,

me011: and I haven't had s- chance to sit down and listen. I was going to do that this afternoon.

mn017: Oh, I haven't - I haven't listened to them either,

mn017: but there must be something wrong, I mean,

mn017: unless our -

me011: Well, Morgan and I were - were having a debate about that.

me011: Whereas I think it- it's probably something pathologic and actually Stephane's results,

me011: I think confirm that. He s- he did

me011: the Aurora system

me011: also got very lousy average error, like fifteen or -

me011: or,

me011: uh, fifteen to twenty percent average?

me011: But then he ran it just on the lapel,

me011: and got about five or six percent word error?

me011: So that - that means to me that somewhere in the other recordings there are some pathological cases. But,

me011: you know, we - th- that may not be true. It may be just some of the segments they're just doing a lousy job on.

me011: So I'll - I'll listen to it and find out since you'd actually split it up by

me013: Right.

me011: segment. So I can actually listen to it.

mn017: Yeah.

me018: Did you run the - Andreas - the r- SRI recognizer on the digits?

mn017: Yeah.

me011: Oh, I thought he had sent that around to everyone, did you just sent that to me?

mn017: No, I d- I didn't.

me011: Oh.

mn017: Since I considered those preliminary, I didn't.

mn017: But, yeah, if you take -

me018: I- it wasn't -

me011: It was bimodal.

mn017: So if you - Yeah, it's actually, um, it - uh - it was trimodal, actually -

me011: Oh, was it trimodal, OK.

me013: Yeah. There's zero, a little bit, and a lot.

mn017: trimodal, so

mn017: there were -

mn017: t-

mn017: there was -

mn017: there was one h- one bump at ze- around zero,

mn017: which were the native speakers,

me013: Yeah.

me013: Yeah.

mn017: the non-pathological native speakers.

me018: Zero percent error?

me013: Y- yeah.

mn017: Then there was another bump at, um,

mn017: oh, like fifteen or something.

me013: Oh was it fifteen?

me018: This is error you're talking about? OK.

mn017: whe-

me011: Yeah.

mn017: Yeah. Those were the non-natives.

me013: Yeah.

mn017: And then there was another distinct bump at,

mn017: like, a hundred,

fe008: Oh, wow!

mn017: which must have been some problem. I can't imagine that -

fe008: Oh, OK.

me001: What is patho- what do you mean by pathological? I'm sorry, I don't -

me011: Just - just something really wrong with -

mn017: In the recording

me001: Oh.

me011: A bug is what I mean, so that it's like -

me001: Oh, OK. I see.

mn017: And there was this one meeting, I forget which one it was, where like,

mn017: uh, six out of the eight channels were all,

mn017: like - had a hundred percent error.

me011: Which probably means like there was a -

me011: th-

me011: the recording

me001: Right.

me011: interface crashed, or there was a short -

me011: you know, someone was jiggling with a cord or,

mn017: But -

me011: uh, I extracted it incorrectly, it was labeled - it was transcribed incorrectly, something

mn017: But -

me001: Mm-hmm.

me011: really bad happened, and I just haven't listened to it yet to find out what it was.

me001: OK.

mn017: So, if I excluded the

mn017: pathological ones,

me013: What we're calling.

mn017: by definition, those that had like over ninety-five percent

mn017: error rate,

mn017: and the non-natives, then the average error rate was like one point four or something,

fe008: Oh. Oh.

me001: Hmm!

mn017: which -

mn017: which seemed reasonable given that,

mn017: you know, the models weren't tuned for -

me013: Yeah.

mn017: for it. And the grammar wasn't tuned either. It was just a.

me018: And it didn't matter whether it was the lapel or whether it was the -

mn017: I haven't split it up that way,

mn017: but it would be -

fe016: But there's no overlap during the digit readings, so it shouldn't really matter.

me001: Right.

mn017: Right.

me018: Yeah.

mn017: So it should -

me013: No, but there's a little difference, and we haven't looked at it for digits, right? And so,

me011: There's a lot. Yeah.

me018: Yeah, so I was curious about that.

me013: cuz - because what he was - what I was saying when I looked at those things is it - it - I was almost gonna call it quadrimodal because - because there was a whole lot of cases where it was zero percent.

mn017: Mm-hmm.

me013: They just plain got it all right.

mn017: Yeah.

me013: And then there - and then there was another bunch that were couple percent or something.

mn017: But if you p- if you actually histogrammed it, and - it was a nice -

me013: Yeah.

mn017: uh, you know, it - it was - zero was the most of them, but then

me011: A normal. Yeah.

mn017: there were -

mn017: the others were sort of decaying from there.

me013: Yeah, yeah.

mn017: And then there was the bump for the non-natives and then the pathological ones, so.

me013: I see. I see.

me011: Yeah, cuz some of our non-natives are pretty non-native. So.

me013: Yeah.

fe008: You - did you have, uh, something in the report about, uh, - about, uh, for- f- uh, forced alignment? Have you - have you started on that?

mn017: Oh, well, yeah, so I've been struggling with the forced alignments. Um.

mn017: So the scheme that I

mn017: drew on the board last time where we tried to, um

mn017: allow reject models for the

mn017: s-

mn017: speech from other

mn017: speakers,

mn017: um,

mn017: most of the time it doesn't work very well.

mn017: So,

mn017: um,

me013: Hmm.

mn017: and the -

mn017: I haven't done - I mean, the only

mn017: way to check this right now was for me to actually

mn017: load these into XWaves and,

mn017: you know, plus the alignments, and s-

mn017: play them and see where the -

mn017: And it looks - And so I looked at

mn017: all of the

mn017: utterances from you, Chuck, in that

mn017: one conversation,

mn017: I don't know which - You probably know which one I mean, it's

mn017: where you were on the lapel

mn017: and Morgan was sitting next to you and we can hear everything Morgan says.

fe008: Hmm.

mn017: But -

mn017: and - and some of what you - I mean, you also

mn017: appear quite a bit in that cross-talk.

mn017: So,

mn017: I actually

mn017: went through all of those, there were I think fifty-five segments,

mn017: um, in - in XWaves, and - and

mn017: sort of did a crude check, and

mn017: more often than not, it - it gets it wrong. So there's either the beginning, mostly the beginning word,

mn017: where th- you, um,

mn017: you know, Chuck talks somewhere into the segment,

mn017: but

mn017: the first,

mn017: um, word of what he says, often

mn017: "I" but it's very reduced "I,"

mn017: that's just aligned

mn017: to the beginning of someone else's speech, uh in that segment, which is cross-talk. So,

mn017: um,

mn017: I'm still tinkering with it, but

mn017: it might well be that we can't get clean alignments out of this - out of those, uh,

mn017: channels, so.

me013: Unless maybe we do this, uh, um, cancellation business.

fe016: Right, but that's - I mean, that was our plan, but it's clear from Dan that this is not something you can do in a short amount of time.

mn017: Yeah, right.

me013: Oh, the short amount of time thing, right.

fe016: So - so we - you know, we had spent a lot of time, um, writing up the HLT paper and we wanted to use that, uh, kind of analysis, but the HLT paper has,

me013: Yeah.

fe016: you know, it's a very crude measure of overlap. It's not really something you could scientifically say is overlap, it's just whether or not the,

fe016: um,

me011: c- High correlation.

fe016: the segments that were all synchronized, whether there was some overlap somewhere.

fe016: And, you know, that pointed out some differences, so he thought well if we can

fe016: do something quick and dirty because Dan said the cross-cancellation, it's not straight-forward. If it were straight-forward then we would try it, but - so, it's sort of

fe016: good to hear that it was not straight-forward, thinking if we can get decent forced alignments,

fe016: then at least we can do sort of a overall report of

fe016: what happens with actual overlap

fe016: in time, but,

fe016: um -

me018: I didn't think that his message said it wasn't straight- forward.

me011: Well if we'd just -

me013: Well

me011: Um-hmm.

me018: I thought he's just saying you have to look over a longer time window when you do it.

fe016: and the - but there are some issues of this timing, um, in the recordings and -

me013: Yeah.

me018: Right. So you just have to look over longer time when you're trying to align the things, you can't - you can't just look -

me011: Well. are you talking about the fact that the recording software doesn't do time-synchronous? Is that what you're referring to?

me011: That seems to me you can do that over the entire file and get a very accurate -

mn017: I don't thi-

mn017: I d-

mn017: I don't think that was the issue. The issue was that you have -

fe016: I - yeah, that was sort of a side issue.

me011: I didn't think so either.

mn017: to - you have have -

mn017: you first have to have a pretty good speech detection on the individual channels.

fe016: And it's dynamic, so I guess it was more dynamic than some simple models

fe016: would be able t- to - so - so there are some things available, and I don't know too much about this area where

fe016: if people aren't moving around much than you could apply them, and it should work pretty well if you took care of this recording time difference.

me011: Right, which should be pretty straight forward.

fe016: Which a- at least is well defined, and

me011: Yeah.

fe016: um, but then if you add the dynamic aspect of adapting distances, then it wasn't -

fe016: I guess it just wasn't something that he could do quickly and not - in time for us to be able to do something by two weeks from now, so. Well less than a week.

fe016: So - um, so I don't know what we can do if anything, that's sort of worth,

fe016: you know, a Eurospeech paper at this point.

me018: Well, Andreas, how well did it work on the non-lapel stuff?

me011: Yeah. That's what I was gonna say. C-

mn017: I haven't checked those yet. It's very tedious to check these.

me018: Mmm.

mn017: Um, we would really need, ideally, a transcriber

mn017: to time mark the -

mn017: you know, the be- at least the beginning and s- ends of contiguous speech.

mn017: Um,

mn017: and,

mn017: you know, then with the time marks, you can do an automatic

mn017: comparison of your -

mn017: of your forced alignments.

me011: Oh, MNCM.

me018: Because - really the - the - at least in terms of how we were gonna use this in our system was

mn017: Mm-hmm.

me018: to get an ideal - an idea, uh, for each channel about the start and end boundaries. We don't really care about like intermediate word boundaries, so -

mn017: No, that's how I've been looking at it. I mean, I don't care that the individual words are aligned correctly, but

fe016: Right.

me018: Yeah.

me018: Yeah.

mn017: you don't wanna,

mn017: uh, infer from the alignment that someone spoke who didn't. so, so -

me018: Right, exactly. So that's why I was wondering if it -

me018: I mean, maybe if it doesn't work for lapel stuff, we can just not use that and -

mn017: Yeah.

mn017: I haven't -

mn017: I ha- just haven't had the time to, um,

mn017: do the same procedure on one of the -

mn017: so I would need a k- I would need a channel that has

mn017: a speaker whose -

mn017: who has a lot of overlap but s-

mn017: you know, is a

mn017: non-lapel mike.

mn017: And, um,

mn017: where preferably, also there's someone sitting next to them

mn017: who talks a lot.

me011: Hmm!

mn017: So,

me011: So a meeting with me in it.

mn017: I - maybe someone can help me find a good candidate and then I would

mn017: be willing to

me018: We c- you know what? Maybe the best way to find that would be to look through these.

mn017: you know, hand-

me018: Cuz you can see the seat numbers, and then you can see what type of mike they were using.

me018: And so we just look for, you know, somebody sitting next to Adam

me018: at one of the meetings -

fe016: Actually y- we can tell from the data that we have, um, yeah, there's a way to tell. It might not be a single person who's always overlapping that person but any number of people, and,

mn017: From the insertions, maybe?

mn017: fr-

mn017: fr- from the -

mn017: Right.

fe016: um, if you align the two

fe016: hypothesis files across the channels, you know, just word alignment, you'd be able to find that.

fe016: So -

fe016: so I guess that's sort of a last -

fe016: ther- there're sort of a few things we could do. One is just do like non-lapels if we can get good enough alignments. Another one was to try to get -

fe016: somehow align Thilo's energy segmentations with

fe016: what we have. But then you have the problem of not knowing where the words are because these meetings were done before that segmentation. But maybe there's something that could be done.

me018: What - what is - why do you need the, um,

me018: the forced alignment for the HLT - I mean for the Eurospeech paper?

fe016: Well,

fe016: I guess I - I wanted to just do something not on

fe016: recognition experiments because that's ju- way too early, but to be able to report,

fe016: you know, actual numbers. Like if we - if we had

fe016: hand-transcribed pe- good alignments or hand-checked alignments, then we could do this paper. It's not that we need it to be automatic.

fe016: But without knowing where the real words are, in time -

me018: So it was to get - it was to get more data and better - to - to squeeze the boundaries in.

fe016: To - to know what an overlap really - if it's really an overlap, or if it's just a -

me018: Ah, OK. Yeah.

fe016: a - a segment correlated with an overlap, and I guess that's the difference to me between like a real paper and a sort of, promissory paper.

fe016: So,

fe016: um, if we d-

fe016: it might be possible to take Thilo's output

fe016: and like if you have, um,

fe016: like right now these meetings are all,

me011: Ugh!

me011: I forgot the digital camera again. Every meeting!

fe016: um,

fe016: you know, they're time-aligned, so if these are two different channels

fe016: and somebody's talking here and somebody else is talking here, just that word,

me011: Mm-hmm.

fe016: if Thilo can tell us that there're boundaries here, we should be able to figure that out because the only thing transcribed in this channel is this word.

fe016: But, um, you know, if there are things -

me011: Two words.

fe016: Yeah, if you have two and they're at the edges, it's like here and here, and there's speech here, then it doesn't really help you, so, um -

me018: Thilo's won't put down two separate marks in that case -

me011: Thilo's will. But.

fe016: Well it w- it would, but, um, we don't know exactly where the words are because the transcriber gave us two words in this time bin and we don't really know,

fe016: I mean, yeah it's -

fe008: Well it's a merging problem. If you had a - if you had a s- if you had a script which would - I've thought about this, um, and I've discussed - I've discussed it with Thilo,

fe016: I mean, if you have any ideas. I would -

fe008: um, the,

fe008: I mean, I - I - in principle I could imagine writing a script which would approximate it to some degree, but there is this problem of slippage, yeah.

me011: Well maybe -

me011: Maybe that will get enough of the cases to be useful.

fe016: Right. I mean, that - that would be really helpful. That was sort of another possibility.

me011: You know s- cuz it seemed like most of the cases are in fact the single word

me011: sorts, or at least a single phrase

mn017: Mmm.

fe008: Well they - they can be stretched. I wouldn't make that generalization cuz sometimes people will say, "And then I" and there's a long pause

me011: in most of the bins.

fe016: Yeah.

fe008: and finish the sentence and - and sometimes it looks coherent and - and the - I mean it's - it's not a simple problem.

fe008: But it's really - And then it's coupled with the problem that sometimes, you know, with - with a fricative you might get the beginning of the word cut off and so it's coupled with the problem that Thilo's isn't

fe008: perfect either. I mean, we've i- th- it's like you have a merging problem plus - so merging plus this problem of, uh, not -

me011: Right.

me011: Hmm!

fe008: y- i- i- if the speech-nonspeech were perfect to begin with, the detector, that would already be an improvement, but that's impossible, you know, i- that's too much to ask.

fe016: Right.

me011: Yes.

fe008: And so i- and may- you know, I mean, it's -

fe008: I think that there always - th- there would have to be some hand-tweaking, but it's possible that a script could be written to merge those two types of things. I've - I've discussed it with Thilo and I mean -

fe008: in terms of not him doing it, but we - we discussed some of the parameters of that and how hard it would be to - in principle - to write something that would do that.

fe016: I mean, I guess in the future it won't be as much as an issue if

fe016: transcribers are using the tightened boundaries to start with, then we have a

fe016: good idea of where the forced alignment is constrained to. So I'm no- I don't know if this-

fe008: Well, it's just, you know, a matter of we had the revolution - we had the revolution of improved, uh, interface, um, one month too late, but it's like,

me011: Oh.

me011: Tools.

fe016: Oh it's - it's a - yeah.

fe008: you know, it's wonderful to have the revolution, so it's just a matter of - of, you know, from now on we'll be able to have things channelized to begin with.

me011: Right. And we'll just have to see how hard that is.

fe008: Yeah, that's right. That's right.

me011: So - so whether the corrections take too much time. I was just thinking about the fact that if Thilo's missed these short segments,

fe016: Yeah.

me011: that might be quite time-consuming for them to insert them.

fe008: Good point.

fe016: But he - he also can adjust this minimum time duration constraint and then what you get is

fe008: Yeah.

me011: Spurious.

fe016: noises mostly, but that might be OK, an-

me011: It might be easier to delete something that's wrong than to insert something that's missing. What do you think, Jane?

fe016: Right. And you can also see in the waveform - exac- yeah.

me013: If you can feel confident that what the - yeah, that there's actually something - that you're not gonna miss something, yeah.

fe016: Yeah.

me011: Yeah. Cuz then - then you just delete it, and you don't have to pick a time.

fe016: I think it's -

fe008: Well the problem is I - you know - I - I - it's a - it's a really good question, and I really find it a pain in the neck to delete things because you have to get the mouse up there

fe008: on the t- on the text line and i- and otherwise you just use an arrow to get down - I mean, i- it depends on how lar- th- there's so many

fe008: extra things that would make it

fe008: one of them harder than the other, or - or vice versa. It's not a simple question. But, you know, I mean, in principle, like, you know, if one of them is easier then to bias it towards whichever one's easier.

me011: Yeah, I guess the semantics aren't clear when you delete a segment, right? Because you would say - You would have to determine what the surroundings were.

fe016: You could just say it's a noise, though, and write, you know, a post-processor will just -

fe016: all you have to do is just -

me011: If it's really a noise.

fe016: or just say it's - just put "X," you know, like "not speech" or something, and then you can get -

fe008: I think it's easier to add than delete, frankly, because you have to, uh, maneuver around on the - on both windows then.

fe016: Yeah, or-

me011: To add or to delete?

fe008: To delete.

me011: OK.

fe016: Anyways, so I - I guess -

me011: That - Maybe that's an interface issue that might be addressable. But I think it's the semantics that are - that are questionable to me, that you delete something - So let's say someone is talking to here,

fe008: It's possible.

me011: and then you have a little segment here.

me011: Well, is that part of the speech? Is it part of the nonspeech?

me011: I mean, w- what do you embed it in?

fe016: There's something nice, though, about keeping,

fe016: and this is probably another discussion, keeping the stuff that Thilo's detector detected as possible speech and just marking it as not speech than

fe016: deleting it. Because then when you align it, then the alignment can - you can put a reject model or whatever, and you're consistent with th- the automatic system, whereas if you delete it -

me011: Oh, I see. So then they could just like put - Oh that's what you meant by just put an "X" there.

me011: Uh, that's an interesting idea.

me011: So - so all they - So that all they would have to do is put like an "X" there.

fe016: Yeah, or some, you know, dummy reject mod-

me011: So blank for -

me011: blank for silence, "S" for speech, "X" for

fe016: whatever, yeah. That's actually a better way to do it cuz the a- the forced alignment will probably be more consistent than -

me011: something else.

fe008: Well, like, I think there's a complication which is that - that you can have speech and noise in s-

fe016: I mean if it's just as easy, but -

fe008: uh, you know, on the same channel, the same speaker, so now sometimes you get a ni- microphone pop and,

fe008: uh, I mean, there're these fuzzy hybrid cases,

fe008: and then the problem with the boundaries that have to be shifted around.

fe008: It's not a simple - not a simple problem.

fe016: Anyway, quick question, though, at a high level do people think,

fe016: let's just say that we're moving to this new era of like using the, um, pre-segmented t- you know, non-synchronous

fe016: conversations, does it make sense to try to take what we have now, which are the ones that, you know, we have recognition on which are synchronous and not time-tightened, and try to

fe016: get something out of those for sort of purposes of illustrating the structure and the nature of the meetings, or is it better to just,

fe016: you know, forget that and tr- I mean, it's -

me011: Well, I think we'll have to, eventually.

me011: And my hope was that we would be able to use the forced alignment to get it. But if we can't -

fe016: Right. That was everybody's hope.

me011: But if we can't, then maybe we just have to -

fe016: And maybe we can for the non-lapel, but is it worth - if we can't then we can fake it even if we're - we report, you know, we're wrong twenty percent of the time or ten percent of the time.

me011: Well, I'm thinking - are you talking about for a paper, or are talking about for the corpus.

fe016: Uh - uh, that's a good question actually.

me011: I mean cuz for the corpus it would be nice if everything were -

fe016: Actually that's a good question because we'd have to completely redo those meetings, and we have like ten of them now.

me011: We wouldn't have to re- do them, we would just have to edit them.

fe008: Well, and also, I mean, I still haven't - I still haven't given up on forced alignment. I think that when Brian comes, this'll be uh an interesting aspect to ask him as well b-

fe016: No, you're right, actually -

me011: When -

fe008: when Brian Kingsbury comes.

me011: Oh, Brian. You s- I thought you said Ryan. And it's like, "Who's Ryan?" OK.

fe008: Yeah, good question. Well, Ryan could come.

fe016: Uh, no, that's a good point, though, because for feature extraction like for prosody or something, I mean, the meetings we have now, it's a good chunk of data -

me011: Yep.

fe016: we need to get a decent f- OK.

fe016: So we should at least try it even if we can't, right?

fe008: That's what my hope has been, and that's what - that's what - you know, ever since the - the February meeting that I transcribed from last year,

fe008: forced alignment has been on the - on the table as a way of cleaning them up later. And - and so I'm hopeful that that's possible. I know that there's complication in the overlap sections and with the lapel mikes, but -

me011: On the table, right?

mn017: There's -

mn017: Yeah.

fe016: I mean, we might be able, at the very worst, we can get transcribers to correct the cases where - I mean, you sort of have a good estimate where these places are because the recognition's so poor.

fe016: Right? And so you're -

me018: Yeah, we were never just gonna go with these as the final alignments. We were always gonna run them past somebody.

fe008: I agree. I agree.

fe016: Yeah.

fe016: So we need some way to push these first chunk of meetings into a state where we get good alignments.

fe008: Absolutely.

mn017: I'm probably going to spend another day or so trying to

mn017: improve things by, um,

mn017: by using,

mn017: um,

mn017: acoustic adaptation.

mn017: Um,

mn017: the -

mn017: Right now I'm using

mn017: the unadapted

mn017: models for the

mn017: forced alignments,

mn017: and it's

mn017: possible that you get

mn017: considerably better results if you, uh, manage to

mn017: adapt the,

mn017: uh,

mn017: phone models to the speaker and the reject model to the -

mn017: to - to all the other speech.

mn017: Um,

mn017: so

me018: Could you - could you at the same time

me018: adapt the reject model to

me018: the speech from all the other channels?

me013: That's what he just said.

me011: That's what he was saying.

mn017: That's what I just said.

fe016: Yeah.

me018: Oh, not just the speech from that - of the other people from that channel,

mn017: Right.

fe016: Right.

me018: but the speech from the a- actual other channels.

mn017: Oh, oh, I see. Um,

me011: I don't think so. I don't think that would work, right? Because you'd - A lot of it's dominated by channel properties.

me013: Oh.

mn017: No, it -

mn017: th-

mn017: Exactly.

mn017: So you want to u-

fe016: But what you do wanna do is take the, even if it's klugey, take the segments - the synchronous segments, the ones from the HLT paper, where only that speaker was talking. Use those for adaptation, cuz if you -

fe016: if you use everything, then you get all the cross-talk in the adaptation, and it's just sort of blurred. And that we know, I mean, we have that. And it's about

mn017: That's a good point.

mn017: Yep.

me018: If you -

mn017: Yeah.

fe016: roughly two-thirds, I mean, very roughly averaged. That's not completely negligible. Like a third of it is bad for adaptation or so.

mn017: Mm-hmm.

me011: Cool. I thought it was higher than that, that's pr-

fe016: It really - it depends a lot. This is just sort of an overall -

mn017: So.

me013: Well I know what we're not turning in to Eurospeech, a redo of the HLT paper.

me011: Right.

me013: That - I don't wanna do that, but.

me011: Yeah, I'm doing that for AVIOS.

fe016: Yeah.

fe016: But I think we're - oh, Morgan's talk went very well, I think. I think Morgan's talk went very well it woke - you know, it was really a well presented - and got people laughing -

me013: Bleep.

me011: Uh, "bleep". Yeah, really.

fe008: Excellent.

mn017: Some good jokes in it?

fe008: Yeah.

me011: Especially the batteried meter popping up, that was hilarious. Right when you were talking about that.

fe016: Yeah.

me013: You know, that wa- that was the battery meter saying that it was fully charged, yeah.

me011: It's full. Yeah.

fe008: You said, "Speaking about energy", or something. That was very nice.

me011: But that was funny.

me011: He - he - he was onto the bullet points about talking about

me013: Yeah.

mn017: Po- low power and -

me011: the - you know - the little hand-held, and trying to get lower power and so on, and Microsoft pops up a little window saying "Your batteries are now fully charged."

me013: Yeah, yeah, yeah.

fe008: That's great.

me011: I'm thinking about scripting that for my talk, you know, put - put a little script in there to say "Your batteries are low" right when I'm saying that.

me013: Yeah.

me013: Yeah. No I mean, i- in - in your case, I mean, you were joking about it, but, I mean, your case

me013: the fact that your talking about similar things at a couple of conferences, it's not -

me013: these are conferences that have d- really different emphases. Whereas HLT and - and Eurospeech, pretty -

me011: Are too close, yeah.

me013: pretty - pretty similar, so I - I - I can't see really

me013: just putting in the same thing, but -

fe016: No, I d- I don't think that paper is really -

fe016: the HLT paper is really more of a introduction-to-the-project paper, and, um -

me013: Yeah.

me011: Yeah, for Eurospeech we want some results if we can get them.

fe016: Well, yeah, it - it's - probably wouldn't make sense, but -

me013: Or some - or some - I mean,

me013: I would see Eurospeech - if we have some Eurospeech papers, these will be paper p- p- uh, submissions. These will be things that are particular things,

me013: aspects of it that we're looking at, rather than, you know, attempt at a global paper about it.

me011: Detail, yeah. Overall.

fe016: Right, right.

fe008: I did go through one of these meetings. I had, uh, one of the transcribers go through and tighten up the bins on one of the, uh, NSA meetings, and then I went through afterwards and double-checked it so that one is really very -

fe016: Oh.

fe008: very accurate. I men- I mentioned the link. I sent - You know that one?

fe016: Oh, so -

me001: The - which one? I'm sorry.

fe008: Um, I'm trying to remember - I don't remember the number off hand. It's one of the NSA's. I sent email before the conference, before last week.

me011: Those are all -

me001: Oh, OK.

fe016: That might - might have been the one - one of the ones that we did.

fe008: Bef- What I mean is Wednesday, Thursday.

me001: Mm-hmm.

fe008: I'm sure that that one's accurate, I've been through it myself.

me001: OK.

fe016: So that might actually be useful but they're all non-native speakers.

mn017: So we could compare before and after and see -

me011: Yeah. Yeah, that's what I was gonna say. The problem with those, they're all German. So.

me001: Yeah, that's the problem with the NSA speakers.

mn017: oh,

fe016: And e- and e- and extremely hard to follow, like word-wise, I bet the transcri- I mean, I have no idea what they're talking about, so, um,

mn017: Darn!

me001: Yeah.

fe008: I corrected it for a number of the words. I'm sure that, um, they're - they're accurate now.

mn017: Uh, actually I have to -

fe016: I mean, this is tough for a language model probably -

mn017: to go.

me001: Right.

fe016: but - but that might be useful just for

me013: Well.

me011: OK, Andreas is leaving - leaving the building.

fe016: speech.

me011: Mm-hmm.

me013: Yeah.

me011: See ya.

me013: See ya.

me011: Um, oh, before you l- go -

me013: I don't think we'll go much longer.

me011: I guess it's alright for you to talk a little without the mike - I noticed you adjusting the mike a lot, did it not fit you well?

me011: Oh.

fe008: Well I won- I noticed when you turned your head, it would - it would tilt.

me011: Maybe it wasn't just tightened enough, or -

fe016: Maybe the - yeah, the s- thing that you have tightened, oh.

me018: Actually if - if you have a larger head, that mike's gotta go farther away which means the - the balance is gonna make it wanna tip down.

me011: OK.

me011: Anyway.

me013: Yeah. OK, see ya.

me011: Cuz, I'm just thinking, you know, we were - we're - we've been talking about changing the mikes,

fe008: Yeah.

me011: uh, for a while, and if these aren't -

me011: acoustically they seem really good, but if they're not comfortable, we have the same problems we have with these stupid things.

fe008: I think it's com- This is the first time I've worn this, I find it very comfortable.

me011: I find it very comfortable too, but, uh, it looked like Andreas was having problems, and I think Morgan was saying it -

me013: Well, but I had it on - I had it on this morning and it was fine.

me018: Can I see that?

me011: Oh, oh you did wear it this morning? OK, it's off, so you can put it on.

me013: Yeah.

me018: I - yeah, I don't want it on, I just -

me018: I just want to, um,

me018: say what I think is a problem with this.

me018: If you are wearing this

me018: over your ears

me018: and you've got it all the way out here,

me018: then the balance is gonna want to pull it this way. Where as if somebody with a smaller head has it back here,

fe016: Yeah.

me011: Right.

me011: It's more balanced.

fe016: So we have to

me018: right? Yeah. Then it - then it falls back this way so it's -

fe008: Oh!

me011: Well wh- what it's supposed to do is the backstrap is supposed to be under your crown, and so that should be - should be -

me011: if it's right against your head there, which is what it's supposed to be, that balances it

fe008: Ah.

me011: so it doesn't slide up. Yep, right - right below - if you feel the back of your head, you feel a little lump,

me018: So this is supposed to be

me018: under that little protuberance.

me011: um, and so it's supposed to be right under that.

me018: Yeah.

me018: So it's really supposed to go more like this than like this. But then isn't that going to - Well, I guess you can control that.

me011: Yes, exactly.

me011: That - that - that tilts, right?

me011: In lots and lots of different ways.

fe016: So I'm not saying anything about bias towards small headsize, but

me011: About heads?

me018: It would be an advantage.

fe016: does seem, uh -

fe008: Well, wonder if it's - if - if he was wearing it over his hair instead of under his hair.

me013: Well, we should -

me011: I think probably it was -

me013: We shou- we should work on compressing the heads, and -

me011: Yeah. It probably just wasn't tight enough to the back of his head.

me011: I mean, so the directions do talk about bending it to your size,

me018: The other thing that would do it would be to hang a five pound weight off the back.

me011: which is not really what we want.

me013: Yeah that's good!

fe016: Right.

fe008: What did you say?

me011: wh-

fe016: A little, um,

me013: Hang a five pound weight off the - off the back. Weight.

me011: We did that -

me018: Hang a five pound weight off the back.

me011: We - at Boeing I used - I was doing augmented reality so they had head-mounts on, and we - we had a little jury-rigged one with a welder's helmet, and we had just a bag

me018: Counter-balance.

me011: with a bunch of marbles in it as a counter-balance.

me013: Or maybe this could be helpful just for evening the conversation between people. If people - those who talk a lot have to wear heavier weights or something, and - and -

me011: Yeah!

me013: um,

me011: Anyway.

me013: um,

me013: so, uh, what was I gonna say? Oh, yeah, I was gonna say, uh, I had these, uh, conversations with NIST folks also while I was there and - and, uh,

me011: Yep.

me013: um, so

me013: they - they have their - their plan for a room, uh, with, um,

me013: mikes in the middle of the table, and, uh, close-mounted mikes, and they're talking about close-mounted and lapels,

me013: just cuz sort of - and the array. Yeah, so they were -

fe016: And arrays, which is the i- interesting - and video, right.

me011: And arrays, yep. And cameras.

me013: And yeah, like multiple - multiple video cameras coverin- covering every - everybody - every place in the room,

me013: uh, the - yeah - the - the mikes in the middle, the head-mounted mikes, the lapel mikes, the array, uh, with - well, there's some discussion of fifty-nine, they might go down to fifty-seven

me011: Fifty-nine elements.

me013: Because, uh,

me013: there is, uh, some pressure from a couple people at the meeting for them to use a KEMAR head.

fe016: Mm-hmm.

me013: I forget what KEMAR, uh, stands for, but what it is is it's dummy head that is very specially designed, and -

me011: Oh, that's right. Yep.

fe016: Right. That's a great idea.

me013: and - and, so what they're actually doing is they're really - there's really two recording systems. So they may not be precisely synchronous, but the-

me013: but there's two - two recording systems, one with, I think, twenty-four channels, and one with sixty-four channels. And the sixty-four channel one is for the array,

me013: but they've got some empty channels there, and anyway they - like they're saying they may

me013: give up a couple or something if - for - for the KEMAR head if they go - go with that.

me011: Right.

me011: Yeah, it is a good idea.

me011: Yeah, h- uh, J- Jonathan Fiscus did say that,

me013: So.

me011: uh, they have lots of software for doing calibration for skew and offset between channels and that they've found that's just not a big deal.

fe016: Mm-hmm

me013: Yeah.

me011: So.

me013: Yeah, I'm not too worried about that. I was thinking -

fe016: But they're still

fe016: planning to do like fake - they have to do something like that, right.

me011: Scenario-based.

me011: Y- right. Their - their

me011: legal issues won't allow them to do otherwise.

me013: Yeah.

me011: But it sounded like they were pretty well thought out and they're - they're gonna be real meetings, it's just that they're with str- with people who would not be meeting otherwise. So.

fe016: Yeah, th- that's true.

fe008: Mm-hmm.

fe016: Mm-hmm.

me018: Did - did they give a talk on this or was this informal?

me011: No. It's just informal.

fe016: No.

me013: No, we just had some discussions, various discussions

fe008: Mm-hmm. Mm-hmm. Yeah.

me011: Yeah, I also sat and chatted with several of the NIST folks. They seemed like a good group.

me013: with them.

me018: What was the, um -

me018: the paper by, um, Lori Lamel that you mentioned?

me013: Um,

fe008: Mmm, yeah.

me013: yeah, we sh- we should just have you - have you read it, but, I mea- ba- i- i- uh, we've all got these little proceedings, but, um,

me013: basically,

me013: it was about, um,

me013: uh, going to a new task where you have insufficient data and using - using data from something else, and adapting, and how well that works.

me013: Uh, so in - in fact it was pretty related to what Liz and Andreas did, uh, except that this was not with meeting stuff, it was with

me011: Right.

me013: uh, like I think they s- didn't they start off with Broadcast News system? And then they went to -

me011: The- their Broadcast News was their acoustic models and then all the other tasks were much simpler.

me013: Yeah.

me011: So they were command and control

me011: and that sort of thing.

me013: TI-digits was one of them,

me011: Yep.

me013: and, uh,

me013: Wall Street Journal.

me018: What was their

me011: Yeah, read Wall Street Journal.

me018: rough - what was their conclusion?

me011: It works.

me013: Yeah.

fe016: Well, it's - it's a good paper, I mean -

me011: Yeah.

me013: Yeah, yeah.

me011: Yeah, that was one of the ones that I liked. That - It not only works, in some cases it was better, which I thought was pretty interesting, but that's cuz they didn't control for parameters.

fe016: Bring the -

me011: So.

me011: You know, the Broadcast News nets were - not nets, acoustic models were a lot more complex.

me013: Probably.

fe016: Right.

me018: Did they ever try going -

me018: going the other direction from simpler task to more complicated tasks, or -?

me011: n-

me011: Not in that paper.

me013: That might be hard.

me011: Yeah, well,

me011: one of the big problems with that is - is often the simpler task isn't fully - doesn't have all the phones in it, and that - that makes it very hard.

me013: Yeah.

me018: Mm-hmm.

me013: Yeah.

me011: But I've done the same thing. I've been using Broadcast News nets for digits,

me013: Yeah.

me011: like for the spr- speech proxy thing that I did? That's what I did.

me013: Yeah, sure.

me011: So. It works.

me013: Yeah.

me013: Yeah, and they have - I mean - they have better adaptation than we had than that - that system, so they -

me011: Yep.

me013: um,

me011: You mean they have some.

me013: yeah, we should probably what would - actually what we should do, uh, I haven't said anything about this, but probably the five of us should

me013: pick out a paper or two

me013: that - that, uh, you know, got our interest, and we should

me013: go around the room at one of the Tuesday lunch meetings and say, you know, what - what was good about the conference, yeah.

me011: Present.

me011: Yep.

me011: Do a trip report.

fe016: Well, the summarization stuff was interesting, I mean, I don't know anything about that field, but for this proposal

fe016: on meeting summarization,

fe016: um, I mean, it's sort of a far cry because they weren't working with meeting type data, but

me011: Right.

fe016: he got sort of an overview on some of the different approaches,

me018: Do you remember who the groups were that we're doing?

fe016: so.

fe016: Well there're - this was the last day, but, I mean, there's - that's a huge field and probably the groups there may not be representative of the field, I - I don't know exactly

me011: A lot of different ones.

fe008: R- I think - Mm-hmm.

fe016: that everyone submits to this particular conference, but

me018: Was - were there folks from BBN presenting?

fe016: yet there was, let's see,

fe016: this was on the last day, Mitre, BBN,

me011: Mitre, BBN, IBM.

fe016: and, um, Prager -

me011: Uh,

fe008: Maryland.

fe016: um,

fe016: I wo- it was - no it was - this was Wednesday morning.

me013: Columbia have anything? No.

me011: Wasn't -

me011: Who - who - who did the order one?

fe016: The sentence ordering one,

fe016: was that Barselou, and these guys?

me011: Ugh! I'm just so bad at that.

fe008: Oh.

fe016: Anyway, I - I - it's in the program, I should have read it to remind myself, but that's sort of useful and I think

fe016: like when Mari and Katrin and Jeff are here it'd be good to figure out some kinds of

fe016: things that we can start doing maybe just on the transcripts cuz we already have -

me013: Mm-hmm.

me011: Yeah, we do have word transcripts. So.

fe016: you know, yeah.

fe008: Well, I like the idea that Adam had of - of, um, z- maybe

fe008: generating minutes based on some of these things that we have because it would be easy to - to - to do that just,

fe016: Right.

fe008: you know, and - and it has to be, though, someone from this group because of the technical nature of the thing.

me011: Someone who actually does take notes, um, I'm very bad at note-taking.

fe016: But I think what's interesting is there's all these different evaluations, like -

me011: I always write down the wrong things.

fe016: just, you know, how do you evaluate whether the summary is good or not, and that's what's - was sort of interesting to me is that there's different ways to do it, and -

fe008: I do take notes.

me011: A judge.

me011: Yep.

me018: Was SRA one of the groups

me018: talking about summarization, no?

fe016: Hm-umm.

fe016: No.

fe008: It was an interesting session. One of those w-

me011: And as I said, I like the Microsoft talk on scaling issues in, uh, word sense disambiguation, that was interesting.

fe016: Yeah.

me013: Yeah, that was an interesting discussion, uh,

me011: The -

me011: It - it - it was the only one -

me013: I

fe016: The data issue comes up all the ti-

me011: It was the only one that had any sort of real disagreement about. So.

me013: Well, I didn't have as much disagreement as I would have liked, but I didn't wanna - I wouldn- I didn't wanna get into it because, uh, you know, it was

me013: the application was one I didn't know anything about, uh, it just would have been, you know, me getting up to be argumentative, but -

me011: Yep.

me013: but, uh, I mean, the missing thi- so - so what they were saying - it's one of these things - is - you know, all you need is more data, sort of -

me013: But I mea- i- wh- it - that's - that's dissing it, uh, improperly, I mean, it was a nice study. Uh, they were doing this - it wasn't word-sense disambiguation, it was - was it w- was it word-sense? Yes.

me011: Well, it sort of was.

fe016: Yeah - yeah - yeah -

me011: But it was - it was a very simple case of "to" versus "too" versus "two" and "there", "their", "they're" -

fe016: And there and their and - and that you could do better with more data, I mean, that's clearly statistically -

me013: Yeah, yeah. OK.

me013: Right.

me011: Yeah.

me013: And so, what they did was they had these different kinds of learning machines,

me013: and they had different amounts of data, and so they did like, you know, eight different methods that everybody, you know, uh, argues about - about, "Oh my - my kind of learning machine is better than your kind of learning machine."

me013: And, uh, they were - started off with a million words that they used, which was evidently a number that a lot of people doing that particular kind of task had been using.

me013: So they went up, being Microsoft, they went up to a billion.

me013: And then they had this log scale showing a - you know, and - and naturally everything gets -

me011: Them being beep, they went off to a billion.

me013: they - well, it's a big company, I didn't - I didn't mean it as a ne- anything negative, but i- i- i-

me011: Yeah.

fe016: You mean the bigger the company the more words they use for training?

me011: Well, I think

me011: the reason they can do that, is that they assumed that text that they get off the web, like from Wall Street Journal, is correct, and edit it.

me011: So that's what they used as training data. It's just saying if it's in this corpus it's correct.

me013: Yeah.

me013: OK. But, I mean, yes. Of course there was the kind of effect that, you know, one would expect that - uh - that you got better and better performance with more and more data.

me013: Um, but the - the real point was that

me013: the - the different learning machines are sort of all over the place, and - and by - by going up significantly in data you can have much bigger effect

me013: then by switching learning machines and furthermore which learning machine was on top kind of depended on where you were in this picture, so, uh,

me018: This was my concern about the recognizer in Aurora.

me013: That -

me018: That the differences we're seeing in the front-end

me013: Yeah.

me011: Are irrelevant.

me018: is b- are irrelevant once you get a real recognizer at the back-end.

me013: Yeah.

fe016: If you add more data? Or -

me013: Yeah.

me018: You know?

fe016: Huh.

me013: Yeah, could well be.

me013: So - so, I mean, that was - that was kind of, you know, it's a good point,

me013: but the problem I had with it was that the implications out of this was that,

me013: uh, the kind of choices you make about learning machines were therefore irrelevant

me013: which is not at - n- t- as for as I know in - in tasks I'm more familiar with is not at all true.

me013: What i- what is - is true is that different learning machines have different properties,

me013: and you wanna know what those properties are. And someone else sort of implied that well we s- you know, a- all the study of learning machine we still don't know what those properties are.

me013: We don't know them perfectly, but we know

me013: that some kinds use more memory and - and some other kinds use more computation and some are - are hav- have limited kind of discrimination, but are just easy to use, and others are -

me018: But doesn't their conclusion just sort of - you could have guessed that before they even started? Because

me018: if you assume that these learning things get better and better and better,

me013: You would guess -

me018: then as you approach - there's a point where you can't get any better, right? You get everything right. So they're all approaching.

me013: Yeah.

fe016: It's just no -

me011: But - No, but there was still a spread. They weren't all up- They weren't converging. They were all still spread.

me013: It w-

me018: But what I'm saying is that th- they have to, as they all get better, they have to get closer together.

me011: But they - Right, right. Sure. But

me011: they hadn't even come close to that point. All the tasks were still improving when they hit a billion.

me013: Yeah.

me018: But they're all going the same way, right? So you have to get closer.

me013: Eventually. O- one would thi-

me011: But they didn't get closer.

me018: Oh they didn't?

me011: They just switched position.

me013: Well -

me013: well that's getting cl- I mean,

me013: yeah, the spread was still pretty wide that's th- that's true, but - but, uh,

me011: Yep.

me013: I think it would be irntu- intu- intuition that this would be the case, but, uh, to really see it and to have the intuition is quite different, I mean, I think somebody w- w-

me013: let's see who was talking about earlier that the effect of having a lot more data is quite different in Switchboard than it is in - in Broadcast News, yeah.

fe016: Well it's different for different tasks. So it depends a lot on whether, you know, it - disambiguation is exactly the case where more data is better, right? You're -

me011: Yeah. It was Liz. Yeah.

me013: Yeah.

fe016: you're - you can assume similar distributions, but if you wanted to do disambiguation on a different type of,

fe016: uh, test data then your training data, then that extra data wouldn't generalize, so.

me011: Right.

me011: But, I think one of their p-

me013: Right.

me011: They - they had a couple points. w- Uh,

me011: I think one of them was that "Well, maybe simpler algorithms and more data are - is better". Less memory, faster operation, simpler.

me011: Right? Because their simplest, most brain-dead algorithm did pretty darn well

me013: Mm-hmm.

me011: when you got - gave it a lot more data. And then also

me011: they were saying, "Well, m- You have access to a lot more data. Why are you sticking with a million words?" I mean, their point was that this million-word corpus that everyone uses is apparently ten or fifteen years old.

me011: And everyone is still using it, so.

me013: Yeah.

me013: But anyway, I - I - I think it's - it's just the - the i- it's - it's - it's not really the conclusion they came to so much,

me011: But we could talk about this stuff, I think this would be fun to do.

me011: Ha!

me013: as the conclusion

me013: that some of the, uh, uh, commenters in the crowd came up with that, uh, you know, this therefore is further evidence that, you know, more data is really all you should care about, and that I thought was just kind of

me011: Right.

me013: going too far the other way, and - and the - the, uh,

me011: Machine-learning.

me013: one - one person ga- g- g- got up and made a - a brief defense, uh, but it was a different kind of grounds, it was that - that, uh,

me013: i- w- the reason people were not using so much data before was not because they were stupid or didn't realize data was important, but in fact th- they didn't have it available.

me013: Um, but the other point to make a- again is that, uh, machine learning still does matter,

me013: but it - it matters more in some situations than in others, and it - and also there's - there's not just mattering

me013: or not mattering, but there's mattering in different ways. I mean, you might be in some situation where

me013: you care how much memory you're using,

me011: Right.

me013: or you care, you know, what recall time is, or you care, you know, and - and -

me011: Or you only have a million words

me013: Yeah, or - or, uh -

me011: for your - some new task.

fe016: Or done another language, or - I mean, you - so there's papers on portability and rapid prototyping and blah-blah-blah, and then there's people saying, "Oh, just add more data." So, these are like two different religions, basically.

me011: Yep.

me013: Yeah.

me011: Right.

me013: Yeah.

me013: And there's cost!

me011: Mm-hmm.

me011: Cost. Yeah. That's a big one.

me013: There's just plain cost, you know, so - so these, I mean th- the - in the - in the speech side, the thing that always occurs to me is that if you -

me013: if you - uh - one person

me013: has a system that requires ten thousand hours to train on,

me013: and the other only requires a hundred,

me013: and they both do about the same because the hundred hour one was smarter,

me013: that's -

me013: that's gonna be better.

me011: Yep.

me013: because people, I mean, there isn't gonna be just one system that people train on and then that's it for the r- for all of time. I mean, people are gonna be

me013: doing other different things, and so it - these - these things matters - matter.

fe008: Yeah, that's it. So, I mean, this was a very provocative slide. She put this up, and it was like this is -

me011: Yeah, so that's one of the slides they put up.

me013: Yeah.

fe008: this p- people kept saying, "Can I see that slide again?" and then they'd make a comment, and one person said,

fe016: Yeah, yeah.

fe008: well-known person said, um, you know, "Before you dismiss forty-five years including my work -"

me011: Forty-five years of research.

fe016: Yeah.

me001: Yeah.

fe016: But th- you know, the same thing has happened in computational linguistics,

fe016: right? You look at the ACL papers coming out, and now there's sort of a turn back towards, OK we've learned statistic - you know, we're basically getting what we expect out of some statistical

fe016: methods, and, you know, the- there's arguments on both sides, so -

me011: Yep. I think the matters is the thing that - that was misleading.

fe008: That was very offending, very offending.

fe016: Yeah, yeah.

me011: Is that - all - all of them are based on all the others, right? Just, you - you can't say -

me013: Right.

me018: Maybe they should have said "focus" or something.

me011: Yeah. I mean, so. - And I'm saying the same thing happened with speech recognition, right? For a long time people were hand-c- coding linguistic rules and then they discovered machine-learning worked better.

me011: And now they're throwing more and more data and worrying - perhaps worrying less and less about, uh, the exact details of the algorithms.

fe016: And - and then you hit this -

me011: Except when they have a Eurospeech paper.

me013: Yeah.

fe008: Yeah.

me011: Anyway.

me013: Anyway, tea is - tea is, uh, starting.

me011: Shall we read some digits?

me011: Are we gonna do one at a time? Or should we read them all agai- at once again.

me013: Let's do it all at once. We - - let's try that again.

fe016: Yes!

fe008: Yeah, that's good.

me011: OK.

fe016: So, and maybe we won't laugh this time also.

me011: So remember to read the transcript number so that, uh, everyone knows that - what it is.

me011: And

me011: ready? Three, two, one.

fe008: Yeah.

fe016: Transcript L twenty-three

me001: Transcript L twenty-four.

me011: L twenty-two.

me013: Transcript L two five.

fe008: Transcript L twenty-eight.

me018: Transcript L dash two seven.

fe016: two two four five two three three nine two seven

fe008: four three five five one O four two three

me001: two zero two one seven four one nine three nine

me011: three zero six nine one three three four three seven seven nine

me013: four three zero three four eight six nine five

me018: three six five five five three three three eight

fe016: seven eight four three two nine four eight seven

fe008: five three nine five six four three eight five five zero zero

me013: seven two seven one six nine nine five nine three

me011: O eight two one six two two one nine seven nine three

me018: two one eight nine nine nine six seven nine four

me001: three nine five seven five six three six seven

fe016: six two two eight four five eight four eight nine

fe016: one three three eight seven five two four O two

me011: four eight O eight six nine four seven three one

me001: six two two eight seven one zero eight nine three

me018: zero seven nine four eight eight nine one O eight

fe008: one eight eight four four three three four zero

me013: four nine one eight five one five one

fe016: O two four four six nine eight nine five eight

me013: six eight eight eight eight four seven three five

me011: one one six seven seven six four three three five

me018: six seven O three nine nine one two seven

fe008: zero two three eight seven eight three three six

me001: nine nine three one nine two three O nine seven

fe016: seven five nine nine six seven eight six eight two

me013: five eight six three eight two one five O

fe008: eight nine seven nine eight one seven four six nine

me011: one nine four zero three six one eight zero

me018: seven one six three six seven two five O two nine eight

me001: five zero zero zero two four eight four two zero zero four

me013: two nine four six four four two nine one

fe016: eight eight two three six two five seven

fe008: zero three two five nine eight two O eight eight

me011: eight O five nine seven five nine two

me018: six three seven eight eight two seven nine two one eight six

fe016: six three O seven one five one six six two

me013: two zero three eight three two zero nine

me001: five five two two nine one five zero two one zero three

fe008: eight two zero three four three two two zero nine

me011: nine three four O eight three O five seven seven

me018: nine six five four five three five five two seven one one

me013: six two nine two four zero two zero one

fe008: three O five six one nine O one one seven

me011: O five three six nine one two nine one nine

me001: five seven three eight seven seven seven seven five eight O nine

me018: six eight one nine six four seven four nine eight

me001: one four three three one six one two

me013: Boy, is that ever efficient.

me011: Yep. That's really fast.

me013: Yeah.

me013: Yeah.