Grad G: Yeah , that 's the problem with the NSA speakers .
PhD D: And e and e and extremely hard to follow , like word - wise ,
Grad E: So .
PhD D: I bet the transcri I mean , I have no idea what they 're talking about ,
Grad G: Yeah .
Postdoc A: I corrected it for a number of the words .
PhD D: so ,
Postdoc A: I 'm sure that , um , they 're they 're accurate now .
PhD D: um ,
PhD F: Uh , actually I have to  to go .
PhD D: I mean , this is tough for a language model probably Grad G: Right .
PhD D: but but that might be useful just for speech .
Professor C: Well .
Grad E: OK , Andreas is leaving leaving the building . Mm - hmm .
Professor C: Yeah .
Grad E: See ya .
Professor C: See ya . I don't think we 'll go much longer .
Grad E: Um , oh , before you l go I guess it 's alright for you to talk a little without the mike I noticed you adjusting the mike a lot , did it not fit you well ? Oh .
Postdoc A: Well I won I noticed when you turned your head , it would it would tilt .
Grad E: Maybe it wasn't just tightened enough , or PhD D: Maybe the yeah , the s thing that you have tightened @ @ ,
PhD B: Actually if if you have a larger head , that mike 's gotta go farther away which means the the balance is gonna make it wanna tip down .
PhD D: oh .
Grad E: OK . Anyway .
Professor C: Yeah . OK , see ya .
Grad E: Cuz , I 'm just thinking , you know , we were we 're we 've been talking about changing the mikes , uh , for a while ,
Grad G :
Postdoc A: Yeah .
Grad E: and if these aren't acoustically they seem really good , but if they 're not comfortable , we have the same problems we have with these stupid things .
Postdoc A: I think it 's com This is the first time I 've worn this , I find it very comfortable .
Grad E: I find it very comfortable too , but , uh , it looked like Andreas was having problems , and I think Morgan was saying it Professor C: Well , but I had it on I had it on this morning and it was fine .
PhD B: Can I see that ?
Grad E: Oh , oh you did wear it this morning ?
Professor C: Yeah .
Grad E: OK , it 's off , so you can put it on .
PhD B: I yeah , I don't want it on , I just I just want to , um , say what I think is a problem with this . If you are wearing this over your ears and you 've got it all the way out here , then the balance is gonna want to pull it this way .
PhD D: Yeah .
Grad E: Right .
PhD B: Where as if somebody with a smaller head has it back here ,
Grad E: It 's more balanced .
PhD B: right ? Yeah .
Postdoc A: Oh !
PhD B: Then it then it falls back this way so it 's PhD D: So we have to
Grad E: Well wh what it 's supposed to do is the backstrap is supposed to be under your crown , and so that should be should be Postdoc A: Ah .
Grad E: if it 's right against your head there , which is what it 's supposed to be , that balances it so it doesn't slide up .
PhD B: So this is supposed to be under that little protuberance .
Grad E: Yep , right right below if you feel the back of your head , you feel a little lump ,
PhD B: Yeah .
Grad E: um , and so it 's supposed to be right under that .
PhD B: So it 's really supposed to go more like this than like this .
Grad E: Yes , exactly .
PhD B: But then isn't that going to Well , I guess you can control that .
Grad E: That that that tilts , right ? In lots and lots of different ways .
PhD D: So I 'm not saying anything about bias towards small headsize ,
Grad E: About heads ?
PhD D: but does seem , uh PhD B: It would be an advantage .
Postdoc A: Well , wonder if it 's if if he was wearing it over his hair instead of under his hair .
Professor C: Well , we should We shou we should work on compressing the heads , and Grad E: I think probably it was Yeah . It probably just wasn't tight enough to the back of his head . I mean , so the directions do talk about bending it to your size , which is not really what we want .
PhD B: The other thing that would do it would be to hang a five pound weight off the back .
Professor C: Yeah
PhD D: Right .
Professor C: that 's good !
Postdoc A: What did you say ?
PhD D: A little ,
Grad E: wh
Professor C: Hang a five pound weight off the off the back .
PhD B: Hang a five pound weight off the back .
PhD D: um ,
Grad E: We did that Professor C: Weight .
Grad E: We at Boeing I used I was doing augmented reality so they had head - mounts on , and we we had a little jury - rigged one with a welder 's helmet ,
PhD B: Counter - balance .
Grad E: and we had just a bag with a bunch of marbles in it as a counter - balance .
Professor C: Or maybe this could be helpful just for evening the conversation between people . If people those who talk a lot have to wear heavier weights or something , and Grad E: Yeah !
Professor C: and um ,
Grad E: Anyway .
Professor C: um , so , uh , what was I gonna say ? Oh , yeah , I was gonna say , uh , I had these , uh , conversations with NIST folks also while I was there and and , uh , um , so they they have their their plan for a room , uh , with , um , mikes in the middle of the table , and , uh , close - mounted mikes ,
Grad E: Yep .
Professor C: and they 're talking about close - mounted and lapels , just cuz
PhD D: And arrays ,
Professor C: sort of and the array .
Grad E: And arrays ,
Professor C: Yeah , so they were PhD D: which is the i interesting Grad E: yep . And cameras .
Professor C: And yeah , like multiple multiple video cameras coverin covering every everybody every place in the room ,
PhD D: and video , right .
Professor C: uh , the yeah the the mikes in the middle , the head - mounted mikes , the lapel mikes , the array , uh , with well , there 's some discussion of fifty - nine ,
Grad E: Fifty - nine elements .
Professor C: they might go down to fifty - seven Because , uh , there is , uh , some pressure from a couple people at the meeting for them to use a KEMAR head . I forget what KEMAR , uh , stands for ,
PhD D: Mm - hmm .
Professor C: but what it is is it 's dummy head that is very specially designed ,
Grad E: Oh , that 's right .
Professor C: and and and , so what they 're actually doing is they 're really there 's really two recording systems .
PhD D: Right .
Grad E: Yep .
PhD D: That 's a great idea .
Professor C: So they may not be precisely synchronous , but the but there 's two two recording systems , one with , I think , twenty - four channels , and one with sixty - four channels . And the sixty - four channel one is for the array , but they 've got some empty channels there , and anyway they like they 're saying they may give up a couple or something if for for the KEMAR head if they go go with that .
Grad E: Right . Yeah , it is a good idea .
Professor C: So .
Grad E: Yeah , h uh , J Jonathan Fiscus did say that , uh , they have lots of software for doing calibration for skew and offset between channels
PhD D: Mm - hmm
Grad E: and that they 've found that 's just not a big deal .
Professor C: Yeah .
Grad E: So .
Professor C: Yeah , I 'm not too worried about that . I was thinking PhD D: But they 're still planning to do like fake Grad E: Scenario - based .
PhD D: they have to do something like that ,
Grad E: Y right .
PhD D: right .
Grad E: Their their legal issues won't allow them to do otherwise .
Professor C: Yeah .
Grad E: But it sounded like they were pretty well thought out
PhD D: Yeah , th that 's true .
Grad E: and they 're they 're gonna be real meetings ,
Postdoc A: Mm - hmm .
Grad E: it 's just that they 're with str with people who would not be meeting otherwise .
PhD B: Did did they give a talk on this or was this informal ?
PhD D: Mm - hmm .
Grad E: So .
PhD D: No .
Grad E: No .
Professor C: No , we just had some discussions , various discussions with them .
Grad E: It 's just informal .
Postdoc A: Mm - hmm . Mm - hmm . Yeah .
Grad E: Yeah , I also sat and chatted with several of the NIST folks . They seemed like a good group .
PhD B: What was the , um the paper by , um , Lori Lamel that you mentioned ?
Professor C: Um , yeah , we sh we should just have you have you read it , but , I mea ba i i uh , we 've all got these little proceedings ,
Postdoc A: Mmm , yeah .
Professor C: but , um , basically , it was about , um , uh , going to a new task where you have insufficient data and using using data from something else , and adapting , and how well that works . Uh , so in in fact it was pretty related to what Liz and Andreas did , uh , except that this was not with meeting stuff , it was with
Grad E: Right .
Professor C: uh , like I think they s didn't they start off with Broadcast News system ? And then they went to Grad E: The - their Broadcast News was their acoustic models and then all the other tasks were much simpler .
Professor C: Yeah .
Grad E: So they were command and control and that sort of thing .
Professor C: TI - digits was one of them , and , uh , Wall Street Journal .
Grad E: Yep .
PhD B: What was their rough what was their conclusion ?
Grad E: Yeah , read Wall Street Journal . It works .
Professor C: Yeah .
PhD D: Well , it 's it 's a good paper , I mean Professor C: Yeah , yeah .
Grad E: Yeah . Yeah , that was one of the ones that I liked .
PhD D: Bring the Grad E: That It not only works , in some cases it was better , which I thought was pretty interesting , but that 's cuz they didn't control for parameters . So .
Professor C: Probably .
Grad E: You know , the Broadcast News nets were not nets ,
PhD D: Right .
PhD B: Did they ever try going going the other direction from simpler task to more complicated tasks ,
Grad E: acoustic models {comment} were a lot more complex .
PhD B: or ?
Grad E: n Not in that paper .
Professor C: That might be hard .
Grad E: Yeah , well , one of the big problems with that is is often the simpler task isn't fully doesn't have all the phones in it ,
Professor C: Yeah .
Grad E: and that that makes it very hard .
PhD B: Mm - hmm .
Professor C: Yeah .
Grad E: But I 've done the same thing . I 've been using Broadcast News nets for digits ,
Professor C: Yeah .
Grad E: like for the spr speech proxy thing that I did ? That 's what I did .
Professor C: Yeah , sure .
Grad E: So . It works .
Professor C: Yeah . Yeah , and they have I mean they have better adaptation than we had than that that system ,
Grad E: Yep .
Professor C: so they um ,
Grad E: You mean they have some .
Professor C: yeah , we should probably what would actually what we should do , uh , I haven't said anything about this , but probably the five of us should pick out a paper or two that that , uh , you know , got our interest , and we should go around the room at one of the Tuesday lunch meetings and say , you know , what what was good about the conference ,
Grad E: Present . Yep . Do a trip report .
Professor C: yeah .
PhD D: Well , the summarization stuff was interesting , I mean , I don't know anything about that field , but for this proposal on meeting summarization , um , I mean , it 's sort of a far cry because they weren't working with meeting type data , but he got sort of an overview on some of the different approaches ,
Grad E: Right .
PhD B: Do you remember who the groups were that we 're doing ?
PhD D: so . Well there 're this was the last day ,
Grad E: A lot of different ones .
Postdoc A: R I think Mm - hmm .
PhD D: but , I mean , there 's that 's a huge field and probably the groups there may not be representative of the field , I I don't know exactly that everyone submits to this particular conference ,
PhD B: Was were there folks from BBN presenting ?
PhD D: but yet there was , let 's see , this was on the last day , Mitre , BBN , and , um , Prager Grad E: Mitre , BBN , IBM . Uh ,
Postdoc A: Maryland .
PhD D: um , I wo it was Professor C: Columbia have anything ? No .
PhD D: no it was Grad E: Wasn't Who who who did the order one ?
PhD D: this was Wednesday morning . The sentence ordering one , was that Barselou , and these guys ?
Grad E: Ugh ! {comment} I 'm just so bad at that .
Postdoc A: Oh .
PhD D: Anyway , I I it 's in the program , I should have read it to remind myself , but that 's sort of useful and I think like when Mari and Katrin and Jeff are here it 'd be good to figure out some kinds of things that we can start doing maybe just on the transcripts cuz we already have Professor C: Mm - hmm .
Grad E: Yeah , we do have word transcripts .
PhD D: you know , yeah .
Grad E: So .
Postdoc A: Well , I like the idea that Adam had of of , um , z maybe generating minutes based on some of these things that we have because it would be easy to to to do that just , you know , and and
PhD D: Right .
Postdoc A: it has to be , though , someone from this group because of the technical nature of the thing .
Grad E: Someone who actually does take notes , um , I 'm very bad at note - taking .
PhD D: But I think what 's interesting is there 's all these different evaluations , like just , you know , how do you evaluate whether the summary is good or not ,
Grad E: I always write down the wrong things .
Postdoc A: I do take notes .
PhD D: and that 's what 's was sort of interesting to me is that there 's different ways to do it ,
Grad E: A judge .
PhD D: and PhD B: Was SRA one of the groups talking about summarization , no ?
Grad E: Yep .
PhD D: Hm - umm . No .
Postdoc A: It was an interesting session . One of those w
Grad E: And as I said , I like the Microsoft talk on scaling issues in , uh , word sense disambiguation ,
PhD D: Yeah .
Grad E: that was interesting .
Professor C: Yeah , that was an interesting discussion ,
Grad E: The Professor C: uh , I
Grad E: It it it was the only one It was the only one that had any sort of real disagreement about .
PhD D: The data issue comes up all the ti
Professor C: Well , I didn't have as much disagreement as I would have liked ,
Grad E: So .
Professor C: but I didn't wanna I wouldn I didn't wanna get into it because , uh , you know , it was the application was one I didn't know anything about ,
Grad E: Yep .
Professor C: uh , it just would have been , you know , me getting up to be argumentative , but but , uh , I mean , the missing thi so so what they were saying it 's one of these things is you know , all you need is more data , sort of But I mea i wh it @ @ that 's that 's dissing it , uh , improperly , I mean , it was a nice study . Uh , they were doing this it wasn't word - sense disambiguation , it was PhD D: Yeah yeah yeah Grad E: Well , it sort of was .
Professor C: was it w was it word - sense ? Yes .
Grad E: But it was it was a very simple case of " to " versus " too " versus " two " and " there " , " their " , " they 're " PhD D: And there and their and Professor C: Yeah , yeah . OK .
PhD D: and that you could do better with more data , I mean , that 's clearly statistically Professor C: Right .
Grad E: Yeah .
Professor C: And so , what they did was they had these different kinds of learning machines , and they had different amounts of data , and so they did like , you know , eight different methods that everybody , you know , uh , argues about about , " Oh my my kind of learning machine is better than your kind of learning machine . " And , uh , they were started off with a million words that they used , which was evidently a number that a lot of people doing that particular kind of task had been using . So they went up , being Microsoft , they went up to a billion . And then they had this log scale showing a you know , and and naturally everything gets Grad E: Them being beep , {comment} they went off to a billion .
Professor C: they well , it 's a big company , I didn't I didn't mean it as a ne anything negative ,
Grad E: Yeah .
Professor C: but i i i
PhD D: You mean the bigger the company the more words they use for training ?
Grad E: Well , I think the reason they can do that , is that they assumed that text that they get off the web , like from Wall Street Journal , is correct , and edit it .
Professor C: Yeah .
Grad E: So that 's what they used as training data . It 's just saying if it 's in this corpus it 's correct .
Professor C: OK . But , I mean , yes . Of course there was the kind of effect that , you know , one would expect that uh that you got better and better performance with more and more data . Um , but the the real point was that the the different learning machines are sort of all over the place , and and by by going up significantly in data you can have much bigger effect then by switching learning machines and furthermore which learning machine was on top kind of depended on where you were in this picture , so ,
PhD B: This was my concern about the recognizer in Aurora .
Professor C: uh , That PhD B: That the differences we 're seeing in the front - end is b
Professor C: Yeah .
Grad E: Are irrelevant .
PhD B: are irrelevant once you get a real recognizer at the back - end .
Professor C: Yeah .
PhD D: If you add more data ? Or PhD B: You know ?
Professor C: Yeah .
PhD D: Huh .
Professor C: Yeah , could well be . So so , I mean , that was that was kind of , you know , it 's a good point , but the problem I had with it was that the implications out of this was that , uh , the kind of choices you make about learning machines were therefore irrelevant which is not at n t as for as I know in in tasks I 'm more familiar with @ @ is not at all true . What i what is is true is that different learning machines have different properties , and you wanna know what those properties are . And someone else sort of implied that well we s you know , a all the study of learning machine we still don't know what those properties are . We don't know them perfectly , but we know that some kinds use more memory and and some other kinds use more computation and some are are hav have limited kind of discrimination , but are just easy to use , and others are PhD B: But doesn't their conclusion just sort of you could have guessed that before they even started ? Because if you assume that these learning things get better and better and better ,
Professor C: You would guess PhD B: then as you approach there 's a point where you can't get any better , right ? You get everything right .
Professor C: Yeah .
PhD D: It 's just no Grad E: But PhD B: So they 're all approaching .
Grad E: No , but there was still a spread . They weren't all up They weren't converging .
PhD B: But what I 'm saying is that th they have to , as they all get better , they have to get closer together .
Professor C: It w
Grad E: They were all still spread . But they Right , right . Sure . But they hadn't even come close to that point . All the tasks were still improving when they hit a billion .
Professor C: Yeah .
PhD B: But they 're all going the same way , right ? So you have to get closer .
Professor C: Eventually . O one would
Grad E: But they didn't get closer .
PhD B: Oh they didn't ?
Professor C: Well Grad E: They just switched position .
Professor C: well that 's getting cl I mean , yeah , the spread was still pretty wide that 's th that 's true ,
Grad E: Yep .
Professor C: but but , uh , I think it would be irntu intu intuition that this would be the case , but , uh , to really see it and to have the intuition is quite different , I mean , I think somebody w w let 's see who was talking about earlier that the effect of having a lot more data is quite different in Switchboard than it is in in Broadcast News ,
PhD D: Well it 's different for different tasks .
Grad E: Yeah . It was Liz . Yeah .
Professor C: yeah .
PhD D: So it depends a lot on whether , you know , it disambiguation is exactly the case where more data is better , right ? You 're you 're you can assume similar distributions ,
Professor C: Yeah .
PhD D: but if you wanted to do disambiguation on a different type of , uh , test data then your training data , then that extra data wouldn't generalize ,
Grad E: Right .
PhD D: so .
Professor C: Right .
Grad E: But , I think one of their p They they had a couple points . w {comment} Uh , I think one of them was that " Well , maybe simpler algorithms and more data are is better " . Less memory , faster operation , simpler . Right ? Because their simplest , most brain - dead algorithm did pretty darn well
Professor C: Mm - hmm .
Grad E: when you got gave it a lot more data . And then also they were saying , " Well , m You have access to a lot more data . Why are you sticking with a million words ? " I mean , their point was that this million - word corpus that everyone uses is apparently ten or fifteen years old . And everyone is still using it , so .
Professor C: Yeah . But anyway , I I I think it 's it 's just the the i it 's it 's it 's not really the conclusion they came to so much , as the conclusion that some of the , uh , uh , commenters in the crowd came up with
Grad E: But we could talk about this stuff , I think this would be fun to do .  Right .
Professor C: that , uh , you know , this therefore is further evidence that , you know , more data is really all you should care about , and that I thought was just kind of going too far the other way ,
Grad E: Machine - learning .
Professor C: and and the the , uh , one one person ga g g got up and made a a brief defense , uh , but it was a different kind of grounds , it was that that , uh , i w the reason people were not using so much data before was not because they were stupid or didn't realize data was important , but in fact th they didn't have it available . Um , but the other point to make a again is that , uh , machine learning still does matter , but it it matters more in some situations than in others , and it and also there 's there 's not just mattering or not mattering , but there 's mattering in different ways . I mean , you might be in some situation where you care how much memory you 're using , or you care , you know , what recall time is ,
Grad E: Right .
Professor C: or you care , you know , and and Grad E: Or you only have a million words for your some new task .
Professor C: Yeah , or or , uh PhD D: Or done another language , or I mean , you so there 's papers on portability and rapid prototyping and blah - blah - blah ,
Grad E: Yep .
Professor C: Yeah .
Grad E: Right .
Professor C: Yeah .
PhD D: and then there 's people saying , " Oh , just add more data . "
Professor C: And there 's cost !
PhD D: So , these are like two different religions , basically .
Grad E: Mm - hmm . Cost .
Professor C: There 's just plain cost ,
Grad E: Yeah . That 's a big one .
Professor C: you know , so so these , I mean th the in the in the speech side , the thing that @ @ always occurs to me is that if you if you uh one person has a system that requires ten thousand hours to train on , and the other only requires a hundred , and they both do about the same because the hundred hour one was smarter , that 's that 's gonna be better . because people , I mean , there isn't gonna be just one system that people train on
Grad E: Yep .
Professor C: and then that 's it for the r for all of time . I mean , people are gonna be doing other different things , and so it these these things matters matter .
Postdoc A: Yeah , that 's it .
Grad E: Yeah , so that 's one of the slides they put up .
Postdoc A: So , I mean , this was a very provocative slide . She put this up , and it was like this is this p people kept saying , " Can I see that slide again ? "
Professor C: Yeah .
PhD D: Yeah ,
Postdoc A: and then they 'd make a comment , and one person said , well - known person said , um , you know , " Before you dismiss forty - five years including my work "
PhD D: yeah .
Grad E: Forty - five years of research .
PhD D: Yeah .
Grad G: Yeah .
PhD D: But th you know , the same thing has happened in computational linguistics , right ? You look at the ACL papers coming out , and now there 's sort of a turn back towards , OK we 've learned statistic you know , we 're basically getting what we expect out of some statistical methods , and , you know , the there 's arguments on both sides ,
Grad E: Yep .
PhD D: so Grad E: I think the matters is the thing that that was misleading .
Postdoc A: That was very offending , very offending .
PhD D: Yeah , yeah .
Grad E: Is that all all of them are based on all the others , right ? Just , you you can't say PhD B: Maybe they should have said " focus " or something .
Professor C: Right .
Grad E: Yeah . I mean , so . And I 'm saying the same thing happened with speech recognition , right ? For a long time people were hand - c coding linguistic rules and then they discovered machine - learning worked better . And now they 're throwing more and more data and worrying perhaps worrying less and less about , uh , the exact details of the algorithms .
PhD D: And and then you hit this Grad E: Except when they have a Eurospeech paper .
Postdoc A: Yeah .
Professor C: Yeah .
Grad E: Anyway .
Professor C: Anyway , tea is tea is , uh , starting .
Grad E: Shall we read some digits ? Are we gonna do one at a time ? Or should we read them all agai at once again .
Professor C: Let 's do it all at once .
Postdoc A: Yeah , that 's good .
Professor C: We @ @ let 's try that again .
PhD D: Yes ! So , and maybe we won't laugh this time also .
Grad E: OK . So remember to read the transcript number so that , uh , everyone knows that what it is . And ready ?
Postdoc A: Yeah .
Grad E: Three , two , one .
Professor C: Boy , is that ever efficient .
Grad E: Yep . That 's really fast .
Professor C: Yeah . Yeah .