Grad H: Yeah , I 've been playing with , um uh , using the close - talking mike to do to try to figure out who 's speaking . So my first attempt was just using thresholding and filtering , that we talked about about two weeks ago , and so I played with that a little bit , and it works O K , except that it 's very sensitive to your choice of your filter width and your threshold . So if you fiddle around with it a little bit and you get good numbers you can actually do a pretty good job of segmenting when someone 's talking and when they 're not . But if you try to use the same paramenters on another speaker , it doesn't work anymore , even if you normalize it based on the absolute loudness .
PhD B: But does it work for that one speaker throughout the whole meeting ?
Grad H: It does work for the one speaker throughout the whole meeting . Um Pretty well .
PhD A: How did you do it Adam ?
Grad H: Pretty well . How did I do it ?
PhD A: Yeah .
Grad H: What do you mean ?
PhD A: I mean , wh what was the Grad H: The algorithm was , uh take o every frame that 's over the threshold , and then median - filter it , and then look for runs .
PhD A: Yeah . Mm - hmm .
Grad H: So there was a minimum run length ,
PhD A: Every frame that 's over what threshold ?
Grad H: so that A threshold that you pick .
PhD A: In terms of energy ? Ah !
Grad H: Yeah .
PhD A: OK .
Postdoc F: Say that again ? Frame over fres threshold .
Grad H: So you take a each frame , and you compute the energy and if it 's over the threshold you set it to one , and if it 's under the threshold you set it to zero , so now you have a bit stream of zeros and ones .
Postdoc F: Hmm . OK .
Grad H: And then I median - filtered that using , um a fairly long filter length . Uh well , actually I guess depends on what you mean by long , you know , tenth of a second sorts of numbers . Um and that 's to average out you know , pitch , you know , the pitch contours , and things like that . And then , uh looked for long runs .
Postdoc F: OK
Grad H: And that works O K , if you fil if you tune the filter parameters , if you tune how long your median filter is and how high you 're looking for your thresholds .
PhD A: Did you ever try running the filter before you pick a threshold ?
Grad H: No . I certainly could though . But this was just I had the program mostly written already so it was easy to do . OK and then the other thing I did , was I took Javier 's speaker - change detector acoustic - change detector , and I implemented that with the close - talking mikes , and unfortunately that 's not working real well , and it looks like it 's the problem is he does it in two passes , the first pass is to find candidate places to do a break . And he does that using a neural net doing broad phone classification and he has the the , uh one of the phone classes is silence . And so the possible breaks are where silence starts and ends . And then he has a second pass which is a modeling a Gaussian mixture model . Um looking for uh whether it improves or or degrades to split at one of those particular places . And what looks like it 's happening is that the even on the close - talking mike the broad phone class classifier 's doing a really bad job .
PhD A: Who was it trained on ?
Grad H: Uh , I have no idea .
PhD A: Hmm .
Grad H: I don't remember . Does an do you remember , Morgan , was it Broadcast News ?
Professor D: I think so , yeah .
Grad H: Um So , at any rate , my next attempt , which I 'm in the midst of and haven't quite finished yet was actually using the uh , thresholding as the way of generating the candidates . Because one of the things that definitely happens is if you put the threshold low you get lots of breaks . All of which are definitely acoustic events . They 're definitely someone talking . But , like , it could be someone who isn't the person here , but the person over there or it can be the person breathing . And then feeding that into the acoustic change detector . And so I think that might work . But , I haven't gotten very far on that . But all of this is close - talking mike , so it 's , uh just just trying to get some ground truth .
PhD E: Only with eh uh , but eh I I I think , eh when when , y I  I saw the the the the speech from PDA and , eh close  talker . I I think the there is a a great difference in the in the signal .
Grad H: Oh , absolutely .
PhD E: Um but eh I but eh I I I mean that eh eh in the in the mixed file you can find , uh zone with , eh great different , eh level of energy .
Grad H: So s my intention for this is is as an aide for ground truth . not PhD E: Um I I think for , eh algorithm based on energy , eh , that um h mmm , more or less , eh , like eh eh , mmm , first sound energy detector .
Grad H: Say it again ?
PhD E: eh nnn . When y you the detect the the the first at at the end of of the detector of , ehm princ um . What is the the name in English ? the the , mmm ,  the de detector of , ehm of a word in the in the s in an isolated word in in the background That , uh
Grad H: I 'm I 'm not sure what you 're saying , can you try PhD E: I mean that when when you use , eh eh any
PhD A: I think he 's saying the onset detector .
PhD E: Yeah .
Grad H: Onset detector , OK .
PhD E: I I think it 's probably to work well eh , because , eh you have eh , in the mixed files a great level of energy . eh and great difference between the sp speaker . And probably is not so easy when you use the the PDA , eh that Because the signal is , eh the in the e energy level .
Grad H: Right .
PhD E: in in that , eh eh speech file is , eh more similar . between the different eh , speaker , um I I think is eh , it will i is my opinion .
Grad H: Right . But different speakers .
PhD E: It will be , eh more difficult to to detect bass - tone energy . the the change . I think that , um
Grad H: Ah , in the clo in the P D A , you mean ?
PhD E: In the PDA .
Grad H: Absolutely .
PhD E: Yeah . Yeah .
Grad H: Yeah , no question . It 'll be much harder .
Professor D: Yeah .
Grad H: Much harder .
PhD E: And the the another question , that when I review the the the work of Javier . I think the , nnn , the , nnn , that the idea of using a neural network to to get a broad class of phonetic , eh from , eh uh a candidate from the the the speech signal . If you have , eh uh , I 'm considering , only because Javier , eh only consider , eh like candidate , the , nnn , eh the silence , because it is the the only model , eh eh , he used that , eh  eh nnn , to detect the the possibility of a a change between the between the speaker ,
Grad H: Right .
PhD E: Um another another research thing , different groups , eh working , eh on Broadcast News prefer to , eh to consider hypothesis eh between each phoneme .
Grad H: Mm - hmm . Yeah , when a phone changes .
PhD E: Because , I I I think it 's more realistic that , uh only consider the  the the the silence between the speaker . Eh there there exists eh silence between between , eh a speaker . is is , eh eh acoustic , eh event , important to to consider .
Professor D: Mm - hmm . Mm - hmm .
PhD E: I I found that the , eh silence in in many occasions in the in the speech file , but , eh when you have , eh eh , two speakers together without enough silence between between them , eh  I think eh is better to use the acoustic change detector basically and I I I IX or , mmm , BIC criterion for consider all the frames in my opinion .
Professor D: Mm - hmm . Yeah , the you know , the reason that he , uh just used silence was not because he thought it was better , it was it was it was the place he was starting .
PhD E: Yeah .
Grad H: Yep .
Professor D: So , he was trying to get something going ,
PhD E: Yeah .
Professor D: and , uh e e you know , as as  as is in your case , if you 're here for only a modest number of months you try to pick a realistic goal ,
PhD E: Yeah , yeah , yeah , yeah . Yeah . Yeah , yeah , yeah , yeah .
Grad H: Do something .
Professor D: But his his goal was always to proceed from there to then allow broad category change also .
PhD E: Uh - huh . But , eh do do you think that if you consider all the frames to apply the the , eh the BIC criterion to detect the the the different acoustic change , eh between speaker , without , uh with , uh silence or with overlapping , uh , I think like like , eh eh a general , eh eh way of process the the acoustic change .
Professor D: Mm - hmm .
PhD E: In a first step , I mean .
Professor D: Mm - hmm .
PhD E: An - and then , eh  eh without considering the you you you , um you can consider the energy like a another parameter in the in the feature vector , eh .
Grad H: Right . Absolutely .
Professor D: Mm - hmm .
PhD E: This this is the idea . And if , if you do that , eh eh , with a BIC uh criterion for example , or with another kind of , eh of distance in a first step , and then you , eh you get the , eh the hypothesis to the this change acoustic , eh  to po process
Grad H: Right .
PhD E: Because , eh eh , probably you you can find the the eh a small gap of silence between speaker with eh eh a ga mmm ,  small duration Less than , eh two hundred milliseconds for example
Professor D: Mm - hmm .
PhD E: and apply another another algorithm , another approach like , eh eh detector of ene , eh detector of bass - tone energy to to consider that , eh that , eh zone . of s a small silence between speaker , or another algorithm to to process , eh the the segment between marks eh founded by the the the BIC criterion and applied for for each frame .
Professor D: Mm - hmm . Mm - hmm .
PhD E: I think is , eh nnn , it will be a an an a more general approach the if we compare with use , eh a neural net or another , eh speech recognizer with a broad class or or narrow class , because , in my opinion eh it 's in my opinion , eh if you if you change the condition of the speech , I mean , if you adjust to your algorithm with a mixed speech file and to , eh to , eh  adapt the neural net , eh used by Javier with a mixed file .
Professor D: Mm - hmm . Mm - hmm .
PhD E: uh With a m mixed file ,
Grad H: With the what file ?
PhD A: " Mixed " .
PhD E: with a the mix , mix .
Postdoc F: " Mixed . "
Grad H: " Mixed ? "
Professor D: Mm - hmm .
PhD E: Sorry . And and then you you , eh you try to to apply that , eh , eh , eh , speech recognizer to that signal , to the PDA , eh speech file , I I think you will have problems , because the the the the condition you you will need t t I I suppose that you will need to to to retrain it .
Professor D: Well , I I Grad H: Oh , absolutely . This is this is not what I was suggesting to do .
Professor D: u Look , I I think this is a One once It 's a I used to work , like , on voiced on voice silence detection , you know , and this is this kind of thing .
PhD E: Really ? Yeah .
Professor D: Um If you have somebody who has some experience with this sort of thing , and they work on it for a couple months , they can come up with something that gets most of the cases fairly easily . Then you say , " OK , I don't just wanna get most of the cases I want it to be really accurate . " Then it gets really hard no matter what you do . So , the p the problem is is that if you say , " Well I I have these other data over here , that I learn things from , either explicit training of neural nets or of Gaussian mixture models or whatever . "
PhD E: Yeah .
Professor D: Uh Suppose you don't use any of those things . You say you have looked for acoustic change . Well , what does that mean ? That that means you set some thresholds somewhere or something ,
PhD E: Yeah .
Professor D: right ? and and so where do you get your thresholds from ?
PhD E: Yeah .
Professor D: From something that you looked at . So you always have this problem , you 're going to new data um H how are you going to adapt whatever you can very quickly learn about the new data ? Uh , if it 's gonna be different from old data that you have ? And I think that 's a problem with this .
Grad H: Well , also what I 'm doing right now is not intended to be an acoustic change detector for far - field mikes . What I 'm doing is trying to use the close - talking mike and just use  Can - and just generate candidate and just try to get a first pass at something that sort of works .
PhD E: Yeah !
PhD A: You have candidates .
PhD G: Actually actually actually PhD E: the candidate .
PhD G: I PhD A: to make marking easier . Yeah .
PhD G: Or Grad H: and I haven't spent a lot of time on it and I 'm not intending to spend a lot of time on it .
PhD G: OK . I um , I , unfortunately , have to run ,
Grad H: So .
PhD G: but , um I can imagine uh building a um model of speaker change detection that takes into account both the far - field and the uh actually , not just the close - talking mike for that speaker , but actually for all of th for all of the speakers .
Grad H: Yep . Everyone else .
Professor D: Yeah .
PhD G: um If you model the  the effect that me speaking has on your microphone and everybody else 's microphone , as well as on that , and you build , um basically I think you 'd you would build a  an HMM that has as a state space all of the possible speaker combinations
Grad H: All the Yep .
PhD E: Yeah .
PhD G: and , um you can control Grad H: It 's a little big .
PhD G: It 's not that big actually , um
Grad H: Two to the N . Two to the number of people in the meeting .
Professor D: But Actually , Andreas may maybe maybe just something simpler but but along the lines of what you 're saying ,
Grad H: Anyway .
PhD E: Yeah .
Professor D: I was just realizing , I used to know this guy who used to build , uh um , mike mixers automatic mike mixers where , you know , t in order to able to turn up the gain , you know , uh as much as you can , you you you lower the gain on on the mikes of people who aren't talking ,
PhD G: Mmm .
PhD E: Yeah {comment} Yeah .
PhD G: Mmm . Mm - hmm .
Professor D: right ? And then he had some sort of reasonable way of doing that ,
PhD G: Mm - hmm .
Professor D: but uh , what if you were just looking at very simple measures like energy measures but you don't just compare it to some threshold overall but you compare it to the energy in the other microphones .
Grad H: I was thinking about doing that originally to find out who 's the loudest , and that person is certainly talking .
Professor D: Yeah .
Grad H: But I also wanted to find threshold uh , excuse me , mol overlap .
Professor D: Yeah .
Grad H: So , not just just the loudest .
PhD E: But , eh
Postdoc F: Mm - hmm .
PhD E: I I Sorry . I I have found that when when I I analyzed the the speech files from the , eh mike , eh from the eh close eh microphone , eh I found zones with a a different level of energy .
PhD G: Sorry , I have to go .
Grad H: OK . Could you fill that out anyway ? Just , put your name in . Are y you want me to do it ? I 'll do it .
PhD A: But he 's not gonna even read that . Oh .
Grad H: I know .
PhD E: including overlap zone . including . because , eh eh depend on the position of the of the microph of the each speaker to , eh , to get more o or less energy i in the mixed sign in the signal . and then , if you consider energy to to detect overlapping in in , uh , and you process the the in the the the speech file from the the the mixed signals . The mixed signals , eh . I I think it 's it 's difficult , um  only to en with energy to to consider that in that zone We have eh , eh , overlapping zone Eh , if you process only the the energy of the , of each frame .
Professor D: Well , it 's probably harder , but I I think what I was s nnn noting just when he when Andreas raised that , was that there 's other information to be gained from looking at all of the microphones and you may not need to look at very sophisticated things ,
PhD E: Yeah .
Professor D: because if there 's  if most of the overlaps you know , this doesn't cover , say , three , but if most of the overlaps , say , are two , if the distribution looks like there 's a couple high ones and and the rest of them are low ,
PhD E: Yeah . Yeah . Yeah . Yeah .
Grad H: And everyone else is low , yeah .
Professor D: you know , what I mean ,
PhD E: Yeah .
Professor D: there 's some information there about their distribution even with very simple measures .
PhD E: Yeah . Yeah .
Professor D: Uh , by the way , I had an idea with while I was watching Chuck nodding at a lot of these things , is that we can all wear little bells on our heads , so that then you 'd know that PhD E: Yeah .
Grad H: Ding , ding , ding , ding .
PhD E: Yeah .
Postdoc F: " Ding " . That 's cute !
PhD B: I think that 'd be really interesting too , with blindfolds . Then Grad H: Nodding with blindfolds ,
PhD B: Yeah . The question is , like whether Grad H: " what are you nodding about ? "
PhD B: Well , trying with and  with and without , yeah .
Grad H: " Sorry , I 'm just I 'm just going to sleep . "
PhD B: But then there 's just one @ @ , like .
Professor D: Yeah .
PhD A: Actually , I saw a uh a woman at the bus stop the other day who , um , was talking on her cell phone speaking Japanese , and was bowing . you know , profusely .
PhD B: Oh , yeah , that 's really common .
PhD C: Yeah .
PhD E: Yeah {comment} Yeah .
PhD A: Just , kept PhD E: Yeah .
Postdoc F: Ah .
Professor D: Wow .
PhD B: It 's very difficult if you try while you 're trying , say , to convince somebody on the phone it 's difficult not to move your hands . Not You know , if you watch people they 'll actually do these things .
Professor D: Mm - hmm ?
PhD B: So . I still think we should try a a meeting or two with the blindfolds , at least of this meeting that we have lots of recordings of
Grad H: Mm - hmm .
PhD B: Um , maybe for part of the meeting , we don't have to do it the whole meeting .
Professor D: Yeah , I think th I think it 's a great idea .
PhD B: That could be fun . It 'll be too hard to make barriers , I was thinking because they have to go all the way
Professor D: W Yeah .
PhD B: you know , I can see Chuck even if you put a barrier here .
Grad H: Well , we could just turn out the lights .
Postdoc F: Actually well also I I can say I made barr barriers for so that the stuff I was doing with Collin wha which just used , um this kind of foam board .
PhD B: Y Yeah ?
Postdoc F: R really inexpensive . You can you can masking tape it together , these are you know , pretty l large partitions .
Professor D: Yeah .
PhD B: But then we also have these mikes , is the other thing I was thinking , so we need a barrier that doesn't disturb the sound ,
Postdoc F: It 's true , it would disturb the , um the the long - range Grad H: The acoustics .
PhD B: um
Professor D: Blindfolds would be good .
Postdoc F: it would Grad H: I think , blindfolds .
PhD B: I mean , it sounds weird but but  you know it 's  it 's cheap and , uh Be interesting to have the camera going .
Professor D: Probably we should wait until after Adam 's set up the mikes , But .
Postdoc F: OK . I think we 're going to have to work on the , uh  on the human subjects form .
PhD A: I 'll be peeking .
Grad H: Yeah , that 's right , we didn't tell them we would be blindfolding .
Professor D: That 's Postdoc F: " Do you mind being blindfolded while you 're interviewed ? "