PhD D: Um , so this is the first thing . Um Yeah , another thing that I it 's important to mention is , um , that this has a this has some additional latency . Um . Because when I do the smoothing , uh , it 's a recursion that estimated the means , so of the g of the gain curve . And this is a filter that has some latency . And I noticed that it 's better if we take into account this latency . So , instead o of using the current estimated mean to , uh , subtract the current frame , it 's better to use an estimate that 's some somewhere in the future . Um PhD A: And that 's what causes the latency ? OK .
PhD B: You mean , the m the mean is computed o based on some frames in the future also ?
Professor C: Mm - hmm .
PhD D: Yeah .
PhD B: Or or no ?
PhD D: It 's the recursion , so it 's it 's the center recursion , right ?
PhD B: Mm - hmm .
PhD D: Um and the latency of this recursion is around fifty milliseconds .
Professor C: One five ?
PhD D :
Professor C: One five ? Five zero ?
PhD D: Five zero ,
Professor C: Five zero .
PhD D: yeah .
Professor C: Yeah .
PhD D: Um ,
PhD B: I 'm sorry ,
PhD D: mmm .
PhD B: why why is that delay coming ? Like , you estimate the mean ?
PhD D: Yeah , the mean estimation has some delay , right ?
PhD B: Oh , yeah .
PhD D: I mean , the the filter that that estimates the mean has a time constant .
PhD B: It isn't OK , so it 's like it looks into the future also . OK .
PhD D: Yeah .
Professor C: What if you just look into the past ?
PhD D: It 's , uh , not as good . It 's not bad .
Professor C: How m by how much ?
PhD D: Um , it helps a lot over the ba the baseline but , mmm Professor C: By how much ?
PhD D: it It 's around three percent , um , relative .
Professor C: Worse .
PhD D: Yeah . Yeah . Um ,
Professor C: Hmm .
PhD D: mmm So , uh Professor C: It 's depending on how all this stuff comes out we may or may not be able to add any latency .
PhD D: Yeah , but Yeah . So , yeah , it depends . Uh , y actually , it 's it 's l it 's three percent . Right . Mmm . Yeah , b but I don't think we have to worry too much on that right now while you kno . Mm - hmm .
Professor C: Um , s Yeah , I mean , I think the only thing is that PhD D: So Professor C: I would worry about it a little .
PhD D: Mm - hmm .
Professor C: Because if we completely ignore latency , and then we discover that we really have to do something about it , we 're going to be find ourselves in a bind .
PhD D: Mm - hmm .
Professor C: So , um , you know , maybe you could make it twenty - five . You know what I mean ?
PhD D: Yeah .
Professor C: Yeah , just , you know , just be be a little conservative
PhD D: Oh yes .
Professor C: because we may end up with this crunch where all of a sudden we have to cut the latency in half or something .
PhD D: s Mm - hmm . Yeah .
Professor C: OK .
PhD D: Um . So , yeah , there are other things in the , um , algorithm that I didn't , uh , @ @ a lot yet ,
PhD A: Oh !
PhD D: which PhD A: Sorry . A quick question just about the latency thing . If if there 's another part of the system that causes a latency of a hundred milliseconds , is this an additive thing ? Or c or is yours hidden in that ?
PhD D: Mm - hmm .
PhD A: Uh PhD D: No , it 's it 's added .
PhD A: It 's additive . OK .
PhD D: Mm - hmm .
PhD B: We can OK . We can do something in parallel also , in some like some cases like , if you wanted to do voice activity detection .
PhD A: Uh - huh .
PhD B: And we can do that in parallel with some other filtering you can do .
PhD D: Mmm .
PhD B: So you can make a decision on that voice activity detection and then you decide whether you want to filter or not .
PhD D: Yeah .
PhD B: But by then you already have the sufficient samples to do the filtering .
PhD A: Mm - hmm .
PhD B: So So , sometimes you can do it anyway .
PhD A: I mean , couldn't , uh I Couldn't you just also I mean , i if you know that the l the largest latency in the system is two hundred milliseconds , don't you couldn't you just buffer up that number of frames and then everything uses that buffer ?
PhD B: Yeah .
PhD A: And that way it 's not additive ?
Professor C: Well , in fact , everything is sent over in buffers cuz of isn't it the TCP buffer some ?
PhD B: You mean , the the data , the super frame or something ?
PhD D: Mm - hmm .
Professor C: Yeah , yeah .
PhD D: Yeah .
PhD B: Yeah , but that has a variable latency because the last frame doesn't have any latency
PhD D: Mm - hmm .
PhD B: and first frame has a twenty framed latency . So you can't r rely on that latency all the time .
Professor C: Yeah .
PhD B: Because I mean the transmission over over the air interface is like a buffer .
PhD D: Yeah .
PhD B: Twenty frame PhD A: Yeah .
PhD B: twenty four frames .
PhD A: Yeah .
PhD B: So But the only thing is that the first frame in that twenty - four frame buffer has a twenty - four frame latency . And the last frame doesn't have any latency .
PhD A: Mm - hmm .
PhD B: Because it just goes as PhD A: Yeah , I wasn't thinking of that one in particular
PhD B: Yeah .
PhD A: but more of , you know , if if there is some part of your system that has to buffer twenty frames , uh , can't the other parts of the system draw out of that buffer and therefore not add to the latency ?
Professor C: Yeah . Yeah . And and that 's sort of one of the all of that sort of stuff is things that they 're debating in their standards committee .
PhD A: Oh ! Hmm .
PhD D: Mm - hmm . Yeah . So , um , there is uh , {comment} these parameters that I still have to to look at . Like , I played a little bit with this overestimation factor , uh , but I still have to to look more at this , um , at the level of noise I add after . Uh , I know that adding noise helped , um , the system just using spectral subtraction without smoothing , but I don't know right now if it 's still important or not , and if the level I choose before is still the right one . Same thing for the shape of the the noise . Maybe it would be better to add just white noise instead of speech shaped noise .
Professor C: That 'd be more like the JRASTA thing in a sense . Yeah .
PhD D: Mm - hmm . Um , yep . Uh , and another thing is to Yeah , for this I just use as noise estimate the mean , uh , spectrum of the first twenty frames of each utterance . I don't remember for this experiment what did you use for these two stage PhD B: I used ten just ten frames . Yeah , because PhD D: The ten frames ?
PhD B: I mean , the reason was like in TI - digits I don't have a lot . I had twenty frames most of the time .
PhD D: Mm - hmm . Um . But , so what 's this result you told me about , the fact that if you use more than ten frames you can improve by t
PhD B: Well , that 's that 's using the channel zero . If I use a channel zero VAD to estimate the noise .
PhD D: Oh , OK .
PhD B: Which PhD D: But this is ten frames plus plus
PhD B: Channel zero dropping .
PhD D: channel PhD B: Hmm .
PhD D: Uh , no , these results with two stage Wiener filtering is ten frames
PhD B: t Oh , this PhD D: but possibly more . I mean , if channel one VAD gives you PhD B: f Yeah . Mm - hmm . Yeah .
PhD D: Yeah . OK . Yeah , but in this experiment I did I didn't use any VAD . I just used the twenty first frame to estimate the noise . And So I expected it to be a little bit better , if , uh , I use more more frames . Um . OK , that 's it for spectral subtraction . The second thing I was working on is to , um , try to look at noise estimation , {comment} mmm , and using some technique that doesn't need voice activity detection . Um , and for this I u simply used some code that , uh , I had from from Belgium , which is technique that , um , takes a bunch of frame , um , and for each frequency bands of this frame , takes a look at the minima of the energy . And then average these minima and take this as an an energy estimate of the noise for this particular frequency band . And there is something more to this actually . What is done is that , uh , these minima are computed , um , based on , um , high resolution spectra . So , I compute an FFT based on the long , uh , signal frame which is sixty - four millisecond PhD A: So you have one minimum for each frequency ?
PhD D: What what I what I d uh , I do actually , is to take a bunch of to take a tile on the spectrogram and this tile is five hundred milliseconds long and two hundred hertz wide .
PhD A: Mmm .
PhD D: And this tile Uh , in this tile appears , like , the harmonics if you have a voiced sound , because it 's it 's the FTT bins . And when you take the m the minima of of these this tile , when you don't have speech , these minima will give you some noise level estimate , If you have voiced speech , these minima will still give you some noise estimate because the minima are between the harmonics . And If you have other other kind of speech sounds then it 's not the case , but if the time frame is long enough , uh , like s five hundred milliseconds seems to be long enough , {comment} you still have portions which , uh , are very close whi which minima are very close to the noise energy .
Professor C: I 'm confused . You said five hundred milliseconds
PhD D: Mmm ?
Professor C: but you said sixty - four milliseconds . Which is which ? What ?
PhD D: Sixty - four milliseconds is to compute the FFT , uh , bins .
Professor C: Yeah ,
PhD D: The the FFT .
Professor C: yeah .
PhD D: Um , actually it 's better to use sixty - four milliseconds because , um , if you use thirty milliseconds , then , uh , because of the this short windowing and at low pitch , uh , sounds , the harmonics are not , wha uh , correctly separated .
Professor C: Mm - hmm .
PhD D: So if you take these minima , it b they will overestimate the noise a lot .
Professor C: So you take sixty - four millisecond F F Ts and then you average them {comment} over five hundred ? Or ? Uh , what do you do over five hundred ?
PhD D: So I take to I take a bunch of these sixty - four millisecond frame to cover five hundred milliseconds ,
Professor C: Ah . OK .
PhD D: and then I look for the minima ,
PhD A: Mmm .
Professor C: I see .
PhD D: on the on on the bunch of uh fifty frames , right ?
Professor C: I see .
PhD D: Mmm . So the interest of this is that , as y with this technique you can estimate u some reasonable noise spectra with only five hundred milliseconds of of signal , so if the the n the noise varies a lot , uh , you can track better track the noise ,
Professor C: Mm - hmm .
PhD D: which is not the case if you rely on the voice activity detector . So even if there are no no speech pauses , you can track the noise level . The only requirement is that you must have , in these five hundred milliseconds segment , {comment} you must have voiced sound at least . Cuz this these will help you to to track the the noise level . Um . So what I did is just to simply replace the VAD - based , uh , noise estimate by this estimate , first on SpeechDat - Car Well , only on SpeechDat - Car actually . And it 's , uh , slightly worse , like one percent relative compared to the VAD - based estimates . Um , I think the reason why it 's not better , is that the SpeechDat - Car noises are all stationary . Um . So , u y y there really is no need to have something that 's adaptive
Professor C: Mm - hmm .
PhD D: and Uh , well , they are mainly stationary . Um . But , I expect s maybe some improvement on TI - digits because , nnn , in this case the noises are all sometimes very variable . Uh , so I have to test it . Mmm .
Professor C: But are you comparing with something e I 'm I 'm p s a little confused again , i it Uh , when you compare it with the V A D - based ,
PhD D: Mm - hmm .
Professor C: VAD - Is this is this the ?
PhD D: It 's It 's the France - Telecom - based spectra , s uh , Wiener filtering and VAD . So it 's their system but just I replace their noise estimate by this one .
Professor C: Oh , you 're not doing this with our system ?
PhD D: In i I 'm not No , no . Yeah , it 's our system but with just the Wiener filtering from their system . Right ? Mmm .
Professor C: OK .
PhD D: Yeah . Actually , th the best system that we still have is , uh , our system but with their noise compensation scheme , right ?
Professor C: Right . But PhD D: So I 'm trying to improve on this , and by by replacing their noise estimate by , uh , something that might be better .
Professor C: OK . But the spectral subtraction scheme that you reported on also re requires a a noise estimate .
PhD D: Yeah . Yeah .
Professor C: Couldn't you try this for that ?
PhD D: But I di
Professor C: Do you think it might help ?
PhD D: Not yet , because I did this in parallel ,
Professor C: I see ,
PhD D: and I was working on one and the other .
Professor C: I see . Yeah .
PhD D: Um ,
PhD B: Yeah .
PhD D: Yeah , for for sure I will . I can try also , mmm , the spectral subtraction .
PhD B: So I 'm also using that n new noise estimate technique on this Wiener filtering what I 'm trying .
Professor C: OK .
PhD B: So I I have , like , some experiments running , I don't have the results .
PhD D: Mm - hmm .
Professor C: Yeah .
PhD B: So .
Professor C: Yeah .
PhD B: I don't estimate the f noise on the ten frames but use his estimate .
Professor C: Yeah .
PhD D: Mm - hmm . Um . Yeah . I , um , also implemented a sp um spectral whitening idea which is in the , um , Ericsson proposal . Uh , the idea is just to um , flatten the log , uh , spectrum , um , and to flatten it more if the the probability of silence is higher . So in this way , you can also reduce somewhat reduce the musical noise and you reduce the variability if you have different noise shapes , because the the spectrum becomes more flat in the silence portions . Um . Yeah . With this , no improvement , uh , but there are a lot of parameters that we can play with and , um Actually , this this could be seen as a soft version of the frame dropping because , um , you could just put the threshold and say that " below the threshold , I will flatten comp completely flatten the the spectrum " . And above this threshold , uh , keep the same spectrum . So it would be like frame dropping , because during the silence portions which are below the threshold of voice activity probability , {comment} uh , w you would have some kind of dummy frame which is a perfectly flat spectrum . And this , uh , whitening is something that 's more soft because , um , you whiten you just , uh , have a function the whitening is a function of the speech probability , so it 's not a hard decision .
Professor C: Mm - hmm .
PhD D: Um , so I think maybe it can be used together with frame dropping and when we are not sure about if it 's speech or silence , well , maybe it has something do with this .
Professor C: It 's interesting . I mean , um , you know , in in JRASTA we were essentially adding in , uh , white uh , white noise dependent on our estimate of the noise .
PhD D: Mm - hmm .
Professor C: On the overall estimate of the noise . Uh , I think it never occurred to us to use a probability in there .
PhD D: Mm - hmm .
Professor C: You could imagine one that that that made use of where where the amount that you added in was , uh , a function of the probability of it being s speech or noise .
PhD D: Mm - hmm . Mm - hmm . Yeah , w Yeah , right now it 's a constant that just depending on the the noise spectrum .
PhD B: There 's Professor C: Yeah .
PhD D: Mm - hmm . Mm - hmm .
Professor C: Cuz that that brings in sort of powers of classifiers that we don't really have in , uh , this other estimate . So it could be it could be interesting .
PhD D: Mm - hmm . Mm - hmm .
Professor C: What what what point does the , uh , system stop recording ? How much PhD A: It 'll keep going till I guess when they run out of disk space ,
Professor C: It went a little long ? I mean , disk PhD A: but I think we 're OK .
PhD D: So .
Professor C: OK .
PhD D: Yeah . Uh Yeah , so there are with this technique there are some I just did something exactly the same as as the Ericsson proposal but , um , the probability of speech is not computed the same way . And I think , i for yeah , for a lot of things , actually a g a good speech probability is important . Like for frame dropping you improve , like you can improve from ten percent as Sunil showed , if you use the channel zero speech probabilities .
Professor C: Mm - hmm . Mm - hmm .
PhD D: For this it might help , um Professor C: Mm - hmm .
PhD D: S so , yeah . Uh , so yeah , the next thing I started to do is to , uh , try to develop a better voice activity detector . And , um I d um yeah , for this I think we can maybe try to train the neural network for voice activity detection on all the data that we have , including all the SpeechDat - Car data . Um And so I 'm starting to obtain alignments on these databases . Um , and the way I mi I do that is that I just use the HTK system but I train it only on the close - talking microphone . And then I aligned I obtained the Viterbi alignment of the training utterances . Um It seems to be , uh i Actually what I observed is that for Italian it doesn't seem Th - there seems to be a problem .
PhD B: No . So , it doesn't seems to help by their use of channel zero or channel one .
PhD D: Well . Because What ?
PhD B: Uh , you mean their d the frame dropping , right ? Yeah , it doesn't PhD D: Yeah . Yeah . So , u but actually the VAD was trained on Italian also ,
PhD B: Italian .
PhD D: so Um , the c the current VAD that we have was trained on , uh , t SPINE , right ?
PhD B: TI - digits .
PhD D: Italian , and TI - digits with noise and PhD B :
PhD D: Uh , yeah . And it seems to work on Italian but not on the Finnish and Spanish data . So , maybe one reason is that s s Finnish and Spanish noise are different . And actually we observed we listened to some of the utterances and sometimes for Finnish there is music in the recordings and strange things , right ?
PhD B: Yeah .
PhD D: Um Yeah , so the idea was to train all the databases and obtain an alignment to train on these databases , and , um , also to , um , try different kind of features , uh , as input to the VAD network . And we came up with a bunch of features that we want to try like , um , the spectral slope , the , um , the degree o degree of voicing with the features that , uh , we started to develop with Carmen , um , e with , uh , the correlation between bands and different kind of features ,
PhD B: Yeah . Mm - hmm .
PhD D: and Yeah .