Professor B: OK . Uh . Got anything to tell us ?
Grad C: Um . Well , I 've been reading some literature about clustering of data . Just , um , I guess , let me put it in context . OK , so we 're talking about discovering intermediate categories to , um to classify . And , uh , I was looking at some of the work that , uh , Sangita was doing on these TRAPS things . So she has , um she has temporal patterns for , um , a certain set of phonemes , from from TIMIT , right ? the most common phonemes . And each one of them has has a a nice pattern over time , a one one second window . And it has has these patterns . Um , so she has , um a TRAP for each one of the phonemes , um , times fifteen , for each of the fifteen critical bands . And , um , she does this agglomerative hierarchical clustering which which basically , um , is a clustering algorithm that , uh , starts with many , many , many different points many different clusters uh , corresponding to the number of data , uh , patterns that you have in the data . And then you have this distance mej metric which , uh , measures how how closely related they are . And you start , um by merging the patterns that are most closely related .
PhD E: And you create a tree .
Grad C: And y yeah , yeah , a dendrogram tree .
PhD E: Mm - hmm .
Grad C: Um .
PhD E: And then you can pick , uh , values anywhere along that tree to fix your set of clusters .
Grad C: Right , usually it 's when , um when the sol similarity measures , um , don't go down as much .
PhD E: Mm - hmm .
Grad C: And so , uh so you stop at that point . And what she found was , sh um , was there were five broad , um broad categories , uh , corresponding to , uh , things like , uh , fricatives and , uh , vocalic , um , and , uh , stops .
Professor B: Mm - hmm .
Grad C: And , uh , one for silence and and another one for schwa schwa sounds . Um , and , um , I was thinking about ways to to generalize this because w you 're it 's sort of like a it 's not a completely automatic way of clustering , because yo beforehand you have these these TRAPS and you 're saying that that these frames correspond to this particular phoneme . Um , and that 's that 's constraining your your clustering to to the set of phonemes that you already have . Um , whereas maybe we want to just take take a look at , um , arbitrary windows in time , um , of varying length , um , and cluster those .
PhD E: Mm - hmm .
Grad C: And I 'm thinking if we if we do that , then we would probably , um , at some point in the clustering algorithm find that we 've clustered things like , OK , thi this is a transition , um , this is a relatively stable stable point .
PhD E: Mm - hmm .
Grad C: Um , and I 'm hoping to find other things of of similarity and maybe use these things as the intermediate , um intermediate categories that , uh , um , I 'll later classify .
PhD E: Mm - hmm .
Professor B: Are you looking at these in narrow bands ?
Grad C: Um , right . F um , I 'm Professor B: Cuz that 's what you 're gonna be using , right ?
Grad C: Yeah , yeah . I I haven't exactly figured out , um , the exact details for that but , uh , the the representation of the data that I was thinking of , was using , um , critical band , um , energies , um , over different lengths of time . So Yeah .
Professor B: Yeah , I mean , it seems somehow that needs th uh , there 's a couple things that I wonder about with this . I mean , so one is is , again , looking at the same representation ,
Grad C: OK .
Professor B: I mean , if you 're going for this sort of thing where you have uh , little detectors that are looking at narrow bands , then what you 're going to be looking for should be some category that you can find with the narrow bands .
Grad C: Mm - hmm .
Professor B: That that seems to be kind of fundamental to it . Um , and then the other thing , uh , is that I wonder about with it , and and don't take this in the wrong way , like I I know what I 'm doing or anything ,
Grad C: Right .
Professor B: but , I mean . Um , just wondering really .
Grad C: Mm - hmm .
Professor B: Um , the sort of standard answer about this sort of thing is that if you 're trying to find the right system in some sense , whether you 're trying by categories or or parameters um , and your goal is discrimination , then having choices based on discrimination as opposed to , um , unsupervised nearness of things , um , is actually better .
Grad C: Hmm .
Professor B: Um , and I don't know if that I mean , since you 're dealing with issues of robustness , you know , maybe maybe this isn't right , but it 'd be something I 'd be concerned about . Because , for instance , you can imagine , uh , uh , i i if you remember from from , uh from your your quals , John Ohala saying that , uh , " buh " {comment} and " puh " {comment} differed , uh , not really cuz of voicing but because of aspiration . I mean , in as far as wha what 's really there in the acoustics .
Grad C: Mm - hmm .
Professor B: So , um , if you looked if you were doing some coarse clustering , you probably would put those two sounds together . And yet , I would gue I would guess that many of your recognition errors were coming from , uh , um , pfft , {comment} screwing up on this distinction .
Grad C: Mm - hmm .
Professor B: So , in fact , it 's a little hard because recognizers , to first order , sort of work . And the reasons we 're doing the things we 're doing is because they don't work as well as we 'd like . And since they sort of work , uh , it means that they are already doing if you go and take any recognizer that 's already out there and you say , " how well is it distinguishing between schwas and stops ? "
Grad C: Mm - hmm .
Professor B: Boy , I bet they 're all doing nearly perfectly on this , right ?
Grad C: Mm - hmm .
Professor B: So these these big categories that differ in huge obvious ways , we already know how to do . So , what are we bringing to the party ? I mean , in fact what we wanna do is have something that , particularly in the presence of noise , uh , is better at distinguishing between , uh , categories that are actually close to one another , and hence , would probably be clustered together .
Grad C: Mmm .
Professor B: So that 's th that 's the hard thing . I mean , I understand that there 's this other constraint that you 're considering , is that you wanna have categories that , uh that would be straightforward for , say , a human being to mark if you had manual annotation . And it 's something that you really think you can pick up . But I think it 's also essential that you wanna look at what are the confusions that you 're making and how can you come up with , uh , categories that , uh , can clarify these confusions .
Grad C: Mm - hmm . Hmm .
Professor B: So , I mean , the standard sort of way of doing that is take a look at the algorithms you 're looking at , but then throw in some discriminative aspect to it .  Y y this is more like , you know , how does LDA differ from PCA ? I mean , they 're the same sort of thing . They 're both orthogonalizing .
Grad C: Right .
Professor B: But , you know and and , um , this is a little harder because you 're not just trying to find parameters . You 're actually trying to find the the the the categories themselves . Uh , so a little more like brain surgery , I think on yourself . Uh . So , uh
Grad C: Yeah .
Professor B: Um , anyway . That 's my thought .
Grad C: OK .
Professor B: You 've been thinking about this problem for a long time actually . I mean , well W actually , you stopped thinking about it for a long time , but you used to think about it a lot .
PhD E: Yeah .
Professor B: And you 've been thinking about it more now ,
PhD D: Yeah .
Professor B: these categories .
PhD E: Yeah .
Professor B: Mm - hmm .
PhD E: I guess I don't I don't um , it 's not clear to me how to reconcile , you know , what you 're saying , which I think is right , with the way I 've been looking at it . That it 's it 's it 's all not very clear to me . But it seems to me that the desire the desirable feature to have is something that , um , is bottom - up . You know , however we do that .
Professor B: Mm - hmm .
PhD E: And and so I guess what I don't understand is how to do that and still be discriminative , because to be discriminative you have to have categories and the only categories that we know of to use are sort of these human human sig significant categories that are significant to humans , like phonemes , things like that .
Professor B: Right .
PhD E: But that 's sort of what you want to avoid . And so that feels I don't know how to get out of this .
Professor B: Well , here 's a here 's a , uh , uh Here 's a generic and possibly useless thought , which is , um , what do you really I mean , in a sense the only s s systems that make sense , uh , are ones that that have something from top - down in th in them . Right ? Because if e even the smallest organism that 's trying to learn to do anything , if it doesn't have any kind of reward for doing or penal penalty for doing anything , then it 's just going to behave randomly .
PhD E: Mm - hmm .
Professor B: So whether you 're talking about something being learned through evolution or being learned through experience , it 's gotta have something come down to it that gives its reward or , you know , at least some reinforcement learning ,
PhD E: Right .
Professor B: right ?
PhD E: So the question is , how far down ?
Professor B: And
PhD E: We could stop at words , but we don't , right ? We go all the way down to phonemes .
Professor B: Right , but I me I I think that maybe in some ways part of the difficulty is is trying to deal with the with these phonemes . You know , and and and i it 's almost like you want categories if if our if our , uh , um , metric of of goodness , uh , i if our PhD E: Mm - hmm .
Professor B: correction if our metric of badness is word error rate then , um , maybe we should be looking at words .
PhD E: Mm - hmm .
Professor B: I mean , for for for very nice , uh , reasons we 've looked for a while at syllables , and they have a lot of good properties , but i i i if you go all the way to words , I mean , that 's really I mean , d w In many applications you wanna go further . You wanna go to concepts or something , or have have have concepts , actions , this sort of thing .
PhD E: Yeah . But words would be a nice Professor B: But , words aren't bad , yeah . And and
PhD E: Yeah , so the common right , the common wisdom is you can't do words because there 's too many of them , right ? So you have to have some smaller set that you can use , uh , and and so everybody goes to phonemes . But the problem is that we we build models of words in terms of phonemes and these models are are really cartoon - ish , right ? So when you look at conversational speech , for example , you don't see the phonemes that you that you have in your word models .
Professor B: Yeah . But but but we 're not trying for models of words here . See , so her here 's maybe where If the issue is that we 're trying to come up with , um , some sort of intermediate categories which will then be useful for later stuff , uh , then maybe it doesn't matter that we can't have enough PhD E: Mm - hmm . Mm - hmm .
Professor B: I mean , what you wanna do is is build up these categories that are that are best for word recognition .
PhD E: Right . Right .
Professor B: And and somehow if that 's built into the loop of what the categories I mean , we do this every day in this very gross way of of running o a thousand experiments
PhD E: Right .
Professor B: because we have fast computers and picking the thing that has the best word error rate .
PhD E: Yeah .
Professor B: In some way I mean , we derive that all the time . In some ways it 's really not {comment} a bad bad thing to do because it tells you in fact how your adjustments at the very low level affect the the final goal .
PhD E: Mm - hmm . Mm - hmm .
Professor B: Um , so maybe there 's a way to even put that in in a much more automatic way ,
PhD E: Right .
Professor B: where you take , you know , something about the error at the level of the word or some other it could be syllable but in some large unit ,
PhD E: Uh - huh .
Professor B: uh , and uh yeah , you may not have word models , you have phone models , whatever , but you sort of don't worry about that , and just somehow feed it back through .
PhD E: Mm - hmm .
Professor B: You know , so that 's , uh , wh what I called a useless comments because I 'm not really telling you how to do it . But I mean , it 's a  it 's it 's , you know it
PhD E: No , but I think the important part in there is that , you know , if you want to be discriminative , you have to have uh , you know , categories .
Professor B: Right .
PhD E: And I think this the important categories are the words , and not the phones .
Professor B: Yeah . Yeah .
PhD E: Maybe . And so Right . If you can put the words in to the loop somehow for determining goodness of your sets of clusters Uh Professor B: Now , that being said , I think that that if you have something that is , um i Once you start dealing with spontaneous speech , all the things you 're saying are are really true .
PhD E: Mm - hmm .
Professor B: If you have read speech that 's been manually annotated , like TIMIT , then , you know , i i you the phones are gonna be right , actually , for the most part .
PhD E: Yeah . Yeah ,
Professor B: So so , uh , it doesn't really hurt them to to do that , to put in discrimination at that level .
PhD E: yeah .
Professor B: Um , if you go to spontaneous speech then it 's it 's trickier and and and , uh , the phones are uh , you know , it 's gonna be based on bad pronunciation models that you have of PhD E :
Professor B: and , um And it won't allow for the overlapping phenomenon
PhD E: Mmm . So it 's almost like there 's this mechanism that we have that , you know , when when we 're hearing read speech and all the phonemes are there you know , we we deal with that , but but when we go to conversational , and then all of a sudden not all the phonemes are there , it doesn't really matter that much to us as humans because we have some kind of mechanism that allows for these word models , whatever those models are , to be munged , you know , and and it doesn't really hurt , and I 'm not sure how  how to build that in . Uh .
Professor B: Yeah , I mean , I guess the other thing i is is to think of a little bit I mean , we when y when you start looking at these kind of results I think it usually is is pretty intuitive , but start looking at um , what are the kinds of confusions that you do make , uh , you know , between words if you want or or or , uh , even phones in in in in read speech , say , uh , when there is noise . You know , so is it more across place or more across manner ? Or is it cor you know , is it ?
Grad C: Mm - hmm .
Professor B: I mean , I know one thing that happens is that you you you , uh , you lose the , um , uh , low energy phones . I mean , if there 's added noise then low energy phones sometimes don't get heard . And if that if that is if it uh , if that turns it into another word or or different you know , or another pair of words or something , then it 's more likely to happen . But , um , I don't know , I w I would I would guess that you 'd Grad C: Mm - hmm .
Professor B: W I don't know . Anyway , that 's PhD E: I think part of the difficulty is that a l a lot of the robustness that we have is probably coming from a much higher level .
Professor B: Mm - hmm .
PhD E: You know , we understand the context of the situation when we 're having a conversation . And so if there 's noise in there , you know , our brain fills in and imagines what what should be there .
Professor B: Well that Grad C: Yeah . We 're we 're doing some sort of prediction of what PhD E: Yeah , exactly .
Professor B: Oh , sure , that 's really big .
Grad C: Yeah .
Professor B: Uh , but I mean , even if you do um , uh , diagnostic rhyme test kind of things , you know , where there really isn't an any information like that , uh , people are still better in noise than they than they are in in , uh uh , than the machines are .
PhD E: Hmm .
Professor B: So , I mean , that 's i Right . We can't we can't get it at all without any language models . Language models are there and important but but , uh Uh . If we 're not working on that then we should work on something else and improve it , but especially if it looks like the potential is there . So Should we do some digits ?
PhD E: Yeah .
Professor B: Since we 're here ?
PhD E: Go ahead , Morgan .
Professor B: OK .
PhD E: OK .
Professor B: That 's all folks .