Grad B: It 's really simple though . So this is the idea . Um . We could pursue , um , if we thought it 's it 's worth it but , uh , I think we we will agree on that , um , to come up with a with a sort of very , very first crude prototype , and do some implementation work , and do some some research , and some modeling . So the idea is if you want to go somewhere , um , and focus on that object down Oh , I can actually walk with this . This is nice . down here . That 's the Powder - Tower . Now , um , we found in our , uh , data and from experiments , that there 's three things you can do . Um , you can walk this way , and come really , really close to it . And touch it . But you cannot enter or do anything else . Unless you 're interested in rock climbing , it won't do you no good standing there . It 's just a dark alley . But you can touch it . If you want to actually go up or into the tower , you have to go this way , and then through some buildings and up some stairs and so forth . If you actually want to see the tower , and that 's what actually most people want to do , is just have a good look of it , take a picture for the family , {comment} you have to go this way , and go up here . And there you have a vre really view It exploded , the during the Thirty years - war . Really uh , interesting sight . And um , these uh these lines are , um , paths ,
Grad E: Mmm .
Grad B: or so That 's ab er , i the street network of our geographic information system . And you can tell that we deliberately cut out this part . Because otherwise we couldn't get our GIS system to take to lead people this way . It would always use the closest point to the object , and then the tourists would be faced , you know , in front of a wall , but it would do them absolutely no good . So , what we found interesting is , first of all , intentions differ . Maybe you want to enter a building . Maybe you want to see it , take a picture of it . Or maybe you actually want to come as close as possible to the building . For whatever reason that may be .
Grad E: What 's it what 's it made out of ?
Grad B: Um , r red limestone .
Grad E: So maybe you would wanna touch it .
Grad B: Yeah , maybe you would want to touch it . Um . Okay , I This , um These intentions , we w w we could , if we want to , call it the the Vista mode , where we just want to eh s get the overview or look at it , the Enter mode , and the , well , Tango mode . I always come up with with silly names . So this " Tango " means , literally translated , " to touch " . So But sometimes the the Tango mode is really relevant in the in the sense that , um , if you want to , uh If you don't have the intention of entering your building , but you know that something is really close to it , and you just want to approach it , or get to that building . Consider , for example , the Post Office in Chicago , a building so large that it has its own zip code . So the entrance could be miles away from the closest point . So sometimes it m m m makes sense maybe to d to distinguish there . So , um , I 've looked , uh , through twenty some Uh , I didn't look through all the data . um , and there there 's uh , a lot more different ways in people uh , the ways people phrase how to g get if they want to get to a certain place . And sometimes here it 's b it 's a little bit more obvious Um . Maybe I should go back a couple of steps and go through the Professor C: No , OK come in , sit down . If you grab yourself a microphone .
Grad B: You need to sign some stuff and read some digits .
Professor C: Well , you can sign afterwards .
Grad B: O or later .
Grad E: You have to al also have to read some digits .
Professor C: Afterwards .
Grad D: OK . {comment} OK . Afterwards is fine .
Grad B: They are uncomfortable . Mm - hmm .
Grad D: Really small ? OK . I see . OK .
Grad B: Yep .
Grad D: Thank you .
Grad B: OK , but that was our idea .
Professor C: And it it it it it also has to be switched on , Nance .
Grad B: Is I I think Grad E: No , that one 's already on , I thought he said .
Professor C: It 's on ? OK , good .
Grad D: OK . It 's on .
Grad E: Yeah .
Grad B: OK . That was the idea . Um , people , when they w when they want to go to a building , sometimes they just want to look at it . Sometimes they want to enter it . And sometimes they want to get really close to it . That 's something we found . It 's just a truism . And the places where you will lead them for these intentions are sometimes ex in incredibly different . I I gave an example where the point where you end up if you want to look at it is completely different from where if you want to enter it . So , this is sort of how people may , uh may phrase those requests to a a a mock - up system at least that 's the way they did it . And we get tons of of these " how do I get to " , " I want to go to " , but also , " give me directions to " , and " I would like to see " . And um , what we can sort of do , if we look closer a closer at the the data That was the wrong one . um , we can look at some factors that may make a difference . First of all , very important , and um , that I 've completely forgot that when we talked . This is of course a crucial factor , " what type of object is it ? " So , some buildings you just don't want to take pictures of . Or very rarely . But you usually want to enter them . Some objects are more picturesque , and you more f more highly photographed . Then of course the the actual phrases may give us some idea of what the person wants . Um . Sometimes I found in the Uh , looking at the data , in a superficial way , I found some s sort of modifiers that that m may also give us a hint , um , " I 'm trying to get to " Nuh ? " I need to get to " . Sort of hints to the fact that you 're not really sightseeing and and just f there for pleasure and so forth and so on . And this leads us straight to the context which also should be considered . That whatever it is you 're doing at the moment may also inter influence the interpretation of of a phrase . So , this is , uh , really uh , uh , uh My suggestion is really simple . We start with , um Now , Let me , uh , say one more thing . What we do know , is that the parser we use in the SmartKom system will never differentiate between any of these . So , basically all of these things will result in the same XML M - three - L structure . Sort of action " go " , and then an object .
Grad D: Mm - hmm .
Grad B: Yeah ? and a source . So it 's it 's it 's way too crude to d capture those differences in intentions . So , I thought , " Mmm ! Maybe for a deep understanding task , that 's a nice sort of playground or first little thing . " Where we can start it and n sort of look " OK , we need , we gonna get those M - three - L structures . The crude , undifferentiated parse . Interpreted input . We may need additional part of speech , or maybe just some information on the verb , and modifiers , auxiliaries . We 'll see . And I will try to to sort of come up with a list of factors that we need to get out of there , and maybe we want to get a g switch for the context . So this is not something which we can actually monitor , now , but just is something we can set . And then you can all imagine sort of a a constrained satisfaction program , depending on on what , um , comes out . We want to have an a structure resulting if we feed it through a belief - net or or something along those lines . We 'd get an inferred intention , we we produce a structure that differentiates between the Vista , the Enter , and the , um , Tango mode . Which I think we maybe want to ignore . But . That 's my idea . It 's up for discussion . We can change all of it , any bit of it . Throw it all away .
Grad F: Now @ @ this email that you sent , actually .
Professor C: What ?
Grad F: Now I remember the email .
Professor C: OK .
Grad E: Huh . Still , I have no recollection whatsoever of the email . I 'll have to go back and check .
Professor C: Not important . So , what is important is that we understand what the proposed task is . And , the the i uh , Robert and I talked about this some on Friday . And we think it 's well - formed . So we think it 's a well - formed , uh , starter task for this , uh , deeper understanding in the tourist domain .
Grad F: So , where exactly is the , uh , deeper understanding being done ? Like I mean , s is it before the Bayes - net ? Is it , uh Professor C: Well , it 's the it 's it 's always all of it . So , in general it 's always going to be , the answer is , everywhere . Uh , so the notion is that , uh , this isn't real deep . But it 's deep enough that you can distinguish between these th three quite different kinds of , uh , going to see some tourist thing . And , so that 's that 's the quote " deep " that we 're trying to get at . And , Robert 's point is that the current front - end doesn't give you any way to Not only doesn't it do it , but it also doesn't give you enough information to do it . It isn't like , if you just took what the front - end gives you , and used some clever inference algorithm on it , you would be able to figure out which of these is going on . So , uh , and this is Bu - I in general it 's gonna be true of any kind of deep understanding , there 's gonna be contextual things , there 're gonna be linguistic things , there 're gonna be discourse things , and they gotta be combined . And , my idea on how to combine them is with a belief - net , although it may turn out that t some totally different thing is gonna work better . Um , the idea would be that you , uh , take your You 're editing your slide ?
Grad B: Yeah . As i a sort of , as I get ideas , uh w uh .
Professor C: Oh .
Grad B: So , discourse I I I thought about that . Of course that needs to sort of go in there .
Professor C: Oh . I 'm sorry . OK . So . This is minutes taking minutes as we go , in his in his own way .
Grad B: Yep .
Professor C: Um , but the p the Anyway . So the thing is , i uh , d naively speaking , you 've you 've got a for this little task , a belief - net , which is going to have as output , the conditional pr probability of one of three things , that the person wants to uh , to View it , to Enter it , or to Tango with it . Um . So that the the output of the belief - net is pretty well formed . And , then the inputs are going to be these kinds of things . And , then the question is there are two questions is , uh , one , where do you get this i {comment} information from , and two , what 's the structure of the belief - net ? So what are the conditional probabilities of this , that , and the other , given these things ? And you probably need intermediate nodes . I we don't know what they are yet . So it may well be that , uh , for example , that , uh , knowing whether Oh , another thing you want is some information abou I think , about the time of day . Now , they may wanna call that part of context .
Grad B: Mm - hmm .
Professor C: But the time of day matters a lot . And , if things are obviously closed , then , you Grad B: People won't want to enter it .
Professor C: Pe - people don't wanna enter them . And , if it 's not obvious , you may want to actually uh , point out to people that it 's closed you know , what they 're g going to is closed and they don't have the option of entering it .
Grad B: s b
Professor C: So another thing that can come up , and will come up as soon as you get serious about this is , that another option of course is to have a more of a dialogue . So if someone says something you could ask them .
Grad E: Yeah .
Professor C: OK . And Now , one thing you could do is always ask them , but that 's boring . And it also w it also be a pain for the person using it . So one thing you could do is build a little system that , said , " whenever you got a question like that I 've got one of three answers . Ask them which one you want . " OK . But that 's , um , not what we 're gonna do .
Grad B: But maybe that 's a false state of the system , that it 's too close to call .
Professor C: Oh yeah . You want the you want the ability to a You want the ability to ask , but what you don't wanna do is onl build a system that always asks every time , and i That 's not getting at the scientific problem ,
Grad B: Mm - hmm .
Professor C: and it 's In general you 're you know , it 's gonna be much more complex than that . a This is purposely a really simple case .
Grad B: Yeah .
Professor C: So , uh Yeah .
Grad B: I have one more point to to Bhaskara 's question . Um , I think also the the the deep understanding part of it is is going to be in there to the extent that we um , want it in terms of our modeling . We can start , you know , basic from human beings , model that , its motions , going , walking , seeing , we can mem model all of that and then compose whatever inferences o we make out of these really conceptual primitives . That will be extremely deep in the in in in my understanding .
Professor C: Yeah . S so so the way that might come up , if you wanna Suppose you wanted to do that , you might say , " Um , as an intermediate step in your belief - net , is there a Source - Path - Goal schema involved ? " OK ? And if so , uh , is there a focus on the goal ? Or is there a focus on the path ? or something . And that could be , uh , one of the conditiona you know , th the In some piece of the belief - net , that could be the the appropriate thing to enter .
Grad F: So , where would we extract that information from ? From the M - three - L ?
Professor C: No . No . See , the M - three - L is not gonna give th What he was saying is , the M - three - L does not have any of that . All it has is some really crude stuff saying , " A person wants to go to a place . "
Grad F: Right .
Grad E: The M - three - L is the old SmartKom output ?
Professor C: Right . M - three well , M - three - L itself refers to Multimedia Mark - up Language .
Grad E: OK . It 's just a language . Right , yeah .
Professor C: So we have th w we we we have to have a better w way of referring to Grad B: The parser output ?
Professor C: Mm - hmm .
Grad B: " Analyzed speech " I think it 's what they call it ,
Professor C: Yeah . The Well , OK .
Grad B: really , oder Professor C: Yeah .
Grad B: o th No , actually , intention lattices is what we 're gonna get .
Professor C: Is - i but they c they call it intention lattice , but tha
Grad B: In - in a intention lattice k Hypothesis .
Professor C: Anyway .
Grad B: They call it intention hypotheses .
Professor C: Right . So , th they 're gonna give us some cr uh or We can assume that y you get this crude information . About intention , and that 's all they 're going to provide . And they don't give you the kind of object , they don't give you any discourse history , if you want to keep that you have to keep it somewhere else .
Grad B: Well , they keep it . We have to request it .
Professor C: Right .
Grad B: Nuh ? But it 's not in there .
Professor C: Well , they they kee they keep it by their lights .
Grad B: Hmm .
Professor C: It may it may or may not be what what we want .
Grad B: Yeah , or i
Professor C: Yeah .
Grad E: So , if someone says , " I wanna touch the side of the Powder - Tower " , that would basically , we need to pop up Tango mode and the and the directions ?
Professor C: If i if Yeah , if it got as simple as that , yeah .
Grad E: Yeah .
Professor C: But it wouldn't .
Grad E: OK . But that doesn't necessarily But we 'd have to infer a Source - Path - Goal to some degree for touching the side , right ?
Grad B: Well Uh , th the there is a p a point there if I understand you . Correct ? Um , because um , sometimes people just say things This you find very often . " Where is the city hall ? " And this do they don't wanna sh see it on a map , or they don't wanna know it 's five hundred yards away from you , or that it 's to the your north . They wanna go there . That 's what they say , is , " Where is it ? " . Where is that damn thing ?
Grad E: And the parser would output Grad B: Well , that 's a a question mark . sh A lot of parsers , um , just , uh That 's way beyond their scope , is of interpreting that . You know ? But um , still outcome w the outcome will be some form of structure , with the town hall and maybe saying it 's a WH focus on the town hall . But to interpret it ,
Grad D: Mm - hmm .
Grad B: you know ? somebody else has to do that job later .
Professor C: Yeah .
Grad E: I 'm just trying to figure out what the SmartKom system would output , depending on these things .
Grad B: Um , it will probably tell you how far away it is , at least that 's That 's even what Deep Map does . It tells you how far away it is , and and shows it to you on a map . Because i we can not differentiate , at the moment , between , you know , the intention of wanting to go there or the intention of just know wanting to know where where it is .
Grad D: People no might not be able to infer that either , right ? Like the fact Like , I could imagine if someone came up to me and asked , " Where 's the city hall ? " , I might say , g ar " Are you trying to get there ? " Because how I describe um , t its location uh , p probably depend on whether I think I should give them , you know , directions now , or say , you know , whatever , " It 's half a mile away " or something like that .
Grad B: Mm - hmm .  It 's a granularity factor ,
Professor C: Yeah .
Grad B: because where people ask you , " Where is New York ? " , you will tell them it 's on the East Coast .
Grad D: Uh - huh . Yeah . Exactly . Right . Right .
Grad B: Y y eh you won't tell them how to get there , ft you know , take that bus to the airport and blah - blah - blah .
Grad D: Yeah .
Grad B: But if it 's the post office , you will tell them how to get there .
Grad D: Right . Mm - hmm .
Grad B: So th They have done some interesting experiments on that in Hamburg as well .
Grad D: Right .
Grad B: So .
Grad D: Right .
Professor C: But i Go go back to the the uh , th
Grad B: So I w this is " onto " is is knowledge about buildings ,
Professor C: Yeah , that slide .
Grad B: their opening times , and then t coupled with time of day , um , this should You know .
Grad D: So that context was like , um , their presumed purpose context , i like business or travel , as well as the utterance context , like , " I 'm now standing at this place at this time " .
Professor C: Yeah , well I think we ought to d a As we have all along , d We we 've been distu distinguishing between situational context , which is what you have as context , and discourse context ,
Grad B: Mm - hmm .
Professor C: which you have as DH ,
Grad B: Nuh .
Professor C: I don't know what the H means .
Grad B: History . Discourse history . Yeah .
Professor C: OK . Whatever . So we can work out terminology later .
Grad B: Yep .
Professor C: So , they 're they 're quite distinct . I mean , you need them both , but they 're quite distinct . And , so what we were talking about doing , a a as a first shot , is not doing any of the linguistics . Except to find out what seems to be useful . So , the the the reason the belief - net is in blue , is the notion would be Uh , this may be a bad dis bad idea , but the idea is to take as a first goal , see if we could actually build a belief - net that would make this three way distinction uh , in a plausible way , given these We have all these transcripts and we 're able to , by hand , extract the features to put in the belief - net . Saying , " Aha ! here 're the things which , if you get them out of out of the language and discourse , and put them into the belief - net , it would tell you which of these three uh , intentions is most likely . " And if to actually do that , build it , um you know , run it y y run it on the data where you hand - transcribe the parameters . And see how that goes . If that goes well , then we can start worrying about how we would extract them . So where would you get this information ? And , expand it to to other things like this . But if we can't do that , then we 're in trouble . I mean th th i i if you can't do this task , um Grad B: We need a different , uh , engine . Machine , I mean .
Professor C: Uh , uh , yeah , or something . Well it i I if it if it 's the belief - nets , we we 'll switch to you know , logic or some terrible thing , but I don't think that 's gonna be the case . I think that , uh , if we can get the information , a belief - net is a perfectly good way of doing the inferential combination of it . The real issue is , do what are the factors involved in determining this ? And I don't know .
Grad B: Hmm . But , only w
Professor C: Hold on a s Hold on a second .
Grad B: Muh .
Professor C: So , I know . Uh , uh , is it clear what 's going on here ?
Grad F: Yep .
Grad D: Um , I missed the beginning , but , um I guess could you back to the slide , the previous one ? So , is it that it 's , um These are all factors that uh , a These are the ones that you said that we are going to ignore now ? or that we want to take into account ? You were saying n
Professor C: Take them into account . But but you don't worry about h
Grad D: Take the the linguistic factors too . Oh , how to extract these features .
Professor C: how to extract them . So , f let 's find out which ones we need first ,
Grad D: OK . Got it .
Professor C: and Grad D: OK . And and it 's clear from the data , um , like , sorta the correct answer in each case .
Professor C: No .
Grad D: But l
Grad B: No . But Grad D: OK .
Professor C: Let 's go back to th Let 's go back to the the the slide of data .
Grad D: That 's that 's the thing I 'm curious ab
Grad B: Um Grad D: Like do we know from the data wh which OK . So Grad B: Not from that data . But , um , since we are designing a a a an , compared to this , even bigger data collection effort , {comment} um , we will definitely take care to put it in there ,
Grad D: Mm - hmm . Mm - hmm .
Grad B: in some shape , way , form over the other ,
Grad D: Mm - hmm .
Professor C: Yeah .
Grad B: to see whether we can , then , get sort of empirically validated data .
Grad D: Right .
Grad B: Um , from this , we can sometimes , you know an and that 's that but that isn't that what we need for a belief - net anyhow ? is sort of s sometimes when people want to just see it , they phrase it more like this ? But it doesn't exclude anybody from phrasing it totally differently , even if they still Grad D: Mm - hmm . Right .
Grad B: you know ?
Grad D: Right .
Grad B: But then other factors may come into play that change the outcome of their belief - net . So , um , this is exactly what Grad D: Right .
Grad B: Because y you can never be sure . And I 'm sure even i the most , sort of , deliberate data collection experiment will never give you data that say , " Well , if it 's phrased like that , the intention is this . "
Grad D: Sure .
Grad B: You know , because then , uh , you Grad D: u u I mean , the only way you could get that is if you were to give th the x subjects a task . Right ? Where you have where your , uh , current goal is to Grad B: We Yeah ! That 's what we 're doing .
Grad D :
Grad B: But but we will still get the phrasing all over the place .
Grad D: So that 's what you want ? OK . So you will know .
Grad B: I 'm sure that , you know Professor C: Yeah .
Grad D: Mm - hmm . The No , that 's fine . I guess , it 's just knowing the intention from the experimental subject .
Professor C: Yeah .
Grad B: Mm - hmm .
Professor C: From that task , yeah . So , uh , I think you all know this , but we are going to actually use this little room
Grad D :
Professor C: and start recording subjects probably within a month or something . So , this is not any lo any of you guys ' worry , except that we may want to push that effort to get information we need . So our job is to figure out how to solve these problems . If it turns out that we need data of a certain sort , then the sort of data collection branch can be , uh , asked to do that . And one of the reasons why we 're recording the meeting for these guys is cuz we want their help when we d we start doing uh , recording of subjects . So , yeah y you 're absolutely right , though . No , you you will not have , and there it is , and , uh But you know , y y the , um Grad D: And I think the other concern that has come up before , too , is if it 's um I don't know if this was collected what situation this data was collected in . Was it is it the one that you showed in your talk ? Like people Grad B: No , no . No .
Grad D: But OK . So was this , like , someone actually mobile , like s using a device ?
Grad B: Uh , N no , no not i it was mobile but not not with a w a real wizard system . So there were never answers .
Grad D: Uh - huh . OK . OK . But , is it I guess I don't know The situation of of collecting th the data of , like Here you could imagine them being walking around the city . as like one situation . And then you have all sorts of other c situational context factors that would influence w how to interpret , like you said , the scope and things like that .
Grad B: Mm - hmm .
Grad D: If they 're doing it in a you know , " I 'm sitting here with a map and asking questions " , I I would imagine that the data would be really different . Um , so it 's just Grad B: Yeah . But It was never th th the goal of that data collection to to serve for sat for such a purpose . So that 's why for example the tasks were not differentiated by intentionality ,
Grad D: Mm - hmm .
Grad B: there was n there was no label ,
Grad D: Mm - hmm .
Grad B: you know , intention A , intention B , intention C .
Grad D: Right .
Grad B: Or task A , B , C . Um I 'm sure we can produce some if we need it , um , that that will help us along those lines .
Grad D: Mm - hmm .
Grad B: But , you know , you gotta leave something for other people to model . So , to Finding out what , you know , situational con what the contextual factors of the situation really are , you know is an interesting s interesting thing .
Grad D: Mm - hmm . Mm - hmm .
Grad B: u u Sort of I 'm , at the moment , curious and I 'm I 'm s w want to approach it from the end where we can s sort of start with this toy system that we can play around with ,
Grad D: Mm - hmm .
Grad B: so that we get a clearer notion of what input we need for that ,
Grad D: Mm - hmm .
Grad B: what suffices and what doesn't . And then we can start worrying about where to get this input , what what do we need , you know Ultimately once we are all experts in changing that parser , for example , maybe , there 's just a couple three things we need to do and then we get more whatever , part of speech and more construction - type - like stuff out of it .