B: because you 're more experienced with running the large vocabulary stuff .
E: OK .
D: So I 'll point you to the web site and the mails corresponding .
B: S
D: So I
E: And it but it 's not ready yet , the system ?
D: Uh , I I think they are still , uh , tuning something on that .
D: So they 're like , d they 're varying different parameters like the insertion penalty and other stuff , and then seeing what 's the performance .
E: Are those going to be parameters that are frozen , nobody can change ?
E: Or ?
D: Uh , w I guess there is , uh , time during which people are gonna make suggestions .
E: Oh , but everybody 's gonna have to use the same values .
D: After that .
E: Oh !
E: Interesting .
E: OK .
D: Yeah , I guess .
D: So these sugges these this , uh , period during which people are gonna make suggestions is to know whether it is actually biased towards any set of features or B: Yeah , so I th th certainly the thing that I would want to know about is whether we get really hurt , uh , on in insertion penalty , language model , scaling , sorts of things .
E: Using our features .
B: Yeah ,
E: Yeah .
B: yeah .
B: Uh , in which case , um , H Hari or Hynek will need to , you know , push the case more about about this .
E: Mm - hmm .
B: Um .
E: And we may be able to revisit this idea about , you know , somehow modifying our features to work with B: Yes . In this case ,
B: that 's right .
E: Yeah .
B: That 's right .
B: Um , some of that may be , uh , a last minute rush thing because if the if our features are changing E: Yeah .
B: Uh .
B: Uh . But , um .
B: Yeah , the other thing is that even though it 's months away , uh , it 's starting to seem to me now like November fifteenth is right around the corner .
B: And , um , if they haven't decided things like this , like what the parameters are gonna be for this , uh , when " deciding " is not just somebody deciding . I mean , in fact there should be some understanding behind the , uh , deciding , which means some experiments and and so forth . It it it seems pretty tight to me .
E: So wha what 's the significance of November fifteenth ?
B: That 's when the evaluation is .
E: OK .
B: Yeah .
B: So , yeah , so
B: after But , you know , they may even decide in the end to push it off .
B: It wouldn't , you know , entirely surprise me .
B: But , uh , due to other reasons , like some people are going away , I 'm I 'm hoping it 's not pushed off for a l a long while .
B: That would be , uh put us in an awkward position .
B: But B: Anyway .
E: OK .
B: Great .
B: Yeah , I think that 'll be helpful .
B: There 's there 's not anybody OGI currently who 's who 's , uh , working with this
B: and and
E: Is is this part of the evaluation just a small part ,
E: or ho how important is this to the overall ?
B: I I think it 's it 's , um it depends how badly you do .
B: I mean , I think that it it is Uh .
D: b
E: This is one of those things that will be debated afterwards ?
B: Yeah .
B: Well , I mean , it 's it 's Conceptually , it my impression , again , you guys correct me if I 'm wrong , but my impression is that , um , they want it as a double check .
B: That you haven't come across you haven't invented features which are actually gonna do badly for a a significantly different task , particularly one with larger vocabulary .
E: Mmm .
B: And , um , but it 's not the main emphasis .
B: I mean , the truth is , most of the applications they 're looking at are pretty small vocabulary .
E: Mmm .
B: So it 's it 's a double check .
B: So they 'll probably assign it some sort of low weight .
E: Seems to me that if it 's a double check , they should give you a one or a zero .
E: Y you passed the threshold or you didn't pass the threshold ,
E: and they shouldn't even care about what the score is .
B: Yeah .
B: But , I mean , we 'll we 'll we 'll see what they come up with .
E: Yeah .
B: Uh , but in in the current thing , for instance , where you have this well - matched , moderately - matched , and and mis highly - mismatched , uh , the emphasis is somewhat on the on the well - matched , but it 's only a a marginal ,
B: right ?
B: It 's like forty , thirty - five , twenty - five , or something like that .
D: Yeah .
B: So you still if you were way , way off on the highly - mismatched , it would have a big effect .
E: Mm - hmm .
B: And , um , it wouldn't surprise me if they did something like that with this .
B: So again , if you 're if you get If it doesn't help you much , uh , for noisy versions of this of large vocabulary data , then , uh , you know , it may not hurt you that much .