PhD A: Um . Yeah . There was a conference call this Tuesday . Um . I don't know yet the  what happened Tuesday , but the points that they were supposed to discuss is still , uh , things like the weights , uh Professor B: Oh , this is a conference call for , uh , uh , Aurora participant sort of thing .
Grad E: For PhD A: Yeah . Yeah .
Professor B: I see .
PhD A: Mmm .
Professor B: Do you know who was who was since we weren't in on it , uh , do you know who was in from OGI ? Was  was was Hynek involved or was it Sunil
PhD A: I have no idea .
Professor B: or ?
PhD A: Mmm , I just Professor B: Oh , you don't know . OK .
PhD A: Yeah .
Professor B: Alright .
PhD A: Um , yeah . So the points were the the weights how to weight the different error rates that are obtained from different language and and conditions . Um , it 's not clear that they will keep the same kind of weighting . Right now it 's a weighting on on improvement .
Professor B: Mm - hmm .
PhD A: Some people are arguing that it would be better to have weights on uh well , to to combine error rates before computing improvement . Uh , and the fact is that for right now for the English , they have weights they they combine error rates , but for the other languages they combine improvement . So it 's not very consistent . Um Professor B: Mm - hmm .
PhD A: Yeah . The , um Yeah . And so Well , this is a point . And right now actually there is a thing also , uh , that happens with the current weight is that a very non - significant improvement on the well - matched case result in huge differences in  in the final number .
Professor B: Mm - hmm .
PhD A: And so , perhaps they will change the weights to PhD C: Hmm .
PhD A: Yeah .
PhD C: How should that be done ? I mean , it it seems like there 's a simple way PhD A: Mm - hmm .
PhD C: Uh , this seems like an obvious mistake or something .
Professor B: Well , I mean , the fact that it 's inconsistent is an obvious mistake .
PhD C: Th - they 're Professor B: But the but , um , the other thing PhD A: In
Professor B: I don't know I haven't thought it through , but one one would think that each It it 's like if you say what 's the what 's the best way to do an average , an arithmetic average or a geometric average ?
PhD C: Mm - hmm .
Professor B: It depends what you wanna show .
PhD A: Mm - hmm .
Professor B: Each each one is gonna have a different characteristic .
PhD A: Yeah .
Professor B: So PhD C: Well , it seems like they should do , like , the percentage improvement or something , rather than the absolute improvement .
PhD A: Tha - that 's what they do .
Professor B: Well , they are doing that .
PhD A: Yeah .
Professor B: No , that is relative . But the question is , do you average the relative improvements or do you average the error rates and take the relative improvement maybe of that ?
PhD A: Yeah . Yeah .
Professor B: And the thing is it 's not just a pure average because there are these weightings .
PhD C: Oh .
Professor B: It 's a weighted average . Um .
PhD A: Yeah . And so when you average the the relative improvement it tends to  to give a lot of of , um , importance to the well - matched case because the baseline is already very good and , um , i it 's PhD C: Why don't they not look at improvements but just look at your av your scores ? You know , figure out how to combine the scores
PhD A: Mm - hmm .
PhD C: with a weight or whatever , and then give you a score here 's your score . And then they can do the same thing for the baseline system and here 's its score . And then you can look at PhD A: Mm - hmm .