Professor B: Good . Um , while we 're still on Aurora stuff maybe you can talk a little about the status with the , uh , Wall Street Journal things for it .
PhD A: So I 've , um , downloaded , uh , a couple of things from Mississippi State . Um , one is their software their , uh , LVCSR system . Downloaded the latest version of that . Got it compiled and everything . Um , downloaded the scripts . They wrote some scripts that sort of make it easy to run the system on the Wall Street Journal , uh , data . Um , so I haven't run the scripts yet . Uh , I 'm waiting there was one problem with part of it and I wrote a note to Joe asking him about it . So I 'm waiting to hear from him . But , um , I did print something out just to give you an idea about where the system is . Uh , they on their web site they , uh , did this little table of where their system performs relative to other systems that have done this this task . And , um , the Mississippi State system using a bigram grammar , uh , is at about eight point two percent . Other comparable systems from , uh  were getting from , uh , like six point nine , six point eight percent . So they 're Professor B: This is on clean test set ?
PhD A: This is on clean on clean stuff . Yeah . They they 've started a table where they 're showing their results on various different noise conditions but they they don't have a whole lot of it filled in and  and I didn't notice until after I 'd printed it out that , um , they don't say here what these different testing conditions are .
Professor B :
PhD A: You actually have to click on it on the web site to see them . So I I don't know what those numbers really mean .
Professor B: What kind of numbers are they getting on these on the test conditions ?
PhD A: Well , see , I was a little confused because on this table , I 'm the they 're showing word error rate . But on this one , I I don't know if these are word error rates because they 're really big . So , under condition one here it 's ten percent . Then under three it goes to sixty - four point six percent .
Professor B: Yeah , that 's probably Aurora .
PhD A: Yeah .
Professor B: I mean PhD A: So m I guess maybe they 're error rates but they 're , uh they 're really high .
Professor B: I I I don't find that surpri
PhD A: So Professor B: I mean , we W what 's what 's some of the lower error rates on on on uh , some of the higher error rates on , uh , some of these w uh , uh , highly mismatched difficult conditions ? What 's a ?
PhD D: Uh . Yeah , it 's around fifteen to twenty percent .
PhD A: Correct ?
PhD D: And the baseline , eh PhD A: Accuracy ?
PhD D: Uh , error rate .
Professor B: Yeah .
PhD D: Twenty percent error rate ,
Professor B: Yeah . So twenty percent error rate on digits .
PhD D: and PhD A: Oh , oh , on digits .
Professor B: So if you 're doing so if you 're doing ,
PhD D: and PhD A: Yeah .
PhD D: On digits .
PhD A: OK .
Professor B: you know ,
PhD D: And this is so so still the baseline .
Professor B: sixty - thousand PhD D: Right ?
PhD A: Yeah .
Professor B: Yeah , and if you 're saying sixty - thousand word recognition , getting sixty percent error on some of these noise condition not at all surprising .
PhD A: Yeah .
PhD D: The baseline is sixty percent also on digits ,
PhD A: Oh , is it ?
PhD D: on the m more mismatched conditions .
PhD A: OK .
Professor B: Yeah .
PhD D: So .
PhD A: So , yeah , that 's probably what it is then . Yeah . So they have a lot of different conditions that they 're gonna be filling out .
Professor B: It 's a bad sign when you looking at the numbers , you can't tell whether it 's accuracy or error rate .
PhD A: Yeah . Yeah . It 's it 's gonna be hard . Um , they 're I I 'm still waiting for them to release the , um , multi - CPU version of their scripts , cuz right now their script only handles processing on a single CPU , which will take a really long time to run . So . But their s
Professor B: This is for the training ?
PhD A: Uh I beli Yes , for the training also . And , um , they 're supposed to be coming out with it any time ,
Professor B: OK .
PhD A: the multi - CPU one . So , as soon as they get that , then I 'll I 'll grab those too
Professor B: OK .
PhD A: and so w
Professor B: Yeah . Cuz we have to get started ,
PhD A: Yeah .
Professor B: cuz it 's cuz , uh ,
PhD A: Yeah . I 'll go ahead and try to run it though with just the single CPU one ,
Professor B: if the PhD A: and I they they , um , released like a smaller data set that you can use that only takes like sixteen hours to train and stuff . So I can I can run it on that just to make sure that the  the thing works and everything .
Professor B: Oh ! Good . Yeah . Cuz we 'll Grad E: Hmm .
Professor B: I guess the actual evaluation will be in six weeks or something . So . Is that about right you think ?
PhD D: Uh , we don't know yet , I I think .
Professor B: Really , we don't know ?
PhD D: Uh - huh . Um .
PhD A: It wasn't on the conference call this morning ?
Professor B: Hmm .
PhD D: No .