jhclark/multeval — Issue #8: Comparing two systems with one hypothesis set each
Author: papajohn
Created: 2011-12-30 19:23:42+00:00
State: open

I would expect the following command line to compare two systems based on only one output each, but instead it crashes:

multeval-0.4$ ./multeval.sh eval --refs example/refs.test2010.lc.tok.en.0 --hyps-baseline example/hyps.lc.tok.en.baseline.opt0 --hyps-sys1 example/hyps.lc.tok.en.sys1.opt0 --metrics bleu
Found existing METEOR installation at ./lib/meteor-1.3
Loading metric: bleu
Found library jBLEU at file:/home/denero/third_party/multeval-0.4/multeval-0.4.jar
Using 6 threads
Reading Hypotheses for system baseline opt run 1 file /home/denero/third_party/multeval-0.4/example/hyps.lc.tok.en.baseline.opt0
Reading Hypotheses for system 1 opt run 1 file /home/denero/third_party/multeval-0.4/example/hyps.lc.tok.en.sys1.opt0
Reading non-laced references file /home/denero/third_party/multeval-0.4/example/refs.test2010.lc.tok.en.0
Collecting sufficient statistics for metric: BLEU
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:571)
at java.util.ArrayList.get(ArrayList.java:349)
at multeval.SuffStatManager.saveStats(SuffStatManager.java:50)
at multeval.MultEvalModule$2.doWork(MultEvalModule.java:287)
at multeval.MultEvalModule$2.doWork(MultEvalModule.java:1)
at multeval.parallel.HypothesisLevelMetricWorkerPool$1.run(HypothesisLevelMetricWorkerPool.java:37)
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:571)
at java.util.ArrayList.get(ArrayList.java:349)
at multeval.SuffStatManager.saveStats(SuffStatManager.java:50)
at multeval.MultEvalModule$2.doWork(MultEvalModule.java:287)
at multeval.MultEvalModule$2.doWork(MultEvalModule.java:1)
at multeval.parallel.HypothesisLevelMetricWorkerPool$1.run(HypothesisLevelMetricWorkerPool.java:37)
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:571)
at java.util.ArrayList.get(ArrayList.java:349)
at multeval.SuffStatManager.saveStats(SuffStatManager.java:50)
at multeval.MultEvalModule$2.doWork(MultEvalModule.java:287)
at multeval.MultEvalModule$2.doWork(MultEvalModule.java:1)
at multeval.parallel.HypothesisLevelMetricWorkerPool$1.run(HypothesisLevelMetricWorkerPool.java:37)
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1

Thanks for the help!