lspecia/QualityEstimation — Issue #2: About DeltaAvg and ties
Author: masterpiga
Created: 2012-02-09 10:51:15+00:00
State: open

Hi all,

I am running the evaluation script by passing it the same file twice, i.e. the ranks and scores are identical across the two files. I would expect the DeltaAvg score to be equal to 1. Instead, I get a value of 0.83. It should be noted that there is no quantile for which the DeltaAvg score equals 1, and actually for q=2 I get: AvgDelta[2]: 0.71 which seems a little bit low considering that the oracle and test files are the same. Is this an expected behavior?

The file that I am using for the evaluation is posted here: https://gist.github.com/1779210

And this is the output that I get:

oracle	:: Ranking: (primary) DeltaAvg = 0.83	(secondary) Spearman-Corr = 1.00
oracle	:: Scoring: (primary) MeanAvgErr = 0.00	(secondary) RootMeanSqrErr = 0.00	LargeErrPerc = 0.0	SmallErrPerc = 100.0

Concerning ties, the instructions say that "no ties are allowed", for simplicity. But I would like to point out the actual point labels are heavily quantized (there are 23 distinct scores in the development set). As a result, the way in which the quality scores are output AND the way in which ties are resolved can both have a major effect on the final evaluation. For example, a system that outputs quality scores with more than one decimal digit can be heavily penalized. I was wondering whether it wouldn't be appropriate to agree on common strategies to format the scores and to sort the segments that are in a tie. Or, as an alternative, decide to allow ties instead.

Thanks, cheers

Daniele