moses-smt/mosesdecoder — pull request #6 by Thomas1205 (2011-12-09 14:56:04+00:00)
Title: Duplicate Removal in MERT

I added duplicate removal to mert. The changes are additions to the code, nothing was changed. Duplicate removal could easily be made optional.

In the later iterations, I do get some speed-ups:

This is how the git-head (pulled a few days before) behaves (Europarl It->En, 500K training sentences, 750 dev sentences, phrase-based MOSES):

starting MERT
Data loaded : [1] seconds
Stopping... : [35] seconds
Data loaded : [1] seconds
Stopping... : [65] seconds
Data loaded : [2] seconds
Stopping... : [102] seconds
Data loaded : [3] seconds
Stopping... : [146] seconds
Data loaded : [3] seconds
Stopping... : [229] seconds
Data loaded : [3] seconds
Stopping... : [298] seconds
Data loaded : [5] seconds
Stopping... : [393] seconds
Data loaded : [4] seconds
Stopping... : [387] seconds
Data loaded : [5] seconds
Stopping... : [394] seconds
Data loaded : [5] seconds
Stopping... : [499] seconds
Data loaded : [6] seconds
Stopping... : [486] seconds
Data loaded : [7] seconds
Stopping... : [541] seconds
Data loaded : [7] seconds
Stopping... : [577] seconds
Data loaded : [8] seconds
Stopping... : [630] seconds
Data loaded : [8] seconds
Stopping... : [655] seconds
Data loaded : [9] seconds
Stopping... : [698] seconds

And here's how my implementation behaves (same task, same system, similar workload):

starting MERT
Data loaded : [1] seconds
Stopping... : [36] seconds
Data loaded : [1] seconds
Stopping... : [65] seconds
Data loaded : [2] seconds
Stopping... : [107] seconds
Data loaded : [3] seconds
Stopping... : [142] seconds
Data loaded : [3] seconds
Stopping... : [222] seconds
Data loaded : [4] seconds
Stopping... : [286] seconds
Data loaded : [4] seconds
Stopping... : [362] seconds
Data loaded : [5] seconds
Stopping... : [363] seconds
Data loaded : [5] seconds
Stopping... : [382] seconds
Data loaded : [6] seconds
Stopping... : [436] seconds
Data loaded : [7] seconds
Stopping... : [413] seconds
Data loaded : [8] seconds
Stopping... : [446] seconds
Data loaded : [8] seconds
Stopping... : [472] seconds
Data loaded : [8] seconds
Stopping... : [496] seconds
Data loaded : [9] seconds
Stopping... : [495] seconds
Data loaded : [9] seconds
Stopping... : [648] seconds
Data loaded : [10] seconds
Stopping... : [485] seconds