Repository: awslabs/sockeye

Issue:
Misalignment of translations
It was not a problem before, but I recently find that there is misalignment (of lines) between the input and the translation output when chunk-size > 1. For example,
- chunk-size=1024, batch-size=16 (misaligned)
- chunk-size=1024, batch-size=1 (misaligned)
- chunk-size=1, batch-size=1 (aligned)

It doesn't always happen, so I haven't found out how it's triggered. I observe this situation when translating large documents (which usually contain empty lines).


Discussion:
Can you share a dataset where we could reproduce the issue?
thanks for reporting! We will look into this. In the mean time let us know if you have any additional information. For example:
* How large is the test set?
* Did you feed inputs and outputs via stdin/stdout or through files?
or if you had an example based on the [sequence copy](https://github.com/awslabs/sockeye/tree/master/tutorials/seqcopy) task that reproduces this issue it would also be great. 
It's a third-party dataset so I cannot share it directly.  The files causing this problem have around 1,300 lines, which is larger than the chunk-size=1024. 24% lines are blank.
I haven't got a chance to do a diagnostic toy test that could reproduces this issue.
Would you be able to reproduce and share an obfuscated version of your dataset with every word replaced by its vocabulary id (or maybe even just a fixed token)?
another relevant question: are you setting a specific maximum output length, or is your model imposing one (e.g. its using learned positional embeddings)? What I am getting at is whether the input sentences are split into multiple chunks or not (i.e. are there exactly 1300 requests, split in chunks of 1024, or more).
You can find a obfuscated sample [here](https://obj.umiacs.umd.edu/mt-data/input.00.id).
I set --max-seq-len=81:80 and I think there's sentences longer than 81 tokens in the sample. I used a RNN model and I didn't use learned positional embeddings.
Thanks. I was able to reproduce this, will debug more.
@artemsok, can you share the permutations? I wonder if it has to do with incorrect sorting of translated chunks. [Line 1170](https://github.com/awslabs/sockeye/blob/7629faead18f49eae503a2ffea9b6057b7297e98/sockeye/inference.py#L1170) of `inference.py` sorts `TranslatedChunk` objects. This is a NamedTuple with an `id` (for the segment) and `chunk_id` (for sub pieces of a split input sentence). But I don't see where a custom sort routine is provided that would actually sort these correctly (i.e., by id and then by chunk_id, both increasing).

Resolving patch:
diff --git a/sockeye/__init__.py b/sockeye/__init__.py
--- a/sockeye/__init__.py
+++ b/sockeye/__init__.py
@@ -11,4 +11,4 @@
 # express or implied. See the License for the specific language governing
 # permissions and limitations under the License.
 
-__version__ = '1.18.37'
+__version__ = '1.18.38'
diff --git a/sockeye/inference.py b/sockeye/inference.py
--- a/sockeye/inference.py
+++ b/sockeye/inference.py
@@ -1103,14 +1103,14 @@ def translate(self, trans_inputs: List[TranslatorInput]) -> List[TranslatorOutpu
 
         # split into chunks
         input_chunks = []  # type: List[TranslatorInput]
-        for input_idx, trans_input in enumerate(trans_inputs, 1):
+        for trans_input in trans_inputs:
             # bad input
             if isinstance(trans_input, BadTranslatorInput):
-                translated_chunks.append(TranslatedChunk(id=input_idx, chunk_id=0, translation=empty_translation()))
+                translated_chunks.append(TranslatedChunk(id=trans_input.sentence_id, chunk_id=0, translation=empty_translation()))
 
             # empty input
             elif len(trans_input.tokens) == 0:
-                translated_chunks.append(TranslatedChunk(id=input_idx, chunk_id=0, translation=empty_translation()))
+                translated_chunks.append(TranslatedChunk(id=trans_input.sentence_id, chunk_id=0, translation=empty_translation()))
             else:
                 # TODO(tdomhan): Remove branch without EOS with next major version bump, as future models will always be trained with source side EOS symbols
                 if self.source_with_eos:


Test patch: