Repository: awslabs/sockeye

Issue:
Sampling nbest list position strangely correlated with quality
I am using Sockeye for producing an nbest list of samples. It appears that the first sample in the resulting nbest list always is the best-quality sample, and then the quality decreases steadily.

More precisely, I mean that if for each output JSON object, if the first string in `translations` is extracted, the corpus BLEU of the resulting set will be highest.

Sample command:

```
OMP_NUM_THREADS=1 python -m sockeye.translate \
                -i $input \
                -o $output \
                -m $model \
                --sample \
                --beam-size 100 \
                --nbest-size 100 \
                --seed $seed \
                --length-penalty-alpha 1.0 \
                --device-ids 0 \
                --batch-size $batch_size
```

If I plot position in nbest list against BLEU:

![Screenshot 2020-11-05 at 10 11 59](https://user-images.githubusercontent.com/12611348/98220969-59899b80-1f4f-11eb-8097-0c5f55acdf89.png)

I was not expecting this to be the case. I thought that the beam is filled with independent samples of comparable quality, perhaps with a large variance. Am I misunderstanding nbest lists of samples somehow?

Edit: Possible lead: I believe that the inference code sorts and normalizes (with `--length-penalty-alpha`) hypotheses when sampling. Is this intended?


Discussion:
I investigated this a bit more, here is a script to reproduce my setup exactly:

https://gist.github.com/bricksdont/1574a5d64136c2816e224abedb9ceb99

The script compares a)  producing 100 samples as an nbest list with b) producing them one-by-one with a different random seed for every call.

**Single samples with varying random seed**

![graph single](https://user-images.githubusercontent.com/12611348/98299923-c8e7a580-1fb8-11eb-8ec0-5909d53d4eca.png)

**Nbest list of samples**

![graph nbest](https://user-images.githubusercontent.com/12611348/98299935-ce44f000-1fb8-11eb-885a-7586997e0690.png)

The single samples are as I expected, and the quality fluctuates between 17 and 20 roughly. The nbest list on the other hand contains samples that lead to a BLEU as high as 25, but also as low as 5.
Hi @bricksdont,
great analysis! I actually was not aware that we are sorting when sampling, but I think in fact we do. Here: https://github.com/awslabs/sockeye/blob/master/sockeye/beam_search.py#L754

With regular beam search, each call to `TopK` creates sorted hypotheses, that is not the case when sampling, i.e. when using `SampleK`. BUT, we do sort the hypotheses right after the beam search loop.

We could skip this in case of sampling I guess, but I wonder if it's really a problem.
Side note: good to see that model score correlates so well with BLEU in your case :)
Hi Felix,

Yes, good to know that model scores work so well!

> We could skip this in case of sampling I guess, but I wonder if it's really a problem.

This was a hard-to-detect issue for me, since I am using the nbest list of samples for further processing. I would suggest that either samples are not sorted or that this is pointed out in the documentation here:

https://awslabs.github.io/sockeye/inference.html

Btw, shuffling the nbest list appears to solve this problem:

**After shuffling translations in nbest list**

![graph shuffled](https://user-images.githubusercontent.com/12611348/98344900-34ab2c00-2014-11eb-851e-aa8b90c378aa.png)

which suggests that sorting is in fact the underlying cause. Sample quality after shuffling is still a bit lower than for 100 independent seeds, but perhaps this can be attributed to randomness.
And just to be sure: `--length-penalty-alpha` does not apply to sampling, does it? I would expect that if sampling is enabled, there is no special treatment (such as dividing by number of tokens) for the accumulated score of a finished sample hypothesis.

**Edit**: I just ran some numbers. No, in an nbest list of samples, distribution of lengths is not affected at all by `--length-penalty-alpha`.
... And a related issue (I hope it's fine to just continue the thread here instead of opening new issues!):

Sampling of a single sample with `--beam-size 1` skips softmax, even though the intention is not to skip it if sampling is enabled.

- If `--sample` is used without an integer, `0` is the constant that is used for this argument (https://github.com/awslabs/sockeye/blob/master/sockeye/arguments.py#L1240)
- The beam search code skips softmax if `not sample`. The boolean value of 0 is False. (https://github.com/awslabs/sockeye/blob/master/sockeye/beam_search.py#L798)

I think that if softmax is skipped, this breaks code in `SampleK`.
Sampled hypotheses are scored like in regular beam search, so the Length Penalty is part of it, afaik.

Thanks for the investigation, do you want to submit a PR?
> Sampled hypotheses are scored like in regular beam search, so the Length Penalty is part of it, afaik.

I did not explain well what I was concerned about, sorry! I mean, length penalty should not affect the length of samples (no problem if it affects the scores).

PR: Sure, should I go for a) mentioning sample order in the docs or b) not sorting samples?
I don't think sorting the samples is a problem so an update to the documentation sounds good to me. Thank you!
Interesting that it is correlated so well!

Normalization happens here: https://github.com/awslabs/sockeye/blob/master/sockeye/beam_search.py#L740  and is applied only to finished hypothesis: https://github.com/awslabs/sockeye/blob/master/sockeye/beam_search.py#L344 .

Resolving patch:
diff --git a/sockeye/__init__.py b/sockeye/__init__.py
--- a/sockeye/__init__.py
+++ b/sockeye/__init__.py
@@ -11,4 +11,4 @@
 # express or implied. See the License for the specific language governing
 # permissions and limitations under the License.
 
-__version__ = '2.3.1'
+__version__ = '2.3.2'
diff --git a/sockeye/beam_search.py b/sockeye/beam_search.py
--- a/sockeye/beam_search.py
+++ b/sockeye/beam_search.py
@@ -795,7 +795,7 @@ def get_beam_search(models: List[SockeyeModel],
 
     inference = None  # type: Optional[_Inference]
     if len(models) == 1:
-        skip_softmax = beam_size == 1 and not output_scores and not sample
+        skip_softmax = beam_size == 1 and not output_scores and sample is None
         if skip_softmax:
             logger.info("Enabled skipping softmax for a single model and greedy decoding.")
         inference = _SingleModelInference(model=models[0],


Test patch: