Use Marian-MT to evaluate translated outputs by printing out per-word log-probility
Author: JunjieHu
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I am going to evaluate the Marian's opus-mt to evaluate the translated output y given the input x. 
I want to get the model's log-probability of y given x, i.e., P(y|x). 
I didn't find any usage example in the document [here](https://huggingface.co/transformers/model_doc/marian.html#multilingual-models).
@sshleifer do you know an example in this case? Thanks!

For example:
Source input (x):  I would like to run this experiment.
Translated output (y) in Chinese: 我     希望   跑        这个    实验   。
The word-level log probability:     -0.3  -0.4   -0.23  -0.43  -0.23 -0.8

BTW, fairseq supports this function in their [comment-line tools](https://fairseq.readthedocs.io/en/latest/command_line_tools.html). 



Comments:
Would love a contribution that implemented it. 
Can you paste a working fairseq command to try to emulate?
Alternatively you could just send a PR.

Hi @sshleifer 

Marian NMT seems to have such functionality as well. Please check their script [here](https://github.com/marian-nmt/marian-examples/blob/master/wmt2017-transformer/run-me.sh#L189).

I modify the [translation code](https://fairseq.readthedocs.io/en/latest/command_line_tools.html#fairseq-generate) in fairseq named as *validate_lm.py*, and run the bash script to get the per-word log-probability.
```bash
python validate_lm.py \
    $binarized_data_dir \
    --source-lang en --target-lang fr \
    --path $path_to_checkpoint \
    --task translation \
    --valid-subset train \
    --max-sentences 16 \
    --nll-file $out_file
```

validate_lm.py
```python
import logging
import sys

import torch

from fairseq import checkpoint_utils, distributed_utils, options, utils, tasks
from fairseq.logging import metrics, progress_bar
from fairseq.options import add_distributed_training_args
from fairseq.criterions.cross_entropy import CrossEntropyCriterion

logging.basicConfig(
    format='%(asctime)s | %(levelname)s | %(name)s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    level=logging.INFO,
    stream=sys.stdout,
)
logger = logging.getLogger('fairseq_cli.validate')


def main(args, override_args=None):
    utils.import_user_module(args)

    assert args.max_tokens is not None or args.max_sentences is not None, \
        'Must specify batch size either with --max-tokens or --max-sentences'

    use_fp16 = args.fp16
    use_cuda = torch.cuda.is_available() and not args.cpu

    if override_args is not None:
        if isinstance(override_args, dict):
            overrides = override_args
        else:
            overrides = vars(override_args)
        print('override_args')
        overrides.update(eval(getattr(override_args, 'model_overrides', '{}')))
    else:
        overrides = None

    # Load ensemble
    logger.info('loading model(s) from {}'.format(args.path))
    task = tasks.setup_task(args)
    models, _model_args = checkpoint_utils.load_model_ensemble(
        [args.path],
        arg_overrides=overrides,
        task=task,
    )
    model = models[0]

    # Move models to GPU
    for model in models:
        if use_fp16:
            model.half()
        if use_cuda:
            model.cuda()

    # Print args
    logger.info('args', args)
    logger.info('overrides', overrides)

    # Build criterion
    # criterion = task.build_criterion(args)
    criterion = CrossEntropyCriterion(task, False)
    criterion.eval()

    for subset in args.valid_subset.split(','):
        try:
            task.load_dataset(subset, combine=False, epoch=1)
            dataset = task.dataset(subset)
        except KeyError:
            raise Exception('Cannot find dataset: ' + subset)

        # Initialize data iterator
        itr = task.get_batch_iterator(
            dataset=dataset,
            max_tokens=args.max_tokens,
            max_sentences=args.max_sentences,
            max_positions=utils.resolve_max_positions(
                task.max_positions(),
                *[m.max_positions() for m in models],
            ),
            ignore_invalid_inputs=args.skip_invalid_size_inputs_valid_test,
            required_batch_size_multiple=args.required_batch_size_multiple,
            seed=args.seed,
            num_workers=args.num_workers,
        ).next_epoch_itr(shuffle=False)
        progress = progress_bar.progress_bar(
            itr,
            log_format=args.log_format,
            log_interval=args.log_interval,
            prefix=f"valid on '{subset}' subset",
            default_log_format=('tqdm' if not args.no_progress_bar else 'simple'),
        )

        fout = open(args.nll_file, 'w')
        nll = []
        for i, sample in enumerate(progress):
            sample = utils.move_to_cuda(sample) if use_cuda else sample
            model.eval()
            with torch.no_grad():
                loss, sample_size, log_output = criterion(model, sample, reduce=False)
                nsentences = log_output['nsentences']
                loss = loss.view(nsentences, -1).tolist()
                for j, sample_id in enumerate(sample['id'].tolist()):
                    loss_j = [ll for ll in loss[j] if ll > 0]
                    avg_nll = sum(loss_j) / len(loss_j)
                    lstr = '\t'.join([f'{ll:.4f}' for ll in loss_j])
                    fout.write(f'{sample_id}\t{avg_nll}\t{lstr}\n')
                    nll.append((sample_id, avg_nll, lstr))
        fout.close()

def cli_main():
    parser = options.get_validation_parser()
    add_distributed_training_args(parser)
    args = options.parse_args_and_arch(parser)

    # only override args that are explicitly given on the command line
    override_parser = options.get_validation_parser()
    group = override_parser.add_argument_group("Valid BW")
    group.add_argument('--nll-file', type=str, default=None)
    add_distributed_training_args(override_parser)
    override_args = options.parse_args_and_arch(override_parser, suppress_defaults=True)
    distributed_utils.call_main(args, main, override_args=override_args)


if __name__ == '__main__':
    cli_main()
```





I'd tinker with that - perhaps in a few days.

I started looking into this but to be able to reproduce any of these examples in order to replicate this in transformers I need to spend hours just to set things up :( It'd have been much easier if such requests came with ready data that we could use right away.

Currently waiting for very sloooooooow downloads of data from statmt.org - will try to setup fairseq for en-fr...

will keep you posted on the progress.

p.s. marianmt I couldn't even build on ubuntu-20.04, so hoping I could sort it out with fairseq

OK, I found an easy way to set up something that your script would run on:

```
mkdir -p data-bin
curl https://dl.fbaipublicfiles.com/fairseq/models/wmt14.v2.en-fr.fconv-py.tar.bz2 | tar xvjf - -C data-bin
curl https://dl.fbaipublicfiles.com/fairseq/data/wmt14.v2.en-fr.newstest2014.tar.bz2 | tar xvjf - -C data-bin
python validate_lm.py data-bin/wmt14.en-fr.joined-dict.newstest2014 --source-lang en --target-lang fr --path data-bin/wmt14.en-fr.joined-dict.transformer/model.pt --task translation --valid-subset test --max-tokens 128
```

(in the future requests please provide something similar so that the devs could quickly reproduce your example. And of course test that it works. Thank you)

Your script won't run as is, had to do some tweaks to it. But it doesn't matter for now.

What it produces is:
```
1596    0.7027008545895418      1.2075  1.4464  1.1499  0.1134  0.1958  0.1032
1615    3.4015886902809145      5.8378  9.5874  0.6482  0.7548  0.1797
2837    1.7323936596512794      0.8643  0.1775  7.3694  0.1477  0.1032
256     1.5152931759754817      0.8425  7.1255  0.8411  0.0599  0.1208  0.1020
556     2.3045735328147807      6.7926  0.0528  1.0420  2.4513  3.3704  0.1184
2922    1.7860032822936773      0.4388  3.8867  1.9373  0.2531  6.5018  0.1011  1.0688  0.1005
612     1.598365530371666       3.2449  2.8535  0.1290  0.1661
605     1.7411062990625699      4.9944  0.1142  0.1147
1481    0.4938086899263518      0.3987  0.5716  1.2258  0.5801  0.4607  0.1157  0.1041
2013    0.7291228858133157      2.0326  0.1758  0.1322  1.7975  0.1314  0.1051
75      1.2818062901496887      0.6200  0.2217  0.0730  0.1697  7.2470  0.4181  0.2230
279     1.4058668922100748      8.1397  1.1370  0.1082  0.1427  0.0871  0.1210  0.1053
2641    0.1703926378062793      0.3318  0.1146  0.2372  0.1261  0.1727  0.1082  0.1022
1120    2.3883045655157833      2.9336  0.8035  1.6985  4.0525  8.6964  3.0033  0.0963  0.1081  0.1024
1031    0.6803936168551445      4.6826  0.7309  0.0864  0.1111  0.1247  0.0863  0.5777  0.0856  0.2151  0.1035
2484    0.17211276292800903     0.3391  0.2407  0.0678  0.4290  0.1203  0.0990  0.1382  0.1095  0.0758  0.1018
2600    1.3488108797797136      4.0854  0.1176  1.0209  0.1188  0.0373  3.7863  0.2754
2814    1.5876335703900881      1.6473  2.3662  0.2776  0.0731  4.8990  1.7478  0.1024
2822    0.6652300975152424      0.6418  0.1277  0.6620  2.6037  0.4055  0.1121  0.1038
2854    0.396458768418857       0.5300  0.1097  1.1606  0.4882  0.2649  0.1183  0.1035
169     1.5903700785711408      4.9675  0.0435  0.1813  1.8350  0.1336  5.2166  0.2192  0.1262
234     2.0860743997618556      2.8854  0.2894  0.3069  0.1739  0.0880  12.6388 0.1717  0.1345
368     0.9114988259971142      0.7726  0.7938  1.8604  3.4764  0.0779  0.0802  0.1271  0.1035
387     0.24928769376128912     0.4698  0.0763  0.2203  0.1345  0.3810  0.4948  0.1152  0.1024
596     0.7321822415105999      5.0991  0.1796  0.1138  0.1113  0.1246  0.0427  0.0759  0.1104
1200    0.587307695299387       1.3917  1.1277  1.4277  0.0914  0.1485  0.1969  0.2117  0.1029
2015    1.1963263098150492      5.4510  1.0897  0.4743  0.0972  0.2240  1.2175  0.8575  0.1595
2216    1.3388446755707264      4.0211  4.2651  0.4503  0.1335  0.0482  0.3432  1.3127  0.1366
2994    0.3156363896559924      0.1788  0.1143  0.1824  1.7381  0.0294  0.0742  0.1021  0.1059
```
I don't suppose this is what you are after, do you?



i think the original poster wants a way to get those log probs for models that are not in fair aew, like MarianMT.

I think I understood that, I'm referring to:

> I modify the translation code in fairseq named as validate_lm.py, and run the bash script to get the per-word log-probability.

which is what I run and displayed the top of the results above. But in the OP it was requested:

> For example:
> Source input (x): I would like to run this experiment.
> Translated output (y) in Chinese: 我 希望 跑 这个 实验 。
> The word-level log probability: -0.3 -0.4 -0.23 -0.43 -0.23 -0.8

So the example code doesn't match the original request and I'm asking to clarify to what output is desired.

In other words - do you want the word and the average log probability of its tokens printed? If yes, I guess we gather the log probs for each token and then somehow keep track of the de-tokenizer and then average the ones that end up comprising each word. I don't have any experience with Kanji - do tokenizers there split the characters into sub-characters, or is it the case where each token is a word, in which case you're asking to just print the log probabilities of each token in the translation output?

Could you redefine your request in terms of latin 2 latin language to remove this possible ambiguity and then once working adapt it to Chinese?  I hope it makes sense where I am not sure what exactly you're after.


@JunjieHu Is this roughly what you are looking for? 
```python
batch = tokenizer.prepare_seq2seq_batch(": I would like to run this experiment.")
model = AutoModelForSeq2SeqLM.from_pretrained('opus-mt/marian-en-zh')
generated_ids = model.generate(batch)
outputs = model(batch.input_ids, labels=generated_ids)
log_probas == outputs.logits[:, generated_ids]
```



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @sshleifer  Sorry for the late response! Yes! This is exactly what I want! Thanks!