BART Summarization : Torchscript Export / Inference Triton Server
Author: anshoomehra
Pull request: no
Labels: name: Migration

# 📚 Migration

@sshleifer maybe you can help ?? (thanks for all your work bud!)

## Information

**Objective** : Performance gain, clocking 1-1.5 sec per transaction at the moment, target : under 100 ms. It seems exporting model via TorchScript & running on Triton Server may be plausible solution.

I am exporting BART Large CNN (for generating Summaries) using TorchScript. I have fine-tuned the model with localized data, but I am unclear on how to use **model.generate(input)** which seems to wrap **model(input)**, whereas **model(input)** is being triggered default at the time of inference from the exported model. For simplicity & ability to reproduce, I pasting the issue/code details as if the model is vanilla pre-trained (and not fine-tuned)

Model: **facebook/bart-large-cnn** 

Language: **English**

The problem arises when using: **torch.jit.trace(<model>, <dummy_input>)**

## Details
1. model.pt gets generated without any issues
2. However, the generated trace is producing output for mask-filling like model(input), what I am hoping to generate is model.**generate**(input).
3. Not sure, how to handle this, at the time of export or later during inference. Can you please help.

Code Block Below:

**Step 1**: Generate model.pt file
```
import torch

from transformers import (
    AutoModelForSeq2SeqLM,
    AutoTokenizer,
    AutoConfig
)

dummy_input = torch.tensor([512 * [1]])

BART_CNN_PATH = 'facebook/bart-large-cnn'
BART_CNN_MODEL = AutoModelForSeq2SeqLM.from_pretrained(BART_CNN_PATH)

BART_CNN_MODEL.eval()

traced_model = torch.jit.trace(BART_CNN_MODEL, dummy_input)
traced_model.save("exportedModelsForTritan/bart_large_cnn_fl/1/model.pt")
```

**Step2**: Inference **(+ Error Details)**

```
BART_CNN_TOKENIZER = AutoTokenizer.from_pretrained(BART_CNN_PATH)
input_tokenized = BART_CNN_TOKENIZER.encode(input_text, return_tensors="pt", max_length=512, truncation=True, padding='max_length')

## Test Inference, If I do not use .generate() code works fine ...
## but then it would attempt mask-filling instead of summaries?...
## With model.generate(input), it returns [1xn] where n is the length of summary
## where-as with mode(input), it's generating tuple with length 3 perhaps logits and possibly hidden state weights..
## which I do not know is of significance for summaries ...
model_output = traced_model.generate(input_tokenized)
```

**Error**: ModuleAttributeError: 'RecursiveScriptModule' object has no attribute 'generate'

**Working Code Prior Export as Reference: <Notice the use of model.generate()>**

```
BART_CNN_PATH = 'facebook/bart-large-cnn'
BART_CNN_MODEL = AutoModelForSeq2SeqLM.from_pretrained(BART_CNN_PATH)
BART_CNN_TOKENIZER = AutoTokenizer.from_pretrained(BART_CNN_PATH)

def bart_cnn_summarize_automl(input_text, num_beams=4, num_words=50):
    input_text = str(input_text)
    input_tokenized = BART_CNN_TOKENIZER.encode(input_text, return_tensors="pt", max_length=512)
    summary_ids = BART_CNN_MODEL.generate(input_tokenized, 
                                       max_length=100, 
                                       min_length=40, 
                                       length_penalty=2.0, 
                                       num_beams=num_beams, 
                                       early_stopping=True)
    output = [BART_CNN_TOKENIZER.decode(id, skip_special_tokens=True, clean_up_tokenization_spaces=False) for id in summary_ids]
    return str(output[0])
```

## Environment info
- Python version:Python 3.6.9
- PyTorch version (GPU?): GPU (T4), 1.7.1
- Docker Image: huggingface/transformers-pytorch-gpu:4.2.1





Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

It seems I have to mimic GenerationMixin.generate() -- advisable? any detailed documentation on 'beam_search' method ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@anshoomehra Were you able to run BART on Triton?

@anshoomehra @moise-g 
Are you able to run BART with Triton, if yes can you please share the details?