Deploying a  pytorch-pretrained-bert on mobile
Author: CowryCode
Pull request: no

### System Info

```shell
I use google colab
```


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code here https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313

### Expected behavior

```shell
I want to export the model for use in a mobile app (flutter). I am new to this and I just can't figure it out. Have tried many recommendations online, but something seems off. Kindly help
```


### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
Hi,

Regarding exporting Transformers models, refer to the guide here: https://huggingface.co/docs/transformers/serialization.

Also cc'ing @hollance who's an expert on mobile ML.

Thanks @NielsRogge I will explore it.

@hollance, I just sent you an email at [mail@hollance.com]

It would be helpful to know what mobile platform you're trying to export to, how you've tried to do the export, and what errors you're running into. There are too many unknowns here to give an answer.

Thanks @hollance for your response, I just saw the message now. I will drop detail of both the code and screenshot shortly 

@hollance below are the codes and errors they threw. What I want to achieve is to use the model on edge device (mobile) developed in flutter. 

**Output of the training**
![image](https://user-images.githubusercontent.com/23655212/176263865-0e1964a6-460d-47cd-bf13-90cd3866e57e.png)
**First Approach I tried**
`model_class = OpenAIGPTDoubleHeadsModel
model = model_class.from_pretrained('./themodel')
Scripted_model = torch.jit.script(model)
opt_model = optimize_for_mobile(Scripted_model)
opt_model._save_for_lite_interpreter("Mobile_model.ptl")`

**Error from Approach 1**
![image](https://user-images.githubusercontent.com/23655212/176264730-60237895-fc57-4607-94c2-f5dbdec9a4c5.png)


**Approach 2**
`from itertools import chain
persona = [["i", "like", "playing", "football", "."],
           ["i", "am", "from", "NYC", "."]]
history = [["hello", "how", "are", "you", "?"],
           ["i", "am", "fine", "thanks", "."]]
reply = ["great", "to", "hear"]
bos, eos, speaker1, speaker2 = "<bos>", "<eos>", "<speaker1>", "<speaker2>"

def build_inputs(persona, history, reply):
    sequence = [[bos] + list(chain(*persona))] + history + [reply + [eos]]
    sequence = [sequence[0]] + [ [speaker2 if (len(sequence)-i) % 2 else speaker1] + s
                                for i, s in enumerate(sequence[1:])]
    words = list(chain(*sequence))                          # word tokens
    segments = [speaker2 if i % 2 else speaker1             # segment tokens
                for i, s in enumerate(sequence) for _ in s]
    position = list(range(len(words)))                      # position tokens
    return words, segments, position, sequence

words, segments, position, sequence = build_inputs(persona, history, reply)

words = tokenizer.convert_tokens_to_ids(words)
segments = tokenizer.convert_tokens_to_ids(segments)

tokenizer_class, model_class = (OpenAIGPTTokenizer, OpenAIGPTDoubleHeadsModel)
tokens_tensor = torch.tensor([words])
segments_tensors = torch.tensor([segments])
model = torch.load('./themodel/pytorch_model.bin')
Scripted_model = torch.jit.trace(model,[tokens_tensor, segments_tensors])
opt_model = optimize_for_mobile(Scripted_model)
opt_model._save_for_lite_interpreter("Mobile_model.ptl")`

**Error of the Second Approach**
![image](https://user-images.githubusercontent.com/23655212/176266493-6d969c7d-bb12-4ba8-b230-4afc818c55ae.png)


The second approach of tracing the model (rather than scripting it), is what I would prefer. However, you need to load the model using `model = OpenAIGPTDoubleHeadsModel.from_pretrained("themodel", torchscript=True)` instead of `torch.load`.

**Thank you @hollance, I really appreciate your effort. I tried it, and below were the error I got.**

**When I changed the model load, I got the error below:**
![image](https://user-images.githubusercontent.com/23655212/176485673-2c69ea09-dc0a-4e4c-8af7-35a60a366cd7.png)

**When I removed the torchscript argument, I got the error below:**
![image](https://user-images.githubusercontent.com/23655212/176486848-6b728116-c1d8-40ba-a333-1e7210a6c401.png)
![image](https://user-images.githubusercontent.com/23655212/176487133-9f825229-4ec3-4fdf-b71a-8f0f22e0573e.png)


That looks like you're not giving it inputs of the correct size. It's a good idea to using your input tensors first in a normal inference call:

```
with torch.no_grad():
    outputs = model(inputs, return_dict=False)
```

where `inputs` are the input tensors this model needs. I expect this to also give an error message, so first make sure that works without problems.

Thank you @hollance for your support, I tried it and that failed. Below is a link to the google colab file perhaps that will explain the scenario better than I can do, I really appreciate your effort to help a newbie in ML. 

https://drive.google.com/file/d/1jKjmO0gh94i57zuPV0rvMyTmRgiyBBFG/view?usp=sharing

Sorry but there's just way too much stuff in that notebook for me to make sense of. Could you create a notebook that has the minimum amount of code in it to reproduce the problem?

Hi @hollance, kindly find below as requested to recreate the problem:
**This is link to the minimized version of the colable file**
https://drive.google.com/file/d/1CTVA6wD26BMHXlU6JY2VJ3S5TW5dI2Ac/view?usp=sharing

**Below is a link to training output (i.e content of "themodel" folder)**
https://drive.google.com/file/d/1ppib_rexC6_XsOlUQeOf3goD-v2uhqad/view?usp=sharing

**Below is a link to the dataset (in case you may want to take a look)**
https://s3.amazonaws.com/datasets.huggingface.co/personachat/personachat_self_original.json


Thank you for your support so far I am grateful. I feel really excited and relieved knowing that I am getting the needed help.

@CowryCode The problem is that the following code doesn't work:

```python
with torch.no_grad():
    outputs = model(tokens_tensor, segments_tensors)
```

The error message is "Index tensor must have the same number of dimensions as input tensor". The same thing happens when you try to do a `torch.jit.trace`. That's why the JIT trace fails.

Now I'm not sure what `segments_tensors` is supposed to be but I think you mean to pass this into the `token_type_ids` argument. However, writing the following doesn't work:

```python
with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
```

This is because the OpenAIGPTDoubleHeadsModel from the pytorch-pretrained-bert package expects there to be a `mc_token_ids` argument. 

Assuming that you actually meant to use OpenAIGPTDoubleHeadsModel from 🤗 Transformers, the above code does work, so I suggest you use that instead.

However, there is another argument, `attention_mask`, in between the `input_ids` and `token_type_ids` arguments. When you call `torch.jit.trace`, you have to supply that attention mask argument too.

The easiest way around this is to create a helper class:

```python
from torch import nn

class Wrapper(nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model

    def forward(self, input_ids, token_type_ids):
        return self.model(input_ids, None, token_type_ids, return_dict=False)
```

And then call it like so:

```python
wrapper = Wrapper(model)
traced_model = torch.jit.trace(wrapper, [tokens_tensor, segments_tensors])
```

This will trace the model into a TorchScript object. Now you can do whatever you need to in order to load it into PyTorch mobile etc.

To verify this traced model gives the same outputs as the original, do this:

```python
with torch.no_grad():
    traced_outputs = traced_model(tokens_tensor, segments_tensors)
```

Then the following should print a very small number (around 1e-6 or 1e-7):

```python
torch.max(torch.abs(outputs[0] - traced_outputs[0])) / torch.max(torch.abs(traced_outputs[0]))
```

P.S. Ideally, you should load the original model as follows, with the `torchscript` argument:

```python
model = OpenAIGPTDoubleHeadsModel.from_pretrained("openai-gpt", torchscript=True)
```


Thank you @hollance, I will explore the solution you gave.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@hollance hi, I wanted to deploy pix2struct into mobile. Do you have advice on which mobile platform, libraries, model format, etc to use and steps on deploying it?