Cannot compile T5 for inferentia
Author: ierezell
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.18.7-arch1-1-x86_64-with-glibc2.34
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The inferentia for Marian (Seq2Seq) is [here](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/src/examples/pytorch/transformers-marianmt.html)

Here is the snippet of code I'm using : 

<details>
<summary>Python Snippet for compiling T5 model</summary>
<br>

```python
import os

import numpy as np
import torch
import torch.neuron
from torch.nn import functional as F
from transformers.generation_utils import GenerationMixin
from transformers.modeling_outputs import BaseModelOutput, Seq2SeqLMOutput
from transformers.modeling_utils import PreTrainedModel
from transformers.models.t5.configuration_t5 import T5Config
from transformers.models.t5.modeling_t5 import T5Model
from transformers.models.t5.tokenization_t5 import T5Tokenizer

model_id = "mrm8488/t5-base-finetuned-question-generation-ap"
num_texts = 1  # Number of input texts to decode
num_beams = 4  # Number of beams per input text
max_encoder_length = 32  # Maximum input token length
max_decoder_length = 32


def infer(model, tokenizer, text):

    # Truncate and pad the max length to ensure that the token size is compatible with fixed-sized encoder (Not necessary for pure CPU execution)
    batch = tokenizer(
        text,
        max_length=max_decoder_length,
        truncation=True,
        padding="max_length",
        return_tensors="pt",
    )
    output = model.generate(
        **batch,
        max_length=max_decoder_length,
        num_beams=num_beams,
        num_return_sequences=num_beams,
    )
    results = [tokenizer.decode(t, skip_special_tokens=True) for t in output]

    print("Texts:")
    for i, summary in enumerate(results):
        print(i + 1, summary)


def reduce(hidden, index):
    _, n_length, _ = hidden.shape

    # Create selection mask
    mask = torch.arange(n_length, dtype=torch.float32) == index
    mask = mask.view(1, -1, 1)

    # Broadcast mask
    masked = torch.multiply(hidden, mask)

    # Reduce along 1st dimension
    summed = torch.sum(masked, 1)
    return torch.unsqueeze(summed, 1)


class NeuronEncoder(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.encoder = model.encoder

    def forward(self, input_ids, attention_mask):
        return self.encoder(input_ids, attention_mask=attention_mask, return_dict=False)


class NeuronDecoder(torch.nn.Module):
    def __init__(self, model, max_length):
        super().__init__()
        self.weight = model.shared.weight.clone().detach()
        self.bias = model.final_logits_bias.clone().detach()
        self.decoder = model.decoder
        self.max_length = max_length

    def forward(self, input_ids, attention_mask, encoder_outputs, index):

        # Build a fixed sized causal mask for the padded decoder input ids
        mask = np.triu(np.ones((self.max_length, self.max_length)), 1)
        mask[mask == 1] = -np.inf
        causal_mask = torch.tensor(mask, dtype=torch.float)

        # Invoke the decoder
        (hidden,) = self.decoder(
            input_ids=input_ids,
            encoder_hidden_states=encoder_outputs,
            encoder_padding_mask=attention_mask,
            decoder_padding_mask=None,
            decoder_causal_mask=causal_mask,
            return_dict=False,
            use_cache=False,
        )

        # Reduce decoder outputs to the specified index (current iteration)
        hidden = reduce(hidden, index)

        # Compute final linear layer for token probabilities
        logits = F.linear(hidden, self.weight, bias=self.bias)
        return logits


class NeuronGeneration(PreTrainedModel, GenerationMixin):
    def trace(
        self, model, num_texts, num_beams, max_encoder_length, max_decoder_length
    ):
        """
        Traces the encoder and decoder modules for use on Neuron.

        This function fixes the network to the given sizes. Once the model has been
        compiled to a given size, the inputs to these networks must always be of
        fixed size.

        Args:
            model (GenerationMixin): The transformer-type generator model to trace
            num_texts (int): The number of input texts to translate at once
            num_beams (int): The number of beams to computer per text
            max_encoder_length (int): The maximum number of encoder tokens
            max_encoder_length (int): The maximum number of decoder tokens
        """
        self.config.max_decoder_length = max_decoder_length

        # Trace the encoder
        inputs = (
            torch.ones((num_texts, max_encoder_length), dtype=torch.long),
            torch.ones((num_texts, max_encoder_length), dtype=torch.long),
        )
        encoder = NeuronEncoder(model)
        self.encoder = torch.neuron.trace(encoder, inputs)

        # Trace the decoder (with expanded inputs)
        batch_size = num_texts * num_beams
        inputs = (
            torch.ones((batch_size, max_decoder_length), dtype=torch.long),
            torch.ones((batch_size, max_encoder_length), dtype=torch.long),
            torch.ones(
                (batch_size, max_encoder_length, model.config.d_model),
                dtype=torch.float,
            ),
            torch.tensor(0),
        )
        decoder = NeuronDecoder(model, max_decoder_length)
        self.decoder = torch.neuron.trace(decoder, inputs)

    # ------------------------------------------------------------------------
    # Beam Search Methods (Copied directly from transformers)
    # ------------------------------------------------------------------------

    def adjust_logits_during_generation(self, logits, cur_len, max_length):
        if cur_len == 1 and self.config.force_bos_token_to_be_generated:
            self._force_token_id_to_be_generated(logits, self.config.bos_token_id)
        elif cur_len == max_length - 1 and self.config.eos_token_id is not None:
            self._force_token_id_to_be_generated(logits, self.config.eos_token_id)
        return logits

    @staticmethod
    def _force_token_id_to_be_generated(scores, token_id) -> None:
        scores[:, [x for x in range(scores.shape[1]) if x != token_id]] = -float("inf")

    # ------------------------------------------------------------------------
    # Encoder/Decoder Invocation
    # ------------------------------------------------------------------------

    def prepare_inputs_for_generation(
        self,
        decoder_input_ids,
        encoder_outputs=None,
        attention_mask=None,
        **model_kwargs,
    ):
        # Pad the inputs for Neuron
        current_length = decoder_input_ids.shape[1]
        pad_size = self.config.max_decoder_length - current_length
        return dict(
            input_ids=F.pad(decoder_input_ids, (0, pad_size)),
            attention_mask=attention_mask,
            encoder_outputs=encoder_outputs.last_hidden_state,
            current_length=torch.tensor(current_length - 1),
        )

    def get_encoder(self):
        """Helper to invoke the encoder and wrap the results in the expected structure"""

        def encode(input_ids, attention_mask, **kwargs):
            (output,) = self.encoder(input_ids, attention_mask)
            return BaseModelOutput(
                last_hidden_state=output,
            )

        return encode

    def __call__(
        self, input_ids, attention_mask, encoder_outputs, current_length, **kwargs
    ):
        """Helper to invoke the decoder and wrap the results in the expected structure"""
        logits = self.decoder(
            input_ids, attention_mask, encoder_outputs, current_length
        )
        return Seq2SeqLMOutput(logits=logits)

    # ------------------------------------------------------------------------
    # Serialization
    # ------------------------------------------------------------------------

    def save_pretrained(self, directory):
        if os.path.isfile(directory):
            print(f"Provided path ({directory}) should be a directory, not a file")
            return
        os.makedirs(directory, exist_ok=True)
        torch.jit.save(self.encoder, os.path.join(directory, "encoder.pt"))
        torch.jit.save(self.decoder, os.path.join(directory, "decoder.pt"))
        self.config.save_pretrained(directory)

    @classmethod
    def from_pretrained(cls, directory):
        config = T5Config.from_pretrained(directory)
        obj = cls(config)
        obj.encoder = torch.jit.load(os.path.join(directory, "encoder.pt"))
        obj.decoder = torch.jit.load(os.path.join(directory, "decoder.pt"))
        return obj

    @property
    def device(self):
        return torch.device("cpu")


model_cpu = T5Model.from_pretrained(model_id)
tokenizer_cpu = T5Tokenizer.from_pretrained(model_id)
model_neuron = NeuronGeneration(model_cpu.config)

# 1. Compile the model
# Note: This may take a couple of minutes since both the encoder/decoder will be compiled
model_neuron.trace(
    model=model_cpu,
    num_texts=num_texts,
    num_beams=num_beams,
    max_encoder_length=max_encoder_length,
    max_decoder_length=max_decoder_length,
)

# 2. Serialize an artifact
# After this call you will have an `encoder.pt`, `decoder.pt` and `config.json` in the neuron_name folder
model_neuron.save_pretrained(neuron_name)
tokenizer.save_pretrained(neuron_name)

model_neuron = NeuronGeneration.from_pretrained(neuron_name)
infer(model_neuron, tokenizer_cpu, sample_text)
```

</details>

To setup the environment : 
1. Create a venv (3.8 seems good for neuron-cc, 3.9 seems not)
2. install neuron-cc `pip install neuron-cc --extra-index-url=https://pip.repos.neuron.amazonaws.com`
3. Install the rest : `pip install pip install torch-neuron sagemaker transformers sentencepiece`
4. run the code above
5. Get a :


<details>
<summary>TraceBack of the compilation</summary>
<br>

```
The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.
Some weights of the model checkpoint at mrm8488/t5-base-finetuned-question-generation-ap were not used when initializing T5Model: ['lm_head.weight']
- This IS expected if you are initializing T5Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.
INFO:Neuron:There are 2 ops of 1 different types in the TorchScript that are not compiled by neuron-cc: aten::embedding, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 627, fused = 587, percent fused = 93.62%
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$641; falling back to native python function call
ERROR:Neuron:No module named 'tensorflow'
Traceback (most recent call last):
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 381, in op_converter
    neuron_function = self.subgraph_compiler(
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/decorators.py", line 67, in trace
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
INFO:Neuron:Number of arithmetic operators (post-compilation) before = 627, compiled = 0, percent compiled = 0.0%
INFO:Neuron:The neuron partitioner created 1 sub-graphs
INFO:Neuron:Neuron successfully compiled 0 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 0.0%
INFO:Neuron:Compiled these operators (and operator counts) to Neuron:
INFO:Neuron:Not compiled operators (and operator counts) to Neuron:
INFO:Neuron: => aten::Int: 49 [supported]
INFO:Neuron: => aten::ScalarImplicit: 2 [supported]
INFO:Neuron: => aten::abs: 1 [supported]
INFO:Neuron: => aten::add: 65 [supported]
INFO:Neuron: => aten::arange: 2 [supported]
INFO:Neuron: => aten::contiguous: 12 [supported]
INFO:Neuron: => aten::div: 2 [supported]
INFO:Neuron: => aten::dropout: 50 [supported]
INFO:Neuron: => aten::embedding: 2 [not supported]
INFO:Neuron: => aten::full_like: 1 [supported]
INFO:Neuron: => aten::gt: 1 [supported]
INFO:Neuron: => aten::linear: 72 [supported]
INFO:Neuron: => aten::log: 1 [supported]
INFO:Neuron: => aten::lt: 1 [supported]
INFO:Neuron: => aten::matmul: 24 [supported]
INFO:Neuron: => aten::mean: 25 [supported]
INFO:Neuron: => aten::min: 1 [supported]
INFO:Neuron: => aten::mul: 53 [supported]
INFO:Neuron: => aten::permute: 1 [supported]
INFO:Neuron: => aten::pow: 25 [supported]
INFO:Neuron: => aten::relu: 12 [supported]
INFO:Neuron: => aten::rsqrt: 25 [supported]
INFO:Neuron: => aten::rsub: 1 [supported]
INFO:Neuron: => aten::size: 14 [supported]
INFO:Neuron: => aten::slice: 4 [supported]
INFO:Neuron: => aten::softmax: 12 [supported]
INFO:Neuron: => aten::sub: 1 [supported]
INFO:Neuron: => aten::to: 41 [supported]
INFO:Neuron: => aten::transpose: 60 [supported]
INFO:Neuron: => aten::type_as: 12 [supported]
INFO:Neuron: => aten::unsqueeze: 5 [supported]
INFO:Neuron: => aten::view: 49 [supported]
INFO:Neuron: => aten::where: 1 [not supported]
Traceback (most recent call last):
  File "compile_model.py", line 232, in <module>
    model_neuron.trace(
  File "compile_model.py", line 128, in trace
    self.encoder = torch.neuron.trace(encoder, inputs)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 184, in trace
    cu.stats_post_compiler(neuron_graph)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 492, in stats_post_compiler
    raise RuntimeError(
RuntimeError: No operations were successfully partitioned and compiled to neuron for this model - aborting trace!
```
</details>

Which seem to say that `.where` is not supported by neuron-cc (and thus inferentia), which mean that we cannot run T5 models fast and cheap...

Could we change the `.where` to have something supported and thus be able to compile models for inferentia ? 
Else should I raise the issue to `neuron-cc` so they can support it ? 

Thanks in advance, 
Have a great day. 

### Expected behavior

Be able to compile a T5 model for inferentia (`.where` being supported)


Note : I also create an issue in the `aws-neuron-sdk` repo : https://github.com/aws/aws-neuron-sdk/issues/440

Comments:
Gently pinging @philschmid here - do you have any ideas maybe?

@Ierezell could you please try to set up your environment as described [here](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/neuron-intro/pytorch-setup/pytorch-install.html#id2). It feels like that you are missing packages to successfully compile the models 

Hello @patrickvonplaten, @philschmid. 
Thanks for the fast reply and help :) 

It's my bad I haven't pasted the good traceback. 

Here is the new one with still the same code, and the `pip freeze` on my local machine, I reinstalled all the dependencies to be sure.
 
Do I need to be on an inferentia ec2 machine to compile my model (here I'm on my local laptop : Linux Lenovo X1 with 1650Gpu ) ?  

<details>
<summary>Pip freeze</summary>
<br>

```
absl-py==1.1.0
astunparse==1.6.3
attrs==21.4.0
cachetools==5.2.0
certifi==2022.6.15
charset-normalizer==2.1.0
decorator==5.1.1
dmlc-nnvm==1.11.0.0+0
dmlc-topi==1.11.0.0+0
dmlc-tvm==1.11.0.0+0
filelock==3.7.1
flatbuffers==2.0
gast==0.5.3
google-auth==2.9.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.47.0
h5py==3.7.0
huggingface-hub==0.8.1
idna==3.3
importlib-metadata==4.12.0
inferentia-hwm==1.11.0.0+0
iniconfig==1.1.1
islpy==2021.1+aws2021.x.16.0.bld0
keras==2.8.0
Keras-Preprocessing==1.1.2
libclang==14.0.1
Markdown==3.3.7
networkx==2.4
neuron-cc==1.11.4.0+97f99abe4
numpy==1.20.0
oauthlib==3.2.0
opt-einsum==3.3.0
packaging==21.3
pluggy==1.0.0
protobuf==3.20.1
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.9
pytest==7.1.2
PyYAML==6.0
regex==2022.6.2
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.8
scipy==1.4.1
sentencepiece==0.1.96
six==1.16.0
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-neuron==2.4.0.0
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.26.0
tensorflow-neuron==2.8.0.2.3.0.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
tokenizers==0.12.1
tomli==2.0.1
torch==1.11.0
torch-neuron==1.11.0.2.3.0.0
tqdm==4.64.0
transformers==4.20.1
typing_extensions==4.3.0
urllib3==1.26.9
Werkzeug==2.1.2
wrapt==1.14.1
zipp==3.8.0
```

</details>


<details>
<summary>Python compile model traceback</summary>
<br>

```
The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.
Some weights of the model checkpoint at mrm8488/t5-base-finetuned-question-generation-ap were not used when initializing T5Model: ['lm_head.weight']
- This IS expected if you are initializing T5Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.
INFO:Neuron:There are 2 ops of 1 different types in the TorchScript that are not compiled by neuron-cc: aten::embedding, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 627, fused = 587, percent fused = 93.62%
2022-07-06 15:58:27.564554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:27.622777: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:27.622940: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:Neuron:Number of neuron graph operations 1637 did not match traced graph 1569 - using heuristic matching of hierarchical information
INFO:Neuron:Compiling function _NeuronGraph$641 with neuron-cc
INFO:Neuron:Compiling with command line: '/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp428tqio_/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp428tqio_/graph_def.neff --io-config {"inputs": {"tensor.1:0": [[1, 32], "int64"], "1:0": [[1, 32, 768], "float32"], "2:0": [[32, 32, 12], "float32"], "3:0": [[1, 32, 768], "float32"]}, "outputs": ["T5Stack_1/T5LayerNorm_79/aten_mul_1/mul:0"]} --verbose 35'
.2022-07-06 15:58:50.575361: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:50.605627: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:50.605816: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
07/06/2022 03:58:52 PM ERROR 2484 [neuron-cc]: Failed to parse model /tmp/tmp428tqio_/graph_def.pb: The following operators are not implemented: {'SelectV2'} (NotImplementedError)

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$641; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp428tqio_/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp428tqio_/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[1, 32], "int64"], "1:0": [[1, 32, 768], "float32"], "2:0": [[32, 32, 12], "float32"], "3:0": [[1, 32, 768], "float32"]}, "outputs": ["T5Stack_1/T5LayerNorm_79/aten_mul_1/mul:0"]}' --verbose 35
Traceback (most recent call last):
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 381, in op_converter
    neuron_function = self.subgraph_compiler(
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/decorators.py", line 219, in trace
    raise subprocess.SubprocessError(
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp428tqio_/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp428tqio_/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[1, 32], "int64"], "1:0": [[1, 32, 768], "float32"], "2:0": [[32, 32, 12], "float32"], "3:0": [[1, 32, 768], "float32"]}, "outputs": ["T5Stack_1/T5LayerNorm_79/aten_mul_1/mul:0"]}' --verbose 35
INFO:Neuron:Number of arithmetic operators (post-compilation) before = 627, compiled = 0, percent compiled = 0.0%
INFO:Neuron:The neuron partitioner created 1 sub-graphs
INFO:Neuron:Neuron successfully compiled 0 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 0.0%
INFO:Neuron:Compiled these operators (and operator counts) to Neuron:
INFO:Neuron:Not compiled operators (and operator counts) to Neuron:
INFO:Neuron: => aten::Int: 49 [supported]
INFO:Neuron: => aten::ScalarImplicit: 2 [supported]
INFO:Neuron: => aten::abs: 1 [supported]
INFO:Neuron: => aten::add: 65 [supported]
INFO:Neuron: => aten::arange: 2 [supported]
INFO:Neuron: => aten::contiguous: 12 [supported]
INFO:Neuron: => aten::div: 2 [supported]
INFO:Neuron: => aten::dropout: 50 [supported]
INFO:Neuron: => aten::embedding: 2 [not supported]
INFO:Neuron: => aten::full_like: 1 [supported]
INFO:Neuron: => aten::gt: 1 [supported]
INFO:Neuron: => aten::linear: 72 [supported]
INFO:Neuron: => aten::log: 1 [supported]
INFO:Neuron: => aten::lt: 1 [supported]
INFO:Neuron: => aten::matmul: 24 [supported]
INFO:Neuron: => aten::mean: 25 [supported]
INFO:Neuron: => aten::min: 1 [supported]
INFO:Neuron: => aten::mul: 53 [supported]
INFO:Neuron: => aten::permute: 1 [supported]
INFO:Neuron: => aten::pow: 25 [supported]
INFO:Neuron: => aten::relu: 12 [supported]
INFO:Neuron: => aten::rsqrt: 25 [supported]
INFO:Neuron: => aten::rsub: 1 [supported]
INFO:Neuron: => aten::size: 14 [supported]
INFO:Neuron: => aten::slice: 4 [supported]
INFO:Neuron: => aten::softmax: 12 [supported]
INFO:Neuron: => aten::sub: 1 [supported]
INFO:Neuron: => aten::to: 41 [supported]
INFO:Neuron: => aten::transpose: 60 [supported]
INFO:Neuron: => aten::type_as: 12 [supported]
INFO:Neuron: => aten::unsqueeze: 5 [supported]
INFO:Neuron: => aten::view: 49 [supported]
INFO:Neuron: => aten::where: 1 [not supported]
Traceback (most recent call last):
  File "compile_model.py", line 232, in <module>
    model_neuron.trace(
  File "compile_model.py", line 128, in trace
    self.encoder = torch.neuron.trace(encoder, inputs)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 184, in trace
    cu.stats_post_compiler(neuron_graph)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 492, in stats_post_compiler
    raise RuntimeError(
RuntimeError: No operations were successfully partitioned and compiled to neuron for this model - aborting trace!

```

</details>

Note that with this simpler snippet (from you blog @philschmid ;) ): 

<details>
<summary>Simpler Python compile model</summary>
<br>

```python
import os
import tensorflow  # to workaround a protobuf version conflict issue
import torch
import torch.neuron
from transformers import AutoTokenizer, AutoModel


# load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, torchscript=True)

# create dummy input for max length 128
dummy_input = "dummy input which will be padded later"
max_length = 128
embeddings = tokenizer(dummy_input, max_length=max_length, padding="max_length",return_tensors="pt")
neuron_inputs = tuple(embeddings.values())

# compile model with torch.neuron.trace and update config
model_neuron = torch.neuron.trace(model, neuron_inputs)
model.config.update({"traced_sequence_length": max_length})

# save tokenizer, neuron model and config for later use
save_dir="tmp"
os.makedirs("tmp",exist_ok=True)
model_neuron.save(os.path.join(save_dir,"neuron_model.pt"))
tokenizer.save_pretrained(save_dir)
model.config.save_pretrained(save_dir)
```

</details>

I got the same traceback with the same SelectV2 error. 

Thanks again for your help. 

Hello @patrickvonplaten, @philschmid. 
Thanks for the fast reply and help :) 

It's my bad I haven't pasted the good traceback. 

Here is the new one with still the same code, and the `pip freeze` on my local machine, I reinstalled all the dependencies to be sure.
 
Do I need to be on an inferentia ec2 machine to compile my model (here I'm on my local laptop : Linux Lenovo X1 with 1650Gpu ) ?  I know I will need to be for inference, but here I just want to compile the model first.

<details>
<summary>Pip freeze</summary>
<br>

```
absl-py==1.1.0
astunparse==1.6.3
attrs==21.4.0
cachetools==5.2.0
certifi==2022.6.15
charset-normalizer==2.1.0
decorator==5.1.1
dmlc-nnvm==1.11.0.0+0
dmlc-topi==1.11.0.0+0
dmlc-tvm==1.11.0.0+0
filelock==3.7.1
flatbuffers==2.0
gast==0.5.3
google-auth==2.9.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.47.0
h5py==3.7.0
huggingface-hub==0.8.1
idna==3.3
importlib-metadata==4.12.0
inferentia-hwm==1.11.0.0+0
iniconfig==1.1.1
islpy==2021.1+aws2021.x.16.0.bld0
keras==2.8.0
Keras-Preprocessing==1.1.2
libclang==14.0.1
Markdown==3.3.7
networkx==2.4
neuron-cc==1.11.4.0+97f99abe4
numpy==1.20.0
oauthlib==3.2.0
opt-einsum==3.3.0
packaging==21.3
pluggy==1.0.0
protobuf==3.20.1
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.9
pytest==7.1.2
PyYAML==6.0
regex==2022.6.2
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.8
scipy==1.4.1
sentencepiece==0.1.96
six==1.16.0
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-neuron==2.4.0.0
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.26.0
tensorflow-neuron==2.8.0.2.3.0.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
tokenizers==0.12.1
tomli==2.0.1
torch==1.11.0
torch-neuron==1.11.0.2.3.0.0
tqdm==4.64.0
transformers==4.20.1
typing_extensions==4.3.0
urllib3==1.26.9
Werkzeug==2.1.2
wrapt==1.14.1
zipp==3.8.0
```

</details>


<details>
<summary>Python compile model traceback</summary>
<br>

```
The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.
Some weights of the model checkpoint at mrm8488/t5-base-finetuned-question-generation-ap were not used when initializing T5Model: ['lm_head.weight']
- This IS expected if you are initializing T5Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing T5Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.
INFO:Neuron:There are 2 ops of 1 different types in the TorchScript that are not compiled by neuron-cc: aten::embedding, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 627, fused = 587, percent fused = 93.62%
2022-07-06 15:58:27.564554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:27.622777: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:27.622940: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:Neuron:Number of neuron graph operations 1637 did not match traced graph 1569 - using heuristic matching of hierarchical information
INFO:Neuron:Compiling function _NeuronGraph$641 with neuron-cc
INFO:Neuron:Compiling with command line: '/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp428tqio_/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp428tqio_/graph_def.neff --io-config {"inputs": {"tensor.1:0": [[1, 32], "int64"], "1:0": [[1, 32, 768], "float32"], "2:0": [[32, 32, 12], "float32"], "3:0": [[1, 32, 768], "float32"]}, "outputs": ["T5Stack_1/T5LayerNorm_79/aten_mul_1/mul:0"]} --verbose 35'
.2022-07-06 15:58:50.575361: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:50.605627: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-06 15:58:50.605816: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
07/06/2022 03:58:52 PM ERROR 2484 [neuron-cc]: Failed to parse model /tmp/tmp428tqio_/graph_def.pb: The following operators are not implemented: {'SelectV2'} (NotImplementedError)

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$641; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp428tqio_/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp428tqio_/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[1, 32], "int64"], "1:0": [[1, 32, 768], "float32"], "2:0": [[32, 32, 12], "float32"], "3:0": [[1, 32, 768], "float32"]}, "outputs": ["T5Stack_1/T5LayerNorm_79/aten_mul_1/mul:0"]}' --verbose 35
Traceback (most recent call last):
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 381, in op_converter
    neuron_function = self.subgraph_compiler(
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/decorators.py", line 219, in trace
    raise subprocess.SubprocessError(
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp428tqio_/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp428tqio_/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[1, 32], "int64"], "1:0": [[1, 32, 768], "float32"], "2:0": [[32, 32, 12], "float32"], "3:0": [[1, 32, 768], "float32"]}, "outputs": ["T5Stack_1/T5LayerNorm_79/aten_mul_1/mul:0"]}' --verbose 35
INFO:Neuron:Number of arithmetic operators (post-compilation) before = 627, compiled = 0, percent compiled = 0.0%
INFO:Neuron:The neuron partitioner created 1 sub-graphs
INFO:Neuron:Neuron successfully compiled 0 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 0.0%
INFO:Neuron:Compiled these operators (and operator counts) to Neuron:
INFO:Neuron:Not compiled operators (and operator counts) to Neuron:
INFO:Neuron: => aten::Int: 49 [supported]
INFO:Neuron: => aten::ScalarImplicit: 2 [supported]
INFO:Neuron: => aten::abs: 1 [supported]
INFO:Neuron: => aten::add: 65 [supported]
INFO:Neuron: => aten::arange: 2 [supported]
INFO:Neuron: => aten::contiguous: 12 [supported]
INFO:Neuron: => aten::div: 2 [supported]
INFO:Neuron: => aten::dropout: 50 [supported]
INFO:Neuron: => aten::embedding: 2 [not supported]
INFO:Neuron: => aten::full_like: 1 [supported]
INFO:Neuron: => aten::gt: 1 [supported]
INFO:Neuron: => aten::linear: 72 [supported]
INFO:Neuron: => aten::log: 1 [supported]
INFO:Neuron: => aten::lt: 1 [supported]
INFO:Neuron: => aten::matmul: 24 [supported]
INFO:Neuron: => aten::mean: 25 [supported]
INFO:Neuron: => aten::min: 1 [supported]
INFO:Neuron: => aten::mul: 53 [supported]
INFO:Neuron: => aten::permute: 1 [supported]
INFO:Neuron: => aten::pow: 25 [supported]
INFO:Neuron: => aten::relu: 12 [supported]
INFO:Neuron: => aten::rsqrt: 25 [supported]
INFO:Neuron: => aten::rsub: 1 [supported]
INFO:Neuron: => aten::size: 14 [supported]
INFO:Neuron: => aten::slice: 4 [supported]
INFO:Neuron: => aten::softmax: 12 [supported]
INFO:Neuron: => aten::sub: 1 [supported]
INFO:Neuron: => aten::to: 41 [supported]
INFO:Neuron: => aten::transpose: 60 [supported]
INFO:Neuron: => aten::type_as: 12 [supported]
INFO:Neuron: => aten::unsqueeze: 5 [supported]
INFO:Neuron: => aten::view: 49 [supported]
INFO:Neuron: => aten::where: 1 [not supported]
Traceback (most recent call last):
  File "compile_model.py", line 232, in <module>
    model_neuron.trace(
  File "compile_model.py", line 128, in trace
    self.encoder = torch.neuron.trace(encoder, inputs)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 184, in trace
    cu.stats_post_compiler(neuron_graph)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 492, in stats_post_compiler
    raise RuntimeError(
RuntimeError: No operations were successfully partitioned and compiled to neuron for this model - aborting trace!

```

</details>

Note that with this simpler snippet (from you blog @philschmid ;) ): 

<details>
<summary>Simpler Python compile model</summary>
<br>

```python
import os
import tensorflow  # to workaround a protobuf version conflict issue
import torch
import torch.neuron
from transformers import AutoTokenizer, AutoModel


# load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, torchscript=True)

# create dummy input for max length 128
dummy_input = "dummy input which will be padded later"
max_length = 128
embeddings = tokenizer(dummy_input, max_length=max_length, padding="max_length",return_tensors="pt")
neuron_inputs = tuple(embeddings.values())

# compile model with torch.neuron.trace and update config
model_neuron = torch.neuron.trace(model, neuron_inputs)
model.config.update({"traced_sequence_length": max_length})

# save tokenizer, neuron model and config for later use
save_dir="tmp"
os.makedirs("tmp",exist_ok=True)
model_neuron.save(os.path.join(save_dir,"neuron_model.pt"))
tokenizer.save_pretrained(save_dir)
model.config.save_pretrained(save_dir)
```

</details>

I got the same traceback with the same SelectV2 error. 

Thanks again for your help. 

@Ierezell i see you have `transformers 4.20.1` and torch 1.11 installed could you try to use the same versions as in the [guide](https://www.philschmid.de/huggingface-sentence-transformers-aws-inferentia) we created?

@philschmid 

Here is the dependencies with `transformers 4.12.3`
```python
python -c "import transformers;print(transformers.__version__)" # 4.12.3
python -c "import torch_neuron;print(torch_neuron.__version__)" #1.9.1.2.3.0.0
```
<details>
<summary>pip freeze</summary>
<br>

```
absl-py==1.1.0
astunparse==1.6.3
attrs==20.3.0
boto3==1.24.24
botocore==1.27.24
cachetools==5.2.0
certifi==2022.6.15
charset-normalizer==2.1.0
click==8.1.3
decorator==5.1.1
dill==0.3.5.1
dmlc-nnvm==1.11.0.0+0
dmlc-topi==1.11.0.0+0
dmlc-tvm==1.11.0.0+0
filelock==3.7.1
flatbuffers==2.0
gast==0.5.3
google-auth==2.9.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.47.0
h5py==3.7.0
huggingface-hub==0.8.1
idna==3.3
importlib-metadata==4.12.0
inferentia-hwm==1.11.0.0+0
iniconfig==1.1.1
islpy==2021.1+aws2021.x.16.0.bld0
jmespath==1.0.1
joblib==1.1.0
keras==2.8.0
Keras-Preprocessing==1.1.2
libclang==14.0.1
Markdown==3.3.7
multiprocess==0.70.13
networkx==2.4
neuron-cc==1.11.4.0+97f99abe4
numpy==1.20.0
oauthlib==3.2.0
opt-einsum==3.3.0
packaging==21.3
pandas==1.4.3
pathos==0.2.9
pluggy==1.0.0
pox==0.3.1
ppft==1.7.6.5
protobuf==3.20.1
protobuf3-to-dict==0.1.5
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.9
pytest==7.1.2
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
regex==2022.6.2
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.8
s3transfer==0.6.0
sacremoses==0.0.53
sagemaker==2.98.0
scipy==1.4.1
six==1.16.0
smdebug-rulesconfig==1.0.1
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-neuron==2.4.0.0
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.26.0
tensorflow-neuron==2.8.0.2.3.0.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
tokenizers==0.10.3
tomli==2.0.1
torch==1.9.1
torch-neuron==1.9.1.2.3.0.0
tqdm==4.64.0
transformers==4.12.3
typing_extensions==4.3.0
urllib3==1.26.9
Werkzeug==2.1.2
wrapt==1.14.1
zipp==3.8.0
```

</details>


which leads to the same traceback (with the "small" snipet above, the same as in your guide): 

<details>
<summary>Traceback</summary>
<br>

```
2022-07-07 11:13:09.200830: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-07 11:13:09.258510: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-07 11:13:09.258664: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:Neuron:There are 3 ops of 1 different types in the TorchScript that are not compiled by neuron-cc: aten::embedding, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md)
INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 293, fused = 285, percent fused = 97.27%
INFO:Neuron:Number of neuron graph operations 837 did not match traced graph 723 - using heuristic matching of hierarchical information
INFO:Neuron:Compiling function _NeuronGraph$347 with neuron-cc
INFO:Neuron:Compiling with command line: '/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp94190olp/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp94190olp/graph_def.neff --io-config {"inputs": {"tensor.1:0": [[1, 128], "int64"], "1:0": [[1, 128, 384], "float32"], "2:0": [[1, 128, 384], "float32"], "3:0": [[1, 128, 384], "float32"]}, "outputs": ["BertEncoder_28/BertLayer_17/BertOutput_5/LayerNorm_7/aten_layer_norm/batchnorm/add_1:0", "BertPooler_29/Tanh_11/aten_tanh/Tanh:0"]} --verbose 35'
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
.2022-07-07 11:13:25.708020: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-07 11:13:25.744793: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-07 11:13:25.744998: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
07/07/2022 11:13:26 AM ERROR 18562 [neuron-cc]: Failed to parse model /tmp/tmp94190olp/graph_def.pb: The following operators are not implemented: {'SelectV2'} (NotImplementedError)

Compiler status ERROR
INFO:Neuron:Compile command returned: 1
WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$347; falling back to native python function call
ERROR:Neuron:neuron-cc failed with the following command line call:
/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp94190olp/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp94190olp/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[1, 128], "int64"], "1:0": [[1, 128, 384], "float32"], "2:0": [[1, 128, 384], "float32"], "3:0": [[1, 128, 384], "float32"]}, "outputs": ["BertEncoder_28/BertLayer_17/BertOutput_5/LayerNorm_7/aten_layer_norm/batchnorm/add_1:0", "BertPooler_29/Tanh_11/aten_tanh/Tanh:0"]}' --verbose 35
Traceback (most recent call last):
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 381, in op_converter
    neuron_function = self.subgraph_compiler(
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/decorators.py", line 219, in trace
    raise subprocess.SubprocessError(
subprocess.SubprocessError: neuron-cc failed with the following command line call:
/mnt/Documents/test/sagemaker_test/.venv/bin/neuron-cc compile /tmp/tmp94190olp/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmp94190olp/graph_def.neff --io-config '{"inputs": {"tensor.1:0": [[1, 128], "int64"], "1:0": [[1, 128, 384], "float32"], "2:0": [[1, 128, 384], "float32"], "3:0": [[1, 128, 384], "float32"]}, "outputs": ["BertEncoder_28/BertLayer_17/BertOutput_5/LayerNorm_7/aten_layer_norm/batchnorm/add_1:0", "BertPooler_29/Tanh_11/aten_tanh/Tanh:0"]}' --verbose 35
INFO:Neuron:Number of arithmetic operators (post-compilation) before = 293, compiled = 0, percent compiled = 0.0%
INFO:Neuron:The neuron partitioner created 1 sub-graphs
INFO:Neuron:Neuron successfully compiled 0 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 0.0%
INFO:Neuron:Compiled these operators (and operator counts) to Neuron:
INFO:Neuron:Not compiled operators (and operator counts) to Neuron:
INFO:Neuron: => aten::Int: 49 [supported]
INFO:Neuron: => aten::add: 21 [supported]
INFO:Neuron: => aten::contiguous: 6 [supported]
INFO:Neuron: => aten::div: 6 [supported]
INFO:Neuron: => aten::dropout: 19 [supported]
INFO:Neuron: => aten::embedding: 3 [not supported]
INFO:Neuron: => aten::gelu: 6 [supported]
INFO:Neuron: => aten::layer_norm: 13 [supported]
INFO:Neuron: => aten::linear: 37 [supported]
INFO:Neuron: => aten::matmul: 12 [supported]
INFO:Neuron: => aten::mul: 1 [supported]
INFO:Neuron: => aten::permute: 24 [supported]
INFO:Neuron: => aten::rsub: 1 [supported]
INFO:Neuron: => aten::select: 1 [supported]
INFO:Neuron: => aten::size: 49 [supported]
INFO:Neuron: => aten::slice: 5 [supported]
INFO:Neuron: => aten::softmax: 6 [supported]
INFO:Neuron: => aten::tanh: 1 [supported]
INFO:Neuron: => aten::to: 1 [supported]
INFO:Neuron: => aten::transpose: 6 [supported]
INFO:Neuron: => aten::unsqueeze: 2 [supported]
INFO:Neuron: => aten::view: 24 [supported]
Traceback (most recent call last):
  File "compile_model_2.py", line 20, in <module>
    model_neuron = torch.neuron.trace(model, neuron_inputs)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 184, in trace
    cu.stats_post_compiler(neuron_graph)
  File "/mnt/Documents/test/sagemaker_test/.venv/lib/python3.8/site-packages/torch_neuron/convert.py", line 492, in stats_post_compiler
    raise RuntimeError(
RuntimeError: No operations were successfully partitioned and compiled to neuron for this model - aborting trace!
```

</details>

Closing: The problem was on my side, thanks anyway for your help and time! :) 

All the "neuron packages" are installable and wheels for 3.8 exists with the repo: `https://pip.repos.neuron.amazonaws.com`
However, the installation leads to the wrong version of TensorFlow (neuron needs <2) that does not exists anymore. 

I cleaned everything, reinstalled with python3.7, and ran out of the box the two scripts above.