The output of IBERT is float32. Am I doing wrong?
Author: kyoungrok0517
Pull request: no

## Environment info
- `transformers` version: 4.5.1
- Platform: Linux-5.8.0-49-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: DDP (pytorch-lightning)

### Who can help
@LysandreJik, @patil-suraj, @patrickvonplaten


## Information

I'm trying IBert. The first output of the model is `float32` so I'm curious why it happens. I set `quant_mode=True`. 

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I'm using MSMARCO (IR dataset)

## To reproduce

Steps to reproduce the behavior:

1. Initialize a model with the command `AutoModel.from_pretrained('kssteven/ibert-roberta-base', quant_mode=True, add_pooling_layer=False)`
2. Check the `dtype` of the model output.

## Expected behavior

The output `dtype` should be `int8`, but I see `float32`


Comments:
The I-BERT framework allows for easy fine-tuning in PyTorch to find the optimal parameters. Once those are found, the model can be deployed in a setup using int-8 capable operations such as TensorRT.

@kssteven418 will be able to explain better than me.

Yes, the current I-BERT implementation (both in HF and Fairseq in my personal repo) only searches for the optimal int8 parameters through quantization-aware training and leaves out the actual model deployment. That is to say, it simulates int8 inference using floating-point representations and operations. One reason we are not supporting int8 execution is that, PyTorch only supports int8 inference via its own quantization APIs. Therefore, the optimal parameters found in the I-BERT framework must be then exported to different frameworks that can support int8 deployment (TensorRT, TVM, etc. are some popular frameworks). We haven't yet open-sourced the code for model deployment.

@kssteven418 Thanks for the answer. What I want is the final output in `int8`. Then will it be ok to just cast the output into `int8` even with pytorch? The whole execution doesn't have to be run in integer mode.

Yes, if you look at the quantization modules, e.g., QuantLinear, there are additional attributes such as `weight_integer` that represent the int8 model parameters in the torch.float. You can cast those numbers to torch.int8, but just make sure that you don't round down the numbers - they must be rounded.

> Yes, if you look at the quantization modules, e.g., QuantLinear, there are additional attributes such as `weight_integer` that represent the int8 model parameters in the torch.float. You can cast those numbers to torch.int8, but just make sure that you don't round down the numbers - they must be rounded.

Thank you very much. This will be the last question I expect :)  

The following is how I defined my network and how the input passes flows. Briefly, I want the output of `IBertModel` to pass one more `QuantLinear` layer to get the final representation, like this. 
1. input -> `IBertModel`
1. `IBertModel` -> `QuantAct` 
1. `QuantAct`  -> `QuantLinear` -> `final_representation`

But the `QuantLinear` module outputs two tensors: `quant_x` and `scaling_factor`. 

**Do I have to deal with both, or can I just use `quant_x` as the final representation?** 

```python
return (
    F.linear(x_int, weight=self.weight_integer, bias=self.bias_integer) * bias_scaling_factor, # use only this?
    bias_scaling_factor, # or do I also need this?
)
```

This is my code.
```python
# Define network
self.pre_linear = QuantAct(self.act_bit, quant_mode=self.quant_mode)

self.linear = QuantLinear(
    self.input_size,
    self.n,
    quant_mode=self.quant_mode,
)

...

# Generate output
x, pre_scaling_factor = self.pre_linear(x)
x, scaling_factor = self.linear(x, pre_scaling_factor)

# x = x * scaling_factor?
```

As you have also noticed, all the quant modules including QuantLinear return two tensors: `quant_x` and `scaling_factor`. Here, `quant_x` / `scaling_factor` represents the quantized (integer) value for the activation - in other words, `quant_x` is the dequantized value. Therefore, you do not have to multiply it with the `scaling_factor`.  

Hi!

I would like to deploy IBERT on a framework like TensorRT. I am a complete beginner in that field and I was wandering if someone could give me some tips on the main steps of how to quantize IBERT? 

Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.