Each IBertLayer receives the same `hidden_states_scaling_factor` in `IBertEncoder.forward()`
Author: mbekmyrz
Pull request: no

**Problem/Question:**
Why each layer receives the same `hidden_states_scaling_factor` in `IBertEncoder.forward()`? https://github.com/huggingface/transformers/blob/fc63914399b6f60512c720959f9182b02ae4a45c/src/transformers/models/ibert/modeling_ibert.py#L584-L590

In fact, `layer_output_scaling_factor` is not even passed as one of the outputs in `IBertLayer.forward()`: https://github.com/huggingface/transformers/blob/fc63914399b6f60512c720959f9182b02ae4a45c/src/transformers/models/ibert/modeling_ibert.py#L532-L537

**Solution:**
in IBertEncoder:
```
layer_outputs, hidden_states_scaling_factor = layer_module(
                hidden_states,
                hidden_states_scaling_factor,
                attention_mask,
                layer_head_mask,
                output_attentions,
            )
```

in IBertLayer:
```
return outputs, layer_output_scaling_factor
```

**Reasoning:**

Following the paper explanations in https://arxiv.org/abs/2101.01321, it seems that equations with quantized inputs should follow the below pattern:

$x_0$ - is real valued input variable
$q_0$ - is integer valued quantized input variable
$S_0$ - is a scaling factor

$x_0 = S_0q_0$ - according to the paper

$i$ - is the layer index
for each layer: $x_i, S_i = Layer(x_{i-1}, S_{i-1})$

where inside a layer: 
$q_{in} = x_{in} / S_{in}$ - get integer values of the layer inputs
$q_{out}, S_{out} = IntFunction(q_{in}, S_{in})$ - some function that performs integer arithmetic
$x_{out} = q_{out} * S_{out}$
and returns $x_{out}, S_{out}$

However, right now, according the code in `modeling_ibert.py`, specifically `IBertLayer` and `IBertEncoder`, the equation looks like this:

for each layer: $x_i = Layer(x_{i-1}, S_0)$ - doesn't even take new scaling factor, just passes the same $S_0$ to each layer

where inside a layer: $q_i = x_i / S_0$

Which seems to be not correct, because each layer is getting the same scaling factor, which is initial $S_0$.

Example with two layers, where each layer is just a Linear layer (or 'QuantLinear'):

given $x_0$, $S_0$ as inputs
Layer 1:
$x_1 = Layer(x_0, S_0) $
inside a layer:
$x_{out} = x_{in} * w_1$ - w is the real valued weights, let's say $v$ and $F$ are the quantized weights and weights scaling factor respectively.
$x_{out} = q_{in} * S_{in} * v_1 * F_1$
$x_{out} = (q_{in} * v_1) * (S_{in} * F_1)$
$x_{out} = q_{out} * S_{out}$
$S_{out} = S_{in} * F_1$
layer outputs are:
$x_1 = (q_0 * v_1) * S_1$
$S_1 = S_0 * F_1$

Layer 2:
$x_2 = Layer(x_1, S_0) $
similarly layer outputs are:
$x_2 = (q_1 * v_2) * S_2$
$S_2 = S_0 * F_2$

Whereas, $S_2$ should be $S_2 = S_1 * F_2$ 


Comments:
cc @kssteven418 as the designer of the model and porter of the code :)

thanks @LysandreJik for reply.

hello @kssteven418, it seems original source code also passes the same scaling factor to each layer:
https://github.com/kssteven418/I-BERT/blob/1b09c759d6aeb71312df9c6ef74fa268a87c934e/fairseq/modules/transformer_sentence_encoder.py#L304-L305

I see that there is an `input_act` inside each `TransformerSentenceEncoderLayer` before applying `self_attn` which is different from huggingface implementation where it has this QuantAct at the end of the layer as `output_activation` in `IBertOutput`. Although, `input_act` in each layer still uses the same scaling factor to calculate quantized integer value: https://github.com/kssteven418/I-BERT/blob/1b09c759d6aeb71312df9c6ef74fa268a87c934e/fairseq/quantization/utils/quant_utils.py#L233


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.