Add initializer_std to TFFunnelModelTester with a default value 0.02
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR sets `initializer_std=0.02` in `TFFunnelModelTester`, so we can use `1e-5` as the threshold in PT/TF equivalence test.
(so the inconsistencies are less likely to be undetected)

TF: @gante @Rocketknight1 


## Details

The `test_pt_tf_model_equivalence` in `test_modeling_tf_common.py` has:

- create Pytorch/TensorFlow models using a config (defined in each TF test script)
- load the Pytorch model to TensorFlow model
- load the TensorFlow model (already changed in the previous step) to the Pytorch model

See
https://github.com/huggingface/transformers/blob/cdc51ffd27f8f5a3151da161ae2b5dbb410d2803/tests/test_modeling_tf_common.py#L359-L366

For `Funnel` model:

- In `test_modeling_tf_funnel.py`, we don't use `initializer_std`.
- In `config_funnel.py`, we have [initializer_std=None](https://github.com/huggingface/transformers/blob/cdc51ffd27f8f5a3151da161ae2b5dbb410d2803/src/transformers/models/funnel/configuration_funnel.py#L124)
- In `modeling_funnel.py`, we have [std = 1.0 if self.config.initializer_std is None](https://github.com/huggingface/transformers/blob/cdc51ffd27f8f5a3151da161ae2b5dbb410d2803/src/transformers/models/funnel/modeling_funnel.py#L780)

Therefore, the Pytorch Funnel model created for the testing uses `std=1.0` to create `FunnelEmbeddings` (and the weights are loaded to TF model). This has the following effect for `TFFunnelForMaskedLM`:

- while all the hidden states have a PT/TF difference in the range `1e-7 ~ 2e-6`, the `logits` will have a larger difference (say `2e-5`) with a higher probability - due to the larger magnitude of embedding weights.

The (extended) PT/TF equivalence test used to search for inconsistency uses `1e-5` as threshold. So far, the equivalence tests failed with this threshold have proven to be due to some PT/TF inconsistency (i.e. not due to randomness). And after some fixes, they all have differences < `1e-5`.

In order to continue to use `1e-5` as threshold (so the inconsistencies are less likely to be undetected), this PR sets `initializer_std=0.02` in `TFFunnelModelTester`.

## Remark

The weight initialization logic for `TFFunnelModel` diverges from the `FunnelModel`, and would be good to fix it in another PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for the PR, @ydshieh!

I was looking at the code, and `initializer_std` is only used in the PT side of Funnel (codebase search [here](https://github.com/huggingface/transformers/search?q=initializer_std)). Do you know how setting this variable actually changes the behavior of the test? I can't think of any way it influences it 🤔 

> Thank you for the PR, @ydshieh!
> 
> I was looking at the code, and `initializer_std` is only used in the PT side of Funnel (codebase search [here](https://github.com/huggingface/transformers/search?q=initializer_std)). Do you know how setting this variable actually changes the behavior of the test? I can't think of any way it influences it 🤔

Hi, @gante 

In order to see the reason, we need to look this part in `test_pt_tf_model_equivalence` inside `test_modeling_tf_common.py`:

https://github.com/huggingface/transformers/blob/cdc51ffd27f8f5a3151da161ae2b5dbb410d2803/tests/test_modeling_tf_common.py#L359-L366

- The line `pt_model = pt_model_class(config)` will be affected by setting or not `initializer_std`.
- `tf_model = transformers.load_pytorch_model_in_tf2_model( ...` means `tf_model` is affected by `pt_model` above


(And as you already see, the weight initialization logic for `TFFunnelModel` diverges from the `FunnelModel` (at least, `initializer_std` is not used for `TFFunnelModel` now). But this is not the concern of this PR, and can be fixed in another PR.)



Oh, I see -- the TF config is passed to PT in that particular test, therefore making use of the variable. Thank you for elaborating :)

In that case, do you know why setting it to `0.02` (and not `1.0`, the default) does the trick? Could it be because TF's default initializer is different?

> Oh, I see -- the TF config is passed to PT in that particular test, therefore making use of the variable. Thank you for elaborating :)
> 
> In that case, do you know why setting it to `0.02` does the trick? Could it be because TF's default initializer is different?

For the test `test_pt_tf_model_equivalence` called by TF model test scripts:
- we use the config prepared in TF test scripts
- the config is used to initialize a PT and a TF model 
- then, we load PT model into TF model

So, TF  initializer plays no role in this test. It is the PT initializer used to create/init the weights, then loaded to TF model.
Since `FunnelModel` uses `initializer_std`, and its default value in `FunnelConfig` is `None` (so `FunnelModel` will use `1.0` for it, see [here](https://github.com/huggingface/transformers/blob/cdc51ffd27f8f5a3151da161ae2b5dbb410d2803/src/transformers/models/funnel/modeling_funnel.py#L780)), it will generate weights for word embeddings with a larger magnitude.

This PR is **NOT** to fix a bug though: Just to use a smaller `initializer_std` to get `smaller word embedding weights` --> in order to keep low error threshold `1e-5` for the PT/TF equivalence test.


More details:

The larger word embeddings will make the difference of `logits` (which uses the shared word embedding, I think) between PT & TF FunnelForMaskedLM larger, more than `1e-5`, and fail the (extended/strict) PT/TF equivalence

@ydshieh thank you for the clarification <3 Are you okay to merge as it is?

Yes, you can go ahead :-) Thanks for reviewing. (This one is tricky: took me quite some time to fully understand the cause - I asked myself all your questions during the process too. )

@ydshieh I've found this: https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L2048

I believe it explains the 0.02 :)

> @ydshieh I've found this: https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L2048
> 
> I believe it explains the 0.02 :)

Yes, that's the usual value for std in transformers models!
But I **think** the default is 0.02 here comes from the fact that the configurations usually have `initializer_range=0.02`, see

https://github.com/huggingface/transformers/blob/60ba48205e4ec070780e8cb8d461421b77432bad/src/transformers/models/bert/configuration_bert.py#L89

(and since BERT is the 1st model in `transformers`, it is continuously used - unless a particular architecture uses different value in the paper)

--

However, for this PR, I choose `0.02` just because this is the common value in the library.
- any value < 0.02, say `1e-3` will make the test pass .
- but we can't set a too small value, otherwise the PT/TF difference (`FunnelForMaskedLM` here) will definitely be small (break the purpose of testing)