Fused_norm_layer_cuda
Author: Sagar1094
Pull request: no

Hi, 
I have created a model on kaggle tpu using run_pretraining.py and it gives me tf_checkpoint file. First of all the model is 12gb in size due to which it is throwing memory error while loading weights into it on kaggle and colab.
Is there a way to reduce the size?

Secondly, I tried performing the operation over a linux server without Gpu so it throws an error no module "fused_norm_layer_cuda" which is obvious but I want to know if there is a way to convert the model from tf to pytorch without GPU, or is there any parameter in BertForPreTraining which can instantiate the model without GPU. Please help. 🤗

Comments:
Hi! What's your model? Do you have the configuration file?

What is the `run_pretraining.py`? This doesn't seem to be one of our scripts.

Hi, extremely sorry for not providing adequate information.

I used google-research/bert for creating the model. The bert repo contains the run_pretraining.py, before that I created the vocab.txt file from my own data and after that I used create_pretraining_data.py and yes I do have the config.json file as well.

I am trying to convert tf_checkpoint file to pytorch_model.bin where I am encountering the issue of memory error on Kaggle and Colab and fused_norm_layer_cuda error on linux server without GPU. Just want to know if it is possible to convert from tf_checkpoint to pytorch without GPU or any way I can reduce the model size to load weights on kaggle or colab without running out of memory

Do you mind sharing the configuration file?

So you pre-trained a model using google-research/bert, and now you're trying to convert it to one of our models, is that correct?

{
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 384,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 256,
  "num_attention_heads": 12,
  "num_hidden_layers": 6,
  "type_vocab_size": 2,
  "vocab_size": 2800000
}
This is the config.json file.

Not exactly, I have used bert repo to create a model which is in tensorflow and have files model.ckpt-100000.data-00000-of-00001 which seems to be in tensorflow but I need model_pytorch.bin so converting from tf_checkpoint file to pytorch using this amazing library. Trying to use the code written in convert_bert_original_tf_checkpoint_to_pytorch.py

Right, you're using the correct script!

Please note, that having a 2_800_000 vocabulary size is absolutely huge!

Unfortunately it's hard to get around memory errors during conversion. The script should work whether you have a GPU available or not. Can you show me the command you used to convert the model on your linux server?

I have little data and many unique terms which are written in English but are addresses from all over India due to which I have to keep the vocabulary size this high and due to which I compromised with the dimensions of the model and even the embedding size as well.

Here is the code snippet:- 

import torch

from pytorch_transformers.modeling_bert import BertConfig, BertForPreTraining, load_tf_weights_in_bert


tf_checkpoint_path="./distlang/"
bert_config_file = "./config.json"
pytorch_dump_path="./distlangpytorch"

config = BertConfig.from_json_file(bert_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = BertForPreTraining(config)

# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)

Its, throwing error while instantiation of the model model = BertForPreTraining(config) 

I have apex installed but without cuda_ext as my server is not having GPU, so it won't install 😅

Hmm I can't reproduce on my setup. Two things:

- Can you update your `transformers` library? It seems that you're using `pytorch-transformers` which is starting to be quite old now. We've patched quite a few bugs since then! > `pip install  -U transformers` and update your imports to `from transformers import`
- Do you mind pasting the stack trace?

If I use thrasformers library will it work fine?
And yes here the stack trace:-
Building PyTorch model from configuration: {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 384,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 256,
  "num_attention_heads": 12,
  "num_hidden_layers": 6,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "vocab_size": 2800000
}

Traceback (most recent call last):
  File "tftopytorch.py", line 18, in <module>
    model = BertForPreTraining(config)
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 761, in __init__
    self.bert = BertModel(config)
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 651, in __init__
    self.embeddings = BertEmbeddings(config)
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/pytorch_transformers/modeling_bert.py", line 240, in __init__
    self.LayerNorm = BertLayerNorm(config.hidden_size, eps=config.layer_norm_eps)
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/apex/normalization/fused_layer_norm.py", line 133, in __init__
    fused_layer_norm_cuda = importlib.import_module("fused_layer_norm_cuda")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'fused_layer_norm_cuda'

Yes, upgrading the library should solve your problems. Since [this commit](https://github.com/huggingface/transformers/commit/98dd19b96b351f481e1268ab6c7b035bb21d106e) we're not using apex for the LayerNorm anymore, so having PyTorch installed on a recent version alongside `transformers` on a recent version should solve your issue!

Turns out the problem is solved. Thanks a lot man for saving the day. Amazing library as well 🤗 really appreciate your quick response and understanding my problem and solving it.

My pleasure :hugs: 

Hey the above problem is solved but I am running into this problem while writing the file to the directory, to my understanding I have to provide the path where I have to save the pyorch model and I am providing a path. Am I missing out something here? 

Here's the snippet of the error the code is same as mentioned above just used transformers instead of pytorch_transformer

Save PyTorch model to ./distlangpytorch/
Traceback (most recent call last):
  File "tftopytorch.py", line 25, in <module>
    torch.save(model.state_dict(), pytorch_dump_path)
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/torch/serialization.py", line 369, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/torch/serialization.py", line 234, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/rohit.anand/envs/data_augment/lib/python3.6/site-packages/torch/serialization.py", line 215, in __init__
    super(_open_file, self).__init__(open(name, mode))
IsADirectoryError: [Errno 21] Is a directory: './distlangpytorch/'

We usually recommend saving using our method `from_pretrained`, as it saves the configuration as well as the model state dict.

Can you try using the following?

```py
model = load_tf_weights_in_bert(model, config, tf_checkpoint_path)
model.save_pretrained(pytorch_dump_path)
```

It worked fine, the model is saved now and the size of the model is 1/3rd the original size which was of the tf_checkpoint. Is the model quantized as well, is it something implemented in the library? Also thanks a ton have been struggling since evening and once posted the issue here got the resolution immediately. 🤗

That's great! The model is not quantized, having a 2/3 reduction is surprising. It's possible that your checkpoint originally had optimizer states which do take quite a big amount of memory. We don't save that in our checkpoints.

Cool, glad I could help!

Hey, its not related to the issue can you please help me with post training quantization. I only am able to see tutorials of dynamic quantization which indeed increases the size of my model. I am not able to find any decent tutorials on post training(static) quantization. Tried torch.quantization.quantize but the method asks for two positional arguments fn_run and fn_args and I am not sure how to define them or create a function to pass to these arguments

So, it turns out I might have figured it out. The model size is 4.3 GB my vocab size is 28_000_00 and hidden layer size is 384. So the vocab part of model turn out to be 4X2800000X384 = 4.3GB. What I don't understand is why the model size only have vocabulary part and no part of weights strange. And is there any free resource where I can train this model for classification you might be aware of 🙈, On colab and kaggle the kernel restarts because they only have 12,16 GB RAM, the model needs a bit more 🙈