What is the input for TFBertForSequenceClassification?
Author: sainimohit23
Pull request: no

# ❓ Questions & Help
What is the input for TFBertForSequenceClassification?
## Details
I have a simple multiclass text data on which I want to train the BERT model.

From docs I have found the input format of data:
```a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: model([input_ids, attention_mask]) or model([input_ids, attention_mask, token_type_ids])```
 
In my understanding:
`input_ids`- tokenized sentences, generated from BERT tokenizer.
`attention_mask`- As name suggests it is attention mask. I should use it to mask out padding tokens. Please correct me if I am wrong.
Now what is `token_type_ids'? is it necessary?

When I tried to print output_shape of the model? I got:
`AttributeError: The layer has never been called and thus has no defined output shape.`
So, let's say my dataset has 5 classes. Does this model expect one-hot encoded vector of shape [BATCH_SIZE, CLASSES] for .fit() method? 

Also if I don't use .from_pretrained() method, will it load an untrained model?

Comments:
Have a look at an example, for instance [`run_tf_glue.py`](https://github.com/huggingface/transformers/blob/master/examples/run_tf_glue.py). To better understand all the arguments, I advise you to read the [documentation](https://huggingface.co/transformers/model_doc/bert.html#bertmodel). You'll find that token_type_ids are

> Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]: 0 corresponds to a sentence A token, 1 corresponds to a sentence B token

So they're only practically useful if your input contains two sequences (for instance if you wish to model some relationship between sentence A and sentence B). In your case, it's probably not needed.

Hi @BramVanroy as you said I tried to run the code from  run_tf_glue.py. Yesterday it was working fine on google colab. But today when I tried to rerun the script. I am getting following error:

```
ImportError                               Traceback (most recent call last)
<ipython-input-7-63fb7d040ab0> in <module>()
      4 import tensorflow_datasets
      5 
----> 6 from transformers import (
      7     BertConfig,
      8     BertForSequenceClassification,

ImportError: cannot import name 'TFBertForSequenceClassification'
```

Looks like there was some issue in colab session. So, closing this.

@sainimohit23  Getting similar issue in local Jupyter notebook. 
"AttributeError: module 'transformers' has no attribute 'TFBertForSequenceClassification' " 
Looks like there is some changes in transformers package.

let me know if this is fixed..


> @sainimohit23 Getting similar issue in local Jupyter notebook.
> "AttributeError: module 'transformers' has no attribute 'TFBertForSequenceClassification' "
> Looks like there is some changes in transformers package.
> 
> let me know if this is fixed..

Are you using the latest version of transformers? Try updating, because it is right there in the source code:

https://github.com/huggingface/transformers/blob/5c3d441ee1dc9150ccaf1075eb0168bbfe28c7f9/src/transformers/modeling_tf_bert.py#L875

@BramVanroy Using latest version of transformers. Double checked. 
Let me know if there is any other issue.

Please find below details useful. 
```
`AttributeError                            Traceback (most recent call last)
<ipython-input-6-5c0ab52ed729> in <module>
----> 1 model = BertForSequenceClassification.from_pretrained('sentiment_model/',from_tf=True)  # re-load
      2 tokenizer = BertTokenizer.from_pretrained('sentiment_model/')

~\AppData\Roaming\Python\Python37\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    485                     from transformers import load_tf2_checkpoint_in_pytorch_model
    486 
--> 487                     model = load_tf2_checkpoint_in_pytorch_model(model, resolved_archive_file, allow_missing_keys=True)
    488                 except ImportError:
    489                     logger.error(

~\AppData\Roaming\Python\Python37\site-packages\transformers\modeling_tf_pytorch_utils.py in load_tf2_checkpoint_in_pytorch_model(pt_model, tf_checkpoint_path, tf_inputs, allow_missing_keys)
    223     # Instantiate and load the associated TF 2.0 model
    224     tf_model_class_name = "TF" + pt_model.__class__.__name__  # Add "TF" at the beggining
--> 225     tf_model_class = getattr(transformers, tf_model_class_name)
    226     tf_model = tf_model_class(pt_model.config)
    227 

AttributeError: module 'transformers' has no attribute 'TFBertForSequenceClassification'`
```

I went through the source code, and this should work _unless_ Tensorflow is not installed in your environment. In such a case, the Tensorflow models are not imported in __init__. Make sure that Tensorflow is installed.

https://github.com/huggingface/transformers/blob/0dbddba6d2c5b2c6fc08866358c1994a00d6a1ff/src/transformers/__init__.py#L287-L313

Hi @BramVanroy, Thanks for the help there was a miss match of tensorflow version, but it looks like the issue is something different. 
`RuntimeError: storage has wrong size: expected -273778883 got 768`

Either fine tuned model is corrupted or other issue. .

Thanks

Can you post the full trace?

@BramVanroy Please find the below details useful. 
Let me know what can be the issue. 
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-d609d3be6585> in <module>
      2 # model = BertForSequenceClassification.from_pretrained('sentiment_model/')
      3 
----> 4 model = BertForSequenceClassification.from_pretrained("sentiment_model/", num_labels=2)
      5 tokenizer = BertTokenizer.from_pretrained('sentiment_model/')

~\AppData\Roaming\Python\Python37\site-packages\pytorch_pretrained_bert\modeling.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    601         if state_dict is None and not from_tf:
    602             weights_path = os.path.join(serialization_dir, WEIGHTS_NAME)
--> 603             state_dict = torch.load(weights_path, map_location='cpu')
    604         if tempdir:
    605             # Clean up temp dir

~\AppData\Roaming\Python\Python37\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    384         f = f.open('rb')
    385     try:
--> 386         return _load(f, map_location, pickle_module, **pickle_load_args)
    387     finally:
    388         if new_fd:

~\AppData\Roaming\Python\Python37\site-packages\torch\serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    578     for key in deserialized_storage_keys:
    579         assert key in deserialized_objects
--> 580         deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
    581         if offset is not None:
    582             offset = f.tell()

RuntimeError: storage has wrong size: expected -273778883 got 768
```



@vijender412 i found this [comment](https://github.com/pytorch/pytorch/issues/12042#issuecomment-426466826) useful 

I don't use Tensorflow, but the documentation suggests that you should load your model like this:

https://github.com/huggingface/transformers/blob/20fc18fbda3669c2f4a3510e0705b2acd54bff07/src/transformers/modeling_utils.py#L366-L368

@ArashHosseini  gone through that but was not able to link my code.

@BramVanroy  
The fine tuned model was saved using'
```
model.save_pretrained('./sentiment_model/')
tokenizer.save_pretrained('./sentiment_model/')
```
And files created were (config.json,pytorch_model.bin,special_tokens_map.json,tokenizer_config.json,vocab.txt) So no checkpoint were created wrt to tensorflow. 

Now as per documentation the loading should be 
```
model = BertForSequenceClassification.from_pretrained("sentiment_model/", num_labels=2)
tokenizer = BertTokenizer.from_pretrained('sentiment_model/')
```
The tokenizer is getting loaded but getting issues while loading model. 
"RuntimeError: storage has wrong size: expected -273778883 got 768"


Then why did you say in your original comment that you had a Tensorflow mismatch? 

I am not sure why this happens. Please open your own topic, and provide all necessary information from the template.

@BramVanroy Earlier I was getting this issue
`AttributeError: module 'transformers' has no attribute 'TFBertForSequenceClassification'`
which got resolved by changing tensorflow version to 2.0. 
**For current issue l will create a new issue after tracing out my code from scratch.**

> @ArashHosseini gone through that but was not able to link my code.
> 
> @BramVanroy
> The fine tuned model was saved using'
> 
> ```
> model.save_pretrained('./sentiment_model/')
> tokenizer.save_pretrained('./sentiment_model/')
> ```
> 
> And files created were (config.json,pytorch_model.bin,special_tokens_map.json,tokenizer_config.json,vocab.txt) So no checkpoint were created wrt to tensorflow.
> 
> Now as per documentation the loading should be
> 
> ```
> model = BertForSequenceClassification.from_pretrained("sentiment_model/", num_labels=2)
> tokenizer = BertTokenizer.from_pretrained('sentiment_model/')
> ```
> 
> The tokenizer is getting loaded but getting issues while loading model.
> "RuntimeError: storage has wrong size: expected -273778883 got 768"

Hi, I meet the same issue(can't load state_dict after saving it), Have you solve it?