Help on training TFBERT to IntegerEncoded sequences
Author: victormaricato
Pull request: no

Hi,

My inputs are Integer Encoded vectors, like:
`[1,2,3,1,2,4,1,2,3,4,2,3,4, ...]`

Where:
`len(inputs) = 1200` & `unique values = 4` (1,2,3,4).

As you can see, this is not a common NLP problem as I have only 4 tokens, instead of a huge vocabulary.
And I could not find any pretrained vocab to tokenize this.

**What I am trying to do**:

I want to fit a BERT model to this sequence data.

**What have I tried**

I am using `tensorflow-2.4.0`, and here is my model:

```
class MyModel(Model):
    def __init__(self):
        super().__init__()
        self._bert = _create_bert_model()
        self._head = Dense(1)

    def call(self, inputs):
        embedding = self._bert(inputs)
        return self._head(embedding)


def _create_bert_model() -> TFBertModel:
    config = BertConfig(vocab_size=4+1)
    return TFBertModel(config)

```
    
As you can see, I want to create an "embedding" using BERT, and pass this to a head (regression).
    
## Issue
    
When I put a breakpoint inside this `call` method, here is what I get:
    
```
# Batch Size: 4 (just for debug)
>> inputs 
<tf.Tensor: shape=(4, 1200), dtype=int32, numpy=
array([[2, 4, 1, ..., 4, 3, 3],
       [2, 1, 4, ..., 1, 2, 4],
       [4, 2, 1, ..., 3, 1, 2],
       [2, 2, 4, ..., 1, 2, 1]], dtype=int32)>

>> self._bert(inputs)
*** tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [512,768] vs. [4,1200,768] [Op:BroadcastTo]
```

## Help

Can anyone provide me any guidance on solving this issue, and procceed with my modelling?

And, is there any tutorial I may find on fitting a HuggingFace Tokenizer/BERT on custom vocabulary?

## Reproduce

```
from transformers import TFBertModel, BertConfig
import numpy as np

x = np.random.randint(1,5,1200)

config = BertConfig(vocab_size=4+1)
model = TFBertModel(config)

model(x)
```

Thank you!

Comments:
I had to define the following parameters on `BertConfig`:

`config = BertConfig(vocab_size=4+1, hidden_size=len(x), max_position_embeddings=len(x))`

My issue is now:

`InvalidArgumentError: Index out of range using input dim 1; input has only 1 dims [Op:StridedSlice] name: tf_bert_model_8/bert/strided_slice/`

The input to BERT (`input_ids`) must be a tensor of shape `(batch_size, sequence_length)` = `(1, 1200)`.

However, currently the shape of your x is only `(1200,`). You should add a batch dimension, like so:

```
import numpy as np

x = np.random.randint(1,5,1200)
x = np.expand_dims(x, axis=0)
```

The embedding layer of BERT will then turn this into a tensor of shape `(batch_size, sequence_length, hidden_size)`, i.e. it will turn each of the 1200 integers into a vector of size `hidden_size`. As you're setting `hidden_size` equal to `len(x)`, this means that each integer will be turned into a vector of size 1200. Is this what you want (seems quite a big vector :p)?

Then the following will work:

```
from transformers import TFBertModel, BertConfig

config = BertConfig(vocab_size=4+1, max_position_embeddings=len(x))
model = TFBertModel(config)

model(x)
```

Btw, please ask questions which are not bugs or feature requests on the [forum](https://discuss.huggingface.co/) rather than here.

Thank you @NielsRogge. I must apologize. Should I cut this issue and paste into the forum?

I think I made a mistake... the vector size should not be that large and won't fit into memory :P

I managed to complete a model forward cycle. But, if you allow me another question:

Should I add an extra integer to the beginning of the sequence (of value != to the existing ones, e.g.: `5`), to act as the <CLS> token?

I am asking this because the `model(inputs)` returns a pooled `(batch_size, 1,hidden_size)` token, instead of `(batch_size, seq_length, hidden_size)`, however I am not sure if I should be passing this to the `Dense(1)` layer in the next step.

> Should I add an extra integer to the beginning of the sequence (of value != to the existing ones, e.g.: `5`), to act as the token?

Each integer acts as a token, so adding one more will increase the number of tokens by one. The `vocab_size` is the total number of tokens for which the model learns an embedding vector.

> I am asking this because the `model(inputs)` returns a pooled `(batch_size, 1,hidden_size)` token, instead of `(batch_size, seq_length, hidden_size)`, however I am not sure if I should be passing this to the `Dense(1)` layer in the next step.

If you use `TFBertModel`, then by default it returns a `TFBaseModelOutputWithPooling` object, with an attribute called `last_hidden_state`. This is a tensor of shape `(batch_size, sequence_length, hidden_size)` and this is probably what you want.

You may close the issue and if you have any further questions, feel free to ask them on the forum, we're happy to help!