Training TFBertForSequenceClassification with DataFrame instead of tensorflow_datasets
Author: konstantin-doncov
Pull request: no
Labels: name: wontfix

I'm trying to fine tune `transformers` with my own dataset in the `csv` file. So, I found [an item in the docs which shows a basic usage example](https://huggingface.co/transformers/training.html#fine-tuning-in-native-tensorflow-2). But the main problem is that this example shows how to use transformers with the `tensorflow_datasets`, but not with the something more real, like with the `pandas` `Dataframe`. So, I have a problem with the `transformers` usage:
```
df = pd.DataFrame({'text': ['SOME ANGRY TEXT!!!', 'Some friendly text :)'], 'label': [1, 0]})

bert_model = transformers.TFBertForSequenceClassification.from_pretrained("bert-base-cased")
bert_tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-cased")

x = bert_tokenizer.batch_encode_plus(
    df.text.values,
    max_length=10,
    add_special_tokens=True,
    pad_to_max_length=True,
)

bert_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['Accuracy'])

bert_history = bert_model.fit(
 x=x,
 y=df.label.values.reshape(-1, 1),
)
```

Output:
```
ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {'(<class \'list\'> containing values of types {"<class \'int\'>"})'}), <class 'numpy.ndarray'>
```
**So, how can I use my custom `pandas` `DataFrame` as x and y or as dataset?**

Also, you need to understand that I found a few questions on the SO which ask almost the same question and has upvotes, [here is one of the last question which I found](https://stackoverflow.com/questions/60463829/training-tfbertforsequenceclassification-with-custom-x-and-y-data). **So, maybe people will find MWE with the `np` or `pd` useful?**

Comments:
I also find out that every example which uses transformers with `binary_crossentropy` loss changes the pretrained model(i.e. adds some layers). Is it necessary? If so, then what minimal working example for this? 

Now I'm just trying to copy [this Kaggle notebook](https://www.kaggle.com/definedennis/pretrained-bert-with-huggingface-tensorflow-2-1), here is my code:
```
df = pd.DataFrame({'text': ['SOME ANGRY TEXT!!!', 'Some friendly text :)'], 'label': [1, 0]})

def create_model():
    bert_model = transformers.TFBertModel.from_pretrained("bert-base-cased")
    
    input_ids = tf.keras.layers.Input(shape=(10,), dtype=tf.int32, name='input_ids')
    token_type_ids = tf.keras.layers.Input((10,), dtype=tf.int32, name='token_type_ids')
    attention_mask = tf.keras.layers.Input((10,), dtype=tf.int32, name='attention_mask')
    
    # Use pooled_output(hidden states of [CLS]) as sentence level embedding
    pooled_output = bert_model({'input_ids': input_ids, 'attention_mask': attention_mask, 'token_type_ids': token_type_ids})[1]
    x = tf.keras.layers.Dropout(rate=0.1)(pooled_output)
    x = tf.keras.layers.Dense(1, activation='sigmoid')(x)
    model = tf.keras.models.Model(inputs={'input_ids': input_ids, 'attention_mask': attention_mask, 'token_type_ids': token_type_ids}, outputs=x)
    return model

bert_model = create_model()
bert_tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-cased")

x = bert_tokenizer.batch_encode_plus(
    df.text.values,
    max_length=10,
    pad_to_max_length=True, 
    return_tensors='tf'
)

bert_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['Accuracy'])

bert_history = bert_model.fit(
    x=x,
    y=df.label.values
)
```

Output:
```
~/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py in __hash__(self)
    724     if (Tensor._USE_EQUALITY and executing_eagerly_outside_functions() and
    725         (g is None or g.building_function)):
--> 726       raise TypeError("Tensor is unhashable. "
    727                       "Instead, use tensor.ref() as the key.")
    728     else:

TypeError: Tensor is unhashable. Instead, use tensor.ref() as the key.
```

How can I fix it?

Now I just use the previous Kaggle notebook, even without code modifications. My code:

```
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from tqdm.notebook import tqdm

import tensorflow as tf
from tensorflow import keras
import tensorflow.keras.backend as K
from tensorflow.keras import layers
from tensorflow.keras.utils import plot_model
from transformers import (
    BertTokenizer,
    TFBertForSequenceClassification,
    TFBertModel,
    BertConfig,
)
tf.__version__

MAX_SEQUENCE_LENGTH = 255
PRETRAINED_MODEL_NAME = 'bert-base-uncased'
BATCH_SIZE = 32

df = pd.read_csv('train.csv')

df.head()

df['target'].value_counts()

df.isnull().sum()

data = df['text'].values
targets = df['target'].values

def create_model():
    bert_model = TFBertModel.from_pretrained(PRETRAINED_MODEL_NAME)
    
    input_ids = layers.Input(shape=(MAX_SEQUENCE_LENGTH,), dtype=tf.int32, name='input_ids')
    token_type_ids = layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32, name='token_type_ids')
    attention_mask = layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32, name='attention_mask')
    
    # Use pooled_output(hidden states of [CLS]) as sentence level embedding
    pooled_output = bert_model({'input_ids': input_ids, 'attention_mask': attention_mask, 'token_type_ids': token_type_ids})[1]
    x = layers.Dropout(rate=0.1)(pooled_output)
    x = layers.Dense(1, activation='sigmoid')(x)
    model = keras.models.Model(inputs={'input_ids': input_ids, 'attention_mask': attention_mask, 'token_type_ids': token_type_ids}, outputs=x)
    return model

tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)
model = create_model()

model.summary()

plot_model(model, to_file='model.png', expand_nested=True, show_shapes=True)

opt = tf.keras.optimizers.Adam(learning_rate=3e-5)
model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])

X_train, X_val, y_train, y_val = train_test_split(data, targets, test_size=0.33, random_state=42, stratify=targets)

X_train = tokenizer.batch_encode_plus(X_train, max_length=MAX_SEQUENCE_LENGTH, pad_to_max_length=True, return_tensors='tf')
X_val = tokenizer.batch_encode_plus(X_val, max_length=MAX_SEQUENCE_LENGTH, pad_to_max_length=True, return_tensors='tf')

history = model.fit(
    x=X_train,
    y=y_train,
    validation_data=(X_val, y_val),
    epochs=3,
    batch_size=BATCH_SIZE
)
```

Output:
```
/usr/lib/python3.8/_collections_abc.py in update(self, other, **kwds)
    835                 self[key] = other[key]
    836         else:
--> 837             for key, value in other:
    838                 self[key] = value
    839         for key, value in kwds.items():

ValueError: too many values to unpack (expected 2)
``` 

You can check [this](https://colab.research.google.com/drive/1DT6lSWRZ3CIIm9noaJxPYjtOavWQB23S?usp=sharing) and [this](https://colab.research.google.com/drive/125jJ0qrXGIe6goNrH_Ja7XPZtYp7nMXU?usp=sharing) gists for reproduction of the errors.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.