ValueError: Cannot reshape a tensor - TFBertForSequenceClassification
Author: DMells
Pull request: no
Labels: name: wontfix

I'm building a multiclass text classification model using Keras and BERT.

To convert my inputs to the required bert format, I'm using the `encode_plus` method found in the BertTokenizer class [found here][1]

The data is a paragraph of sentences per feature, and has a single label (of 45 labels in total)

**The code to convert the inputs is :**

    def create_input_array(df, tokenizer):
    sentences = df.text.values
    labels = df.label.values

    input_ids = []
    attention_masks = []
    token_type_ids = []

    # For every sentence...
    for sent in sentences:
        # `encode_plus` will:
        #   (1) Tokenize the sentence.
        #   (2) Prepend the `[CLS]` token to the start.
        #   (3) Append the `[SEP]` token to the end.
        #   (4) Map tokens to their IDs.
        #   (5) Pad or truncate the sentence to `max_length`
        #   (6) Create attention masks for [PAD] tokens.

        encoded_dict = tokenizer.encode_plus(
            sent,  # Sentence to encode.
            add_special_tokens=True,  # Add '[CLS]' and '[SEP]'
            max_length=128,  # Pad & truncate all sentences.
            pad_to_max_length=True,
            return_attention_mask=True,  # Construct attn. masks.
            return_tensors='tf',  # Return tf tensors.
        )

        # Add the encoded sentence to the list.
        input_ids.append(encoded_dict['input_ids'])

        # And its attention mask (simply differentiates padding from non-padding).
        attention_masks.append(encoded_dict['attention_mask'])

        token_type_ids.append(encoded_dict['token_type_ids'])

    return [np.asarray(input_ids, dtype=np.int32),
            np.asarray(attention_masks, dtype=np.int32),
            np.asarray(token_type_ids, dtype=np.int32)]

**The model in it's most basic form which still reproduces the error:**

    model = TFBertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels = labellen,
    output_attentions = False, 
    output_hidden_states = False
    )

**Compile and fit:**

    optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3, epsilon=1e-08, clipnorm=1.0)
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
    model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
    model.fit(x_train, y[:100], epochs=1, batch_size=3)

**The error when I run this :**

    

> ValueError: Cannot reshape a tensor with 768 elements to shape
> [1,1,128,1] (128 elements) for '{{node
> tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape}}
> = Reshape[T=DT_FLOAT, Tshape=DT_INT32](tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape/ReadVariableOp,
> tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape/shape)'
> with input shapes: [768], [4] and with input tensors computed as
> partial shapes: input[1] = [1,1,128,1].

I understand that BERT converts every token into a 768 value array, but that is the only knowledge I have of that particular number, so I'm stuck on how to proceed.

I would also appreciate your thoughts on whether TFBertForSequenceClassification is appropriate for paragraph classification.

Many thanks.

  [1]: https://huggingface.co/transformers/main_classes/tokenizer.html


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.