Issue 1:
System Info

transformers version: 4.22.1

Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.28

Python version: 3.9.5

Who can help?

@Rocketknight1

Information

The official example scripts

My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)

My own task or dataset (give details below)

Reproduction

Using tf.keras.Model version of gpt2, and training as casual language modelling.

gpt2_model.compile(..., metrics=[PerplexityMetric()]) #using inner loss (hf_compute_loss)
gpt2_model.fit(...)

The problem occurs when metric is calculated. I did not come up with an idea to use inner loss of transformers in tf.keras.metric.Metric (if anyone knows how it's good to rid of duplicate calculation) . So calculating loss as written in def hf_compute_loss in TFCausalLanguageModelingLoss. PerplexityMetric as:

class PerplexityMetric(tf.keras.metrics.Metric):
def __init__(self, name='perplexity', **kwargs):
super(PerplexityMetric, self).__init__(name=name, **kwargs)
self.cross_entropy = tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True, reduction=tf.keras.losses.Reduction.NONE
)
self.perplexity = self.add_weight(name='tp', initializer='zeros')

def _calculate_perplexity(self, real, pred):
unmasked_loss = self.cross_entropy(tf.nn.relu(real), pred)
# make sure only labels that are not equal to -100 affect the loss
loss_mask = tf.cast(real != -100, dtype=unmasked_loss.dtype)
masked_loss = unmasked_loss * loss_mask
reduced_masked_loss = tf.reduce_sum(masked_loss) / tf.reduce_sum(
loss_mask
)
loss_ = tf.reshape(reduced_masked_loss, (1,))
perplexity = tf.math.exp(loss_[-1])
return perplexity

def update_state(self, y_true, y_pred, sample_weight=None):
perplexity = self._calculate_perplexity(y_true, y_pred)
self.perplexity.assign(perplexity)

def result(self):
return self.perplexity

def reset_state(self):
# reset at the start of each epoch.
self.perplexity.assign(0.0)

Expected behavior

Both inner loss calculation and the loss calculation in the metric are same. What I expect is that calculated losses are the same in training output. Last float within brackets is loss calculated in PerplexityMetric() (loss_).

1/7 [===>..........................] - ETA: 2s - loss: 8.0234 - perplexity: 1708.3318[7.44036245]
2/7 [=======>......................] - ETA: 2s - loss: 8.1277 - perplexity: 1703.3676[7.78889942]
3/7 [===========>..................] - ETA: 1s - loss: 8.2354 - perplexity: 2413.6597[7.75492764]
4/7 [================>.............] - ETA: 1s - loss: 8.3068 - perplexity: 2333.0405[7.91714859]
5/7 [====================>.........] - ETA: 0s - loss: 8.3837 - perplexity: 2743.9358[7.48942709]
6/7 [========================>.....] - ETA: 0s - loss: 8.3680 - perplexity: 1789.0269[7.8069911]

I did not understand the difference between loss values. Is there any wrong calculation in PerplexityMetric or is this a buggy behaviour?

Issue 2:
GPU:v100

run run_glue.py with the command in the README:

python ./examples/run_glue.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $GLUE_DIR/$TASK_NAME \
--max_seq_length 128 \
--per_gpu_eval_batch_size=8 \
--per_gpu_train_batch_size=8 \
--learning_rate 2e-5 \
--num_train_epochs 3.0 \
--output_dir /tmp/$TASK_NAME/

and i also run with --fp16, but spend the same time and gpu memory, why fp16 does not work