Fix Language Modeling Loss
Author: CatalinVoss
Pull request: yes

This fixes the language modeling loss setup for GPT and GPT-2. Minimizing the loss would previously destroy the language model within a few steps. I believe that both loss computations were incorrect, since they computed the cross-entropy without shifting the logits. Given the masking setup here, we want the ith logits to act as predictors of the (i+1)st token label (not the ith).

When I was debugging this, to be sure that there's no other issue, I checked that the logits coming out of the OpenAI tensorflow and the pytorch implementation appeared to be the same (kind of a blackbox test). We can import the original model as `tf_model` and compare:

```
# Construct input sample of two 512 token 1's
batch_size = 1
batch = [[1,2]*512]*batch_size
batch = np.array(batch)

# Attempt to seed everything
seed = 42
seed_all(seed)
np.random.seed(seed)
tf.set_random_seed(seed)

# PyTorch Implementation
model = GPT2LMHeadModel.from_pretrained('gpt2')
torch_batch = torch.tensor(batch)
loss = model(torch_batch, lm_labels=torch_batch)
print(loss.detach().numpy())

logits, presents = model(torch_batch)
print(logits.detach().numpy())

# TensorFlow implementation
with tf.Session() as sess:
    hparams = tf_model.default_hparams()
    hparams.override_from_dict({
                                  "n_vocab": 50257,
                                  "n_ctx": 1024,
                                  "n_embd": 768,
                                  "n_head": 12,
                                  "n_layer": 12
                                })

    # Set up graph and loss as I believe is correct similar to https://github.com/nshepperd/gpt-2
    context = tf.placeholder(tf.int32, [batch_size, None])
    output = tf_model.model(hparams=hparams, X=context)
    logits = output['logits']
    loss = tf.reduce_mean(
        tf.nn.sparse_softmax_cross_entropy_with_logits(
            labels=context[:, 1:], logits=logits[:, :-1]))
    train_vars = [v for v in tf.trainable_variables() if 'model' in v.name]

    # Load checkpoint
    ckpt = tf.train.latest_checkpoint('gpt_tf/models/117M')
    saver = tf.train.Saver(var_list=train_vars)
    saver.restore(sess, ckpt)

    # Run!
    tf_logits, tf_loss = sess.run((logits, loss), feed_dict={context: batch})
    print(tf_loss)
    print(tf_logits)

    # Other differences
    param_optimizer = list(model.named_parameters())
    print("torch:")
    print([n for n, p in param_optimizer])

    print("tensorflow:")
    print([v.name for v in tf.trainable_variables()])

    # They look like the same 147 params...
```

*This gave us for torch:*

- Loss: `11.067907`

- Logits:
	
	```
	[[[ -32.901043  -31.20237   -34.66221  ...  -39.486702  -39.87312
	    -32.238667]
	  [ -55.52076   -53.428535  -56.4767   ...  -68.153885  -66.77085
	    -58.600616]
	  [ -59.22766   -58.769135  -54.14502  ...  -64.58172   -65.165565
	    -57.34685 ]
	  ...
	  [-261.01627  -245.20702  -258.9687   ... -285.7149   -292.39874
	   -260.91663 ]
	  [-256.27637  -251.34431  -242.03348  ... -280.47235  -287.56
	   -256.33374 ]
	  [-261.22495  -245.68788  -258.8527   ... -286.35617  -292.90662
	   -261.41626 ]]]
	```

*...and for TensorFlow:*

- Loss: `0.019036641 `

- Logits:

	```
	[[[ -32.9011    -31.202427  -34.662262 ...  -39.486755  -39.87316
	    -32.238716]
	  [ -55.52075   -53.42854   -56.476707 ...  -68.153885  -66.770874
	    -58.60062 ]
	  [ -59.227722  -58.76918   -54.14507  ...  -64.58176   -65.165596
	    -57.346878]
	  ...
	  [-261.01633  -245.20723  -258.96875  ... -285.71472  -292.3988
	   -260.91663 ]
	  [-256.27637  -251.3442   -242.03352  ... -280.4723   -287.56
	   -256.33374 ]
	  [-261.22498  -245.68774  -258.85263  ... -286.3562   -292.90668
	   -261.41626 ]]]
	```

Shifting the logits so that tokens < n predict the nth token like in the TF example above should fix this.

Comments:
I fixed the test failure above in `472857c`. Note that this occurred because I was running torch > 1.0. They fixed the contiguous view issue in https://github.com/pytorch/pytorch/issues/3653. In my view, it would probably make sense to update torch to >1.0 and remove `contiguous()` calls where possible throughout this codebase.

Hi @CatalinVoss,
This looks good, thanks!

We'll try to keep backward compatibility up to pytorch 0.4.1 for the moment since quite a lot of downstream libraries use this package (like AllenNLP, fair, ParlAI...). So it's good to have the `contiguous()` calls, thanks for that.

Do you want to add a simple script showcasing finetuning GPT-2 on some dataset (open question, you don't need to)?

OK, thanks! That all makes sense. Yes, I can add a demo finetuning script, but I probably won't get to it immediately, so probably best to do it in another PR, if that's OK.

Ok good to merge, thanks @CatalinVoss!