Is there any way that I can directly feed the hidden output of the embedding layer into each of the transformer's layer?
Author: h56cho
Pull request: no
Labels: name: PyTorch

name: Usage

Hello,

For an original sequence ```X``` that has length of ```n```, I am interested in feeding the embedding of the original sequence ```X``` (```E```) as an input to the self-attention block of each layer of ```GPT2LMHeadModels``` (here, layer = self-attention block + feedforward block), and examine the layer output generated by ```E```.

Is there any way that I can carry out this task with HuggingFace ```GPT2LMHeadsModel``` transformers?

Thank you,

Comments:
How about manually handling the embeddings and attention layers?

```py
from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

sequence = tokenizer.encode("Try this out", return_tensors="pt")
embeds = model.get_input_embeddings()(seq)
first_layer_output, first_layer_attentions = model.transformer.h[0](embeds)
```

Hello,

Thank again for your reply!

1. Just to make sure that I am understanding this correctly, is the line
```python
model.transformer.h[0]
```
used to access the first layer of the transformer? so that I can access the second layer, third layer, etc., with ```model.transformer.h[1],  model.transformer.h[2]``` and so on?

2. To access the output head of the transformer, do I simply do:
```python
model.transformer.h[last index]
````
?

Thank you!

Yes, in GPT-2 the layers can be accessed via the `h` attribute. You're correct in your assumption regarding accessing the second and third layers.

This gives you the output of the MLP, which is of dimension `(batch_size, sequence_length, hidden_dim)`.

Hello,

Thank you for your reply.

I am having some trouble understanding the MLP function, which is found [here](https://github.com/huggingface/transformers/blob/73028c5df0c28ca179fbe565482a9c2143787f61/src/transformers/modeling_gpt2.py#L200).

Q1. For MLP, why are we setting the n_state to be equal to 3072, which is 4 * n_embd?
Q2. Below is the forward function for the MLP class:
```python
    def forward(self, x):
        h = self.act(self.c_fc(x))
        h2 = self.c_proj(h)
        return self.dropout(h2)
```
in the forward function above, what exactly do the lines ``` h = self.act(self.c_fc(x))``` and ``` h2 = self.c_proj(h)``` do?

Thank you,

> Yes, in GPT-2 the layers can be accessed via the `h` attribute. You're correct in your assumption regarding accessing the second and third layers.
> 
> This gives you the output of the MLP, which is of dimension `(batch_size, sequence_length, hidden_dim)`.

How would you feed input directly into a particular Keras Bert layer? Is there a way to automatically feed inputs at one layer, and have the rest be processed starting at that layer?

Purpose: I would like to feed the hidden states of one transformer, into another, so I would need to bypass the inputID->embedding layer. 

I did some tinkering and tried this 

```
testt = tf.random.uniform([3, 5,768], minval=-1, maxval=1, dtype=tf.dtypes.float32, seed=None, name=None)
model.layers[0].encoder.layer[3]((testt, None, None))
```

Seems promising, since output shapes are (3, 5, 768). 

Edit:

Maybe I can create a new model from these individual layers. 

```
testt = tf.random.uniform([3, 5,768], minval=-1, maxval=1, dtype=tf.dtypes.float32

def get_new_model():
    inputHiddenVals = tf.keras.Input(shape=[None, 768], dtype=tf.float32, name='input_Q',
                                    batch_size=None) 

    hidden1 = model.layers[0].encoder.layer[3]((inputHiddenVals, None, None))
    hidden2 = model.layers[0].encoder.layer[4]((hidden1[0], None, None))
    hidden3 = model.layers[0].encoder.layer[5]((hidden2[0], None, None))
    modelNew = tf.keras.Model(inputs=inputHiddenVals, outputs=hidden3)
    return modelNew

nModel = get_new_model()
nModel(testt)

```
Seems to work


Update, doesn't seem to work. The copied layers have parameters missing. 

```
from transformers import TFBertModel, AutoModel, TFRobertaModel
import tensorflow as tf
import tensorflow_addons as tfa

tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
import os
from copy import deepcopy

logger = tf.get_logger()
logger.info(tf.__version__)

def get_mini_models():
    tempModel = TFRobertaModel.from_pretrained('bert-base-uncased', from_pt=True)

    layer9 = deepcopy(tempModel.layers[0].encoder.layer[8])
    layer10 = deepcopy(tempModel.layers[0].encoder.layer[9])

    inputHiddenVals = tf.keras.Input(shape=[None, 768], dtype=tf.float32, name='input_Q',
                                    batch_size=None) 

    hidden1 = layer9((inputHiddenVals, None, None), training=True)
    hidden2 = layer10((hidden1[0], None, None), training=True)
    modelNew = tf.keras.Model(inputs=inputHiddenVals, outputs=hidden2)

    del tempModel

    return modelNew

@tf.function
def loss_fn(_, probs):
    bs = tf.shape(probs)[0]
    labels = tf.eye(bs, bs)
    return tf.losses.categorical_crossentropy(labels,
                                              probs,
                                              from_logits=True)

model = get_mini_models()
# model.layers[2].trainable = False
model.compile(loss=loss_fn,
                optimizer=tfa.optimizers.AdamW(weight_decay=1e-4, learning_rate=1e-5, 
                                                epsilon=1e-06))

tempModel = TFRobertaModel.from_pretrained('bert-base-uncased', from_pt=True)
layer9 = deepcopy(tempModel.layers[0].encoder.layer[8])

for i, var in enumerate(model.weights):
    print(model.weights[i].name)
```

> tf_roberta_model/roberta/encoder/layer_._8/attention/self/query/kernel:0
> tf_roberta_model/roberta/encoder/layer_._8/attention/self/query/bias:0
> tf_roberta_model/roberta/encoder/layer_._8/attention/self/key/kernel:0
> tf_roberta_model/roberta/encoder/layer_._8/attention/self/key/bias:0
> tf_roberta_model/roberta/encoder/layer_._8/attention/self/value/kernel:0
> tf_roberta_model/roberta/encoder/layer_._8/attention/self/value/bias:0

It's missing a layer, and not even all the weights for the first layer were transferred

```
for i, var in enumerate(layer9.weights):
    print(layer9.weights[i].name)
```

> tf_roberta_model_1/roberta/encoder/layer_._8/attention/self/query/kernel:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/self/query/bias:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/self/key/kernel:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/self/key/bias:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/self/value/kernel:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/self/value/bias:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/output/dense/kernel:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/output/dense/bias:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/output/LayerNorm/gamma:0
> tf_roberta_model_1/roberta/encoder/layer_._8/attention/output/LayerNorm/beta:0
> tf_roberta_model_1/roberta/encoder/layer_._8/intermediate/dense/kernel:0
> tf_roberta_model_1/roberta/encoder/layer_._8/intermediate/dense/bias:0
> tf_roberta_model_1/roberta/encoder/layer_._8/output/dense/kernel:0
> tf_roberta_model_1/roberta/encoder/layer_._8/output/dense/bias:0
> tf_roberta_model_1/roberta/encoder/layer_._8/output/LayerNorm/gamma:0
> tf_roberta_model_1/roberta/encoder/layer_._8/output/LayerNorm/beta:0

Here's a colab notebook if you want to play around with it

https://colab.research.google.com/drive/1XoESTWyo4qr4uApIai7Ac4tUDAeLDEI-?usp=sharing