ValueError: Layer weight shape (30522, 768) not compatible with provided weight shape torch.Size([1, 15, 3072])
Author: mathshangw
Pull request: no
Labels: name: TensorFlow

 I tried to use word embedding from using BERT as an embedding layer using Keras like that 
```
    inputs2 = Input(shape=(max_length,)) 
    sent = Embedding(vocab_size, 768, mask_zero=True)(inputs2)
    model.layers[1].set_weights([embedding_matrix])    
    model.layers[1].trainable = False
```
but got the following error 
 ```
  model.layers[1].set_weights([embedding_matrix])
  File "/home/user/.local/lib/python3.6/site-packages/keras/engine/base_layer.py", line 1801, in set_weights
    'shape %s' % (ref_shape, weight_shape))
ValueError: Layer weight shape (30522, 768) not compatible with provided weight shape torch.Size([1, 15, 3072])

```

Comments:
Hi @mathshangw, is there a way for you to share more? We don't have knowledge of most of the variables you're showing here.

Hi @mathshangw! Thank you for sharing your issue with us.

I see two potential issues, given the information you gave us so far:
1. the `inputs2` variable in your initial script contains a pytorch tensor. I am not sure whether Keras accept pytorch tensors as inputs, try to convert it to a TF tensor. Alternatively, you can change Bert to be a tensorflow model, by changing  `AutoModel` to `TFAutoModel` (then the output would be a TF tensor);
2. The output of your `get_bert_embed_matrix` function is already an embedding, so you don't need an Embedding layer. You can use that as an input to a downstream model.

Let me know if it helps.

Finally -- I know this is your first issue here, but check our [guide for issues](https://github.com/huggingface/transformers/blob/master/ISSUES.md) if you haven't so far. It helps you find the answer faster (if it exists out there) and it helps us and the community to deliver the best experience possible (if it doesn't) 🤗 

Thanks a lot for replying .. so excuse me does this function write for return word-embedding using BERT ? 

Yes, the concatenation of the last 4 hidden states, returned by your `get_bert_embed_matrix` function, is one of the most well-known methods ([table 7 in the original BERT paper](https://arxiv.org/pdf/1810.04805.pdf)) to obtain word-level embeddings for BERT :) 

> Hi @mathshangw! Thank you for sharing your issue with us.
> 
> I see two potential issues, given the information you gave us so far:
> 
>     1. the `inputs2` variable in your initial script contains a pytorch tensor. I am not sure whether Keras accept pytorch tensors as inputs, try to convert it to a TF tensor. Alternatively, you can change Bert to be a tensorflow model, by changing  `AutoModel` to `TFAutoModel` (then the output would be a TF tensor);
> 
>     2. The output of your `get_bert_embed_matrix` function is already an embedding, so you don't need an Embedding layer. You can use that as an input to a downstream model.
> 
> 
> Let me know if it helps.
> 
> Finally -- I know this is your first issue here, but check our [guide for issues](https://github.com/huggingface/transformers/blob/master/ISSUES.md) if you haven't so far. It helps you find the answer faster (if it exists out there) and it helps us and the community to deliver the best experience possible (if it doesn't) hugs

thanks a lot but how can i use the function as an input to a downstream model , please 

You can do as follows (there are actually many ways to do it):
1. Ensure you get the output of `get_bert_embed_matrix` as a tf.Tensor ([docs](https://www.tensorflow.org/api_docs/python/tf/Tensor))
2. Define your model using the sequential API ([docs](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential))
3. Ensure the first later is a `Flatten` keras layer, so it handles the input shapes for you (see the examples in the [docs](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten))
4. Define the rest of your downstream model

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.