TFT5: get_input_embeddings() and get_output_embeddings()
Author: parthe
Pull request: no

In class TFT5Model

1. The get_input_embeddings() and get_output_embeddings() methods do not have any documentation provided in them
2. Furthermore, the get_output_embeddings provides the same output as the get_input embeddings. This needs to be resolved. Or flagged with a NotImplementedError

Comments:
From what I'm seeing, the `TFT5Model` **does** have [documentation](https://huggingface.co/transformers/model_doc/t5.html#transformers.TFT5Model.get_input_embeddings) for `get_input_embeddings` and `get_output_embeddings`.

I believe the output embeddings and input embeddings should actually be the same. The embeddings are shared between input and output. Wdyt @patrickvonplaten? 

Agree that the documentation is not the greatest, could definitely be improved :-). 

The idea is that both `get_input_embeddings()` and `get_output_embeddings` return the **same** (this should be made clearer in the docs) embeddings matrix of dimension Vocab_size x Hidden_size. 

Now, to make the embeddings matrix work for both input and output, we need to be able to get a Vocab_size -> Hidden_size mapping (for the input embeddings) and Hidden_size -> Vocab_size mapping (for the output embeddings). In TF, we use a trick here, by wrapping the embedding in this layer: https://github.com/huggingface/transformers/blob/c53cc018de70436196858ca91c1a34f1b8947028/src/transformers/modeling_tf_utils.py#L1521

And then by calling the embedding with different modes ("linear" and "embedding"), we get the correct mapping. See https://github.com/huggingface/transformers/blob/c53cc018de70436196858ca91c1a34f1b8947028/src/transformers/modeling_tf_t5.py#L1074 for example. 

So IMO, the code is fine, I agree with you @parthe, that the documention should be cleaner and explain the logic I explained here a bit. 

If you feel like it @parthe, it would be amazing if you could open a PR to straighten up the documentation here (the docstring). 

Hi - I think I have related questions since I couldn't find answers in the documentation for get_input_embeddings().... 

I've been using the approach [here](https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/) to access the hidden states to obtain embeddings (I've also updated it to be based on `transformers` in my own notebook instead of `pytorch-pretrained-bert`) . I was wondering how the output of get_input_embeddings maps to the output of the hidden states there? I've not been able to figure that out. Also, what would be the advantage of using one over the other?

Thanks!

So I would recommend that you take the `hidden_states` by setting `config.output_hidden_states=True` (Note: this API will be changed soon, see PR: #4538). 
Then you can map the `hidden_states` to `lm_logits` (non normalized scores for each word in the vocab) using:

```python 
embed_tokens = self.get_output_embeddings()
lm_logits = embed_tokens(<your_hidden_states>, mode="linear")
```

Let me know if this isn't clear :-) 

Hi @patrickvonplaten, referring to the quote below (from this [comment](https://github.com/huggingface/transformers/issues/3852#issuecomment-618852195)):
> The idea is that both `get_input_embeddings()` and `get_output_embeddings` return the **same** (this should be made clearer in the docs) embeddings matrix of dimension Vocab_size x Hidden_size.
> 
> Now, to make the embeddings matrix work for both input and output, we need to be able to get a Vocab_size -> Hidden_size mapping (for the input embeddings) and Hidden_size -> Vocab_size mapping (for the output embeddings). In TF, we use a trick here, by wrapping the embedding in this layer:
> 
> https://github.com/huggingface/transformers/blob/c53cc018de70436196858ca91c1a34f1b8947028/src/transformers/modeling_tf_utils.py#L1521
> 
> And then by calling the embedding with different modes ("linear" and "embedding"), we get the correct mapping. See
> 
> https://github.com/huggingface/transformers/blob/c53cc018de70436196858ca91c1a34f1b8947028/src/transformers/modeling_tf_t5.py#L1074
> 
> for example.

Does this only apply to TFT5Model or is the same across all models which has `get_input_embeddings` and `get_output_embeddings` method?

It should be the same across models that share input and output embeddings :-)