[doc] transformers.PreTrainedTokenizer.encode() doesn't get resolved to its doc
Author: stas00
Pull request: no

There are a few 

> See: transformers.PreTrainedTokenizer.encode()

in the docstrings, but they don't resolve to anything in the online docs, since the `encode` method is in `PreTrainedTokenizerBase`
as it can be seen: https://huggingface.co/transformers/internal/tokenization_utils.html#transformers.tokenization_utils_base.PreTrainedTokenizerBase.encode

Should sphinx be able to resolve inheritance and still point to the right doc, or must the docs be modified to say:

> See: transformers.PreTrainedTokenizerBase.encode()

instead?

There are 117 of these.

Example:

https://huggingface.co/transformers/model_doc/t5.html#transformers.T5Model.forward

> Indices can be obtained using T5Tokenizer. See transformers.PreTrainedTokenizer.encode() and transformers.PreTrainedTokenizer.__call__() for detail.

the `encode` method doesn't get a link.

-----------------------

Also it seems that all modules have this

> `What are input IDs? <../glossary.html#input-ids>`__

but not where `input_ids` are documented, instead after the mask - shouldn't that line be 2 records up?

![snapshot_18](https://user-images.githubusercontent.com/10676103/106406034-e5934e00-63ec-11eb-8ddd-7a56d4e70732.png)



@sgugger 

Comments:
You're right, but changing the class won't help us as `PreTrainedTokenizerBase` is not documented either. A quick fix would be to just add the `encode` method to the doc in `PreTrainedTokenizer` so that the reference gets resolved.

> Also it seems that all modules have this
> What are input IDs? <../glossary.html#input-ids>__
> but not where input_ids are documented, instead after the mask - shouldn't that line be 2 records up?

It seems to be only in the T5 model from a quick look. This is missing in the `input_ids` arg (but it should also be in the `decoder_input_ids` args). Did you find other models where it's missing?

Can do a quick PR to fix those tomorrow morning.

> You're right, but changing the class won't help us as PreTrainedTokenizerBase is not documented either.

Isn't this the documentation?

https://huggingface.co/transformers/internal/tokenization_utils.html#transformers.tokenization_utils_base.PreTrainedTokenizerBase.encode

> It seems to be only in the T5 model from a quick look. This is missing in the input_ids arg (but it should also be in the decoder_input_ids args). Did you find other models where it's missing?

It's hard to devise a full-proof detector, because the input varies, but here is a quick attempt that catches a few of missing ones:
```
# from the root of the repo
grep -Inr -A30 'input_ids (:obj:' src/transformers/models/ | \
perl -ne '$x .= $_; END { for (split /--/, $x) { s/attention_mask.*//msg; print if !/What are input IDs/ } }'
```
Basically I'm trying to match every instance of the `*_input_ids` doc entries (assuming they all have the same pattern), I dump the subsequent text and then I check whether there is a matching "What are input IDs" in the next few lines. I also snip out any text after `attention_mask` to avoid overlap with entries like `decoder_input_ids`, which may have this pointer.

It dumps output where it's most likely missing, like in this entry:
```
src/transformers/models/t5/modeling_tf_t5.py:942:        decoder_input_ids (:obj:`tf.Tensor` of shape :obj:`(batch_size, target_sequence_length)`, `optional`):
src/transformers/models/t5/modeling_tf_t5.py-943-            Provide for sequence to sequence training. T5 uses the :obj:`pad_token_id` as the starting token for
src/transformers/models/t5/modeling_tf_t5.py-944-            :obj:`decoder_input_ids` generation. If :obj:`past_key_values` is used, optionally only the last
src/transformers/models/t5/modeling_tf_t5.py-945-            :obj:`decoder_input_ids` have to be input (see :obj:`past_key_values`).
src/transformers/models/t5/modeling_tf_t5.py-946-
src/transformers/models/t5/modeling_tf_t5.py-947-            To know more on how to prepare :obj:`decoder_input_ids` for pretraining take a look at `T5 Training
src/transformers/models/t5/modeling_tf_t5.py-948-            <./t5.html#training>`__. If :obj:`decoder_input_ids` and :obj:`decoder_inputs_embeds` are both unset,
src/transformers/models/t5/modeling_tf_t5.py-949-            :obj:`decoder_input_ids` takes the value of :obj:`input_ids`.
```

The detected chunks are just double newline separated. There are probably a few false positives, but most seem to be true positives. You have the file and the line for the context.

And in many places where "What are input IDs" are, in the same place the corresponding entry for attention is missing.

Also note I only scanned under `/models/`, there is more in non-model files, but I think it's by design.



> Isn't this the documentation?
>https://huggingface.co/transformers/internal/tokenization_utils.html#transformers.tokenization_utils_base.PreTrainedTokenizerBase.encode

Ah yes, but this is under the "internal" tools, so let's have the subclasses show the documentation since I doubt users will go that far down.

Will try your magic perl, thanks!

Well, I meant that the xref link could link to that page. So it's not about users browsing to it, but sphinx resolving to that doc. 

Unless I'm missing something and you are talking about something else.