Descriptions of shared models and interaction with contributors
Author: chinisan
Pull request: no
Labels: name: wontfix

## 🚀 Feature

It would be nice to have more room for descriptions from the contributors of shared models since at the moment one can only guess from the title as to what the model does and what was improved from existing models. 

Additionally, ways of interactions with the contributors such as comments and upvotes would be helpful in further improving these models. This would be a good indication of what models work well, and contributors can clarify how the model was trained so the same work doesn't have to be done twice

## Motivation

While looking for an AlbertForQuestionAnwering model, I discovered that transformers doesn't provide a pretrained model at the moment (see (https://github.com/huggingface/transformers/issues/1979)) even though it is mentioned here (https://huggingface.co/transformers/model_doc/albert.html#albertforquestionanswering)

Within the shared models, I tried out 2 models that mention albert and squad, but I couldn't get either to run on this simple example:

```
tokenizer = AutoTokenizer.from_pretrained("ktrapeznikov/albert-xlarge-v2-squad-v2", cache_dir=model_directory)
model = AutoModel.from_pretrained("ktrapeznikov/albert-xlarge-v2-squad-v2", cache_dir=model_directory)`
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
input_ids = tokenizer.encode(question, text)
print(input_ids)
token_type_ids = [0 if i <= input_ids.index(3) else 1 for i in range(len(input_ids))]
start_scores, end_scores = model(torch.tensor([input_ids]), token_type_ids=torch.tensor([token_type_ids]))
all_tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]))
````

The example runs but doesn't give me an answer. Same with replydotai/albert-xxlarge-v1-finetuned-squad2

Now, I am not sure if I just made a mistake or if the model does not do question answering. Could it be that these models were only finetuned on the language in the squad dataset? But that wouldn't make a lot of sense. 

Having a better description of the models could have helped and in a comment section, I could have talked directly to the contributor.

I would appreciate any help in getting a Albert for question answering running.

Thanks!

## Additional context
Python 3.7.3
transformers 2.3.0
Pytorch 1.3.1
<!-- Add any other context or screenshots about the feature request here. -->


Comments:
In a sense, this is related to the discussion that we had over at https://github.com/huggingface/transformers/pull/2281#issuecomment-570574944. The [answer](https://github.com/huggingface/transformers/pull/2281#issuecomment-571418343) by @julien-c was that they are aware of the difficulties and sensitivities that custom models bring, but that they are still figuring out how they want to approach this.

An up-voting system seems a good idea, and I also want to emphasise that good documentation for each model is paramount, explaining how it was trained (hyperparameters, data, task), and how it possibly differs from the "standard" or "official" implementation.

If I am not missing anything here, `AutoModel` will only load the language model, but not a model for question answering (incl. the prediction head).

Try exchanging the line where you load the model with: 
```
model = AutoModelForQuestionAnswering.from_pretrained("ktrapeznikov/albert-xlarge-v2-squad-v2", cache_dir=model_directory)
```

So right now, I guess we need to know what type of community model we are loading?!
It would be helpful to have some information in the stored configs to infer what task the model was trained on. I found the param `finetuning_task` there, but it's `null` for all models I have checked. 



 

@tholor Thanks that worked! 

Hi all, 

- the sample code on the model pages should indeed showcase the `AutoModelXXX` variant that uses the head(s) defined in the weights. Fix coming soon ⚠️.
- For model description, you can already add a **`README.md`** file to your shared folder on S3 and it will be rendered on your model's page: see e.g. https://huggingface.co/dbmdz/bert-base-german-uncased from @stefan-it 
   - You can use this file as a model card to describe your model, which datasets did you train on, eval results, etc.
   - We'll also add metadata such as language, downstream task, etc. which will let us filter results on the models listing page (e.g. "find models for QA in 🇳🇱")

- Finally, we are thinking of storing the README.md files inside a `model_cards/` folder inside the transformers repo itself. i.e. use git and GitHub to let the community collaborate on model READMEs, not just the model author.

The kind of editing rules we could put into place would be :
- anyone can propose a PR anywhere.
- on "main" canonical models we (HuggingFace and/or maintainers) validate the PRs.
- on "user" models the model's author(s) decide. (they are pinged automatically, and they can validate/refuse)

Thoughts?

Sounds great, @julien-c !

I like the idea of having a `readme.md`. Given the variety of tasks, it might be difficult to press everything into a structured config / modelcard format. Nevertheless, I would appreciate having the most important metadata (language, downstream task, training data, performance) in a config and maybe even making it a requirement for upload. Otherwise, it might become easily a big mess after a while and comparison will become more difficult (e.g. eval results).

Regarding the git workflow for the readme: I like it. If I got it right, people would still be able to upload an initial readme via CLI and only subsequent changes are managed via git? Otherwise, it could slow down the upload of new models a lot.

Being relatively strict, as @tholor suggests, seems like a good idea to prevent a forest of random models (see what I did there?). As metadata, I think at least language should definitely be required.

I very much like the idea of being able to filter by metadata. One could imagine scenarios where you want to filter by language, trained head, upload date, architecture, and so on.

The difference between canonical and user models should be made very clear, though. Take for example the models that were explicitly added to the repo (bert-base-german-dbmdz-cased for instance) but that were created by users. Are those canonical (it seems like that because it's part of `BERT_PRETRAINED_MODEL_ARCHIVE_MAP`)?

Looking forward to it @julien-c!

Hi 🤗

my thoughts on this issue:

I really like that an author of a model can upload a README file via `transformers-cli` interface. This really speeds up a change/additions to the README - and won't require a review process (so I'm not stealing someone's time for a simple README change). On the other side there's no detailed overview of trained models (languages, training data, results), except I visit all model pages.

So here's my suggestion (and this hopefully fixes the model name issues that were discussed by @BramVanroy ):

Whenever a user model is added to the `*_PRETRAINED_MODEL_ARCHIVE_MAP` the model alias must be identical to the S3 model name. E.g. I would rename `bert-base-german-dbmdz-cased` to `dbmdz/bert-base-german-cased`. This may break backward compatibility (but I could live with that).

Adding a new user model to `*_PRETRAINED_MODEL_ARCHIVE_MAP` is done in a PR and this PR requires an additional README file for a detailed model description (location: `model_cards/`). 

We should really define a kind of template for that README that includes all relevant information like language, training data and results on downstream tasks.

I could also image a kind of json-based model card, that will be parsed by the Hugging Face model hub page, so that we can search for models/languages.

@stefan-it That solves the issue I had in part, indeed. The question remains (as was discussed elsewhere), which models go into the `_PRETRAINED_MODEL_ARCHIVE_MAP` and is it even still necessary if you can then download the model via `user/weights-name`? I would then remove those user models from the archive map, and only make them available through the `user/weights-name` directive. That way it is clear that the `_PRETRAINED_MODEL_ARCHIVE_MAP` contains canonical models that HuggingFace added themselves. Any and all other models should then be loaded through the user approach. What do you think?

I think that's a good idea :+1: 

In addition to the `model_cards` folder it would be great to have an overview page of all available user models (that were added via PR) to e.g. to find all available BERT models.

I'm thinking of this kind of overview page:

http://pfliu.com/ner/ner.html

with filters like architecture, language or amount of training data 🤔

> * the sample code on the model pages should indeed showcase the `AutoModelXXX` variant that uses the head(s) defined in the weights

Should now be implemented on the website, please let us know if you see anything fishy. Thanks!

([example model page](https://huggingface.co/Musixmatch/umberto-commoncrawl-cased-v1))

It's really starting to take shape, I like it!

Some overall suggestions design-wise:

- there is overflow on the website's x-axis, even on my 2560x1440 monitor
- the column width of the website is very small. So small even that a single `.from_pretrained(...)` command doesn't fit in one line. Perhaps make the width larger, but only for the code blocks?
- in the usage section, code seems to be styled twice (once as a block, and once as marked text), same for citation section
- some more vertical space between code blocks would be nice
- code is overflowing their containers (should be manageable when making code blocks wider + using `pre {max-width: 100%; overflow-x: auto}`
- on mobile: the tables overflow, too
- on mobile: the model name/heading is too large which causes an overflow

Mobile testing is probably quite important. The most new models that I come into contact with are from twitter, so I open the model cards on my phone. I suppose many others, too.

Great job already, I really like it!



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I just re-read this thread and it feels like even though there's still lots of stuff todo, we're starting to get there!

=> https://huggingface.co/

Thanks so much for your ideas in this thread, they were super helpful. 

I'll close this particular issue as "completed", but would love your feedback and ideas for the next steps

cc @LysandreJik @thomwolf