Decision transformer gym
Author: edbeeching
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Added the Decision Transformer Model to the transformers library (https://github.com/kzl/decision-transformer)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@LysandreJik

Comments:
I have:
- Removed the (temporary) changes that were made on the gpt2 model
- Added zeroed position encodings to the call to gpt2 forward, so they are ignored. (as these are implemented in the DT model)
- Fixed issues related to black line length (my IDE was set to the default I think)
- Removed datasets submodule
- Added and fixed many tests for the ModelTesterMixin. I skip:
  - test_generate_without_input_ids
  - test_pruning
  - test_resize_embeddings
  - test_head_masking
  - test_attention_outputs
  - test_hidden_states_output
  - test_inputs_embeds
  - test_model_common_attributes
- Updated some of the doc strings etc.

Still Todo:

- Example file / notebook. Ideally this would show a model checkpoint being loaded and then demonstrate the performance in an RL environment. Is this what you are thinking? I have checkpoints saved from the author's implementation, but I will need to modify them so the state_dict matches this implementation (it will be missing the gpt2 position encoding weights, for example)
- Integration test. Similar issue as the above.

Please let me know if you have any comments, questions or ideas related to the commits I have just added.


Really cool to see the first RL model in Transformers here! Agree with @LysandreJik in that we can just set the position ids to a 0 valued tensor so that the model doesn't need to be adapted. 

For me, it would be important to understand how the model would be used in practice so that we can better understand how to best implement the model in Transformers. @edbeeching, do you think it could be possible to add an integration test that verifies that a **minimal** use case of the model works as expected? E.g. maybe we could show how the user would use this implementation to retrieve an action state `a_t` from `s_{t-1}`, `r_{t-1}` and `a_{t-1}` and then how `a_t` would be further treated for the next time step? So a code-snippet showing a two-step inference with some comments/code explaining where the reward and state come from as well. 

Very exciting!

Thanks for the comments, the model is now loaded from a pretrained checkpoint. I have set the seed and have added a comparison between the output and the expected output. Let me know of any other comments or additions.

Sorry some files got committed unintentially, I have removed them.
Regarding the model checkpoint files, the authors are from Berkley, but I trained the models myself with their codebase (they did not share their checkpoints). So I don't know who the rightful "authors" of the checkpoints are, I am not bothered either way. Let me know what you think.

> checkpoint

I see! Yeah since you've trained them, then I think it's totally fine to just publish them under your name on the Hub! So I think the PR is good to go for me :-)

Think we still got some `decision-transformer-gym-halfcheetah-expert` folders though ;-) 

There are five conflicts to resolve, could you take care of this before we do a final review @edbeeching ?

Ok I went through the conflicts.
One final question, it appears that the test files have been reorganized since I first forked the transformers repo. They now appear to be nested each in their own subdirectory. Should add a directory for my test to reflect this change in organization?

_The documentation is not available anymore as the PR was closed or merged._