Implement PyTorch and/or TensorFlow sequence classification architectures for causal language models
Author: LysandreJik
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

The architecture `GPT2ForSequenceClassification` was added in #7501 in PyTorch. It would be great to have it in TensorFlow (cf. issues #7622), but it would also be great to have it for other causal models: ~OpenAI GPT~, ~CTRL~ (PR opened @elk-cloner), ~TransfoXL~ (PR opened @spatil6)

Below is a list of items to follow to make sure the integration of such an architecture is complete:

- Implement `XXXForSequenceClassification` in `modeling_xxx.py` or `TFXXXForSequenceClassification` in `modeling_tf_xxx.py
- Test that architecture in `tests/test_modeling_xxx.py` or `tests/test_modeling_tf_xxx.py`
- Add that architecture to `__init__.py` and ` docs/source/model_doc/xxx.rst`.

Taking a look at the code changes in #7501 would be a good start.

A very good first issue to get acquainted with the library and its architectures!


Comments:
Hi @LysandreJik is this issue still open? I'll like to pick it up

I believe @fmcurti is working on the OpenAI GPT implementation, but both CTRL and TransfoXL are still open! Would love a PR!

Hi Lysandre, thanks for assigning this issue to me. I've been trying to
setup transformers on my local machine (Windows 10). I've been having
several issues with the setup.
Is there any guide I could follow?

Thanks much

On Fri, 9 Oct 2020 at 12:38, Lysandre Debut <notifications@github.com>
wrote:

> Assigned #7623 <https://github.com/huggingface/transformers/issues/7623>
> to @pasDamola <https://github.com/pasDamola>.
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/7623#event-3859742526>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGGRMAH2H4Q5JYISSW23W3LSJ3Y4HANCNFSM4SGSU6FQ>
> .
>


Sure, have you taken a look at the [`CONTRIBUTING.md` document](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md)? What issues have you been having?

Yes I have.
When I run `pip install -e ".[dev]"`, I always encounter this error. I'm also running it in anaconda environment

![Screenshot (406)](https://user-images.githubusercontent.com/26023424/95594786-529c6580-0a43-11eb-821b-ec0aacfff4e7.png)

`

I believe the repo cannot be installed from conda as of now, can you use a pip virtual environment?

Alright, I'll try that now

Still having the same error I had while in conda. I'm trying to install tensorflow locally and retry this again

Hi @LysandreJik , I'm still having the same errors running on a pip virtual environment

Do you manage to install `TensorFlow` in your pip environment?

Hi @LysandreJik not yet. I get a similar error. I'm trying to look for solutions on the internet

![Screenshot (407)](https://user-images.githubusercontent.com/26023424/95607857-3b19a880-0a54-11eb-9842-38b46764a449.png)


Hi @LysandreJik – has anyone picked up the CTRL or TransfoXL architectures yet? I'd love to take a crack at one of them if available. Thank you!

No, feel free to take a crack at it! Let me know and I'll put you in the issue description.

is there anybody working on these ? @LysandreJik 

I believe CTRL and TransfoXL are still available. Feel free to open a PR!

Hi @LysandreJik , 

As this Feature request is closed

Do we need TF implementation of causal models GPT-1, Transfoxl and CTRL?
I'm ready to contributed for that as well.



That would be very welcome @spatil6!

Ok thanks @LysandreJik.

I'm waiting for this PR #8714 to get merge.
Once done, I'll raise PR for these models as well.