seq2seq model with transformer
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I am urgently looking for a sequence to sequence model with transformer with script to
finetuning and training, I appreciate telling me which of the implementations in this repo could do a sequence to sequence model?
thanks
Best regards
Julia

Comments:
Are you looking for LSTM/RNN-based seq2seq architectures or Transformer-based architectures? This repository does not host any LSTM/RNN architectures.

You can find information on all our (transformer) [models here](https://huggingface.co/pytorch-transformers/pretrained_models.html), and [examples using them here](https://huggingface.co/pytorch-transformers/examples.html).

I am looking for transformer based, pretrained model, I am not sure which of  the implemented models in this repo I can use for seq2seq model? thanks for your help

The models hosted on this repo unfortunately probably cannot be used in a traditional sequence-to-sequence manner like translation (if that's what you have in mind).

yes, exactly, I am looking for such models, even gpt model cannot be used
for this purpose?
or gpt2 by conditioning?
Are you aware of clean implementation for seq2seq model with any of these
pretrained models hosted in your repo?
thanks.

On Thu, Aug 8, 2019 at 5:10 PM Lysandre Debut <notifications@github.com>
wrote:

> The models hosted on this repo unfortunately probably cannot be used in a
> traditional sequence-to-sequence manner like translation (if that's what
> you have in mind).
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZM7GUYVJHJ6YKKQK3PDQDQZM5A5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD335YCY#issuecomment-519560203>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AM3GZM2PJO6J63NC4DPTFW3QDQZM5ANCNFSM4IKIFMCQ>
> .
>


Hi @juliahane, maybe take a look at `fairseq`

Hi
Thanks, Do you mind also suggest me a good implementation with lstm for
seq2seq model, I need some implementation with high quality of decoding,
thanks.

On Thu, Aug 8, 2019 at 6:52 PM Julien Chaumond <notifications@github.com>
wrote:

> Hi @juliahane <https://github.com/juliahane>, maybe take a look at fairseq
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZG265MHPY53WI2HNTQDRFLDA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD34HSYQ#issuecomment-519600482>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AM3GZMYF5YYWWJMDVOXH3LTQDRFLDANCNFSM4IKIFMCQ>
> .
>


Hi
I found FairSeq implementation not really clean and modular code. Are you
aware of more work which extend BERT, GPT, ... to a language model with
decoder?
thanks
Julia

On Thu, Aug 8, 2019 at 9:07 PM julia hane <juliahane123@gmail.com> wrote:

> Hi
> Thanks, Do you mind also suggest me a good implementation with lstm for
> seq2seq model, I need some implementation with high quality of decoding,
> thanks.
>
> On Thu, Aug 8, 2019 at 6:52 PM Julien Chaumond <notifications@github.com>
> wrote:
>
>> Hi @juliahane <https://github.com/juliahane>, maybe take a look at
>> fairseq
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZG265MHPY53WI2HNTQDRFLDA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD34HSYQ#issuecomment-519600482>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AM3GZMYF5YYWWJMDVOXH3LTQDRFLDANCNFSM4IKIFMCQ>
>> .
>>
>


Then you should have a look at the "Cross-lingual Language Model Pretraining" from Lample and Conneau: https://arxiv.org/abs/1901.07291

Implementation of supervised and unsupervised NMT can be found here: https://github.com/facebookresearch/XLM#iii-applications-supervised--unsupervised-mt :)

Hi
thanks a lot. I was wondering if you could also suggest me a good
implementation for seq2seq with LSTMs in pytorch with good accuracy.
I have a deadline and I cannot find any, I really appreciate your help.
thanks
Julia

On Thu, Aug 8, 2019 at 11:41 PM Stefan Schweter <notifications@github.com>
wrote:

> Then you should have a look at the "Cross-lingual Language Model
> Pretraining" from Lample and Conneau: https://arxiv.org/abs/1901.07291
>
> Implementation of supervised and unsupervised NMT can be found here:
> https://github.com/facebookresearch/XLM#iii-applications-supervised--unsupervised-mt
> :)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZM3URUZ6GWYI7A4WPFTQDSHKBA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD347NXQ#issuecomment-519698142>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AM3GZM2PRHPLQBS7XBGCUBLQDSHKBANCNFSM4IKIFMCQ>
> .
>


Hey Julia, without a specific task in mind I can't think of anything relevant, but browsing [paperswithcode.com with a seq2seq search](https://paperswithcode.com/search?q_meta=&q=seq2seq) yields quite a few interesting results.

Hi
My task is a autoencoding text. So encoding and decoding it in one
language. Thanks

On Wed, Aug 14, 2019, 5:18 PM Lysandre Debut <notifications@github.com>
wrote:

> Hey Julia, without a specific task in mind I can't think of anything
> relevant, but browsing paperswithcode.com with a seq2seq search
> <https://paperswithcode.com/search?q_meta=&q=seq2seq> yields quite a few
> interesting results.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZ3PDZUSW7QWNQV3CDQEQO2XA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JEKTA#issuecomment-521291084>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AM3GZM62USCZO4Z5UGUTOBTQEQO2XANCNFSM4IKIFMCQ>
> .
>


I was wondering if you could tell me which of these are a fast sequence to
sequence implementation,
this is really hard for me to figure out which one to use. thanks

On Wed, Aug 14, 2019 at 5:19 PM julia hane <juliahane123@gmail.com> wrote:

> Hi
> My task is a autoencoding text. So encoding and decoding it in one
> language. Thanks
>
> On Wed, Aug 14, 2019, 5:18 PM Lysandre Debut <notifications@github.com>
> wrote:
>
>> Hey Julia, without a specific task in mind I can't think of anything
>> relevant, but browsing paperswithcode.com with a seq2seq search
>> <https://paperswithcode.com/search?q_meta=&q=seq2seq> yields quite a few
>> interesting results.
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZ3PDZUSW7QWNQV3CDQEQO2XA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JEKTA#issuecomment-521291084>,
>> or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AM3GZM62USCZO4Z5UGUTOBTQEQO2XANCNFSM4IKIFMCQ>
>> .
>>
>


I did checked this implementations you sent me, I honestly cannot find a
single good seq2seq one with lstm, and I really appreciate your help

On Wed, Aug 14, 2019 at 5:39 PM julia hane <juliahane123@gmail.com> wrote:

> I was wondering if you could tell me which of these are a fast sequence to
> sequence implementation,
> this is really hard for me to figure out which one to use. thanks
>
> On Wed, Aug 14, 2019 at 5:19 PM julia hane <juliahane123@gmail.com> wrote:
>
>> Hi
>> My task is a autoencoding text. So encoding and decoding it in one
>> language. Thanks
>>
>> On Wed, Aug 14, 2019, 5:18 PM Lysandre Debut <notifications@github.com>
>> wrote:
>>
>>> Hey Julia, without a specific task in mind I can't think of anything
>>> relevant, but browsing paperswithcode.com with a seq2seq search
>>> <https://paperswithcode.com/search?q_meta=&q=seq2seq> yields quite a
>>> few interesting results.
>>>
>>> —
>>> You are receiving this because you were mentioned.
>>> Reply to this email directly, view it on GitHub
>>> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZ3PDZUSW7QWNQV3CDQEQO2XA5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JEKTA#issuecomment-521291084>,
>>> or mute the thread
>>> <https://github.com/notifications/unsubscribe-auth/AM3GZM62USCZO4Z5UGUTOBTQEQO2XANCNFSM4IKIFMCQ>
>>> .
>>>
>>


@juliahane `fairseq` has an example of how to use a LSTM (encoder & decoder) for a seq2seq model:

https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html

Additionally, you could also check out Joey NMT, which has a very nice and clear codebase:

https://github.com/joeynmt/joeynmt

Hi
Thanks, Fairseq to me is not following a good coding practice although
Facebook has published it,
but the second one looks much better, thank you.

I was wondering if you could tell me if torchtext is faster than using
dataloader in pytorch for seq2seq applications?
I wonder how torchtext impact the speed and if  this is really better than
dataloader

thanks

On Fri, Aug 16, 2019 at 1:42 PM Stefan Schweter <notifications@github.com>
wrote:

> @juliahane <https://github.com/juliahane> fairseq has an example of how
> to use a LSTM (encoder & decoder) for a seq2seq model:
>
> https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html
>
> Additionally, you could also check out Joey NMT, which has a very nice and
> clear codebase:
>
> https://github.com/joeynmt/joeynmt
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/988?email_source=notifications&email_token=AM3GZMZGVNRAT6FJLOEZAB3QE2HB3A5CNFSM4IKIFMC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4OMY7A#issuecomment-521981052>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AM3GZM3YNYMZ32LXZCCPSA3QE2HB3ANCNFSM4IKIFMCQ>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Merging with #1506