Related to abstractive text summarization
Author: laibamehnaz
Pull request: no
Labels: name: wontfix

I was wondering if we could use XLM or XLM-R for abstractive text summarization.

Comments:
If you are looking for summerization in non-English languages you can try using `MBartForConditionalGeneration`, or multilingual Bert using the `EncoderDecoder` framework. Not sure if xlm-r is yet supported in `EncoderDecoder`

Looking at xlm-r source code it seems that it can be easily added in EncoderDecoder as it subclasses Roberta which is supported in EncoderDecoder

Alright. I can try mBART and mBERT. 
What I was wondering about XLM was, if we could use it in a language modeling setting for this task, like how we use GPT for any seq2seq task.

Sending both text and the summary together and calculating the loss only over the summaries.

Not sure if that'll work since it's trained with MLM and encoder only with bi-directional attention. What you described above will need a causal LM with unidirectional attention.

EncoderDecoder class allows you to use encoder only models as both encoder and decoder and fine-tune for seq-2-seq task. Here's an example of Roberta2Roberta fine-tuned on CNN dm https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16

Makes sense. Saw XLMWithLMHead in https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py, so just got curious.

> Not sure if that'll work since it's trained with MLM and encoder only with bi-directional attention. What you described above will need a causal LM with unidirectional attention.



> EncoderDecoder class allows you to use encoder only models as both encoder and decoder and fine-tune for seq-2-seq task. Here's an example of Roberta2Roberta fine-tuned on CNN dm https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16

Oh thank you so much!

Also what you said is doable, xlm-r can be used like a causal LM by configuring the attention mask. Might not give the best results though. See how RobertaForCausalLM is implemented. 

> Makes sense. Saw XLMWithLMHead in https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py, so just got curious.
> 
> > Not sure if that'll work since it's trained with MLM and encoder only with bi-directional attention. What you described above will need a causal LM with unidirectional attention.

Aah Sorry, typo, I meant XLM-R, not xlm

> Also what you said is doable, xlm-r can be used like a causal LM by configuring the attention mask. Might not give the best results though. See how RobertaForCausalLM is implemented.

Ohh, sure. Will check it out.

Also, what will be the best way to finetune T5  in a multi-task setting.

Also, are there any models we can use for code-switched data.

> Also, are there any models we can use for code-switched data.

Not too familiar with this, but seen few models on model hub and they used Bert.
https://huggingface.co/sagorsarker/codeswitch-hineng-ner-lince

> Also, what will be the best way to finetune T5 in a multi-task setting.

If you can cast all your tasks in text-2-text format then multi-task training can be done simply using task pre-fixes as shown in the paper. Also I think the performance will depend upon the tasks and datasets so some experimentation is necessary.  Most important thing when doing multi-task is how you sample examples from different tasks. See section 3.5.2 of T5 paper.

Also the best place to ask this question would be
https://discuss.huggingface.co/t/t5-finetuning-tips/684

Alright, thank you so much for the help !!

I tried using Xlmr2Xlmr but seems that regardless of what input I provide I get the same output; I checked to see the is_decoder flag is set to true in the decoder. This issue persists throughout the finetuning process

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.