Fine-tuning model and Generation
Author: antmarakis
Pull request: no
Labels: name: wontfix

Hello!

I am beginner and I just wanted to run some experiments, but I've hit a road block. I am trying to generate text using `run_generator.py` after I fine-tune a model on my data using `simple_lm_finetuning.py`. I've looked around a bit, and I'm not sure how to go about this, or if this is possible at all. I don't see an option for `run_generator` to use BERT models, and I'm not sure how to bridge the two scripts.

Basically what I want to do is to fine-tune a model on my data and then generate text. Can this be done with `run_generator` and `simple_lm_finetuning`?

Thank you!

---

EDIT:

Forgot to add my code:

```
python pytorch-transformers/examples/lm_finetuning/simple_lm_finetuning.py \
    --train_corpus data.txt \
    --bert_model bert-base-uncased \
    --do_lower_case \
    --output_dir finetuned_lm/ \
    --do_train

python pytorch-transformers/examples/run_generation.py \
    --model_type=transfo-xl \
    --length=20 \
    --model_name_or_path='finetuned_lm'
```

Comments:
Having the same question - how to use bert for generation? 

the same problem， how to train my own data for text generation?

We'll add an example for fine-tuning this month.

@thomwolf as I read in other issues, BERT model cannot be used to generate text directly (your reply https://github.com/huggingface/pytorch-transformers/issues/401#issuecomment-477111518).
What exact examples are you planning to add? Thanks.

@Bruno-bai did you figure out how to train own data?

Not really. Would appreciate a tutorial:)

On Mon, Aug 19, 2019 at 5:22 AM Vedang Mandhana <notifications@github.com>
wrote:

> @Bruno-bai <https://github.com/Bruno-bai> did you figure out how to train
> own data?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/874?email_source=notifications&email_token=AHZ2KL5Q2RIKUTGARMX6EY3QFINYTA5CNFSM4IGHELH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4RUR4I#issuecomment-522406129>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHZ2KL3FS6CDZQN75KNXVPDQFINYTANCNFSM4IGHELHQ>
> .
>


this would be very useful example to have. to finetune gpt2, xlnet, ... and run generation from the finetuned model. Don't know whether bert supports generation or not, but the ones that do..

I am too struggling with similar problem. I want to train a non-english (hindi) language model on my custom dataset and use it for text generation. From what I understood, BERT sucks at text generation as it uses MLM for training. The ones that do well (gpt,trans-xl,xlnet) don't have a pretrained multilingual model available.

@Bruno-bai @sakalouski are you looking for training own data for language generation? Coz I have done it for classification and can help with that.

Hi @thomwolf 

> We'll add an example for fine-tuning this month.

Has this example been added yet?

Thanks

Hi @amin-nejad, the example has been added and is available [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_lm_finetuning.py).

Thanks @LysandreJik. Will this also work with Transformer-XL if we just modify the source code to include the Transformer-XL Config, LMHeadModel and Tokenizer as a model class? Or will it require more substantial changes?

Using `run_lm_finetuning.py` seemingly works for Transformer-XL if we additionally import the Transformer-XL Config, LMHeadModel and Tokenizer and modify the `MODEL_CLASSES` to include them. We also need to provide the `block_size` as a command line parameter. Training curves look reasonable and decoding also happens without errors using `run_generation.py` but the model output is pretty much always just a bunch of equals signs e.g. `= = = = = = = = =` etc. for me at least anyway. Clearly more substantial changes are required to `run_lm_finetuning.py`  to make it work. If anyone knows what/why, please let me know

One thing we should do (maybe when we have some bandwidth for that with @LysandreJik) is to push back a PR to PyTorch repo to add an option to have biases on all clusters of PyTorch's Adaptive Softmax so we can rely on the official Adaptive Softmax implementation instead of having our own.

That would make the job of maintaining and upgrading Transformer-XL a lot easier as it's currently the most cumbersome code base to maintain.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.