How to contribute to “Write with transformer”?
Author: mauceri
Pull request: no
Labels: name: wontfix

name: Write With Transformer

## 🚀 I would like to contribute to a French version of this App

I’m French, I write short stories, and I’m also a software engineer

## Motivation

I’ll retire in 6 months and I wanted to build such an app before I stumbled on your demo.

## Additional context

https://www.linkedin.com/in/mauceri/


Comments:
What is it that you can contribute? The only (yet impressive) thing that is going on is language modeling. Can you contribute a pre-trained French model for one of the frameworks? That's (as far as I know) the only way to contribute. 

Thanks Bram, I’m going to investigate what the cost could be for XLNet on clevergrid https://www.clevergrid.io/?pk_campaign=ga-gpu-1&pk_source=adwords&pk_medium=sem&pk_content=gpuasaservicefr&gclid=CjwKCAjwibzsBRAMEiwA1pHZrvm8ozRMrbcDR7YoYiKqsq6gEnPo9AecJwjKzBxa8L-4_hB6ny4uARoCwfMQAvD_BwE

Envoyé de mon iPad

> Le 28 sept. 2019 à 09:44, Bram Vanroy <notifications@github.com> a écrit :
> 
> What is it that you can contribute? The only (yet impressive) thing that is going on is language modeling. Can you contribute a pre-trained French model for one of the frameworks? That's (as far as I know) the only way to contribute.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.


Hi all,
We (ovh) are open to calculate it for free.

Not sure if a new French language model is still necessary after Camembert has been introduced.

That's awesome news, @jqueguiner! Let us know if we can help.

@BramVanroy To work well with Write With Transformer, we would want more like a FR-pretrained GPT-2-like model. CamemBERT wouldn't do on generation out of the box.

See also the more specific issue: https://github.com/huggingface/transformers/issues/1356

For generation CamemBERT is of no use I think...

Envoyé de mon iPad

> Le 22 nov. 2019 à 14:02, Bram Vanroy <notifications@github.com> a écrit :
> 
> ﻿
> Not sure if a new French language model is still necessary after Camembert has been introduced.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.


Yes, CamemBERT is awesome, but for WWT we need a FR-Pretrained GPT-2 model!

Envoyé de mon iPad

> Le 22 nov. 2019 à 14:55, Julien Chaumond <notifications@github.com> a écrit :
> 
> ﻿
> That's awesome news, @jqueguiner! Let us know if we can help.
> 
> @BramVanroy To work well with Write With Transformer, we would want more like a FR-pretrained GPT-2-like model. CamemBERT wouldn't do on generation out of the box.
> 
> See also the more specific issue: #1356
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.


Camembert doesnt offer generation, only syntax analysis and masking due to the nature of the network. Multiple mask generation (<mask><mask><mask>) gives really uggly results as you cna test here : https://market-place.ai.ovh.net/#!/apis/43323c37-59e7-4092-b23c-3759e7c09288/pages/94d31892-4e64-446f-9318-924e64346f9e

IMO we should start training using OSCAR dataset 
https://traces1.inria.fr/oscar/

@julien-c yes we can start with a collab GPT2 french training ipynb together then I'll prepare the env for a DGX1 or something similar. I didn't train a GPT2 before. IS it scaling over multiple GPU's ? do we need horovod adaptation ?

Oops, sorry everyone. I thought this was a general French model question. My bad. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.