Is there an implementation of BERT architecture in PyTorch that I can modify here?
Author: abhisheknovoic
Pull request: no

Hello Team,

  Firstly, thanks for this amazing repo.

I am doing my own research and I want access to a native implementation of BERT in PyTorch so I can modify the architecture and play with it by including a few of my own modules.

Is that possible with the codebase in HuggingFace repo here?

Thanks

Comments:
Hi! Yes, you can modify the BERT architecture as you please, it's self contained. It's in the [modeling_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py) file.

Thanks @LysandreJik , can you also confirm if this implementation supports multi gpu training? 

This implementation is a PyTorch model, so it supports everything a PyTorch model can do :) GPU, Multi-GPU, TPU, you name it.

Hello @LysandreJik  and team, I am looking at the script `run_language_modeling.py` at https://github.com/huggingface/transformers/tree/master/examples/language-modeling . 

I saw that the example uses WikiText-2 dataset for example. If I want to fine-tune BERT on my own dataset, how should the dataset be structured? Should I mask the words myself or is there some DataLoader that will do that?

I downloaded the WikiText data and I saw an example chunk of text is 
```
 = Robert <unk> =

 Robert <unk> is an English film , television and theatre actor . He had a guest @-@ starring role on the television series The Bill in 2000 . This was followed by a starring role in the play Herons written by Simon Stephens , which was performed in 2001 at the Royal Court Theatre . He had a guest role in the television series Judge John <unk> in 2002 . In 2004 <unk> landed a role as " Craig " in the episode " Teddy 's Story " of the television series The Long Firm ; he starred alongside actors Mark Strong and Derek Jacobi . He was cast in the 2005 theatre productions of the Philip Ridley play Mercury Fur , which was performed at the Drum Theatre in Plymouth and the <unk> <unk> Factory in London . He was directed by John <unk> and starred alongside Ben <unk> , Shane <unk> , Harry Kent , Fraser <unk> , Sophie Stanton and Dominic Hall .
 In 2006 , <unk> starred alongside <unk> in the play <unk> written by Mark <unk> . He appeared on a 2006 episode of the television series , Doctors , followed by a role in the 2007 theatre production of How to Curse directed by <unk> <unk> . How to Curse was performed at Bush Theatre in the London Borough of <unk> and Fulham . <unk> starred in two films in 2008 , <unk> <unk> by filmmaker Paris <unk> , and <unk> Punch directed by <unk> Blackburn . In May 2008 , <unk> made a guest appearance on a two @-@ part episode arc of the television series Waking the Dead , followed by an appearance on the television series <unk> in November 2008 . He had a recurring role in ten episodes of the television series <unk> in 2010 , as " <unk> Fletcher " . <unk> starred in the 2011 film <unk> directed by Paris <unk> .

 = = Career = =
```

In my case I have a large set of text files. Just text files with free text inside it. Can someone point me to a document/resource that lets me understand how should the input be for masked language modelling pretraining using BERT?

I plan to use the `https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py` file contents, modify the layers a bit based on my architecture decisions and train it on my own dataset using masked language modeling where random words are masked and I predict them back.

Any help is appreciated. Thanks

Hi @abhisheksgumadi, this is a very interesting and rather broad question. Could you ask it on the forums over on https://discuss.huggingface.co? Thanks a lot!