Building a State-of-the-art Text Classifier for any language you want!
By: Pratyush Parashar
Tags: NLP, Naturallanguageprocessing, Machine Learning, Fastai, Deep Learning

Overview

The idea is to first train(or use a pretrained model, if available) a language model on the wikipedia dataset of that language that can accurately predict the next word given a set of words( kind of what the keyboards on our phones do when they suggest a word). We then use that model to classify reviews, tweets, articles etc and amazingly with a few tweaks, we can build a state-of-the-art model for text classification in that language. For purpose of this article, I’ll be building the language model for Hindi language and use it to classify reviews/articles.

The method used in building this model is ULMFiT(Universal Language Model Fine-Tuning for Text Classification). The underlying concepts behind ULMFit are intricate and complex and the explaining its details would be better in a different article. However, the fastaiv1 library makes the task of language modelling and text classification quite easy and straightforward(requires less than 20 lines of code!!).

ULMFiT is also described by Jeremy Howard in in his mooc’s and can be found at these links. All the code mentioned below is available on my github.

The Wikipedia Dataset

We start off by downloading and cleaning the Wikipeda articles written in Hindi.

%reload_ext autoreload

%autoreload 2

%matplotlib inline # importing the required libraries

from fastai import *

from fastai.text import * torch.cuda.set_device(0)

# each lang has its code which is defined here(under the colunmn

# 'wiki': # Initializing variables# each lang has its code which is defined here(under the colunmn# 'wiki': https://meta.wikimedia.org/wiki/List_of_Wikipedias data_path = Config.data_path()

lang = 'hi'

name = f'{lang}wiki'

path = data_path/name

path.mkdir(exist_ok=True, parents=True) # create directory

lm_fns = [f'{lang}_wt', f'{lang}_wt_vocab']

Now let’s download the articles from wikipedia. Wikipedia mantains a List of Wikipedias that contains information regarding the number of articles present in a particular language, number of edits, and depth. The ‘depth’ column (defined as [Edits/Articles] × [Non-Articles/Articles] × [1 − Stub-ratio] ) is a rough indicator of a Wikipedia’s quality, showing how frequently its articles are updated. The higher the depth, the greater would be the quality of the articles.

def get_wiki(path,lang):

name = f'{lang}wiki'

if (path/name).exists():

print(f"{path/name} already exists; not downloading")

return xml_fn = f"{lang}wiki-latest-pages-articles.xml"

zip_fn = f"{xml_fn}.bz2"

print("downloading...")

download_url(f'

print("unzipping...")

bunzip(path/zip_fn) if not (path/xml_fn).exists():print("downloading...")download_url(f' https://dumps.wikimedia.org/{name}/latest/{zip_fn}' , path/zip_fn)print("unzipping...")bunzip(path/zip_fn) with working_directory(path):

print("extracting...")

os.system("python -m wikiextractor.WikiExtractor --processes 4 --no_templates " +

f"--min_text_length 1800 --filter_disambig_pages --log_file log -b 100G -q {xml_fn}")

shutil.move(str(path/'text/AA/wiki_00'), str(path/name))

shutil.rmtree(path/'text') def split_wiki(path,lang):

dest = path/'docs'

name = f'{lang}wiki'

if dest.exists():

print(f"{dest} already exists; not splitting")

return dest

title_re = re.compile(rf'<doc id="\d+" url="

lines = (path/name).open()

f=None dest.mkdir(exist_ok=True, parents=True)title_re = re.compile(rf' https://{lang}.wikipedia.org/wiki\?curid=\d+ " title="([^"]+)">')lines = (path/name).open()f=None for i,l in enumerate(lines):

if i%100000 == 0: print(i)

if l.startswith('<doc id="'):

# Since "/" is a directory seperator, we replace those with _ when # saving the file names.

title = title_re.findall(l)[0].replace('/','_')

if len(title)>150: continue # if title_length>150 skip # article.

if f: f.close()

f = (dest/f'{title}.txt').open('w')

else: f.write(l)

f.close()

return dest

The get_wiki function downloads the xml version of all the articles as zip file. We then extract the contents using bunzip and run Wikiextractor (remember to run !pip install wikiextractor first) to clean the files we extracted. We clean it because the xml file contains some xml and metadata which is not great for a language model as we don’t want the model to predict the next token in an xml file but to predict it in standard text file. We only select articles that have a minimum text length of 1800.

We then call split_wiki which splits the single wikipedia file into a seperate file per article. Each article is seperated with

<doc id=”14" url=”https://hi.wikipedia.org/wiki?curid=14" title=”दैनिक पूजा”>`

so we just compile a regular expression and search for those. After running this, the path should contain the following files(-models):

[PosixPath('/home/jupyter/.fastai/data/hiwiki/hiwiki'),

PosixPath('/home/jupyter/.fastai/data/hiwiki/log'),

PosixPath('/home/jupyter/.fastai/data/hiwiki/docs'),

PosixPath('/home/jupyter/.fastai/data/hiwiki/models'),

PosixPath('/home/jupyter/.fastai/data/hiwiki/hiwiki-latest-pages-articles.xml'),

PosixPath('/home/jupyter/.fastai/data/hiwiki/hiwiki-latest-pages-articles.xml.bz2')]

We are now ready to load the data and train our language model.

Create and train a Language model

We load the data into a fastai databunch and create our model.

data = (TextList.from_folder(dest)

.split_by_rand_pct(0.1, seed=42)#10% data used for validation

.label_for_lm()

.databunch(bs=bs, num_workers=1))

data.save(f'{lang}_databunch') # save databunch.

len(data.vocab.itos), len(data.train_ds) learn = language_model_learner(data, AWD_LSTM, drop_mult=0.5, pretrained=False).to_fp16()

We use the language_model_learner class from fastai and use an AWD-LSTM with all dropouts multiplied by 0.5 . We also use mixed precision to_fp16() in order to quickly train our model. More details on that here.

Now we set the learning rate(usually 0.01 is fine but it varies with batch_size) and train our language model for 20 epochs. You could also use learn.lr_find and then learn.recorder.plot() to find the best learning rate.

lr = 1e-2

lr *= bs/48 # scale LR by batchsize.

learn.unfreeze()

learn.fit_one_cycle(20, lr, moms=(0.8, 0.7))

We then save our model after training is complete. It’s pretty easy to acheive upto 40% accuracy for any language model. Since the number of hindi articles available on wikipedia are less the model finds it harder to genralize( for context, there are about 10 times as many articles in English than in Hindi).

mdl_path = path/'models'

mdl_path.mkdir(exist_ok=True)

learn.to_fp32().save(mdl_path/lm_fns[0], with_opt=False)

learn.data.vocab.save(mdl_path/(lm_fns[1] + '.pkl'))

Now we come to the sentiment analysis part.We have 3 available classfication datasets in hindi to check where we stand: BBC articles, IITP movie reviews and IITP product reviews. I’ll only be using the BBC dataset here to perform sentiment analysis. You can have a look at the repo where I have evaluated the model on other datasets mentioned above.

Fine-Tuning the Language model

We load up the train, valid and test sets using pandas. What we now do is use this new dataset to further fine-tune the language model. Why is this an important step? Well, there may be words in the classification dataset that show up a lot in reviews/articles but don’t show up as much in Wikipedia articles(eg: blockbuster movie names, actors, actresses, etc). To take full advantage of that we use the train, valid and test datasets to finetune the language model. This doesn’t constitute as cheating because we’re using only the review column of the datasets and they’re being used for a model that predicts the next word and not whether a review is positive, negative or neutral. This kind of training is referred to as semi-supervised learning. We can use all the unlabelled data that we have here.

train_df = pd.read_csv('data/bbc-articles-hi/hi-train.csv', header=None)

test_df = pd.read_csv('data/bbc-articles-hi/hi-test.csv', header=None)

df = pd.concat([train_df,test_df], sort=False)

We combine the train and test dataframes and load them up as a fastai databunch to be fed into the model.

data_lm = (TextList.from_df(df, path, cols=1) # the article col.

.split_by_rand_pct(0.1, seed=42)

.label_for_lm()

.databunch(bs=bs, num_workers=1))

We then define the language model learner, load up the previously saved one, and train it for a couple epochs. The language_model_learner class has a parameter pretrained_fnames where we pass the model and its vocab.

learn_lm = language_model_learner(data_lm, AWD_LSTM, pretrained_fnames=lm_fns, drop_mult=1.0)

lr = 1e-3

lr *= bs/48

# finetuning the language model

learn_lm.fit_one_cycle(2, lr*10, moms=(0.8,0.7))

learn_lm.unfreeze()

learn_lm.fit_one_cycle(8, lr, moms=(0.8,0.7))

# saving the model and the encoder.

learn_lm.save(f'{lang}fine_tuned')

learn_lm.save_encoder(f'{lang}fine_tuned_enc')

We are now ready to feed the data into the encoder and classify the articles.

NOTE: If your instance runs out of memory, try reducing the batch_size or restarting the kernel. You could also make use of garbage collector in python by assigning variables you longer need to None and then calling gc.collect().

Training the classifier

data_clas = (TextList.from_df(train_df, path, vocab=data_lm.vocab, cols=1)

.split_by_rand_pct(0.1, seed=42)

.label_from_df(cols=0)

.databunch(bs=bs, num_workers=1)) # saving the databunch so you could just load it up again when re- # running the notebook.

data_clas.save(f'{lang}_textlist_class')

data_clas = load_data(path, f'{lang}_textlist_class', bs=bs, num_workers=1)

We can see that our dataset has 14 classes which are:

You can also call data_clas.show_batch() to view your data. I’ve used 3 metrics in the notebook, namely accuracy, f1 score and Mathews Correlation(mcc). F1 score and mcc, both have their pros and cons and so I tried using both after reading this article. By default f1 score has the average parameter set to ‘binary’ and since this is a multiclass classification, we set it to ‘micro’.

We now load up the classifier and set it to fp16() as it helps in training quickly(mixed precision).