Turkish NER Model Training using Spark NLP, with the help of NLU.
By: Murat Gunay
Tags: Deep Learning, Spark Nlp, Python, Nlu, NLP

Labeled Data Set

The raw text data set I used is compiled from Turkish news and consists of 27.556 sentences. The labeled entities are PERSON, LOCATION, and ORGANIZATION.

Data (text) Preparation and Creating CoNLL File

In this topic, I will cover all steps quickly to create my CoNLL file for NER training.

I use Google Colab (stands for Collaboratory) which is a product from Google Research.

Colab allows anybody to write and execute arbitrary python code through the browser and is especially well suited to machine learning, data analysis, and education.

More technically, Colab is a hosted Jupyter notebook service that requires no setup to use, while providing free access to computing resources including GPUs.

First, we initialize the Java environment, PySpark, Spark NLP, and NLU :

import os # Install java

! apt-get update -qq

! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"

os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]

! java -version # Install pyspark

! pip install --ignore-installed pyspark==2.4.4 # Install Spark NLP

! pip install --ignore-installed spark-nlp # Install Spark NLU

! pip install nlu > /dev/null

Second, Starting the Spark session. I customize my spark session to fit my use-case which requires high driver memory and Kryo buffer capacity :

from pyspark.sql import SparkSession spark = SparkSession.builder \

.appName("Spark NLP") \

.master("local[*]") \

.config("spark.driver.memory", "20G") \

.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \

.config("spark.kryoserializer.buffer.max", "2000M") \

.config("spark.driver.maxResultSize", "20G") \

.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.6.2").getOrCreate()

Then, Loading the necessary packages :

from pyspark.ml import Pipeline

from sparknlp.annotator import *

from sparknlp.common import *

from sparknlp.base import *

import pandas as pd

from sparknlp.training import CoNLL

import pyspark.sql.functions as F

import nlu

from sklearn.metrics import classification_report

Downloading Data Set

Now we need to load the data set (I can not share the labeled data set). After downloading it, let’s read it and start to get it ready for the CoNLL format.

with open("Turkish_NER_Tagged.txt", "r", encoding="utf-8") as file:

text = file.read() text[0:100]

Output :

Again, Data Cleaning

As we can see in the sample of this data set, this corpus is actually tokenized and separated by spaces.

The labels are in the HTML tag format. In order to capture the whole tag and the labeled tokens, we should remove the spaces inside the tags as follows.

<b_enamex TYPE= → <b_enamexTYPE=

→ We need to separate the document sentence by sentence due to the creating correct form of the CONLL file. So we should read the file line by line.

with open("Turkish_NER_Tagged.txt", "r", encoding="utf-8") as file:

text_ln = file.readlines() len(text_ln) >> 27556 # the number of sentences

I remove the spaces as I mentioned above and collect the all sentences line by line into a list named text.

text = [i.replace(‘<b_enamex TYPE=’, ‘<b_enamexTYPE=’) for i in text_ln]

NLU Time!

It is time to use the NLU library to create a pandas DataFrame for preparing the dataset to create a CoNLL file easily. Follow the steps carefully.

pipe = nlu.load("tokenize") df = pipe.predict(text, output_level="token") # We need token columns in our DataFrame, so we set "token" for output level df # Great! We have 513216 tokens in "token" column and

# "origin_index" represents the sentences.

Output :

This is an NLU miracle! We have created pandas DataFrame in which all tokens are in a column. This will be very useful for creating the CoNLL file format. Ok, keep focusing on the codes line by line.

df["sent_id"] = df.index # We create a new column named "sent_id" to use later. We will use this column for separating sentences by a blank line as in CoNLL file format. df.reset_index(drop = True, inplace = True) df["ner"] = "O"

# We create a new column to collect the ner labels into it

Let’s start to extract and collect the NER labels into the “ner” column.

count = 0 for tok in df.token.values: if ('<b_enamexTYPE="LOCATION">' in tok):

if ('<e_enamex>' in tok) : df.ner.iloc[count] = "B-LOC"

if ('<e_enamex>' not in tok) : df.ner.iloc[count] = "BB-LOC" if ('<b_enamexTYPE="ORGANIZATION">' in tok):

if ('<e_enamex>' in tok): df.ner.iloc[count] = "B-ORG"

if ('<e_enamex>' not in tok): df.ner.iloc[count] = "BB-ORG" if ('<b_enamexTYPE="PERSON">' in tok):

if ('<e_enamex>' in tok): df.ner.iloc[count] = "B-PER"

if ('<e_enamex>' not in tok): df.ner.iloc[count] = "BB-PER" if ('<b_enamexTYPE=' not in tok) and ('<e_enamex>' in tok):

df.ner.iloc[count] = "I-XXX"

count+=1

We have filled the ner column with B-Labels regarding label tags, but we need to correct the I-Label formats as well. In this code snippet above, BB- represents the beginning of NER chunks. So, we can specify the NER category based on label tags.

for indx in range(len(df.ner.values)): if df.ner.iloc[indx] == "BB-PER" :

count = 0

while df.ner.iloc[indx] != "I-XXX":

indx += 1

count += 1 for i in range(indx, indx-count,-1):

df.ner.iloc[i] = "I-PER" if df.ner.iloc[indx] == "BB-ORG" :

count = 0

while df.ner.iloc[indx] != "I-XXX":

indx += 1

count += 1 for i in range(indx, indx-count,-1):

df.ner.iloc[i] = "I-ORG" if df.ner.iloc[indx] == "BB-LOC" :

count = 0

while df.ner.iloc[indx] != "I-XXX":

indx += 1

count += 1 for i in range(indx, indx-count,-1):

df.ner.iloc[i] = "I-LOC"

Now, it is time to get rid of the NER labels (HTML Tags) in the token column. After removing HTML NER tags, I will create two pos columns (pos1-pos2) that we need in the CoNLL files.

rep_dic = {'<b_enamexTYPE="LOCATION">':"", '<b_enamexTYPE="ORGANIZATION">':"", '<b_enamexTYPE="PERSON">':"", '<e_enamex>':""} df.replace({"token" : rep_dic, "ner" : {"BB-" : "B-"}}, regex=True, inplace=True) df["pos1"] = "NN"

df["pos2"] = "NN" # It doesn't matter what the pos tag is.

# We choose to fill it with "NN". It is just for

# CoNLL file format. df = df[["token", "pos1", "pos2", "ner", "sent_id"]]

Yes, we are getting closer to the end. Now, we should put empty lines between each sentence to fit the CoNLL file format.

df_dummy = pd.DataFrame()

df_blank = pd.DataFrame({"token":[np.nan], "pos1":[np.nan], \

"pos2":[np.nan], "ner" : [np.nan], \

"sent_id":[np.nan]}) for i in range(df.sent_id.nunique()):

df_part = df[df["sent_id"] == i]

df_dummy = pd.concat([df_dummy, df_blank, df_part], ignore_index=True) df_conll = df_dummy[["token","pos1","pos2","ner"]] df_conll.columns = ["-DOCSTART-", "-X-", "-X-", "O"] df_conll.head(10)

Output :

Finally! It is ready. Let’s write the df_conll to disk as a CoNLL file.

df_conll.to_csv("Turkish_NER.conll", index=False, sep=" ") # CONLL file is ready to use!..

Training Turkish NER Model with GloVe

Yes, we have once again observed that data cleaning and preparation continues to take up most of our time. This is the nature of this business, as stated in this article. [3]

ML/DL Process is a small part of our business.

Finally, we produced our CONLL file for use. Let’s start to train our model using Multi-Language GloVe Embeddings("glove_840B_300")

Loading, Exploring, and Splitting the CoNLL File

data = CoNLL().readDataset(spark, "Turkish_NER.conll")

data.show()

Output :

CoNLL() Function Transforms the Data Set into Annotated DataFrame which is ready to be trained.

Yes, we can start to set up the NER training pipeline step by step. At first, we will split 80% of our data set randomly as “train_data” to train our models, and the remaining 20% ​​as “test_data” to measure the performance of our trained model.

train_data, test_data = data.randomSplit([0.8, 0.2], seed = 100) train_data.select(F.explode(F.arrays_zip('token.result','label.result')).alias("cols")).select(F.expr("cols['0']").alias("token"), F.expr("cols['1']").alias("ground_truth")).groupBy('ground_truth').count().orderBy('count', ascending=False).show(100,truncate=False)

Output :

Setting Pipeline

I use GloVe Multi-language Word Embeddings (“glove_840B_300”) which is very successful in NER training.

glove_embeddings = WordEmbeddingsModel() \

.pretrained('glove_840B_300','xx')\

.setInputCols(["document",'token'])\

.setOutputCol("embeddings")\

.setCaseSensitive(True) nerTagger = NerDLApproach()\

.setInputCols(["sentence", "token", "embeddings"])\

.setLabelColumn("label")\

.setOutputCol("ner")\

.setMaxEpochs(2)\

.setLr(0.002)\

.setPo(0.005)\

.setBatchSize(16)\

.setRandomSeed(0)\

.setVerbose(1)\

.setValidationSplit(0.2)\

.setEvaluationLogExtended(True) \

.setEnableOutputLogs(True)\

.setIncludeConfidence(True)\

.setOutputLogsPath('ner_logs') ner_pipeline_glove = Pipeline(stages=[

glove_embeddings,

nerTagger

])

The parameters in nerTagger and customizing policy are explained extensively in this article [1]. It can be helpful for the best understanding.

We have set our pipeline for NER training. Now we have to fit our pipeline to train_data to create the NER model.

Training and Saving Our NER Model

ner_model_glove = ner_pipeline_glove.fit(train_data) ner_model_glove.stages[1].write().overwrite().save("Tr_NER_Glove_20201019")

That’s all. Everything was just for these two lines of codes. Let’s take a look at the performance of our NER Model.

Performance Metrics

test_data = glove_embeddings.transform(test_data) predictions = ner_model_glove.transform(test_data) preds_df = predictions.select(F.explode(F.arrays_zip('token.result',\

'label.result','ner.result')).alias("cols")) \

.select(F.expr("cols['0']").alias("token"), \

F.expr("cols['1']").alias("ground_truth"), \

F.expr("cols['2']").alias("prediction")).toPandas() print (classification_report(preds_df['ground_truth'], \ preds_df['prediction'], digits=4))

Output :

The results are promising.

Let’s Make Some Predictions with Real Turkish Text

First, we have to set the prediction pipeline.

document = DocumentAssembler()\

.setInputCol("text")\

.setOutputCol("document") sentence = SentenceDetector()\

.setInputCols(['document'])\

.setOutputCol('sentence') token = Tokenizer()\

.setInputCols(['sentence'])\

.setOutputCol('token') glove_embeddings = WordEmbeddingsModel() \

.pretrained('glove_840B_300','xx')\

.setInputCols(["sentence",'token'])\

.setOutputCol("embeddings")\

.setCaseSensitive(True) loaded_ner_model = NerDLModel.load("Tr_NER_Glove_20201019")\

.setInputCols(["sentence", "token", "embeddings"])\

.setOutputCol("ner") converter = NerConverter()\

.setInputCols(["document", "token", "ner"])\

.setOutputCol("ner_span") ner_prediction_pipeline = Pipeline(

stages = [document,

sentence,

token,

glove_embeddings,

loaded_ner_model,

converter])

Ok, secondly, I have set the prediction pipeline and now I can fit it to an empty data frame to create a prediction model.

empty_data = spark.createDataFrame([['']]).toDF("text") prediction_model = ner_prediction_pipeline.fit(empty_data) prediction_model.transform(empty_data)

Finally, we can predict real Turkish text using Light Pipeline and ner_highlighter function.

from sparknlp.base import LightPipeline light_model = LightPipeline(prediction_model) !wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Public/utils/ner_highlighter.py import ner_highlighter

Prediction Samples

text = “Oğlum Kerem çok iyi bir çocuktur. İstanbul Kadıköy sahilinde yaşamaktadır. Kerem, Power FM Radyosu’nda çalışmaktadır.”

text = “İlk defa Güneydoğu Anadolu bölgesinde düzenlenecek olan Uluslararası Nöroloji Kongresi başkanlığını, bu yıl Esra Günay üstlenecek.”

text = “Türkiye Futbol Federasyonu başkanı Nihat Özdemir, Kocaeli Gölcük’e doğru yola çıktı.”

text = “Güzellikler Partisi İl Başkanlığı toplantısı bu yıl Ankara Çankaya merkezde icra edildi. Toplantı başkanlığını Ali Veli yürüttü.”

result = light_model.fullAnnotate(text)[0] ner_highlighter.chunk_highlighter(result, entity_column="ner_span")

Outputs :

Predictions are not bad at all.

text = “Şehzadeler ilçesindeki Rehberlik ve Araştırma Merkezi Özel Eğitim Hizmetleri Bölümü’nde görevli rehberlik ve danışmanlık öğretmeni Sabri Bozkurt, fazla kiloları nedeniyle sorun yaşayınca, bir arkadaşının tavsiyesiyle bisikletle spor yapmaya başladı. Bozkurt, zamanla tutkusu haline gelen bisiklet sporuyla 2,5 senede 112 kilodan 94 kiloya düştü. Özellikle koronavirüs süresince 7 ayda 6 bin kilometre pedal çeviren Bozkurt, Gökdoğanlar Bisiklet Kulübü adında bir kulüp kurarak, grup üyeleriyle birlikte Ege bölgesi başta olmak üzere Türkiye’nin farklı şehirlerine gezi düzenliyor.”

result = light_model.fullAnnotate(text)[0] ner_highlighter.chunk_highlighter(result, entity_column="ner_span")

Output :

Conclusion

In this post, I have trained a Turkish NER model using Spark NLP, NLU with multi-language GloVe Word Embeddings. The average f1 score is about %89 which is so good for prediction.

References

[1] Veysel Kocaman, Named Entity Recognition (NER) with BERT in Spark NLP, (2020)

[2] Christian Kasim Loan, NER model with ELMo Embeddings in 5 minutes in Spark-NLP, (2020)

[3] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Franc¸ois Crespo, Dan Dennison, Hidden Technical Debt in Machine Learning Systems