spaCy NER Model to Identify Scientific Datasets — Coleridge Initiative
By: Bao Tram Duong
Tags: Coleridge Initiative, Flatiron School, NLP, Kaggle Competition, Spacy Ner Train Python

INTRODUCTION

In this notebook, we will be training spaCy NER to identify DATASET entities from a body of text , a task known as named-entity recognition (NER). NER is a fundamental Natural Language Processing (NLP) task to extract entities of interest.

DATA SOURCE

This project uses dataset from the Kaggle competition Coleridge Initiative where data scientists are challenged to show how publicly funded data are used to serve science and society. The results will show how public data are being used in science and help the government make wiser, more transparent public investments. It will help move researchers and governments from using ad-hoc methods to automated ways of finding out what datasets are being used to solve problems, what measures are being generated, and which researchers are the experts.

METHODOLOGY

In this competition, natural language processing (NLP) is used to automate the discovery of how scientific data are referenced in publications. Utilizing the full text of scientific publications from numerous research areas gathered from CHORUS publisher members and other sources, data scientists will identify data sets that the publications’ authors used in their work.

The objective of the competition is to identify the mention of datasets within scientific publications. The goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques.

Disclaimer: This is NOT the solution for this competition. Please check out the Leaderboard for winning notebooks. In this blog, my focus is text processing and sequence labeling: how to look for scientific articles in a body of text, process the raw text and subsequently classify them with predefined labels.

I attempted:

n-gram models (blog, n-gram notebook)

Recurrent neural network (RNN) deep learning models (blog, Bidirectional LSTM notebook, GRU notebook)

Convolutional neural network (CNN) deep learning models (blog, sep-CNN notebook)

spaCy NER (blog, spaCy notebook)

Summary of everything: blog, notebook

For this part of the project, we will discuss spaCy NER method.

What is spaCy?

spaCy is an open-source library for advanced Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more. It is designed specifically to build applications that process and understand large volumes of text. It can be used to build information extraction or natural language understanding systems, or to pre-process text for deep learning.

Parts-of-Speech (PoS) Tagging

Let’s use a sample sentence from our training dataset:

'using rich data from early childhood longitudinal study we estimate the effects of prekindergarten on children s school readiness.'

By using .load(‘en_core_web_sm') , spaCy returns a Language object containing all components and data needed to process text.

Calling the nlp object on a string of text will return a processed Doc that provides word types, like the parts of speech, and how the words are related to each other such as below:

TEXT POS DEP using VERB advcl

rich ADJ amod

data NOUN dobj

from ADP prep

early ADJ amod

childhood NOUN nmod

longitudinal ADJ amod

study NOUN pobj

we PRON nsubj

estimate VERB ROOT

the DET det

effects NOUN dobj

of ADP prep

prekindergarten ADJ pobj

on ADP prep

children NOUN poss

s PART case

school NOUN compound

readiness NOUN pobj

. PUNCT punct

Other spaCy functions are:

LEMMA TAG SHAPE ALPHA STOP use VBG xxxx 1 1

rich JJ xxxx 1 0

datum NNS xxxx 1 0

from IN xxxx 1 1

early JJ xxxx 1 0

childhood NN xxxx 1 0

longitudinal JJ xxxx 1 0

study NN xxxx 1 0

-PRON- PRP xx 1 1

estimate VBP xxxx 1 0

the DT xxx 1 1

effect NNS xxxx 1 0

of IN xx 1 1

prekindergarten JJ xxxx 1 0

on IN xx 1 1

child NNS xxxx 1 0

s POS x 1 0

school NN xxxx 1 0

readiness NN xxxx 1 0

. . . 0 0

Named Entity Recognition (NER)

For this particular task, we will not use POS Tagging but instead use NER, or Named Entity Recognition.

Commonly NER tasks consist of recognizing persons’ names, locations (geographic objects), organizations, phone numbers, cardinal, GPE. The existing core NLP models from spaCy are trained to recognize various entities as given below.

Training spaCy NER with Custom Entities

Our aim is to further train this model to incorporate for our own custom DATASET entities present in our training dataset. To do this we have to go through the following steps:

Generate sentences with DATASET entities Train the general model that is capable of distinguishing between entities and non-entities Annotate the new dataset using this model Replacement of the ent tags with DATASET Train this entity-specific model

DATA ANNOTATION: GETTING SENTENCES & LABELS

For the full EDA of this dataset, see notebook.

Transforming Data into spaCy Compliant Format

For training a spaCy model for customized named entity recognition, the data must be transformed to the following format:

[(input text, entities:

[(start_index, end_index, entity_name), start_index, end_index, entity_name), ...]), ....]

Using the following code, we read each individual publication and break it down into sentences using sent_tokenize function from the nltk (Natural Language Toolkit), which is a leading platform for building Python programs to work with human language data. For each sentence, we will use .search to search for matching dataset_title, dataset_label, cleaned_label in each of the sentences. When a match is found, it will be returned as a tuple containing starting and ending index of the matched string and labeled as DATASET .

To process the sentence, we use 2 functions clean_text and shorten_sentences to process the text. Since we need to preserve the exact words for dataset_title , the text cleaning process is simple with removing special characters and lower-casing. Then we break the sentences into shorter sentences.

The first step is to build a universal model that is capable of distinguishing between entities and non-entities. Since this is a supervised learning problem, we need to get a list of training data called DATA , that is annotated manually. We will go publication by publication and label each string that match with our dataset_title, dataset_label, cleaned_label as DATASET .

We generated 64,512 sentences with the matched strings and 50,303 sentences without the matched strings.

Text with entities: 64512

Text without entities: 50303

The output will have the required format for spaCy, for instance:

[('text this study used data from the national education longitudinal study nels 88 to examine the effects of dual enrollment programs for high school students on college degree attainment.',

{'entities': [(35, 72, 'DATASET')]}), ('the study also reported whether the impacts of dual enrollment programs were different for first generation college students versus students whose parents had attended at least some college.',

{'entities': []}), ('section title degree attainment text any college degree attainment the study author collected information on college degree attainment from the fourth follow up of the national education longitudinal study collected in 2000.',

{'entities': [(168, 205, 'DATASET')]}), ('only students who attended college were included in the study.',

{'entities': []}), ('section title bachelor s degree attainment text the study author collected information on bachelor s degree attainment from the fourth follow up of the national education longitudinal study collected in 2000.',

{'entities': [(152, 189, 'DATASET')]})]

Let’s put it into a dataframe:

MODELING

After preprocessing the data and having prepared it to train, we need to further add the vocabulary of new entities in the model ner pipeline. The entire code for training is:

Break down of train_spacy function

Load the model, or create an empty model using spacy.blank with the ID of desired language, in this case, en for English. Because there is no model trained previously, a blank English model will be created and we have to add the DATASET entity recognizer to the pipeline.

#setting up the pipeline and entity recognizer if model is not None:

print(f"training existing model")

nlp = spacy.load(model)

print("Model is Loaded '%s'" % model)

else:

print(f"Creating new model") nlp = spacy.blank('en') if 'ner' not in nlp.pipe_names:

ner = nlp.create_pipe('ner')

nlp.add_pipe(ner, last = True)

else:

ner = nlp.get_pipe('ner')

2. Add the new entity label to the entity recognizer using the add_label method. This is where we add DATASET .

#based on template, get labels and save those for further training

for _, annotations in DATA:

for ent in annotations.get("entities"):

ner.add_label(ent[2])

3. If an existing model is being used, we have to disable all other pipeline components during training using nlp.disable_pipes. This way, only the entity recognizer gets trained.

The core spaCy models have three pipelines: tagger , parser , and ner . We need to disable tagger and parser pipelines, since we will only be training the ner pipe, although, one can train all the other pipelines simultaneously.

#get names of other pipes to disable them during training

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != ‘ner’]

with nlp.disable_pipes(*other_pipes): if model is None:

optimizer = nlp.begin_training() else:

optimizer = nlp.entity.create_optimizer()

4. nlp.update steps through the words of the input. At each word, it makes a prediction. It then consults the annotations, to see whether it was right. If it was wrong, it adjusts its weights so that the correct action will score higher next time.

tags = dict()

for itn in range(iterations):

print("Starting iteration " + str(itn))

random.shuffle(DATA)

losses = {}

batches = minibatch(DATA, size = compounding(4.0, 16.0, 1.001))



#type 2 with mini batch

for batch in batches:

texts, annotations = zip(*batch)

golds = annotations

nlp.update(texts,

golds,

drop = 0.2,

losses = losses,

sgd = optimizer) print(losses)

4. Save the trained model using nlp.to_disk .

#save

nlp.meta["name"] = "dataset_entity_extractor" nlp.to_disk("dataset_entity_extractor") import pickle



pickle.dump(nlp, open( "dataset_entity_extractor.p", "wb" ))

5. Lastly, we train with 30 epochs.

downloads = en_core_web_sm

training existing model

Model is Loaded 'en_core_web_sm'

Starting iteration 0

{'ner': 1263480.8545014686}

Starting iteration 1

{'ner': 1239889.9346434528}

Starting iteration 2

{'ner': 1236714.1569774537}

Starting iteration 3

{'ner': 1224203.932969632}

Starting iteration 4

{'ner': 1220028.6853948643}

Starting iteration 5

{'ner': 1219100.5855520354}

Starting iteration 6

{'ner': 1219885.752013701}

Starting iteration 7

{'ner': 1227414.530614621}

Starting iteration 8

{'ner': 1229595.3504867007}

Starting iteration 9

{'ner': 1226151.068897839}

Starting iteration 10

{'ner': 1229808.705460116}

Starting iteration 11

{'ner': 1229848.655951032}

Starting iteration 12

{'ner': 1225323.3275067185}

Starting iteration 13

{'ner': 1224625.4582372657}

Starting iteration 14

{'ner': 1231397.426613342}

Starting iteration 15

{'ner': 1230273.062327528}

Starting iteration 16

{'ner': 1233129.9568728018}

Starting iteration 17

{'ner': 1234482.223875716}

Starting iteration 18

{'ner': 1228235.1531106292}

Starting iteration 19

{'ner': 1236455.0306296044}

Starting iteration 20

{'ner': 1236151.556702537}

Starting iteration 21

{'ner': 1236109.1513396176}

Starting iteration 22

{'ner': 1230541.210079721}

Starting iteration 23

{'ner': 1241770.394253696}

Starting iteration 24

{'ner': 1238087.6685350395}

Starting iteration 25

{'ner': 1238092.9928586548}

Starting iteration 26

{'ner': 1229801.0468574367}

Starting iteration 27

{'ner': 1231733.8357542714}

Starting iteration 28

{'ner': 1230138.431220992}

Starting iteration 29

{'ner': 1220608.3220826644}

Explore spaCy

As we can see, our model identify and highlight the desired match string with DATASET using displacy.render

Another example:

We can also visualize the sentence: