Web Scraping and NLP for Automated e-Learning Content Creation
By: Erdem Isbilen
Tags: Online Learning, Spacy, Web Scraping, Scrapy, Content Creation

Photo by JESHOOTS.COM on Unsplash

If you aim to develop an e-learning platform, the first difficulty you have to overcome will probably be creating the course content. Unfortunately, creating course content is the most challenging part of the development process as it consumes a lot of time and energy. Fortunately, this process can be automated with the help of web scraping and natural language processing.

In this post, I will show you step by step how such automation can be achieved using Scrapy and spaCy. In the first step, I will use the Scrapy to scrape news articles from the web. In the second step, I will process the text data by spaCy to convert the news articles to something directly usable in an English e-learning platform.

The result of the above steps will be a JSON file containing 2000 questions in two categories which are ‘Put the words in correct order’ and ‘Find the missing word’. The content of this JSON file, “simple_sentences.json”, then will be used in my demo web application, lingomoo.

Photo by Erdem Isbilen — lingomoo demo web app — “Find the missing word” question type

If you want to see how all can be put together in a real-life scenario, check out my lingomoo demo application here. In addition to the demo application, I provided the source code in my GITHUB repository.

Let’s start with the first step.

Step 1 - Scraping the News Articles

The main purpose of this step is to generate large amounts of text data that we can process and filter according to the task at hand. Daily news articles on the web are the best resource of proper text data containing correctly written sentences with no spelling errors. So, it is a good fit for our purpose as we are planning to convert these sentences into the questions to use them in our English e-learning platform.

We can scrape as many articles as we would like, thanks to the online news providers storing the articles of past years in their online accessible repositories. In my case, I scraped 2000 articles from TRT World website to obtain the large text data required for later steps. Below you can find the JSON file generated after scraping process. The actual JSON file, “sports_article_body.json”, is stored in my GITHUB repository.

Photo by Erdem Isbilen — “sports_article_body.json” structure

As the main subject of this post is the second step, I will not give further details about the actual scraping process. You can check out my related post below to better understand the details related to web scraping.

Step 2 — Processing the Raw Text with spaCy

This step will be involved with filtering and processing the text data according to the task at hand. There will be several sub-steps which are listed below;

Splitting the news articles into the sentences

Classifying the sentences based on their complexity and grammatical structure so that we can filter out the content we don’t need.

Splitting the sentences into the words so that we can form our questions using individual words for “Put the words in correct order” question type.

Defining a “missing word” for each sentence together with answer options for the “Find the missing word” question type.

Storing the questions in a JSON file

Photo by Erdem Isbilen — lingomoo demo web app — “Put the words in correct order” question type

Splitting the News Articles Into the Sentences

In this step, we will simply read from the JSON file created in the previous step and split each article into sentences.

# json_to_txt.py

# Reads news articles from a JSON file

# Splits the content into sentences

# Write each sentence into a text file import json

import spacy

import re # Loads the spaCy small English language model

nlp = spacy.load('en_core_web_sm') # Remove several special characters from the text

def remove_special_chars(text):

bad_chars = ["%", "#", '"', "*"]

for i in bad_chars:

text = text.replace(i, '')

return text # Split the text into the sentences

def split_sentences(document):

sentences = [sent.string.strip() for sent in document.sents]

return sentences # Write the sentences in article body into a text file

def article_body_write(line):

doc = nlp(line)

sentences = split_sentences(doc)

sentence_index = 0



for sentence in sentences:

sentence_index +=1

print("Sentence #" + str(sentence_index) + "-" * 20)

print(sentence)



text_file.write(sentence + '

') # Write the article title into a text file

def article_title_write(line):

print("Title" + line[0])

text_file.write(line[0] + '

') # Loop through the articles in JSON file

# Write sentences into a text file

with open('/Users/erdemisbilen/Language/sports_article_body.json')

as json_file: data = json.load(json_file) with open("sports_article_all.txt", "w") as text_file:

for p in data:

article_title = p['article_title']

article_body = p['article_body']

article_body = remove_special_chars(article_body)

article_title_write(article_title)

article_body_write(article_body) text_file.close()

After we run this python script, we will have a text file, “sports_article_all.txt”, containing 40K sentences ready for processing and filtering.

Classifying the Sentences Based on Their Complexity and Grammatical Structure

At this point, as we have all the 40K sentences in a text file, we can now process and filter them to generate the questions for our English e-learning application.

I will have a basic filtering process that ignores the sentences having more than 9 words and sentences having less than 4 words. That is because, after 9 words, it is not practical to show the sentence as a question on mobile devices without using considerably small font sizes. Another disadvantage of long sentences is that they are difficult to grasp by beginner and elementary level users.

If the sentence has less than 10 words, then I calculate the sentence difficulty considering the sentence length, average word length and the number of polysyllabic words. This is required to put the sentences in order, based on their complexity so that we can provide them to the user accordingly.

LONG_WORDS_LIMIT = 5

POLYSYLLABIC_WORDS_LIMIT = 4 def calculate_sentence_length(words):

return (len(words)) def calculate_word_character_length_mavrg(words):

total_char = 0

for word in words:

total_char += len(word)

return (round(total_char/len(words),1)) def calculate_number_of_long_words (words):

total_long_word = 0

for word in words:

if len(word) > LONG_WORDS_LIMIT:

total_long_word += 1

return (total_long_word) def calculate_number_of_polysyllabic_words(words):

total_polysyllabic_words = 0

for word in words:

if len(word) > POLYSYLLABIC_WORDS_LIMIT:

total_polysyllabic_words += 1

return (total_polysyllabic_words)

Then, I will classify the sentences based on their grammatical structure with spaCy’ rule-based matcher function. As an example, using a set of spaCy’ Part-of-Speech tags, we can define a rule to classify the sentences having future tense or sentences having present tense. By doing this, we can further refine our content. You can find all the rules below that I created to classify the content based on their grammatical structure.

conjuction_rule_1 = [{'POS': 'CCONJ'}] prepositions_sub_conj_rule_1 = [{'POS': 'ADP'}] comparative_adjective_rule_1 = [{'TAG': 'JJR'}]

superlative_adjective_rule_1 = [{'TAG': 'JJS'}]

comparative_adverb_rule_1 = [{'TAG': 'RBR'}]

superlative_adverb_rule_1 = [{'TAG': 'RBS'}] there_is_are_rule_1 = [{'TEXT': 'There', 'TEXT': 'there'},{'TEXT': 'is', 'TEXT': 'are'}]

models_rule_1 = [{'TAG': 'MD'},{'TAG': 'VB'} ]

to_infinitive_rule_1 = [{'TAG': 'TO'},{'TAG': 'VB'}]

gerunds_rule_1 = [{'TAG': 'VB'},{'TAG': 'VBG'}]

passive_rule_1 = [{'DEP': 'auxpass'}, {'TAG':'VBN'}] subject_rule_1 = [{'DEP': 'csubj'}]

subject_rule_2 = [{'DEP': 'nsubj'}]

object_rule_1 = [{'DEP': 'dobj'}] #Future Tenses With Auxiliaries

future_tense_rule_1 = [{'TEXT': 'will'}, {'TAG':'VB'}]

future_tense_rule_2 = [{'TEXT': 'shall'}, {'TAG':'VB'}] #Morphological Present Tenses

simple_present_rule_1 = [{'POS': 'PRON'}, {'TAG': 'VBP','TAG': 'VBZ' }, {'TAG':'VB', 'TAG':'VBG', 'TAG':'VBN', 'OP': '!'}]

present_continuous_rule_1 = [{'TAG': 'VBZ', 'DEP': 'aux'}, {'TAG':'VBG', 'DEP': 'ROOT'}]

present_perfect_rule_1 = [{'TAG': 'VBZ', 'DEP': 'aux'}, {'TAG':'VBN', 'DEP': 'ROOT'}]

present_perfect_continuous_rule_1 = [{'TAG': 'VBZ', 'DEP': 'aux'}, {'TAG':'VBN'}, {'TAG':'VBN', 'DEP': 'ROOT'}] #Morphological Past Tenses

simple_past_rule_1 = [{'POS': 'PRON'}, {'TAG': 'VBD'},{'TAG':'VB', 'TAG':'VBG', 'TAG':'VBN' , 'OP': '!'}] #Possessive Endings the boy's ball

possesive_ending_rule_1 = [{'TAG':'POS'}] #Possessive Pronoun

possesive_pronoun_rule_1 = [{'TAG':'PRP$'}]

Splitting the Sentences Into the Words

I will split the sentences into the words with keeping the punctuation, as I want to display the words together with punctuation marks. This will help the user to better understand the structure of the sentence with the help of the punctuation marks.

def simple_tokenization(line):

sentence_length = 0

linguistic_features = []

tokenizer = Tokenizer(nlp.vocab)

toks = tokenizer(line)



for token in toks:

print(token.text)

if not "

" in token.text:

sentence_length += 1

linguistic_features.append({'word': token.text})



return(linguistic_features, sentence_length)

Defining a “missing word” for Each Sentence Together with Answer Options

For the ‘Find the missing word’ question type, we have to define the missing word of the sentence with the options for the answer. To do this, I will first define a VERB or an ADJ in the sentence as a missing word. Then I will find the vector representation of this word with the help of the spaCy pre-trained NLP model. Next, using the nlp.vocab.vectors.most_similar() function, I will find the 15 most similar words to the missing word. I will choose 9th and 14th of them as options for the answers.

def define_missing_word(my_doc):

missing_word=""

option1=""

option2="" for token in my_doc:

if token.is_punct != True and token.is_alpha == True :

if (token.pos_ == "VERB" or token.pos_ == "ADJ" or token.pos_ ==

"ADV") and (random.choice(RANDOM_LIST)==1 or missing_word==""):



w_id = nlp.vocab.strings[token.text]

w_vector = nlp.vocab.vectors[w_id]

most_similar = nlp.vocab.vectors.most_similar

(w_vector.reshape(1,300), n=15)

missing_word = token.text

option1 = nlp.vocab.strings[most_similar[0][0][9]].lower()

option2 = nlp.vocab.strings[most_similar[0][0][14]].lower() return(missing_word,option1,option2)

Storing the questions in a JSON file

The final step is to store all the information we generated in a JSON for later use in our demo web application.

json_data.append({

'sentence': line,

'sentence_length': sentence_length,

'difficulty_score': difficulty_score,

'missing_word': missing_word,

'option1': option1,

'option2': option2,

'word_character_length_mavrg': word_character_length_mavrg,

'number_of_long_words': number_of_long_words,

'named_entities': named_entities,

'linguistic_features': linguistic_features}) with open('simple_sentences.json', 'w') as outfile:

json.dump(json_data, outfile)

Full code, “nlp_simple.py”, is provided in my GITHUB repository.

Summary

The output of the “nlp_simple.py” script is a JSON file containing 2000 questions. These questions should be reviewed before serving them to the user as there can be some errors occurred throughout the process.

In this post, I tried to explain how I manage automating the e-learning content creation for my English e-learning demo application.

I hope you enjoyed my article and I hope it helps you to build your e-learning application.