How to Automate Content Creation Process for an E-Learning Platform
By: Erdem Isbilen
Tags: E Learning Solutions, Spacy, NLP, Content Creation, Data Science

How to Automate Content Creation Process for an E-Learning Platform

Generate 2000 questions in 5 minutes

Photo by Erdem Isbilen — lingomoo demo web app

If you aim to develop an e-learning platform, the first difficulty you have to overcome will probably be creating the course content. Unfortunately, creating course content is the most challenging part of the development process as it consumes a lot of time and energy. Fortunately, this process can be automated with the help of web scraping and natural language processing.

In this post, I will show you step by step how such automation can be achieved using spaCy. I will process the text data by spaCy to convert any given raw text data to something directly usable in an English e-learning platform.

The result of the above steps will be used in my demo web application, lingomoo.

Photo by Erdem Isbilen — lingomoo web app

If you want to see how all can be put together in a real-life scenario, check out my lingomoo demo application here. In addition to the demo application, I provided the source code in my GITHUB repository.

Let’s start with the first step.

Step 1 — Raw Text Data Generation

The main purpose of this step is to generate large amounts of text data that we can process and filter according to the task at hand.

To do this, you can scrape text content from websites under a public licence. As this is not the main subject of the article, I will not give further detail on this.

Below you can find the JSON file containing raw text data required. The actual JSON file, “sports_article_body.json”, is stored in my GITHUB repository.

Photo by Erdem Isbilen — “sports_article_body.json” structure

Step 2 — Processing the Raw Text with spaCy

This step will be involved with filtering and processing the text data according to the task at hand. There will be several sub-steps which are listed below;

Splitting the news articles into the sentences

Classifying the sentences based on their complexity and grammatical structure so that we can filter out the content we don’t need.

Splitting the sentences into the words so that we can form our questions using individual words for “Put the words in correct order” question type.

Defining a “missing word” for each sentence together with answer options for the “Find the missing word” question type.

Storing the questions in a JSON file

Splitting the News Articles Into the Sentences

In this step, we will simply read from the JSON file created in the previous step and split each article into sentences.

# json_to_txt.py

# Reads news articles from a JSON file

# Splits the content into sentences

# Write each sentence into a text fileimport json

import spacy

import re# Loads the spaCy small English language model

nlp = spacy.load('en_core_web_sm')# Remove several special characters from the text

def remove_special_chars(text):

bad_chars = ["%", "#", '"', "*"]

for i in bad_chars:

text = text.replace(i, '')

return text# Split the text into the sentences

def split_sentences(document):

sentences = [sent.string.strip() for sent in document.sents]

return sentences# Write the sentences in article body into a text file

def article_body_write(line):

doc = nlp(line)

sentences = split_sentences(doc)

sentence_index = 0



for sentence in sentences:

sentence_index +=1

print("Sentence #" + str(sentence_index) + "-" * 20)

print(sentence)



text_file.write(sentence + '

')# Write the article title into a text file

def article_title_write(line):

print("Title" + line[0])

text_file.write(line[0] + '

')# Loop through the articles in JSON file

# Write sentences into a text file

with open('/Users/erdemisbilen/Language/sports_article_body.json')

as json_file: data = json.load(json_file) with open("sports_article_all.txt", "w") as text_file:

for p in data:

article_title = p['article_title']

article_body = p['article_body']

article_body = remove_special_chars(article_body)

article_title_write(article_title)

article_body_write(article_body) text_file.close()

After we run this python script, we will have a text file, “sports_article_all.txt”, containing 40K sentences ready for processing and filtering.

Classifying the Sentences Based on Their Complexity and Grammatical Structure

At this point, as we have all the 40K sentences in a text file, we can now process and filter them to generate the questions for our English e-learning application.

I will have a basic filtering process that ignores the sentences having more than 9 words and sentences having less than 4 words. That is because, after 9 words, it is not practical to show the sentence as a question on mobile devices without using considerably small font sizes. Another disadvantage of long sentences is that they are difficult to grasp by beginner and elementary level users.

If the sentence has less than 10 words, then I calculate the sentence difficulty considering the sentence length, average word length and the number of polysyllabic words. This is required to put the sentences in order, based on their complexity so that we can provide them to the user accordingly.

LONG_WORDS_LIMIT = 5

POLYSYLLABIC_WORDS_LIMIT = 4def calculate_sentence_length(words):

return (len(words))def calculate_word_character_length_mavrg(words):

total_char = 0

for word in words:

total_char += len(word)

return (round(total_char/len(words),1))def calculate_number_of_long_words (words):

total_long_word = 0

for word in words:

if len(word) > LONG_WORDS_LIMIT:

total_long_word += 1

return (total_long_word)def calculate_number_of_polysyllabic_words(words):

total_polysyllabic_words = 0

for word in words:

if len(word) > POLYSYLLABIC_WORDS_LIMIT:

total_polysyllabic_words += 1

return (total_polysyllabic_words)

Then, I will classify the sentences based on their grammatical structure with spaCy’ rule-based matcher function. As an example, using a set of spaCy’ Part-of-Speech tags, we can define a rule to classify the sentences having future tense or sentences having present tense. By doing this, we can further refine our content. You can find all the rules below that I created to classify the content based on their grammatical structure.

conjuction_rule_1 = [{'POS': 'CCONJ'}]prepositions_sub_conj_rule_1 = [{'POS': 'ADP'}]comparative_adjective_rule_1 = [{'TAG': 'JJR'}]

superlative_adjective_rule_1 = [{'TAG': 'JJS'}]

comparative_adverb_rule_1 = [{'TAG': 'RBR'}]

superlative_adverb_rule_1 = [{'TAG': 'RBS'}]there_is_are_rule_1 = [{'TEXT': 'There', 'TEXT': 'there'},{'TEXT': 'is', 'TEXT': 'are'}]

models_rule_1 = [{'TAG': 'MD'},{'TAG': 'VB'} ]

to_infinitive_rule_1 = [{'TAG': 'TO'},{'TAG': 'VB'}]

gerunds_rule_1 = [{'TAG': 'VB'},{'TAG': 'VBG'}]

passive_rule_1 = [{'DEP': 'auxpass'}, {'TAG':'VBN'}]subject_rule_1 = [{'DEP': 'csubj'}]

subject_rule_2 = [{'DEP': 'nsubj'}]

object_rule_1 = [{'DEP': 'dobj'}]#Future Tenses With Auxiliaries

future_tense_rule_1 = [{'TEXT': 'will'}, {'TAG':'VB'}]

future_tense_rule_2 = [{'TEXT': 'shall'}, {'TAG':'VB'}]#Morphological Present Tenses

simple_present_rule_1 = [{'POS': 'PRON'}, {'TAG': 'VBP','TAG': 'VBZ' }, {'TAG':'VB', 'TAG':'VBG', 'TAG':'VBN', 'OP': '!'}]

present_continuous_rule_1 = [{'TAG': 'VBZ', 'DEP': 'aux'}, {'TAG':'VBG', 'DEP': 'ROOT'}]

present_perfect_rule_1 = [{'TAG': 'VBZ', 'DEP': 'aux'}, {'TAG':'VBN', 'DEP': 'ROOT'}]

present_perfect_continuous_rule_1 = [{'TAG': 'VBZ', 'DEP': 'aux'}, {'TAG':'VBN'}, {'TAG':'VBN', 'DEP': 'ROOT'}]#Morphological Past Tenses

simple_past_rule_1 = [{'POS': 'PRON'}, {'TAG': 'VBD'},{'TAG':'VB', 'TAG':'VBG', 'TAG':'VBN' , 'OP': '!'}]#Possessive Endings the boy's ball

possesive_ending_rule_1 = [{'TAG':'POS'}]#Possessive Pronoun

possesive_pronoun_rule_1 = [{'TAG':'PRP$'}]

Splitting the Sentences Into the Words

I will split the sentences into the words with keeping the punctuation, as I want to display the words together with punctuation marks. This will help the user to better understand the structure of the sentence with the help of the punctuation marks.

def simple_tokenization(line):

sentence_length = 0

linguistic_features = []

tokenizer = Tokenizer(nlp.vocab)

toks = tokenizer(line)



for token in toks:

print(token.text)

if not "

" in token.text:

sentence_length += 1

linguistic_features.append({'word': token.text})



return(linguistic_features, sentence_length)

Defining a “missing word” for Each Sentence Together with Answer Options

For the ‘Find the missing word’ question type, we have to define the missing word of the sentence with the options for the answer. To do this, I will first define a VERB or an ADJ in the sentence as a missing word. Then I will find the vector representation of this word with the help of the spaCy pre-trained NLP model. Next, using the nlp.vocab.vectors.most_similar() function, I will find the 15 most similar words to the missing word. I will choose 9th and 14th of them as options for the answers.

def define_missing_word(my_doc):

missing_word=""

option1=""

option2="" for token in my_doc:

if token.is_punct != True and token.is_alpha == True :

if (token.pos_ == "VERB" or token.pos_ == "ADJ" or token.pos_ ==

"ADV") and (random.choice(RANDOM_LIST)==1 or missing_word==""):



w_id = nlp.vocab.strings[token.text]

w_vector = nlp.vocab.vectors[w_id]

most_similar = nlp.vocab.vectors.most_similar

(w_vector.reshape(1,300), n=15)

missing_word = token.text

option1 = nlp.vocab.strings[most_similar[0][0][9]].lower()

option2 = nlp.vocab.strings[most_similar[0][0][14]].lower() return(missing_word,option1,option2)

Storing the questions in a JSON file

The final step is to store all the information we generated in a JSON for later use in our demo web application.

json_data.append({

'sentence': line,

'sentence_length': sentence_length,

'difficulty_score': difficulty_score,

'missing_word': missing_word,

'option1': option1,

'option2': option2,

'word_character_length_mavrg': word_character_length_mavrg,

'number_of_long_words': number_of_long_words,

'named_entities': named_entities,

'linguistic_features': linguistic_features}) with open('simple_sentences.json', 'w') as outfile:

json.dump(json_data, outfile)

Full code, “nlp_simple.py”, is provided in my GITHUB repository.

Summary

The output of the “nlp_simple.py” script is a JSON file containing 2000 questions. These questions should be reviewed before serving them to the user as there can be some errors occurred throughout the process.

In this post, I tried to explain how I manage automating the e-learning content creation for my English e-learning demo application.

I hope you enjoyed my article and I hope it helps you to build your e-learning application.