Web-Scraping and Pre-Processing for NLP | by James Briggs | Towards Data Science

For Natural Language Processing, clean data is important. Even more so when that data is coming from the web. In this article we will go through a real example of web scraping and data pre-processing for a Stoic philosophy text generator.
The data we will be using is Epistulae Morales Ad Lucilium (Moral Letters to Lucilius) written by exiled Roman senator Seneca during the late Stoa.
The letters are sourced from WikiSource here. This page consists of a list of all 124 letters. Where each letter is contained in it’s own page.
First, we must extract the HTML from this contents page. For this we use Python’s requests and BeautifulSoup libraries.
This gives us a BeautifulSoup object, which contains the raw html we have given via html. Let’s take a look what this looks like.
In this, we must extract the local paths to each letter. The BeautifulSoup object allows us to extract all <a> elements with soup.find_all('a'). However, this returns all <a> elements, so we then need to filter for just those which link to the letters.
We do this using regular expressions, which is incredibly simple, we build our regex to search for anything that begins with Letter followed by one or more spaces \s+ and finally ending with one to three digits \d{1,3}. This gives us re.compile(r"^Letter\s+\d{1,3}$").
By applying this regex to the list of <a> elements provided by BeautifulSoup, we will get the following.
Now we need to define a function to parse the HTML from each page. This is actually very straight-forward, as the letter text is all contained in the only <p> elements on the page.
So, similar to before, we extract all <p> elements. We then do a small amount of formatting to make the text more readable, before returning the letter text.
With our pull_letter function and letters_regex, we can read all letters, which we will place in moral_letters.
At this point we have all the data we need, neatly stored in moral_letters.
Formatting our data into a NLP friendly format is much simpler to write, albeit much more abstracted.
We will need to convert the text we have at the moment into numbers, creating a numeric representation of our data, which we will call data_idx.
To do this we will create a char2idx dictionary, which stands for character-to-index. This, as is given away by the name, converts characters 'a', 'b', 'c' to indices 0, 1, 2.
Every character in char2idx must also map to a unique index. For this we must create a set of all characters in the data (a set is simply list unique values). This set of characters is known as a vocabulary, which we will define as vocab.
So now, let’s read the first letter from Seneca in our new format.
Although it seems like nonsense, this is exactly what we want when formatting text data for NLP.
Now all that remains is slicing and shuffling our data, then it is ready to be input into a model. In our case we are using tensorflow.
TensorFlow allows us to to use tf.data.Dataset, an API we can use to simplify the dataset transformation process. To create a dataset from our Numpy array data_idx, we use tf.data.Dataset.from_tensor_slices.
During training, we will only look at one segment of text at once. To split the data into sequences we use the Dataset .batch method.
As this is a text generator, our target data will simply consist of the input data, shifted one character forward. For this we will define a function called split_xy.
Finally, we create batches of 64 sequences, again using .batch, which are then shuffled using .shuffle.
Inside dataset, we have 175 batches (due to len(txt) / (SEQLEN * BATCHSIZE)). Every single batch contains an input and target array, built by the split_xy function.
This data is now fully prepared, ready to be input into a model for training.
Despite the outward appearance of complexity. The implementation of machine learning is no longer an gargantuan task, reserved for the brightest minds of our time.
We are now living in an age where we can teach a computer to reproduce philosophy from some of the greatest minds in human history. Not only can we do it, but it is incredibly easy to do.
It is a fascinating time, and one which I hope we all make the most of.
Thanks for reading!
I’ve written about designing neural networks to reproduce Stoic philosophy before. If you’re interested, you can read about it here:
towardsdatascience.com
For anyone that is curious, here are my (attempted) Latin translations of the article headers.Epistulae: LettersUnum Epistula Legimus: We read one letterOmnis Epistulae: All lettersPraeparatio: Preparation (eg preparing the data)Simplex Munditiis: Elegance through simplicityQui Conclusioni: Here concludes / the conclusion