Web Scraping News Articles to Build an NLP Data Pipeline | by Erdem Isbilen | Towards Data Science

Although it is ok to experiment with the ready-to-use datasets, generating your NLP data pipeline further improves your skills and gives you more freedom in your project choices.
In this article, I will try to explain my NLP data workflow from start to finish. I listed below, the three open-source Python frameworks that I used in my workflow;
Scrapy for extracting the raw text data from the web
spaCy for cleaning and normalizing the text
Tensorflow 2.0 for constructing the data pipeline
The complete workflow will be explained in 3 easy to follow steps. Full source code is provided in my GitHub repository.
Let’s start with the first step.
I decided to scrape news articles from the TRT World website to experiment with several NLP algorithms and data pipeline concepts using the text data scraped.
I aimed to scrape about 2000–3000 articles and store them in a JSON file.
To do this, I created a Scrapy project and generated 2 spiders; one for extracting the article links, another one for extracting the article headlines, and bodies using the links captured in the previous step.
Let’s install the Scrapy and start our Scrapy project.
# Install the scrapy$ pip install scrapy# Start web scraping project with scrapys$ scrapy startproject TRTWorld$ cd TRTWorldTRTWorld $ scrapy genspider Articles trtworld.comTRTWorld $ scrapy genspider ArticleScraper trtworld.com
Our first spider which is “Articles.py” will get the article links by visiting 500 TRTWorld web pages. It will be extracting the href information of the article links available at each page and storing them in a JSON file.
# Spider 1 # Articles.py which scrape article links# importsimport scrapyfrom scrapy.http import Requestfrom TRTWorld.items import TrtworldItemclass ArticlesSpider(scrapy.Spider): name = 'Articles' allowed_domains = ['trtworld.com'] start_urls = ['http://trtworld.com/']def start_requests(self):# Hardcoded URL that contains TURKEY related subjects  url="https://www.trtworld.com/turkey?page={}"link_urls = [url.format(i) for i in range(0,500)]# Loops through 500 pages to get the article links  for link_url in link_urls:print(link_url)# Request to get the HTML content    request=Request(link_url, cookies={'store_language':'en'},     callback=self.parse_main_pages)yield requestdef parse_main_pages(self,response):item=TrtworldItem()# Gets HTML content where the article links are stored  content=response.xpath('//div[@id="items"]//div[@class="article-   meta"]')# Loops through the each and every article link in HTML 'content'  for article_link in content.xpath('.//a'):# Extracts the href info of the link to store in scrapy item   item['article_url'] =       article_link.xpath('.//@href').extract_first()item['article_url'] =      "https://www.trtworld.com"+item['article_url']yield(item)def parse(self, response):  pass
After we finalized our first spider, we can now run it with the below command to generate “article_links” JSON file.
TRTWorld $ scrapy crawl -o article_links.json -t json Articles
The next step is to scrape the news articles using the links stored in the JSON file. To do this, let’s create our second spider which is “ArticleScraper”.
# Spider 2# ArticleScraper.py which scrape article headlies and bodies# importsimport scrapyfrom scrapy.http import Requestfrom TRTWorld.items import TrtworldItemimport jsonclass ArticlescraperSpider(scrapy.Spider): name = 'ArticleScraper' allowed_domains = ['trtworld.com'] start_urls = ['http://trtworld.com/']def start_requests(self):    # Open the JSON file which contains article links  with open('/Users/erdemisbilen/Angular/TRTWorld  /article_links.json') as json_file:      data = json.load(json_file)      for p in data:    print('URL: ' + p['article_url'])# Request to get the HTML content    request=Request(p['article_url'],               cookies={'store_language':'en'},               callback=self.parse_article_page)    yield requestdef parse_article_page(self,response):item=TrtworldItem() a_body=""# Extracts the article_title and stores in scrapy item item['article_title']=response.xpath('//h1[@class="article- title"]/text()').extract();# Extracts the article_description and stores in scrapy item item['article_description']=response.xpath('//h3[@class="article- description "]/text()').extract();# Extracts the article_body in <p> elements for p in response.xpath('//div[@class="contentBox bg-w  noMedia"]//p/text()').extract():      a_body=a_body+p   item['article_body']= a_body   yield(item)def parse(self, response):  pass
Below command runs the ArticleScraper spider and generates a JSON file containing the 3000 news articles. You can see below the content of the JSON file that the spider produced.
TRTWorld $ scrapy crawl -o article_body.json -t json ArticleScraper
Now that you have around 3000 articles stored in our JSON file, we can start to consider using them in our experimental NLP studies.
To use any text data in NLP applications, we have to convert the text into the numbers as computers have difficulty to understand the words.
Before doing this, we should clean and normalize our text. This step transforms our text into a more simple and structured form so that machine learning algorithms can perform efficiently and better.
In our case, we have news articles containing well-structured sentences so we may not need to apply all of the pre-processes listed below. See an example article below.
Let’s start with installing the ‘spaCy’ and required dependencies of it.
# Install the spaCypip install -U spacy# Install the spaCy Lemmatizationpip install -U spacy-lookups-data# Install the spaCy English Modelpython -m spacy download en_core_web_sm
To clean and normalize the text, we will be applying below processes into our text:
Sentence Segmentation
At first, we will be splitting each article into the sentences. Then we will be cleaning and normalizing each sentence with the help of the spaCy library.
# Splitting text into sentences using spaCydef split_sentences(document): sentences = [sent.string.strip() for sent in doc.sents] return sentences
Removing Stopwords
Stopwords are the most commonly used words in a language and do not help in NLP tasks such as sentiment analysis or text classification. So, you may consider to remove them from the text to increase the speed and accuracy of the model.
In our case, I used spaCy’s built-in stopwords. You can customize the default stopwords according to your domain-specific requirements.
# Removes stopwords from a sentence using spaCy (token.is_stop)def remove_stopwords(sentence): sentence = nlp(sentence) processed_sentence = ' '.join([token.text for token in sentence if token.is_stop != True ]) return processed_sentence# Removes stopwords from spaCy default stopword listnlp.Defaults.stop_words -= {"my_stopword_1", "my_stopword_2"}# Adds custom stopwords into spaCy default stopword listnlp.Defaults.stop_words |= {"my_stopword_1", "my_stopword_2"}# Prints spaCy default stopwordsprint(nlp.Defaults.stop_words)
Removing Punctuations, Quotes, Brackets, Currency Chars and Digits
Oftentimes, we just want words in our NLP pipeline, which means that we have to remove the punctuation and other special characters from the sentence including the digits.
# Removes punctuation and special chars from a sentence using spaCy def remove_punctuation_special_chars(sentence): sentence = nlp(sentence) processed_sentence = ' '.join([token.text for token in sentence  if token.is_punct != True and      token.is_quote != True and      token.is_bracket != True and      token.is_currency != True and      token.is_digit != True]) return processed_sentence# spaCy - List of special charecters to be removed_currency = r"\$ £ € ¥ ฿ US\$ C\$ A\$ ₽ ریال ₴"_punct = (    r"... ...... , : ; \! \? ¿ ؟ ¡ \( \) \[ \] \{ \} < > _ # \* & 。 ? ! , 、 ; : ~ · । ، ۔ ؛ ٪" )_quotes = r'\' " ” “ ` ‘  ́ ’ ‚ , „ » « 「 」 『 』 ( ) 〔 〕 【 】 《 》 〈 〉'
Lemmatization
Lemmatization is reverting the word to its base form. This is achieved by reducing the inflectional and derivationally related forms of a word to its base form.
Lemmatization aims to reduce the vocabulary count and to normalize the words.
# Lemmatization process with spaCydef lemmatize_text(sentence):    sentence = nlp(sentence)    processed_sentence = ' '.join([word.lemma_ for word in     sentence])        return processed_sentence
You can see above how the sentences are modified after the Lemmatization and stop word removal.
Depending on the NLP task in hand, you may consider not to apply some of the cleaning and normalization processes, as there will be some level of information loss in each of the processes described above.
In some cases, we may want to see the word frequencies to understand the content of a text. ‘spaCy’ provides easy to apply tools to achieve this. Below, you can see the complete Python script containing the text pre-processing and word frequency calculation functions.
# JSONtoTXT.py# Reads news articles from a JSON file# Splits the content into sentences# Cleans and normalizes the content# Write each processed sentence into a text fileimport jsonimport spacyfrom spacy.lang.en import English # updatedfrom spacy.lang.en.stop_words import STOP_WORDSfrom collections import Counterimport re# Loads the spaCy small English language modelnlp = spacy.load('en_core_web_sm')# Removes stopwords from spaCy default stopword listnlp.Defaults.stop_words -= {"my_stopword_1", "my_stopword_2"}# Adds custom stopword into spaCy default stopword listnlp.Defaults.stop_words |= {"my_stopword_1", "my_stopword_2"}print(nlp.Defaults)# Calculates the frequency of words in a documentdef word_frequency(my_doc):# all tokens that arent stop words or punctuations words = [token.text for token in my_doc if token.is_stop != True  and token.is_punct != True]# noun tokens that arent stop words or punctuations nouns = [token.text for token in my_doc if token.is_stop != True  and token.is_punct != True and token.pos_ == "NOUN"]# verb tokens that arent stop words or punctuations verbs = [token.text for token in my_doc if token.is_stop != True  and token.is_punct != True and token.pos_ == "VERB"]# five most common words word_freq = Counter(words) common_words = word_freq.most_common(5) print("---------------------------------------") print("5 MOST COMMON TOKEN") print(common_words) print("---------------------------------------") print("---------------------------------------")# five most common nouns noun_freq = Counter(nouns) common_nouns = noun_freq.most_common(5) print("5 MOST COMMON NOUN") print(common_nouns) print("---------------------------------------") print("---------------------------------------")# five most common verbs verb_freq = Counter(verbs) common_verbs = verb_freq.most_common(5) print("5 MOST COMMON VERB") print(common_verbs) print("---------------------------------------") print("---------------------------------------")# Removes stopwords from a sentence using spaCy (token.is_stop)def remove_stopwords(sentence): sentence = nlp(sentence) processed_sentence = ' '.join([token.text for token in sentence if  token.is_stop != True ]) return processed_sentence# Removes punctuation and special chars from a sentence using spaCydef remove_punctuation_special_chars(sentence): sentence = nlp(sentence) processed_sentence = ' '.join([token.text for token in sentence   if token.is_punct != True and      token.is_quote != True and      token.is_bracket != True and      token.is_currency != True and      token.is_digit != True]) return processed_sentence# Lemmatization process with spaCydef lemmatize_text(sentence):    sentence = nlp(sentence)    processed_sentence = ' '.join([word.lemma_ for word in     sentence])    return processed_sentencedef remove_special_chars(text): bad_chars = ["%", "#", '"', "*"]  for i in bad_chars:   text = text.replace(i, '') return text# Splitting text into sentences using spaCydef split_sentences(document): sentences = [sent.string.strip() for sent in doc.sents] return sentencessentence_index = 0with open('/Users/erdemisbilen/TFvenv/articles_less.json') as json_file: data = json.load(json_file)   with open("article_all.txt", "w") as text_file:  for p in data:   article_body = p['article_body']   article_body = remove_special_chars(article_body)doc = nlp(article_body)sentences = split_sentences(doc)   word_frequency(doc)for sentence in sentences:    sentence_index +=1    print("Sentence #" + str(sentence_index) + "-----------------")    print("Original Sentence               : " + sentence)    sentence = remove_stopwords(sentence)    sentence = remove_punctuation_special_chars(sentence)    print("Stopwors and Punctuation Removal: " + sentence)    sentence = lemmatize_text(sentence)    print("Lemmitization Applied           : " + sentence)    text_file.write(sentence + '\n')   text_file.close()
You can see below, what the script produces after processing the news articles.
Now that we cleaned and normalized our text as well as splitting it into sentences, it is time to construct a data pipeline with Tensorflow 2.0.
In many cases, feeding the text content directly into the NLP model is not an efficient way of managing the data input process.
Tensorflow ‘tf.data API’ provides better performance with considering flexibility and efficiency. I encourage you to watch this video if you would like to understand why ‘tf.data’ is better than the conventional data pipelines.
I will be using Tensorflow “tf.data.TextLineDataset API” to construct my NLP data pipeline.
Let’s start with installing the “TensorFlow” and “tensorflow-datasets”.
# Install the tensorflow with pip$ pip install tensorflow# Install the tensorflow-datasets with pip$ pip install tensorflow-datasets
I will mainly follow the “Load text” tutorial provided by the TensorFlow to develop my data pipeline.
First, we will use “tf.data.TextLineDataset” with the “articlesTXT.txt” file we generated in previous steps to construct our dataset.
Although we have only one txt file in our data pipeline, the code below provides the ability to load and label several txt files.
parent_dir = "/Users/erdemisbilen/Angular/TRTWorld/articlesTXT"FILE_NAMES = ['article_all.txt']BUFFER_SIZE = 2000BATCH_SIZE = 128TAKE_SIZE = 200def labeler(example, index):  return example, tf.cast(index, tf.int64)labeled_data_sets = []for i, file_name in enumerate(FILE_NAMES):  lines_dataset = tf.data.TextLineDataset(os.path.join(parent_dir,   file_name))  labeled_dataset = lines_dataset.map(lambda ex: labeler(ex, i))  labeled_data_sets.append(labeled_dataset)all_labeled_data = labeled_data_sets[0]for labeled_dataset in labeled_data_sets[1:]:  all_labeled_data = all_labeled_data.concatenate(labeled_dataset)  all_labeled_data = all_labeled_data.shuffle(    BUFFER_SIZE, reshuffle_each_iteration=False)
Then we will use “tfds.features.text.Tokenizer” to split the sentences into the tokens and construct our vocabulary set.
tokenizer = tfds.features.text.Tokenizer()vocabulary_set = set()for text_tensor, _ in all_labeled_data:  some_tokens = tokenizer.tokenize(text_tensor.numpy())  vocabulary_set.update(some_tokens)vocab_size = len(vocabulary_set)print("Vocabulary size.   :" + str(vocab_size))print("-------------------------------")print(vocabulary_set)print("-------------------------------")
Once we created our vocabulary set, the next step is to encode each word in our vocabulary by assigning a unique integer value for each of the words in our vocabulary set. This is an index-based encoding process which maps each word with a unique index number.
encoder = tfds.features.text.TokenTextEncoder(vocabulary_set)example_text = next(iter(all_labeled_data))[0].numpy()print(example_text)encoded_example = encoder.encode(example_text)print(encoded_example)
Then, we can encode the whole dataset using a map function.
def encode(text_tensor, label):  encoded_text = encoder.encode(text_tensor.numpy())  return encoded_text, labeldef encode_map_fn(text, label):  encoded_text, label = tf.py_function(encode,                                        inp=[text, label],                                        Tout=(tf.int64, tf.int64))  return encoded_text, labelall_encoded_data = all_labeled_data.map(encode_map_fn)
Our dataset contains sentences with varying length. Now, the final step is to pad each dataset item to a certain size vector as many NLP models use fixed-length vector dimensions.
Meantime, we will be batching the content of the dataset with a batch size of 128. This will be the number of dataset items we will input to the model for each iteration of the training process.
train_data = all_encoded_data.skip(TAKE_SIZE).shuffle(BUFFER_SIZE)train_data = train_data.padded_batch(BATCH_SIZE, padded_shapes=([200],()))test_data = all_encoded_data.take(TAKE_SIZE)test_data = test_data.padded_batch(BATCH_SIZE, padded_shapes=([200],()))sample_text, sample_labels = next(iter(test_data))sample_text[0], sample_labels[0]
After padding, all sentences in our dataset represented as a vector (with size 200) as shown below.
Now that our data pipeline is ready, we can start building an LSTM model to test our data pipeline.
#Training a LSTM model to test the data pipelinevocab_size += 1model = tf.keras.Sequential()model.add(tf.keras.layers.Embedding(vocab_size, 64))model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)))for units in [64, 64]:  model.add(tf.keras.layers.Dense(units, activation='relu'))# Output layer. The first argument is the number of labels.model.add(tf.keras.layers.Dense(3, activation='softmax'))optimizer = tf.keras.optimizers.Adam(learning_rate=0.005, amsgrad=True)model.compile(optimizer= optimizer,              loss='sparse_categorical_crossentropy',              metrics=['accuracy'])log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)model.fit(train_data, epochs=10, steps_per_epoch=4, validation_data=test_data, callbacks=[tensorboard_callback])eval_loss, eval_acc = model.evaluate(test_data)print('\nEval loss: {:.3f}, Eval accuracy: {:.3f}'.format(eval_loss, eval_acc))
You can see above the data is fed into the model successfully with the data pipeline we constructed.
In this post, I tried to explain my way of building an NLP data pipeline from scratch. I hope my article helps you to build your NLP application.