How to Get Started with Spark NLP in 2 Weeks — Part I | by Mustafa Aytuğ Kaya | Towards Data Science

If you want to make a head start in enterprise NLP, but have no clue about Spark, this article is for you. I have seen many colleagues wanting to step to this domain but disheartened due to the initial learning overhead that comes with Spark. It may seem inconspicuous at first glance since Spark code is a bit different than your regular Python script. However, Spark and Spark NLP basics aren’t really hard to learn. If you axiomatically accept this assertion, I will show you how easy the basics are and will provide a road map to pave the way to learn key elements, which will satisfy most use cases of an intermediate level practitioner. Due to impeccable modularity that comes with Spark NLP pipelines, for an average learner, -mark my words- two weeks will be enough to build basic models. Roll up your sleeves, here we start!
Why Spark NLP?Supply and Demand is the answer: It Is The Most Widely Used Library In Enterprises! Here are a few reasons why. Common NLP packages today have been designed by academics and they favor ease of prototyping over runtime performance, eclipsing scalability, error handling, target frugal memory consumption and code reuse. Although some libraries like ‘the industrial-strength NLP library — spaCy’ might be considered an exception (since they are designed to get things done rather than doing research), they may fall short of enterprise targets when it comes to dealing with data in volume.
We are going to have a different strategy here. Rather than following the crowds in the routine, we will use basic libraries to brush up ‘basics’ and then jump directly to address the enterprise sector. Our final aim is to target the niche by building continental pipelines, which are impossible to resolve with standard libraries, albeit their capacity in their league.
If you are not convinced yet, please read this article for benchmarking and comparison with spaCy, which will give you five good reasons to start with Spark NLP. First of all Spark NLP has the innate trait of scalability it inherits from Spark, which was primarily used for distributed applications, it is designed to be scalable. Spark NLP benefits from this since it can scale on any Spark cluster as well as on-premise and with any cloud provider. Furthermore, Spark NLP optimizations are done in such a way that it could run orders of magnitude faster than what the inherent design limitations of legacy libraries allow. It provides the concepts of annotators and it includes more than what other NLP libs include. It includes sentence detection, tokenization, stemming, lemmatization, POS Tagger, NER, dependency parse, text matcher, date matcher, chunking, context-aware spell checking, sentiment detector, pre-trained models, and training models with very high accuracy according to academic peer-reviewed results. Spark NLP also includes production-ready implementation of BERT embeddings for named entity recognition. For example, it makes much fewer errors on NER compared to spaCy, which we tested in the second part of this article. Also, worthy of notice, Spark NLP includes features that provide full Python API, supports training on GPU, user-defined deep learning networks, Spark, and Hadoop.
The library comes with a production-ready implementation of BERT embeddings and uses transfer learning for data extraction. Transfer learning is a highly-effective method of extracting data that can leverage even small amounts of data. As a result, there’s no need to collect large amounts of data to train SOTA models.
Also, John Snow labs Slack channel provides top tier support that is beneficial because developers and new learners tend to band together and create resources that every one of them can benefit from. You will get answers to your questions right from the developers with dedication. I have been there a few times, can attest that they are quick and accurate in their response. Additionally, anyone finding themselves stuck can quickly get help from people that have had similar problems through Stack Overflow or similar platforms.
Famous “Facts” About Spark That Are Wrong
- Spark is cluster computing so it can’t be run on local machines
Wrong! You can run Spark on your local machine, and each CPU core will be used to the core! That’s how it looks like when Spark NLP is in action:
What’s even more, you can run Spark on GPU.
- Spark is yet another language to learn!
Well, if you know SQL, PySpark and Spark NLP is not going to feel like another language at all. SQL and Regex are all languages on their own, but it wasn’t hard to learn their basics, right?
- Spark is built for big data, so Spark NLP is only good for big data.
Yes, there is a significant overhead used for Spark internals, but Spark NLP introduces a ‘Light Pipeline’ for smaller datasets.
At first, Spark seems like another challenging language to learn. Spark isn’t the easiest library to comprehend, but what we will be doing in the NLP domain has been skillfully crafted in Spark NLP’s infrastructure provided by a simple API that can be easily interacted with.
While being one of the sharpest pencils in a data scientist’s toolbox, Pandas uses only a single CPU core, and in essence, it is not fast or robust enough to handle bigger datasets. Spark was designed to hurdle those deficiencies using cluster computing, and you can even run it on your local machine, assigning as many CPU cores as you want! Unfortunately, even though it runs on a local machine, pip install Pyspark is not enough to set it up, and a series of dependencies must be installed on PC and Mac. For those who want to jump into action as quickly as possible, I recommend the use of Google Colab.
1) Setting up on Mac or LinuxTo utilize Spark NLP, Apache Spark version 2.4.3 and higher must be installed. Assuming that you haven’t installed Apache Spark yet, let’s start with Java installation at first. Just go to the official website and from “Java SE Development Kit 8u191”, and install JDK 8. A necessary command-line script can be found in detail here.
2) Setting up Spark on a PC can be a little bit tricky. Having tried many methods, I found this article to be the only one that works. It also includes a clear and concise video. In addition to the provided documentation, this blog page also helped me.
3) Unless you have to run Spark on your local machine, the easiest way to get started with PySpark is using Google Colab. Since it is essentially a Jupyter Notebook that runs on Google server, you don’t need to install anything in our system locally.
To run spark in Colab, first, we need to install all the dependencies in the Colab environment such as Apache Spark 2.3.2 with Hadoop 2.7, Java 8, and Findspark to locate the spark in the system. Please refer to this article for further details.
import os# Install java! apt-get install -y openjdk-8-jdk-headless -qq > /dev/nullos.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]! java -version# Install pyspark! pip install --ignore-installed -q pyspark==2.4.4# Install Spark NLP! pip install --ignore-installed -q spark-nlp==2.4.5
For PC users who don’t want to run their notebooks on Colab, I recommend installing Spark NLP on a dual boot Linux system or using WSL 2, however, this benchmarking article reflects some performance loss with WSL 2. To be more precise, I had my best Spark NLP experience after a dual boot Ubuntu 20.04 installation.
Day 2: Spark Basics, RDD Structure, and NLP Baby Steps with Spark
As previously mentioned, Apache Spark is a distributed cluster computing framework that is highly efficient for large data sets using in-memory computations for lightning-fast data processing. It is also considered to be more efficient than MapReduce for the complex application running on Disk.
While libraries like pandas are adequate for most everyday operations, big data requires a cluster computing framework due to the volume, variety, and velocity, which are three defining properties or dimensions of big data.
At the center of Spark is the Spark Core — the foundation of the overall project, on top of which the rest of Spark libraries are built. Spark SQL lets you query structured data inside Spark programs, using either SQL or a familiar DataFrame API.
MLlib fits into Spark’s APIs and interoperates with NumPy in Python and R libraries. You can use any Hadoop data source, making it easy to plug into Hadoop workflows. Spark excels at iterative computation, enabling MLlib to run fast — up to 100x faster than MapReduce.
Spark Mllib contains the legacy API built on top of RDDs. Although I find Spark Mllib and RDD structure easier to use as a Python practitioner, as of Spark 2.0, the RDD-based APIs in the Spark.MLlib package has entered maintenance mode. The primary Machine Learning API for Spark is now the DataFrame-based API in the Spark ML package.
Transformations create new RDDs and actions to perform calculations. Lambda, Map, and Filter are some of the basic functions. Real datasets are generally key, value pairs similar to Python dictionaries but are represented like tuples. Please observe the code below for creating bigrams and word counts to see how similar to Python code it is. I would highly recommend this course for further details.
import resentences = sc.textFile(“ulyses.txt”) \ .glom() \ .map(lambda x: “ “.join(x)) \ .flatMap(lambda x: x.split(“.”)).map(lambda x: re.sub(‘[^a-zA-Z\s]’,’’,x))bigrams = sentences.map(lambda x:x.split()) \ .flatMap(lambda x: [((x[i],x[i+1]),1) for i in range(0,len(x)-1) \ if all([x[i].lower() not in stop_words,x[i+1].lower() not in stop_words])])freq_bigrams = bigrams.reduceByKey(lambda x,y:x+y) \ .map(lambda x:(x[1],x[0])) \ .sortByKey(False)In [1]:freq_bigrams.take(10)Out[1]:(1, ('Author', 'James')),       (1, ('Joyce', 'Release')),       (1, ('EBook', 'Last')),       (1, ('Last', 'Updated')),       (1, ('December', 'Language')),       (1, ('English', 'Character')),       (1, ('Character', 'set')),       (1, ('set', 'encoding')),       (1, ('UTF', 'START')),       (1, ('II', 'III'))]
# word count exampletext_file = sc.textFile(“example_text1.txt”)counts_rdd = text_file.flatMap(lambda line: line.split(“ “)) \ .map(lambda word: (word, 1)) \ .reduceByKey(lambda a, b: a + b)# print the word frequencies in descending ordercounts_rdd.map(lambda x: (x[1], x[0])) \ .sortByKey(ascending=False)\ .collect()[:10]
Output
Out [3]: [(988, 'the'),          (693, 'and'),          (623, 'of'),          (604, 'to'),          (513, 'I'),          (450, 'a'),          (441, 'my'),          (387, 'in'),          (378, 'HAMLET'),          (356, 'you')]
Day 3/4: Feature Engineering / Cleaning Data with Pyspark
Spark ML provides higher-level API built on top of DataFrames for constructing ML pipelines, standardizing APIs for machine learning algorithms to make it easier to combine multiple algorithms into a single workflow. Here we will cover the key concepts introduced by the Spark ML API.
Machine learning steps can be applied to from Spark SQL to support a variety of data types under a unified Dataset concept. A SchemaRDD can be created either implicitly or explicitly from a regular RDD.
A Transformer is an abstraction which includes feature transformers and learned models, implementing a method transform() which converts one SchemaRDD into another, generally by appending one or more columns.
An Estimator abstracts the concept of a learning algorithm or any algorithm which fits or trains on data, implementing a method fit() which accepts a SchemaRDD and produces a Transformer. For example, a learning algorithm such as LogisticRegression is an Estimator, and calling fit() trains a LogisticRegressionModel, which is a Transformer.
In machine learning, it is common to run a sequence of algorithms to process and learn from data such as splitting each document text into words, converting words into a numerical feature vector, and training a prediction model using the feature vectors and labels. Spark ML represents such workflows as Pipelines, which consist of a sequence of Transformers and Estimators to be run in a specific order, calling transform() and fit() methods respectively, to produce a Transformer (which becomes part of the PipelineModel, or fitted Pipeline), and that Transformer’s transform() method is called on the dataset.
A Pipeline is an Estimator. Thus, after a Pipeline’s fit() method runs, it produces a PipelineModel which is a Transformer. Once the PipelineModel’s transform() method is called on a test dataset, the data passes through the Pipeline in order, updating the data set as each stage’s transform() method updates it and passes it to the next stage.
Pipelines and PipelineModels help to ensure that training and test data go through identical feature processing steps. Below is an example of a pipeline model. Please note that data preprocessing is carried out on a Pandas Dataframe.
#define regex pattern for preprocessingpat1 = r’@[A-Za-z0–9_]+’pat2 = r’https?://[^ ]+’combined_pat = r’|’.join((pat1,pat2))www_pat = r’www.[^ ]+’negations_dic = {“isn’t”:”is not”, “aren’t”:”are not”, “wasn’t”:”was not”, “weren’t”:”were not”, “haven’t”:”have not”,”hasn’t”:”has not”,”hadn’t”:”had not”,”won’t”:”will not”, “wouldn’t”:”would not”, “don’t”:”do not”, “doesn’t”:”does not”,”didn’t”:”did not”, “can’t”:”can not”,”couldn’t”:”could not”,”shouldn’t”:”should not”,”mightn’t”:”might not”, “mustn’t”:”must not”}neg_pattern = re.compile(r’\b(‘ + ‘|’.join(negations_dic.keys()) + r’)\b’)def pre_processing(column):   first_process = re.sub(combined_pat, ‘’, column)   second_process = re.sub(www_pat, ‘’, first_process)   third_process = second_process.lower()   fourth_process = neg_pattern.sub(lambda x:\    negations_dic[x.group()], third_process)   result = re.sub(r’[^A-Za-z ]’,’’,fourth_process)   return result.strip()df[‘text’]=df.iloc[:,:].text.apply(lambda x: pre_processing(x))df=df[~(df[‘text’]==’’)]# shuffle the datadf = df.sample(frac=1).reset_index(drop=True)# set the random seed and split train and test with 99 to 1 rationp.random.seed(777)msk = np.random.rand(len(df)) < 0.99train = df[msk].reset_index(drop=True)test = df[~msk].reset_index(drop=True)# save both train and test as CSV filestrain.to_csv(‘pyspark_train_data.csv’)test.to_csv(‘pyspark_test_data.csv’)
tokenizer = [Tokenizer(inputCol='text',outputCol='words')]ngrams = [NGram(n=i, inputCol='words', outputCol='{}_grams'.format(i)) for i in range(1,4)]cv =[CountVectorizer(vocabSize=5460,inputCol='{}_grams'.format(i),\           outputCol='{}_tf'.format(i)) for i in range(1,4)]idf = [IDF(inputCol='{}_tf'.format(i),\        outputCol='{}_tfidf'.format(i), minDocFreq=5) \        for i in range(1,4)]assembler = [VectorAssembler(inputCols=['{}_tfidf'.format(i)\            for i  in range(1,4)], outputCol='features')]label_stringIdx = [StringIndexer(inputCol='sentiment', \                    outputCol='label')]lr = [LogisticRegression()]pipeline = Pipeline(stages=tokenizer+ngrams+cv+idf+assembler+\           label_stringIdx+lr)model = pipeline.fit(train_set)predictions = model.transform(test_set)
In PySpark, interactions with SparkSQL through DataFrame API and SQL queries are priority subjects to learn. Pyspark.sql module must be perused for a better understanding of basic operations. DataFrame transformations and actions are not too hard to construct programmatically and operations on DataFrames can also be done using SQL queries. If you have time to take only one course, please spend it on Feature Engineering with Pyspark course by John Hogue. This course is also recommended, time permitting (Day 5).
Day 6/7: Practice Makes Perfect
The best way to learn is practice. You will need a hardened skill set for next week. Try to deep dive into Pyspark modules, since you will be extensively using them. To re-iterate “First Steps With PySpark and Big Data Processing”, please follow these exercises. It is a recommended practice to download some datasets from Kaggle or Project Gutenberg and solve pre-processing problems, as well as others. Please follow this excellent notebook from the creators of Spark NLP. For additional resources, please observe this notebook and this article.
Next Up: Part II — How to Wrap Your Head Around Spark NLP. In this part, we will understand “Annotators /Transformers in Spark NLP” and emphasize “Text Preprocessing with Spark”, “Pretrained Models” and “Text Classifiers” using numerous notebooks. Moreover, we will run a complete pipeline with spaCy and SparkNLP and compare the results.