How to Wrap Your Head Around Spark NLP | by Mustafa Aytuğ Kaya | Towards Data Science

Welcome to the second part of the Spark NLP article. In the first part, the objective was to present an ice breaker for NLP practitioners and warm-up minds towards Spark NLP. The strongest bias against any Spark-based library comes from the school of thought that states “Spark code is a bit different than your regular Python script”. To fight this prejudice, learning strategies were shared, and if you have followed them, you are ready for the next level.
In this part of the article, we will compare spaCy to Spark NLP and dive deeper into Spark NLP modules and pipelines. I created a notebook that uses both spaCy and Spark NLP for doing the same things for picture-perfect comparison. SpaCy performs well, but in terms of speed, memory consumption, and accuracy Spark NLP outperforms it. Therefore, in terms of ease of use, once initial friction of Spark is overcome, I found it to be at least on par with spaCy, thanks to the convenience that comes with pipelines.
Usage of smart, comprehensive notebooks, prepared by Spark NLP creators, which provide numerous examples for real-life scenarios together with the repo I created for practice is highly recommended to excel in the required skill sets.
Day 8/9: Understanding Annotators /Transformers in Spark NLP and Text Preprocessing with Spark
Built natively on Apache Spark and TensorFlow, Spark NLP library provides simple, performant as well as accurate NLP notations for machine learning pipelines which can scale easily in a distributed environment. This library reuses the Spark ML pipeline along with integrating NLP functionality.
The library covers many common NLP tasks, including tokenization, stemming, lemmatization, part of speech tagging, sentiment analysis, spell checking, named entity recognition, all of which are included as open-source and can be used by training models with your data. Spark NLP’s annotators utilize rule-based algorithms, machine learning, and some Tensorflow running under the hood to power specific deep learning implementations.
In Spark NLP, all Annotators are either Estimators or Transformers just like Spark ML, consisting of two types: AnnotatorApproach and AnnotatorModel. Any annotator training on a DataFrame that produces a model is an AnnotatorApproach. Those transforming a DataFrame into another DataFrame through some models are AnnotatorModels (e.g. WordEmbeddingsModel). Normally, an annotator does not take Model suffix if it doesn’t rely on a pre-trained annotator while transforming a DataFrame (e.g. Tokenizer). Here is a list of Annotators and their descriptions:
To get through the NLP process, we need to get raw data preprocessed. In addition to SQL filters, transformations, and user-defined functions, Spark NLP comes with powerful tools for the task. DocumentAssembler is a special transformer that creates the first annotation of type Document which may be used by follow up annotators in the pipeline.
TokenAssembler reconstructs a Document type annotation from tokens, usually after these have been normalized, lemmatized, normalized, spell checked, etc, to use this document annotation in further annotators.
Doc2Chunk converts Document type annotations into Chunk type with the contents of a chunk Col, while Chunk2Doc converts a CHUNK type column back into DOCUMENT. Useful when trying to re-tokenize or do further analysis on a CHUNK result.
Finisher outputs annotation(s) values into a string for ease of use. Once we have our NLP pipeline ready to go, we might want to use our annotation results somewhere else where it is practically available.
As stated earlier, a Pipeline is specified as a sequence of stages, and each stage is either a Transformer or an Estimator. These stages are run in order, and the input DataFrame is transformed as it passes through each stage.
And here is how this pipeline is coded in Spark NLP.
from pyspark.ml import Pipelinedocument_assembler = DocumentAssembler()\ .setInputCol(“text”)\ .setOutputCol(“document”)sentenceDetector = SentenceDetector()\ .setInputCols([“document”])\ .setOutputCol(“sentences”)tokenizer = Tokenizer() \ .setInputCols([“sentences”]) \ .setOutputCol(“token”)normalizer = Normalizer()\ .setInputCols([“token”])\ .setOutputCol(“normal”)word_embeddings=WordEmbeddingsModel.pretrained()\ .setInputCols([“document”,”normal”])\ .setOutputCol(“embeddings”)nlpPipeline = Pipeline(stages=[ document_assembler,  sentenceDetector, tokenizer, normalizer, word_embeddings, ])pipelineModel = nlpPipeline.fit(df)
When we fit() on the pipeline with Spark data frame (df), its text column is fed into DocumentAssembler() transformer at first and then a new column “document” is created in Document type (AnnotatorType). As mentioned before, this transformer is the initial entry point to Spark NLP for any Spark data frame. Then its document column is fed into SentenceDetector() (AnnotatorApproach) and the text is split into an array of sentences and a new column “sentences” in Document type is created. Then the “sentences” column is fed into Tokenizer() (AnnotatorModel) and each sentence is tokenized and a new column “token” in Token type is created. And so on. Please refer to the Spark NLP workshop preprocessing notebook and my repo for seeing annotators and transformers in action.
Day 10: RegexRegular expressions are essentially a tiny, highly specialized programming language embedded inside Python and made available through the re module. Using this little language, you specify the rules for the set of possible strings that you want to match; this set might contain English sentences, or e-mail addresses, clinical entities, or anything you like. You can also use regexes to modify a string or to split it apart in various ways.
Regex is one of the major building blocks of NLP, which we will be using not only for preprocessing but also as a part of RegexMatcher in the Spark NLP pipeline. As it can sometimes be quite complicated, I recommend this course by Maria Eugenia Inzaugarat to have a quick grasp on advanced topics, and please refer to the official python documentation as well as the brilliant python regex how to tutorial. Another key component I used for mastering regex was Regex101, a very powerful API that enables you to test almost any regular expression on text chunks.
Most of the time Regex is overlooked. Trust me: you will need it and you will use it. The more you know, the better.
Day 11: Pretrained Models
It was previously mentioned that trained annotators are called AnnotatorModel and the goal here is to transform one DataFrame into another through the specified model (trained annotator). Spark NLP offers pre-trained models in many languages and all you need to do is load the pre-trained model and configure its parameters accordingly. Why worry about training a new model from scratch when you can enjoy the direct application of pre-trained SOTA algorithms with a single transform()method? In the official documentation, you can find detailed information regarding how these models are trained by using which algorithms and datasets. Sample notebooks can be found here.
MODEL_NAME='sentimentdl_use_twitter'documentAssembler = DocumentAssembler()\    .setInputCol("text")\    .setOutputCol("document")use = UniversalSentenceEncoder.pretrained(name="tfhub_use", lang="en")\ .setInputCols(["document"])\ .setOutputCol("sentence_embeddings")sentimentdl = SentimentDLModel.pretrained(name=MODEL_NAME, lang="en")\    .setInputCols(["sentence_embeddings"])\    .setOutputCol("pr_sentiment")nlpPipeline = Pipeline(      stages = [          documentAssembler,          use,          sentimentdl      ])empty_df = spark.createDataFrame([['']]).toDF("text")pipelineModel = nlpPipeline.fit(empty_df)result = pipelineModel.transform(spark_df1)
Day 12/13: Text Classification
There are several text classification options in Spark NLP:
Text preprocessing in Spark NLP and using ML algorithms from Spark ML
Text preprocessing and word embeddings (Glove, Bert, Elmo) in Spark NLP and ML algorithms from Spark ML
Text preprocessing and sentence embeddings (Universal Sentence Encoders) in Spark NLP and ML algorithms from Spark ML
Text preprocessing and ClassifierDL (TensorFlow based DL architecture) in Spark NLP
As discussed thoroughly in this article about Spark NLP, all these text processing steps before ClassifierDL can be implemented in a pipeline that is specified as a sequence of stages, and each stage is either a Transformer or an Estimator. These stages are run in order, and the input DataFrame is transformed as it passes through each stage. That is, the data are passed through the fitted pipeline in order. Each stage’s transform() method updates the dataset and passes it to the next stage. With the help of Pipelines, we can ensure that training and test data go through identical feature processing steps.
You can build a text classifier with Bert, Elmo, Glove and Universal Sentence Encoders in Spark NLP with this ClassiferDL. Sample classifier notebooks can be found here and here.
document_assembler = DocumentAssembler() \    .setInputCol("comment_text") \    .setOutputCol("document")\    .setCleanupMode('shrink')tokenizer = Tokenizer() \  .setInputCols(["document"]) \  .setOutputCol("token")    normalizer = Normalizer() \    .setInputCols(["token"]) \    .setOutputCol("normalized")\    .setLowercase(True)stopwords_cleaner = StopWordsCleaner()\      .setInputCols("normalized")\      .setOutputCol("cleanTokens")\      .setCaseSensitive(False)lemma = LemmatizerModel.pretrained('lemma_antbnc') \    .setInputCols(["cleanTokens"]) \    .setOutputCol("lemma")bert = BertEmbeddings.pretrained('bert_base_uncased', 'en') \      .setInputCols("document", "lemma") \      .setOutputCol("embeddings")\      .setPoolingLayer(0) # default 0embeddingsSentence = SentenceEmbeddings() \      .setInputCols(["document", "embeddings"]) \      .setOutputCol("sentence_embeddings") \      .setPoolingStrategy("AVERAGE")classsifierdl = ClassifierDLApproach()\  .setInputCols(["sentence_embeddings"])\  .setOutputCol("prediction")\  .setLabelColumn("toxic")\  .setMaxEpochs(5)\  .setEnableOutputLogs(True)\  .setBatchSize(32)\  .setValidationSplit(0.1)\  .setDropout(0.75)\  .setLr(0.0035)\  #.setOutputLogsPath('logs')clf_pipeline = Pipeline(    stages=[document_assembler,             tokenizer,            normalizer,            stopwords_cleaner,             lemma,             bert,            embeddingsSentence,            classsifierdl           ])clf_pipelineModel = clf_pipeline.fit(sdf)
Day 13–14: SpaCy or Spark NLP — A Benchmarking Comparison
In this section, we will get a lot of exercise coding! We will be using a library consisting of seven different classics downloaded from Gutenberg.org. The corpus consists of approximately 4.9 M characters and 97 thousand sentences. We will build the same output data frames using similar processes both by spaCy and Spark NLP and then compare the results.
Spacy is one of the best-documented libraries I have seen and I was pleasantly surprised that now they include a free course, which I have taken previously. A good chance to brush up your skills, if required.
The whole notebook of the comparison and the corpus data can be found in my GitHub repo. Let’s start by examining the spaCy way.
### 2.1 Clean Tabs and Whitespaceclean_shrink = lambda text : text.replace(r'\n|\t|\s+',' ').replace('\s+',' ').strip()df.loc[:,'document']=df.text.map(clean_shrink)### 2.2 Tokenizersentence_tokenizer = lambda sent : [token for token in nlp(sent)]df.loc[:,'token']=df.document.map(sentence_tokenizer)### 2.3 Normalizernormalizer = lambda tokens : [re.sub(punct,'',token.text)  for token in tokens if re.sub(punct,'',token.text) != '']df.loc[:,'normalized']=df.token.map(normalizer)### 2.4 Remove Stop Wordsnormalizer_and_stop = lambda tokens : [re.sub(punct,'',token.text)  for token in tokens if re.sub(punct,'',token.text) != '' and not token.is_stop]df.loc[:,'cleanTokens']=df.token.map(normalizer_and_stop)### 2.5 Lemmatizenormalizer_and_stop_lemma = lambda tokens : [re.sub(punct,'',token.lemma_)  for token in tokens if re.sub(punct,'',token.text) != '' and not token.is_stop]df.loc[:,'lemma']=df.token.map(normalizer_and_stop_lemma)### 2.6 Stemmerstemmer = PorterStemmer()stems = lambda tokens : [stemmer.stem(token.text) if len(tokens)>0 else [] for token in tokens]df.loc[:,'stem']=df.token.map(stems)### 2.7 Part of Speech Taggingnormalizer_and_stop_pos = lambda tokens : [re.sub(punct,'',token.pos_)  for token in tokens if re.sub(punct,'',token.text) != '' and not token.is_stop]df.loc[:,'pos']=df.cleanTokens.map(normalizer_and_stop_pos)### 2.8 Token Assemblertoken_assembler = lambda tokens : " ".join(tokens)df.loc[:,'clean_text']=df.cleanTokens.map(token_assembler)### 2.9 Taggertagger = lambda text : [(ent.text, ent.label_) for ent in nlp(text).ents]df.loc[:,'ner_chunks']=df.loc[:,'document'].map(tagger)### 2.10 Regex Parsernoun_chunker = lambda text : [(chnk,(chnk[0].pos_,chnk[1].pos_,chnk[2].tag_ ))for chnk in nlp(text).noun_chunks if len(chnk.text.split())==3\                              and  chnk.text.replace(' ','').isalpha()   and chnk[0].pos_ == 'DET'and chnk[1].pos_ == 'ADJ' and chnk[2].tag_ in ['NN','NNP']                             ]df.loc[:,'RegexpParser'] =df.loc[:,'document'].map(noun_chunker)
Let’s take a brief pause here and observe the Regex Parser output. We asked for the noun chunker to return chunks that consist of a Determiner, an Adjective, and a Noun (proper, singular or plural).
[chunk for chunk in df.RegexpParser.values if chunk!=[]]
The results look pretty good.
[[(My dear Robinson, ('DET', 'ADJ', 'NNP'))], [(this accidental souvenir, ('DET', 'ADJ', 'NN'))], [(a great deal, ('DET', 'ADJ', 'NN'))], [(a great amount, ('DET', 'ADJ', 'NN'))], [(the local hunt, ('DET', 'ADJ', 'NN')),  (some surgical assistance, ('DET', 'ADJ', 'NN'))], [(a remarkable power, ('DET', 'ADJ', 'NN'))], [(a good deal, ('DET', 'ADJ', 'NN'))], [(a fresh basis, ('DET', 'ADJ', 'NN')),  (this unknown visitor, ('DET', 'ADJ', 'NN'))], [(the obvious conclusion, ('DET', 'ADJ', 'NN'))], [(their good will, ('DET', 'ADJ', 'NN'))], [(my dear Watson, ('DET', 'ADJ', 'NNP')),  (a young fellow, ('DET', 'ADJ', 'NN'))], [(a favourite dog, ('DET', 'ADJ', 'NN'))], [(the latter part, ('DET', 'ADJ', 'NN'))], [(that local hunt, ('DET', 'ADJ', 'NN'))], [(a heavy stick, ('DET', 'ADJ', 'NN'))], [(a professional brother, ('DET', 'ADJ', 'NN'))], [(the dramatic moment, ('DET', 'ADJ', 'NN'))], [(a long nose, ('DET', 'ADJ', 'NN'))],................
Let’s continue building our blocks.
### 2.11 N-Gram Generatorngram_generator = lambda input_list: [*zip(*[input_list[i:] for i in range(n)])]n=3df.loc[:,'triGrams'] = df.loc[:,'token'].map(ngram_generator)### 2.12 Word2Vec Embeddingsvector = lambda tokens: [(token.text, token.has_vector, token.vector, token.is_oov) for token in tokens]df.loc[:,'vectors'] = df.loc[:,'token'].map(vector)### 2.13 Regex Matcherrules = r'''\b[A-Z]\w+ly\b|Stephen\s(?!Proto|Cardinal)[A-Z]\w+|Simon\s[A-Z]\w+'''regex_matchers = lambda text : re.findall(rules,text)df.loc[:,'Regex_matches'] =df.loc[:,'document'].map(regex_matchers)
Let’s take a look at regex matches
df.Regex_matches[df.Regex_matches.map(len)>1]
Here are the results:
13123                      [Polly, Polly]25669                      [Sally, Sally]27262                      [Sally, Sally]27273                      [Polly, Sally]27340                      [Polly, Sally]28311                      [Pelly, Dolly]42016                      [Feely, Feely]49802                    [Finally, Emily]52129                    [Lively, Lively]58295                      [Silly, Milly]62141                      [Silly, Milly]64811                       [Only, Molly]71650                        [Hely, Daly]74427                      [Healy, Dolly]77404                      [Molly, Milly]77437                      [Milly, Molly]81557                     [Molly, Reilly]84023          [Szombathely, Szombathely]89594                       [Healy, Joly]92206    [Simon Dedalus, Stephen Dedalus]92980      [Firstly, Nelly, Nelly, Nelly]93046               [Szombathely, Karoly]94402       [Reilly, Simon Dedalus, Hely]94489    [Stephen Dedalus, Simon Dedalus]
Let’s Venture Into The Characters...
Now that we have a dataset with many features, we have a plethora of options to dive into. Let’s examine the characters that are in the books...Let’s find NER Chunks that have a ‘PERSON’ tag, consisting of 2 words.
flatlist = lambda l : [re.sub("[^a-zA-Z\s\']","",item[0]).title().strip()  for sublist in l for item in sublist if item[1]=='PERSON' and len(item[0].split())==2]ner_chunks = df.ner_chunks.to_list()names=(flatlist(ner_chunks))len(sorted(names))
The code above returned 4832 names, which looks a bit suspicious since this number is high. Let’s inspect the result of the Counter object:
('St Clare', 306), ('Buck Mulligan', 96), ('Aunt Chloe', 76), ('Martin Cunningham', 74), ('Masr George', 45), ('Ned Lambert', 44), ('Solomon Northup', 43), ('Tom Sawyer', 42), ('Aunt Sally', 37), ('Uncle Tom', 37), ('Ben Dollard', 36), ('Myles Crawford', 35), ('Blazes Boylan', 28), ('Uncle Abram', 24), ('J J', 22),......  ('L L', 5), ('Mark Twain', 5), ("Tom Sawyer'S", 5), ('Chapter Xi', 5), ('Chapter Xix', 5), ('Dey Wuz', 5), ('George Jackson', 5), ('Levi Bell', 5), ('King Lear', 5), ('Simon Legree', 5), ('Garrett Deasy', 5), ('A E', 5), ('S D', 5), ('Josie Powell', 5), ('Mrs Purefoy', 5), ('Ben Howth', 5), ('Bald Pat', 5), ('Barney Kiernan', 5), ('Michael Gunn', 5), ('C C', 5),
Many tags are inaccurate, unfortunately. Please observe some chapter titles as well as capitalized initials that are inaccurately returned as PER tags.
In writing the above code, mappings were used to ensure fast pacing, and the Spark NLP pipeline that will be implemented below was mimicked. Once we run a similar code in Spark NLP, we will compare results in terms of memory usage, speed, and accuracy.
Time to do things Spark NLP way!
documentAssembler = DocumentAssembler()\.setInputCol(“text”)\.setOutputCol(“document”)\.setCleanupMode(“shrink”)sentenceDetector = SentenceDetector().\    setInputCols(['document']).\    setOutputCol('sentences')tokenizer = Tokenizer() \    .setInputCols(["sentences"]) \    .setOutputCol("token")ngrams = NGramGenerator() \            .setInputCols(["token"]) \            .setOutputCol("ngrams") \            .setN(3) \            .setEnableCumulative(False)\            .setDelimiter("_") # Default is spacenormalizer = Normalizer() \    .setInputCols(["token"]) \    .setOutputCol("normalized")\    .setLowercase(False)\    .setCleanupPatterns(["[^\w\d\s\.\!\?]"])stopwords_cleaner = StopWordsCleaner()\      .setInputCols("normalized")\      .setOutputCol("cleanTokens")\      .setCaseSensitive(False)\lemma = LemmatizerModel.pretrained('lemma_antbnc') \    .setInputCols(["cleanTokens"]) \    .setOutputCol("lemma")stemmer = Stemmer() \    .setInputCols(["token"]) \    .setOutputCol("stem")pos = PerceptronModel.pretrained("pos_anc", 'en')\      .setInputCols("clean_text", "cleanTokens")\      .setOutputCol("pos")chunker = Chunker()\    .setInputCols(["sentences", "pos"])\    .setOutputCol("chunk")\    .setRegexParsers(["<DT>+<JJ>*<NN>"])  ## Determiner - adjective - singular nountokenassembler = TokenAssembler()\    .setInputCols(["sentences", "cleanTokens"]) \    .setOutputCol("clean_text")\tokenizer2 = Tokenizer() \    .setInputCols(["clean_text"]) \    .setOutputCol("token2")glove_embeddings = WordEmbeddingsModel.pretrained('glove_100d')\          .setInputCols(["document", "lemma"])\          .setOutputCol("embeddings")\          .setCaseSensitive(False)onto_ner = NerDLModel.pretrained("onto_100", 'en') \          .setInputCols(["document", "token", "embeddings"]) \          .setOutputCol("ner")ner_converter = NerConverter() \  .setInputCols(["sentences", "token", "ner"]) \  .setOutputCol("ner_chunk")rules = r'''\b[A-Z]\w+ly\b, starting with a capital letter ending with 'ly'Stephen\s(?!Proto|Cardinal)[A-Z]\w+, followed by "Stephen"Simon\s[A-Z]\w+, followed by "Simon"'''with open('ulyses_regex_rules.txt', 'w') as f:        f.write(rules)regex_matcher = RegexMatcher()\    .setInputCols('sentences')\    .setStrategy("MATCH_ALL")\    .setOutputCol("regex_matches")\    .setExternalRules(path='./ulyses_regex_rules.txt', delimiter=',')
Now that our pieces are ready, let’s define the assembly line.
nlpPipeline = Pipeline(stages=[ documentAssembler, sentenceDetector, tokenizer, ngrams, normalizer, stopwords_cleaner, lemma, stemmer, tokenassembler, tokenizer2, pos, chunker, glove_embeddings, onto_ner, ner_converter, regex_matcher  ])empty_df = spark.createDataFrame([[‘’]]).toDF(“text”)pipelineModel = nlpPipeline.fit(empty_df)lib_result = pipelineModel.transform(library)
Let’s check regex matches according to our search criteria:- A whole word that begins with a capital letter and ends with ‘ly’,- ‘Stephen’ that is not followed by ‘Cardinal’ or ‘Proto’, but followed by a word that starts with a capital letter.- ‘Simon’ is followed by a word that starts with a capital letter.We are looking for at least two occurrences in each sentence...
match_df.filter(F.size('finished_regex_matches')>1).show(truncate = 50)
Let’s take a look at the results...
The chunker annotator in our pipeline is going to return chunks that consist of a Determiner, an Adjective, and a singular Noun.
lib_result.withColumn(    "tmp",     F.explode("chunk")) \    .select("tmp.*").select("begin","end","result","metadata.sentence").show(20,truncate = 100)
Here are the top 20 rows of the chunker results.
Let’s Venture Into The Characters...Spark NLP Way.
Here we examine the characters that are in the books...This time we will be using Spark NLP mechanics. Please note differences in accuracy as compared to spaCy.
l = result_ner.filter(result_ner.ner_label == "PERSON").select(F.expr("ner_chunk")).collect()names = list([re.sub("[^a-zA-Z\s\']","",l_[0]).title() for l_ in l if l_[0].replace(' ','').isalpha() and len(l_[0].strip().split())==2 and "’" not in l_[0]])len(set(names))
This time, the number of characters we have are limited to 1284. Let’s take a look at the most common 350 names.
('Buck Mulligan', 93), ('Aunt Chloe', 82), ('Martin Cunningham', 71), ('Bayou Boeuf', 48), ('Aunt Sally', 39), ('Ned Lambert', 39), ('Mary Jane', 38), ('Solomon Northup', 36), ('John Thornton', 34), ('Myles Crawford', 33), ('Ben Dollard', 31), ('Sherlock Holmes', 30), ('Tom Sawyer', 30), ('John Eglinton', 29), ('Nosey Flynn', 28), ('Corny Kelleher', 27), ('Mrs Breen', 27), ('Father Conmee', 26), ('Uncle Tom', 25), ('John Wyse', 24), ('Henry Baskerville', 23), ('Uncle Abram', 22), ('Blazes Boylan', 19), ('Bob Doran', 18), ('Davy Byrne', 18), ('Coombe Tracey', 17), ('Aunt Phebe', 17), ('Simon Dedalus', 17),........................ ('Sandy Hill', 8), ('Theophilus Freeman', 8), ('Father Cowley', 8), ('Gregory B', 7), ('George Harris', 7), ('Rachel Halliday', 7), ('George Shelby', 7), ('Anne Hampton', 7), ('Peter Tanner', 7), ('Almidano Artifoni', 7), ('Hugo Baskerville', 6), ('Laura Lyons', 6), ('Aunt Polly', 6), ('Peter Wilks', 6),.........................
The names list looks much more accurate. No wonder why Spark NLP is enterprise preferred!
Let’s Talk about Resource Consumption:The system used for this study is an 8 core Intel(R) Core(TM) i7–9700K CPU @ 3.60GHz with 32820MB Memory. The operating system is Ubuntu 20.04.
Spark NLP uses less memory and runs twice as fast when compared to spaCy. This fact, being coupled with higher accuracy of the Spark NLP provides good reasons to master this library!
That’s all for now. The main purpose of this article was to give momentum to intermediate level practitioners a guide on how to achieve their goals in the NLP domain using Spark. Some Spark NLP concepts may be difficult to learn in the beginning, however, the articles, courses, and sample notebooks presented in this paper will provide a good starting point. Initial overhead with Spark may be high, but spaCy — Spark NLP comparison shows the difference. It is not easy, but not too hard either to master Spark NLP...and it is definitely a skill you will have shining on your resume if you aim for the enterprise domain!
Please follow the link to the first part of the article.