Package: nlu
NLU: The Power of Spark NLP, the Simplicity of PythonJohn Snow Labs' NLU is a Python library for applying state-of-the-art text mining, directly on any dataframe, with a single line of code.
As a facade of the award-winning Spark NLP library, it comes with1000+of pretrained models in100+, all production-grade, scalable, and trainable, witheverything in 1 line of code.NLU in ActionSee how easy it is to use any of thethousandsof models in 1 line of code, there are hundreds oftutorialsandsimple examplesyou can copy and paste into your projects to achieve State Of The Art easily.NLU & Streamlit in ActionThis 1 line let's you visualize and play with1000+ SOTA NLU & NLP modelsin200languagesstreamlitrunhttps://raw.githubusercontent.com/JohnSnowLabs/nlu/master/examples/streamlit/01_dashboard.pyNLU provides tight and simple integration into Streamlit, which enables building powerful webapps in just 1 line of code which showcase the.
View theNLU&Streamlit documentationorNLU & Streamlit examples section.
The entire GIF demo andAll NLU resources overviewTake a look at our official NLU page:https://nlu.johnsnowlabs.com/for user documentation and examplesRessourceDescriptionInstall NLUJust runpip install nlu pyspark==3.0.2The NLU NamespaceFind all the names of models you can load withnlu.load()Thenlu.load(<Model>)functionLoad any of the1000+ models in 1 lineThenlu.load(<Model>).predict(data)functionPredict onStrings,List of Strings,Numpy Arrays,Pandas,ModinandSpark DataframesThenlu.load(<train.Model>).fit(data)functionTrain a text classifier for2-Class,N-ClassesMulti-N-Classes,Named-Entitiy-RecognitionorParts of Speech TaggingThenlu.load(<Model>).viz(data)functionVisualize the results ofWord Embedding Similarity Matrix,Named Entity Recognizers,Dependency Trees & Parts of Speech,Entity Resolution,Entity LinkingorEntity Status AssertionThenlu.load(<Model>).viz_streamlit(data)functionDisplay an interactive GUI which lets you explore and test every model and feature in NLU in 1 click.General ConceptsGeneral concepts in NLUThe latest release notesNewest features added to NLUOverview NLU 1-liners examplesMost common used models and their resultsOverview NLU 1-liners examples for healthcare modelsMost common used healthcare models and their resultsOverview of all NLU tutorials and Examples100+ tutorials on how to use NLU on text datasets for various problems and from various sources like Twitter, Chinese News, Crypto News Headlines, Airline Traffic communication, Product review classifier training,Connect with us on SlackProblems, questions or suggestions? We have a  very active and helpful community of over 2000+ AI enthusiasts putting NLU, Spark NLP & Spark OCR to good useDiscussion ForumMore indepth discussion with the community? Post a thread in our discussion ForumJohn Snow Labs MediumArticles and Tutorials on the NLU, Spark NLP and Spark OCRJohn Snow Labs YoutubeVideos and Tutorials on the NLU, Spark NLP and Spark OCRNLU WebsiteThe official NLU websiteGithub IssuesReport a bugGetting Started with NLUTo get your hands on the power of NLU, you just need to install it via pip and ensure Java 8 is installed and properly configured. CheckoutQuickstart for more infospipinstallnlupyspark==3.0.2Loading and predicting with any model in 1 line pythonimportnlunlu.load('sentiment').predict('I love NLU! <3')Loading and predicting with multiple models in 1 lineGet 6 different embeddings in 1 line and use them for downstream data science tasks!nlu.load('bert elmo albert xlnet glove use').predict('I love NLU! <3')What kind of models does NLU provide?NLU provides everything a data scientist might want to wish for in one line of code!NLU provides everything a data scientist might want to wish for in one line of code!1000 + pre-trained models100+ of the latest NLP word embeddings ( BERT, ELMO, ALBERT, XLNET, GLOVE, BIOBERT, ELECTRA, COVIDBERT) and different variations of them50+ of the latest NLP sentence embeddings ( BERT, ELECTRA, USE) and different variations of them100+ Classifiers (NER, POS, Emotion, Sarcasm, Questions, Spam)300+ Supported LanguagesSummarize Text and Answer Questions with T5Labeled and Unlabeled Dependency parsingVarious Text Cleaning and Pre-Processing methods like Stemming, Lemmatizing, Normalizing, Filtering, Cleaning pipelines and moreClassifiers trained on many different datasetsChoose the right tool for the right task! Whether you analyze movies or twitter, NLU has the right model for you!trec6 classifiertrec10 classifierspam classifierfake news classifieremotion classifiercyberbullying classifiersarcasm classifiersentiment classifier for moviesIMDB Movie Sentiment classifierTwitter sentiment classifierNER pretrained on ONTO notesNER trainer on CONLLLanguage classifier for 20 languages on the wiki 20 lang dataset.Utilities for the Data Science NLU applicationsWorking with text data can sometimes be quite a dirty job. NLU helps you keep your hands clean by providing components that take away from data engineering intensive tasks.Datetime MatcherPattern MatcherChunk MatcherPhrases MatcherStopword CleanersPattern CleanersSlang CleanerWhere can I see all models available in NLU?For NLU models to load, seethe NLU Namespaceor theJohn Snow Labs Modelshubor gostraight to the source.Supported Data TypesPandas DataFrame and SeriesSpark DataFramesModin with Ray backendModin with Dask backendNumpy arraysStrings and lists of stringsOverview of all tutorials using the NLU-LibraryIn the following tabular, all available tutorials using NLU are listed. These tutorials will help you learn the
usage of the NLU library and on how to use it for your own tasks. Some of the tasks NLU does are
translating from any language to the english language, lemmatizing, tokenizing, cleaning text from
Symbol or unwanted syntax, spellchecking, detecting entities, analyzing sentiments and many more!{:.table2}Tutorial DescriptionNLU Spells UsedOpen In ColabDataset and Paper ReferencesAlbert Word Embeddings with NLUalbert,sentiment pos albert emotionAlbert-Paper,Albert on Github,Albert on TensorFlow,T-SNE,T-SNE-Albert,Albert_EmbeddingBert Word Embeddings with NLUbert,pos sentiment emotion bertBert-Paper,Bert Github,T-SNE,T-SNE-Bert,Bert_EmbeddingBIOBERT Word Embeddings with NLUbiobert,sentiment pos biobert emotionBioBert-Paper,Bert Github,BERT: Deep Bidirectional Transformers,Bert Github,T-SNE,T-SNE-Biobert,Biobert_EmbeddingCOVIDBERT Word Embeddings with NLUcovidbert,sentiment covidbert posCovidBert-Paper,Bert Github,T-SNE,T-SNE-CovidBert,Covidbert_EmbeddingELECTRA Word Embeddings with NLUelectra,sentiment pos  en.embed.electra emotionElectra-Paper,T-SNE,T-SNE-Electra,Electra_EmbeddingELMO Word Embeddings with NLUelmo,sentiment pos elmo emotionELMO-Paper,Elmo-TensorFlow,T-SNE,T-SNE-Elmo,Elmo-EmbeddingGLOVE Word Embeddings with NLUglove,sentiment pos glove emotionGlove-Paper,T-SNE,T-SNE-Glove,Glove_EmbeddingXLNET Word Embeddings with NLUxlnet,sentiment pos  xlnet emotionXLNet-Paper,Bert Github,T-SNE,T-SNE-XLNet,Xlnet_EmbeddingMultiple Word-Embeddings and Part of Speech in 1 Line of codebert electra elmo glove xlnet albert posBert-Paper,Albert-Paper,ELMO-Paper,Electra-Paper,XLNet-Paper,Glove-PaperNormalzing with NLUnorm-Detect sentences with NLUsentence_detector.deep,sentence_detector.pragmatic,xx.sentence_detectorSentence DetectorSpellchecking with NLUn.a.n.a.-Stemming with NLUen.stem,de.stem-Stopwords removal with NLUstopwordsStopwordsTokenization with NLUtokenize-Normalization of Documentsnorm_document-Open and Closed book question answering with Google's T5en.t5,answer_questionT5-Paper,T5-ModelOverview of every task available with T5en.t5.baseT5-Paper,T5-ModelTranslate between more than 200 Languages in 1 line of code with Marian Modelstr.translate_to.fr,en.translate_to.fr,fr.translate_to.he,en.translate_to.deMarian-Papers,Translation-Pipeline (En to Fr),Translation-Pipeline (En to Ger)BERT Sentence Embeddings with NLUembed_sentence.bert,pos sentiment embed_sentence.bertBert-Paper,Bert Github,Bert-Sentence_EmbeddingELECTRA Sentence Embeddings with NLUembed_sentence.electra,pos sentiment embed_sentence.electraElectra Paper,Sentence-Electra-EmbeddingUSE Sentence Embeddings with NLUuse,pos sentiment use emotionUniversal Sentence Encoder,USE-TensorFlow,Sentence-USE-EmbeddingSentence similarity with NLU using BERT embeddingsembed_sentence.bert,use en.embed_sentence.electra embed_sentence.bertBert-Paper,Bert Github,Bert-Sentence_EmbeddingPart of Speech tagging with NLUposPart of SpeechNER Aspect Airline ATISen.ner.aspect.airlineNER Airline Model,Atis intent DatasetNLU-NER_CONLL_2003_5class_examplenerNER-PipleNamed-entity recognition with Deep Learning ONTO NOTESner.ontoNER_OntoAspect based NER-Sentiment-Restaurantsen.ner.aspect_sentiment-Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Chinesezh.segment_words,zh.pos,zh.ner,zh.translate_to.enTranslation-Pipeline (Zh to En)Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Japaneseja.segment_words,ja.pos,ja.ner,ja.translate_to.enTranslation-Pipeline (Ja to En)Detect Named Entities (NER), Part of Speech Tags (POS) and Tokenize in Koreanko.segment_words,ko.pos,ko.ner.kmou.glove_840B_300d,ko.translate_to.en-Date Matchingmatch.datetime-Typed Dependency Parsing with NLUdepDependency ParsingUntyped Dependency Parsing with NLUdep.untyped-E2E Classification with NLUe2ee2e-ModelLanguage Classification with NLUlang-Cyberbullying Classification with NLUclassify.cyberbullyingCyberbullying-ClassifierSentiment Classification with NLU for TwitteremotionEmotion detectionFake News Classification with NLUen.classify.fakenewsFakenews-ClassifierIntent Classification with NLUen.classify.intent.airlineAirline-Intention classifier,Atis-DatasetQuestion classification based on the TREC dataseten.classify.questionsQuestion-ClassifierSarcasm Classification with NLUen.classify.sarcasmSarcasm-ClassifierSentiment Classification with NLU for Twitteren.sentiment.twitterSentiment_Twitter-ClassifierSentiment Classification with NLU for Moviesen.sentiment.imdbSentiment_imdb-ClassifierSpam Classification with NLUen.classify.spamSpam-ClassifierToxic text classification with NLUen.classify.toxicToxic-ClassifierUnsupervised keyword extraction with NLU using the YAKE algorithmyake-Grammatical Chunk Matching with NLUmatch.chunks-Getting n-Grams with NLUngram-Assertionen.med_ner.clinical en.assert,en.med_ner.clinical.biobert en.assert.biobert, ...Healthcare-NER,NER_Clinical-Classifier,Toxic-ClassifierDe-Identification Model overviewmed_ner.jsl.wip.clinical en.de_identify,med_ner.jsl.wip.clinical en.de_identify.clinical, ...NER-ClinicalDrug Normalizationnorm_drugs-Entity Resolutionmed_ner.jsl.wip.clinical en.resolve_chunk.cpt_clinical,med_ner.jsl.wip.clinical en.resolve.icd10cm, ...NER-Clinical,Entity-Resolver clinicalMedical Named Entity Recognitionen.med_ner.ade.clinical,en.med_ner.ade.clinical_bert,en.med_ner.anatomy,en.med_ner.anatomy.biobert,  ...-Relation Extractionen.med_ner.jsl.wip.clinical.greedy en.relation,en.med_ner.jsl.wip.clinical.greedy en.relation.bodypart.problem, ...-Visualization of NLP-Models with Spark-NLP and NLUner,dep.typed,med_ner.jsl.wip.clinical resolve_chunk.rxnorm.in,med_ner.jsl.wip.clinical resolve.icd10cmNER-Piple,Dependency Parsing,NER-Clinical,Entity-Resolver (Chunks) clinicalNLU Covid-19 Emotion ShowcaseemotionEmotion detectionNLU Covid-19 Sentiment ShowcasesentimentSentiment classificationNLU Airline Emotion DemoemotionEmotion detectionNLU Airline Sentiment DemosentimentSentiment classificationBengali NER Hindi Embeddings for 30 Modelsbn.ner,bn.lemma,ja.lemma,am.lemma,bh.lemma,en.ner.onto.bert.small_l2_128,..Bengali-NER,Bengali-Lemmatizer,Japanese-Lemmatizer,Amharic-LemmatizerEntity Resolutionmed_ner.jsl.wip.clinical en.resolve.umls,med_ner.jsl.wip.clinical en.resolve.loinc,med_ner.jsl.wip.clinical en.resolve.loinc.biobert-NLU 20 Minutes Crashcourse - the fast Data Science routespell,sentiment,pos,ner,yake,en.t5,emotion,answer_question,en.t5.base...T5-Model,Part of Speech,NER-Piple,Emotion detection,Spellchecker,Sentiment classificationChapter 0: Intro: 1-linerssentiment,pos,ner,bert,elmo,embed_sentence.bertPart of Speech,NER-Piple,Sentiment classification,Elmo-Embedding,Bert-Sentence_EmbeddingChapter 1: NLU base-features with some classifiers on testdataemotion,yake,stemEmotion detectionChapter 2: Translation between 300+ languages with Mariantr.translate_to.en,en.translate_to.fr,en.translate_to.heTranslation-Pipeline (En to Fr),Translation (En to He)Chapter 3: Answer questions and summarize Texts with T5answer_question,en.t5,en.t5.baseT5-ModelChapter 4: Overview of T5-Tasksen.t5.baseT5-ModelGraph NLU 20 Minutes Crashcourse - State of the Art Text Mining for Graphsspell,sentiment,pos,ner,yake,emotion,med_ner.jsl.wip.clinical, ...Part of Speech,NER-Piple,Emotion detection,Spellchecker,Sentiment classificationHealthcare with NLUmed_ner.human_phenotype.gene_biobert,med_ner.ade_biobert,med_ner.anatomy,med_ner.bacterial_species,...-Part 0: Intro: 1-linersspell,sentiment,pos,ner,bert,elmo,embed_sentence.bertBert-Paper,Bert Github,T-SNE,T-SNE-Bert,Part of Speech,NER-Piple,Spellchecker,Sentiment classification,Elmo-Embedding,Bert-Sentence_EmbeddingPart 1: NLU base-features with some classifiers on Testdatayake,stem,ner,emotionNER-Piple,Emotion detectionPart 2: Translate between 200+ Languages in 1 line of code with Marian-Modelsen.translate_to.de,en.translate_to.fr,en.translate_to.heTranslation-Pipeline (En to Fr),Translation-Pipeline (En to Ger),Translation (En to He)Part 3: More Multilingual NLP-translations for Asian Languages with Marianen.translate_to.hi,en.translate_to.ru,en.translate_to.zhTranslation (En to Hi),Translation (En to Ru),Translation (En to Zh)Part 4: Unsupervise Chinese Keyword Extraction, NER and Translation from chinese newszh.translate_to.en,zh.segment_words,yake,zh.lemma,zh.nerTranslation-Pipeline (Zh to En),Zh-LemmatizerPart 5: Multilingual sentiment classifier training for 100+ languagestrain.sentiment,xx.embed_sentence.labse train.sentimentn.a.Sentence_Embedding.LabsePart 6: Question-answering and Text-summarization  with T5-Modellanswer_question,en.t5,en.t5.baseT5-PaperPart 7: Overview of all tasks available with T5en.t5.baseT5-PaperPart 8: Overview of some of the Multilingual modes with State Of the Art accuracy (1-liner)bn.lemma,ja.lemma,am.lemma,bh.lemma,zh.segment_words, ...Bengali-Lemmatizer,Japanese-Lemmatizer,Amharic-LemmatizerOverview of some Multilingual modes avaiable with State Of the Art accuracy (1-liner)bn.ner.cc_300d,ja.ner,zh.ner,th.ner.lst20.glove_840B_300D,ar.nerBengali-NERNLU 20 Minutes Crashcourse - the fast Data Science route--Need help?Ping us on SlackPost an issue on GithubSimple NLU DemosNLU different output levels DemoFeatures in NLU OverviewTokenizationTrainable Word SegmentationStop Words RemovalToken NormalizerDocument NormalizerStemmerLemmatizerNGramsRegex MatchingText Matching,ChunkingDate MatcherSentence DetectorDeep Sentence Detector (Deep learning)Dependency parsing (Labeled/unlabeled)Part-of-speech taggingSentiment Detection (ML models)Spell Checker (ML and DL models)Word Embeddings (GloVe and Word2Vec)BERT Embeddings (TF Hub models)ELMO Embeddings (TF Hub models)ALBERT Embeddings (TF Hub models)XLNet EmbeddingsUniversal Sentence Encoder (TF Hub models)BERT Sentence Embeddings (42 TF Hub models)Sentence EmbeddingsChunk EmbeddingsUnsupervised keywords extractionLanguage Detection & Identification (up to 375 languages)Multi-class Sentiment analysis (Deep learning)Multi-label Sentiment analysis (Deep learning)Multi-class Text Classification (Deep learning)Neural Machine TranslationText-To-Text Transfer Transformer (Google T5)Named entity recognition (Deep learning)Easy TensorFlow integrationGPU SupportFull integration with Spark ML functions1000 pre-trained models in +200 languages!Multi-lingual NER models: Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Hewbrew, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Urdu and moreNatural Language inferenceCoreference resolutionSentence CompletionWord sense disambiguationClinical entity recognitionClinical Entity LinkingEntity normalizationAssertion Status DetectionDe-identificationRelation ExtractionClinical Entity ResolutionCitationWe have published apaperthat you can cite for the NLU library:@article{KOCAMAN2021100058,title={Spark NLP: Natural language understanding at scale},journal={Software Impacts},pages={100058},year={2021},issn={2665-9638},doi={https://doi.org/10.1016/j.simpa.2021.100058},url={https://www.sciencedirect.com/science/article/pii/S2665963821000063},author={Veysel Kocaman and David Talby},keywords={Spark, Natural language processing, Deep learning, Tensorflow, Cluster},abstract={Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}}}