Package: nlu-spark23
NLU: The Power of Spark NLP, the Simplicity of PythonJohn Snow Labs' NLU is a Python library for applying state-of-the-art text mining, directly on any dataframe, with a single line of code.
As a facade of the award-winning Spark NLP library, it comes with hundreds of pretrained models in tens of languages - all production-grade, scalable, and trainable.Project's WebsiteTake a look at our official Spark NLU page:https://nlu.johnsnowlabs.com/for user documentation and examplesNLU in actionFeaturesTokenizationTrainable Word SegmentationStop Words RemovalToken NormalizerDocument NormalizerStemmerLemmatizerNGramsRegex MatchingText MatchingChunkingDate MatcherSentence DetectorDeep Sentence Detector (Deep learning)Dependency parsing (Labeled/unlabeled)Part-of-speech taggingSentiment Detection (ML models)Spell Checker (ML and DL models)Word Embeddings (GloVe and Word2Vec)BERT Embeddings (TF Hub models)ELMO Embeddings (TF Hub models)ALBERT Embeddings (TF Hub models)XLNet EmbeddingsUniversal Sentence Encoder (TF Hub models)BERT Sentence Embeddings (42 TF Hub models)Sentence EmbeddingsChunk EmbeddingsUnsupervised keywords extractionLanguage Detection & Identification (up to 375 languages)Multi-class Sentiment analysis (Deep learning)Multi-label Sentiment analysis (Deep learning)Multi-class Text Classification (Deep learning)Neural Machine TranslationText-To-Text Transfer Transformer (Google T5)Named entity recognition (Deep learning)Easy TensorFlow integrationGPU SupportFull integration with Spark ML functions+710 pre-trained models in +192 languages!+450 pre-trained pipelines in +192 languages!Multi-lingual NER models: Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Hewbrew, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, and Urdu.Getting Started with NLUTo get your hands on the power of NLU, you just need to install it via pip and ensure Java 8 is installed and properly configured. CheckoutQuickstart for more infospipinstallnlupyspark==2.4.7Loading and predict with any model in 1 line pythonimportnlunlu.load('sentiment').predict('I love NLU! <3')Loading and predict with multiple models in 1 lineGet 6 different embeddings in 1 line and use them for downstream data science tasks!nlu.load('bert elmo albert xlnet glove use').predict('I love NLU! <3')What kind of models does NLU provide?NLU provides everything a data scientist might want to wish for in one line of code!NLU provides everything a data scientist might want to wish for in one line of code!1000 + pre-trained models100+ of the latest NLP word embeddings ( BERT, ELMO, ALBERT, XLNET, GLOVE, BIOBERT, ELECTRA, COVIDBERT) and different variations of them50+ of the latest NLP sentence embeddings ( BERT, ELECTRA, USE) and different variations of them100+ Classifiers (NER, POS, Emotion, Sarcasm, Questions, Spam)300+ Supported LanguagesSummarize Text and Answer Questions with T5Labeled and Unlabeled Dependency parsingVarious Text Cleaning and Pre-Processing methods like Stemming, Lemmatizing, Normalizing, Filtering, Cleaning pipelines and moreClassifiers trained on many different different datasetsChoose the right tool for the right task! Whether you analyze movies or twitter, NLU has the right model for you!trec6 classifiertrec10 classifierspam classifierfake news classifieremotion classifiercyberbullying classifiersarcasm classifiersentiment classifier for moviesIMDB Movie Sentiment classifierTwitter sentiment classifierNER pretrained on ONTO notesNER trainer on CONLLLanguage classifier for 20 languages on the wiki 20 lang dataset.Utilities for the Data Science NLU applicationsWorking with text data can sometimes be quite a dirty Job. NLU helps you keep your hands clean by providing lots of components that take away data engineering intensive tasks.Datetime MatcherPattern MatcherChunk MatcherPhrases MatcherStopword CleanersPattern CleanersSlang CleanerWhere can I see NLUs entire offer?Checkout theNLU Namespacefor everything that NLU has to offer!Supported Data TypesPandas DataFrame and SeriesSpark DataFramesModin with Ray backendModin with Dask backendNumpy arraysStrings and lists of stringsCheckout the following notebooks for examples on how to work with NLU.NLU Demos on DatasetsKaggle Twitter Airline Sentiment Analysis NLU demoKaggle Twitter Airline Emotion Analysis NLU demoKaggle Twitter COVID Sentiment Analysis NLU demoKaggle Twitter COVID Emotion Analysis nlu demoNLU component examplesCheckout the following notebooks for examples on how to work with NLU.NLU Training ExamplesBinary Class Text Classification training2 class Finance News sentiment classifier training2 class Reddit comment sentiment classifier training2 class Apple Tweets sentiment classifier training2 class IMDB Movie sentiment classifier training2 class twitter classifier trainingMulti Class Text Classification training5 class WineEnthusiast Wine review classifier training3 class Amazon Phone review classifier training5 class Amazon Musical Instruments review classifier training5 class Tripadvisor Hotel review classifier training5 class Phone review classifier trainingMulti Label Text  Classification trainingTrain Multi Label Classifier on E2E dataset DemoTrain Multi Label  Classifier on Stack Overflow Question Tags dataset DemoNamed Entity Recognition training (NER)NER Training examplePart of Speech tagger training (POS)POS Training exampleNLU Applications ExamplesSentence Similarity with Multiple Sentence Embeddings6 Wordembeddings in 1 line with T-SNE plottingNLU Demos on DatasetsKaggle Twitter Airline Sentiment Analysis NLU demoKaggle Twitter Airline Emotion Analysis NLU demoKaggle Twitter COVID Sentiment Analysis NLU demoKaggle Twitter COVID Emotion Analysis nlu demoNLU examples grouped by componentThe following are Collab examples which showcase each NLU component and some applications.Named Entity Recognition (NER)NER pretrained on ONTO NotesNER pretrained on CONLLTokenize, extract POS and NER in ChineseTokenize, extract POS and NER in KoreanTokenize, extract POS and NER in JapaneseAspect based sentiment NER sentiment for restaurantsPart of speech (POS)POS pretrained on ANC datasetTokenize, extract POS and NER in ChineseTokenize, extract POS and NER in KoreanTokenize, extract POS and NER in JapaneseSequence2SequenceTranslate between 192+ languages with marianTry out the 18 Tasks like Summarization Question Answering and more on T5T5 Open and Closed Book question answering tutorialClassifiersUnsupervised Keyword Extraction with YAKEToxic Text ClassifierTwitter Sentiment ClassifierMovie Review Sentiment ClassifierSarcasm Classifier50 Class Questions Classifier300 Class Languages ClassifierFake News ClassifierE2E ClassifierCyberbullying ClassifierSpam ClassifierEmotion ClassifierWord EmbeddingsBERT, ALBERT, ELMO, ELECTRA, XLNET, GLOVE at once with t-SNE plottingBERT Word Embeddings and t-SNE plottingALBERT Word Embeddings and t-SNE plottingELMO Word Embeddings and t-SNE plottingXLNET Word Embeddings and t-SNE plottingELECTRA Word Embeddings and t-SNE plottingCOVIDBERT Word Embeddings and t-SNE plottingBIOBERT Word Embeddings and t-SNE plottingGLOVE Word Embeddings and t-SNE plottingSentence EmbeddingsBERT Sentence Embeddings and t-SNE plottingELECTRA Sentence Embeddings and t-SNE plottingUSE Sentence Embeddings and t-SNE plottingSentence EmbeddingsBERT Sentence Embeddings and t-SNE plottingELECTRA Sentence Embeddings and t-SNE plottingUSE Sentence Embeddings and t-SNE plottingDependency ParsingUntyped Dependency ParsingTyped Dependency ParsingText Pre Processing and CleaningTokenizationStopwords removalStemmingLemmatizationNormalizingSpell checkingSentence DetectingNormalize documentsChunkersN GramEntity ChunkingMatchersDate MatcherNeed help?Ping us on SlackPost an issue on GithubSimple NLU DemosNLU different output levels Demo