Package: nlp-preprocessing-wrappers
NLP Preprocessing WrappersPreprocessing WrappersHow to useInstallInstall the library fromPyPI:pipinstallnlp-preprocessing-wrappersUsageNLP Preprocessing Wrappers is a Python library that provides a set of preprocessing wrappers for Stanza and
spaCy, providing a unified API for both libraries, making them interchangeable.Let's start with a simple example. Here we are using theSpacyTokenizerwrapper to preprocess a text:fromnlp_preprocessing_wrappersimportSpacyTokenizerspacy_tokenizer=SpacyTokenizer(language="en",return_pos_tags=True,return_lemmas=True)tokenized=spacy_tokenizer("Mary sold the car to John.")forwordintokenized:print("{:<5}{:<10}{:<10}{:<10}".format(word.index,word.text,word.pos,word.lemma))"""0    Mary       PROPN      Mary1    sold       VERB       sell2    the        DET        the3    car        NOUN       car4    to         ADP        to5    John       PROPN      John6    .          PUNCT      ."""You can load any model from spaCy, with its canonical name,en_core_web_sm, or with a simple alias, as
we did here, likeen. By default, the simpler alias loads the smaller version of each model. For a complete
list of available models, seespaCy documentation.In the very same way, you can load any model from Stanza using theStanzaTokenizerwrapper:fromnlp_preprocessing_wrappersimportStanzaTokenizerstanza_tokenizer=StanzaTokenizer(language="en",return_pos_tags=True,return_lemmas=True)tokenized=stanza_tokenizer("Mary sold the car to John.")forwordintokenized:print("{:<5}{:<10}{:<10}{:<10}".format(word.index,word.text,word.pos,word.lemma))"""0    Mary       PROPN      Mary1    sold       VERB       sell2    the        DET        the3    car        NOUN       car4    to         ADP        to5    John       PROPN      John6    .          PUNCT      ."""For more simple scenarios, you can use theWhiteSpaceTokenizerwrapper, which will just split the text
by whitespace:fromnlp_preprocessing_wrappersimportWhitespaceTokenizerwhitespace_tokenizer=WhitespaceTokenizer()tokenized=whitespace_tokenizer("Mary sold the car to John .")forwordintokenized:print("{:<5}{:<10}".format(word.index,word.text))"""0    Mary1    sold2    the3    car4    to5    John6    ."""FeaturesComplete preprocessing pipelineSpacyTokenizerandStanzaTokenizerprovide a unified API for both libraries, exposing most of their
features, like tokenization, Part-of-Speech tagging, lemmatization and dependency parsing. You can activate
and deactivate any of these usingreturn_pos_tags,return_lemmasandreturn_deps. So, for example,StanzaTokenizer(language="en",return_pos_tags=True,return_lemmas=True)will return a list ofTokenobjects, with theposandlemmafields filled.whileStanzaTokenizer(language="en")will return a list ofTokenobjects, with only thetextfield filled.GPU supportWithuse_gpu=True, the library will use the GPU if it is available. To set up the environment for the GPU,
refer to theStanza documentationand thespaCy documentation.APITokenizersSpacyTokenizerclassSpacyTokenizer(BaseTokenizer):def__init__(self,language:str="en",return_pos_tags:bool=False,return_lemmas:bool=False,return_deps:bool=False,split_on_spaces:bool=False,use_gpu:bool=False,):StanzaTokenizerclassStanzaTokenizer(BaseTokenizer):def__init__(self,language:str="en",return_pos_tags:bool=False,return_lemmas:bool=False,return_deps:bool=False,split_on_spaces:bool=False,use_gpu:bool=False,):WhitespaceTokenizerclassWhitespaceTokenizer(BaseTokenizer):def__init__(self):Sentence SplitterSpacySentenceSplitterclassSpacySentenceSplitter(BaseSentenceSplitter):def__init__(self,language:str="en",model_type:str="statistical"):