Package: deepnlp-cerelab
DeepNLPThis is a new natural language processing library based on modern deep learning methods. The library focus on basic NLP tasks such as: POS (part of speech), NER (named entity recognition) and DP (dependency parsing). The main language is English but we are working hard to support Vietnamese and others in the near future.Installation ðŸ”¥This repository is tested on python 3.7+ and Tensorflow 2.8+Deepnlp can be installed using pip as follows:pip install deepnlp-cerelabDeepnlp can also be installed from source with the following commands:git clone https://github.com/hieupth/deepnlp.git
cd deepnlp/
pip install -e .Tutorials ðŸ¥®1. Sentence Segmentation2. Word Tokenizer3. Install and load pretrained model and vocabs4. POS Tagging5. Named Entity Recognition6. Dependency Parsing7. Multil Task8. Clear Cache9. List of pretrained models1. Sentence SegmentationUsage>>>importdeepnlp>>>text="""\Mr. Smith bought cheapsite.com for 1.5 million dollars, i.e. he paid a lot for it. Did he mind? Adam Jones Jr. thinks he didn't. In any case, this isn't true... Well, with a probability of .9 it isn't.""">>>deepnlp.sentence_tokenize(text)['Mr. Smith bought cheapsite.com for 1.5 million dollars, i.e. he paid a lot for it.','Did he mind?',"Adam Jones Jr. thinks he didn't.","In any case, this isn't true...","Well, with a probability of .9 it isn't.",'']2. Word TokenizeUsage>>>importdeepnlp>>>text="I have an apple.">>>deepnlp.word_tokenize(text)['I','have','an','apple','.']3. Install and load pretrained model and vocabsInstall pretrained model and vocabs>>>importdeepnlp>>>deepnlp.download('deepnlp_eng')Or you can also install pretrained model and vocabs independently of each other>>>importdeepnlp>>>deepnlp.download_model('deepnlp_eng')>>>deepnlp.download_vocabs('deepnlp_eng')Load models and vocabs>>>importdeepnlp>>>model=deepnlp.load_model('deepnlp_eng')>>>vocabs=deepnlp.load_vocabs('deepnlp_eng',task='multi')# pos, ner, dp4. POS TaggingWithPosTaggerclass>>>importdeepnlp>>>model=deepnlp.PosTagger('deepnlp_eng')>>>modelmodel_name:deepnlp_eng,vocab_name:deepnlp_eng,tokenizer_name:distilroberta-base>>>output=model.inference('I have an apple.',device='cpu')# default device = 'cpu'>>>output<deepnlp.utils.data_struct.TokenClassificationDataat0x7fbc3ddbab90>>>>output.value(){'Sequence':'I have an apple.','Inference':{'I':{'score':0.9175689,'label':'PRP'},'have':{'score':0.9232193,'label':'VBP'},'an':{'score':0.9158458,'label':'DT'},'apple':{'score':0.86957675,'label':'NN'},'.':{'score':0.8892631,'label':'.'}}}>>>deepnlp.print_out([output])Ihaveanapple.1IPRP2haveVBP3anDT4appleNN5..Withpipelineclass>>>importdeepnlp>>>model=deepnlp.load_model('deepnlp_eng')>>>pipeline=deepnlp.pipeline(model,task='pos_tagger')>>>output=pipeline("I have an apple.",device='cpu')# default device = 'cpu'>>>deepnlp.print_out([output])Ihaveanapple.1IPRP2haveVBP3anDT4appleNN5..5. Named Entity RecognitionWithNerTaggerclass>>>importdeepnlp>>>model=deepnlp.NerTagger('deepnlp_eng')>>>output=model.inference('Please confirm your song choice: Same Old War, playing on the kitchen speaker',device='cpu')# default device = 'cpu'output<deepnlp.utils.data_struct.TokenClassificationDataat0x7f69d9504750>>>>output.value(){'Sequence':'Please confirm your song choice: Same Old War, playing on the kitchen speaker','Inference':{'Same':{'score':0.922773,'label':'B-MISC'},'Old':{'score':0.9353856,'label':'I-MISC'},'War':{'score':0.92017937,'label':'I-MISC'}}}>>>deepnlp.print_out([output],del_prefix_ner=False)# if you set del_prefix_ner= True, B-MISC or I-MISC will become MISCPleaseconfirmyoursongchoice:SameOldWar,playingonthekitchenspeaker1PleaseO2confirmO3yourO4songO5choiceO6SameB-MISC7OldI-MISC8WarI-MISC9,O10playingO11onO12theO13kitchenO14speakerOWithpipelineclass>>>importdeepnlp>>>model=deepnlp.load_model('deepnlp_eng')>>>pipeline=deepnlp.pipeline(model,task='ner_tagger')>>>output=pipeline("Please confirm your song choice: Same Old War, playing on the kitchen speaker")>>>deepnlp.print_out([output],del_prefix_ner=True,device='cpu')# default device = 'cpu'Pleaseconfirmyoursongchoice:SameOldWar,playingonthekitchenspeaker1PleaseO2confirmO3yourO4songO5choiceO6SameMISC7OldMISC8WarMISC9,O10playingO11onO12theO13kitchenO14speakerO6. Dependency ParsingWithDPParserclass>>>importdeepnlp>>>model=deepnlp.DPParser('deepnlp_eng')>>>output=model.inference("I have an apple.",device='cpu')# default device = 'cpu'>>>output<deepnlp.utils.data_struct.ParserDataat0x7f69da3125d0>>>>output.value(){'Sequence':'I have an apple.','Inference':{'xpos':['PRP','VBP','DT','NN','.'],'head':[2,0,4,2,2],'rela':['nsubj','root','det','obj','punct']}}>>>deepnlp.print_out([output])Ihaveanapple.1IPRP2nsubj2haveVBP0root3anDT4det4appleNN2obj5..2punctWithpipelineclass>>>importdeepnlp>>>model=deepnlp.load_model('deepnlp_eng')>>>pipeline=deepnlp.pipeline(model,task='dp_parser')>>>output=pipeline("I have an apple.",device='cpu')# default device = 'cpu'>>>deepnlp.print_out([output])Ihaveanapple.1IPRP2nsubj2haveVBP0root3anDT4det4appleNN2obj5..2punct7. Multi TaskWithMultiTask>>>importdeepnlp>>>model=deepnlp.MultiTask('deepnlp_eng')>>>output=model.inference("Please confirm your song choice: Same Old War, playing on the kitchen speaker",device='cpu')# default device = 'cpu'>>>output<deepnlp.utils.data_struct.MultiDataat0x7f69da8f7650>>>>deepnlp.print_out([output])Pleaseconfirmyoursongchoice:SameOldWar,playingonthekitchenspeaker1PleaseUHO2discourse2confirmVBO0root3yourPRP$O5nmod:poss4songNNO5compound5choiceNNO2obj6SameJJMISC8amod7OldNNPMISC8compound8WarNNPMISC2obj9,,O2punct10playingVBGO2advcl11onINO14case12theDTO14det13kitchenNNO14compound14speakerNNO10oblWithpipeline>>>importdeepnlp>>>model=deepnlp.load_model('deepnlp_eng')>>>pipeline=deepnlp.pipeline(model,task='multi')>>>output=pipeline("Please confirm your song choice: Same Old War, playing on the kitchen speaker",device='cpu')# default device = 'cpu'>>>deepnlp.print_out([output])Pleaseconfirmyoursongchoice:SameOldWar,playingonthekitchenspeaker1PleaseUHO2discourse2confirmVBO0root3yourPRP$O5nmod:poss4songNNO5compound5choiceNNO2obj6SameJJMISC8amod7OldNNPMISC8compound8WarNNPMISC2obj9,,O2punct10playingVBGO2advcl11onINO14case12theDTO14det13kitchenNNO14compound14speakerNNO10obl8. Clear CacheRemove pretrained model and vocabsdeepnlp_eng>>>deepnlp.clear_cache('deepnlp_eng')Or>>>deepnlp.clear_model('deepnlp_eng')>>>deepnlp.clear_vocabs('deepnlp_eng')9. List of pretrained modelsdeppnlp_eng- support for English:download pretrained model-download vocabsdeepnlp_vie- support for Vietnamese: Will be updated in the futureLicenseApache 2.0 License.Copyright © 2022Hieu Pham. All rights reserved.