Package: plane
PlanePlaneis a tool for shaping wood using muscle power to force the cutting blade over the wood surface.fromWikipediaThis package is used for extracting or replacing specific parts from text, like URL, Email, HTML tags, telephone numbers and so on. Also supports punctuation normalization and removement.See the fullDocuments.InstallPython3.xonly.pippipinstallplaneInstall from sourcepythonsetup.pyinstallFeaturesno other dependenciesbuild-in regex patterns:plane.pattern.Regexcustom regex patternspattern combinationextract, replace patternssegment sentencechain function calls:plane.plane.Planepipeline:plane.PipelineUsageQuick startUse regex toextractorreplace:fromplaneimportEMAIL,extract,replacetext='fake@no.com & fakefake@nothing.com'emails=extract(text,EMAIL)# this return a generator objectforeinemails:print(e)>>>Token(name='Email',value='fake@no.com',start=0,end=11)>>>Token(name='Email',value='fakefake@nothing.com',start=14,end=34)print(EMAIL)>>>Regex(name='Email',pattern='([a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-]+)',repl='<Email>')replace(text,EMAIL)# replace(text, Regex, repl), if repl is not provided, Regex.repl will be used>>>'<Email> & <Email>'replace(text,EMAIL,'')>>>' & 'patternRegexis a namedtuple with 3 items:namepattern: Regular Expressionrepl: replacement tag, this will replace matched regex when usingreplacefunction# create new patternfromplaneimportbuild_new_regexcustom_regex=build_new_regex('my_regex',r'(\d{4})','<my-replacement-tag>')Also, you can build new pattern from default patterns.Attention: this should only be used for language range.fromplaneimportextract,build_new_regex,CHINESE_WORDSASCII=build_new_regex('ascii',r'[a-zA-Z0-9]+',' ')WORDS=ASCII+CHINESE_WORDSprint(WORDS)>>>Regex(name='ascii_Chinese_words',pattern='[a-zA-Z0-9]+|[\\U00004E00-\\U00009FFF\\U00003400-\\U00004DBF\\U00020000-\\U0002A6DF\\U0002A700-\\U0002B73F\\U0002B740-\\U0002B81F\\U0002B820-\\U0002CEAF\\U0002CEB0-\\U0002EBEF]+',repl=' ')text="自然语言处理太难了！who can help me? (╯▔🔺▔)╯"print(' '.join([t.valuefortinlist(extract(text,WORDS))]))>>>"自然语言处理太难了 who can help me"fromplaneimportCHINESE,ENGLISH,NUMBERCN_EN_NUM=sum([CHINESE,ENGLISH,NUMBER])text="佛是虚名，道亦妄立。एवं मया श्रुतम्। 1999 is not the end of the world. "print(' '.join([t.valuefortinextract(text,CN_EN_NUM)]))>>>"佛是虚名，道亦妄立。 1999 is not the end of the world."Default Regex:DetailsURL: only ASCIIEMAIL: local-part@domainTELEPHONE: like xxx-xxxx-xxxxSPACE:,\t,\n,\r,\f,\vHTML: HTML tags, Script part and CSS partASCII_WORD: English word, numbers,<tag>and so on.CHINESE: all Chinese characters (only Han and punctuations)CJK: all Chinese, Japanese, Korean(CJK) characters and punctuationsTHAI: all Thai and punctuationsVIETNAMESE: all Vietnames and punctuationsENGLISH: all English chars and punctuationsNUMBER: 0-9Regex namereplaceURL'<URL>'EMAIL'<Email>'TELEPHONE'<Telephone>'SPACE' 'HTML' 'ASCII_WORD' 'CHINESE' 'CJK' 'segmentsegmentcan be used to segment sentence, English and Numbers like 'PS4' will be keeped and others like Chinese '中文' will be split to single word format['中', '文'].fromplaneimportsegmentsegment('你看起来guaiguai的。<EOS>')>>>['你','看','起','来','guaiguai','的','。','<EOS>']punctuationpunc.removewill replace all unicode punctuations to' 'or something you send to this function as paramterrepl.punc.normalizewill normalize some Unicode punctuations to English punctuations.Attention: '+', '^', '$', '~' and some chars are not punctuation.fromplaneimportpunctext='Hello world!'punc.remove(text)>>>'Hello world '# replace punctuation with special stringpunc.remove(text,'<P>')>>>'Hello world<P>'# normalize punctuationspunc.normalize('你读过那本《边城》吗？什么编程？！人生苦短，我用 Python。')>>>'你读过那本(边城)吗?什么编程?!人生苦短,我用 Python.'Chain functionPlanecontainsextract,replace,segmentandpunc.remove,punc.normalize, and these methods can be called in chain. Sincesegmentreturns list, it can only be called in the end of the chain.Plane.textsaves the result of processed text andPlane.valuessaves the result of extracted strings.fromplaneimportPlanefromplane.patternimportEMAILp=Plane()p.update('My email is my@email.com.').replace(EMAIL,'').text# update() will init Plane.text and Plane.values>>>'My email is .'p.update('My email is my@email.com.').replace(EMAIL).segment()>>>['My','email','is','<Email>','.']p.update('My email is my@email.com.').extract(EMAIL).values>>>[Token(name='Email',value='my@email.com',start=12,end=24)]PipelineYou can usePipelineif you like.segmentandextractcan only present in the end.fromplaneimportPipeline,replace,segmentfromplane.patternimportURLpipe=Pipeline()pipe.add(replace,URL,'')pipe.add(segment)pipe('http://www.guokr.com is online.')>>>['is','online','.']