Package: lextok
lextokRule-based tokenizer and pattern matching for basic Philippine entities using spacy.[!IMPORTANT]
Should be used in tandem withdoclexQuickstartpoetryenvuse3.11.6# 3.12 not yet supportedpoetryinstall
poetryshell
python-mspacydownloaden_core_web_sm# base modelRationaleBeforeimportspacynlp=spacy.load("en_core_web_sm")# no modifications to the modeldoc1=nlp("Sec. 36(b)(21)")fortokenindoc1:print(f"{token.text=}{token.pos_=}{token.ent_type_=},{token.i=}")"""token.text='Sec' token.pos_='PROPN' token.ent_type_='ORG' token.i=0token.text='.' token.pos_='PUNCT' token.ent_type_='' token.i=1token.text='36(b)(21' token.pos_='NUM' token.ent_type_='CARDINAL' token.i=2token.text=')' token.pos_='PUNCT' token.ent_type_='' token.i=3"""Afterfromlextokimportlextoklex=lextok()# inclusion of custom tokenizer, attribute and entity rulerdoc2=lex("Sec. 36(b)(21)")fortokenindoc2:print(f"{token.text=}{token.pos_=}{token.ent_type_=}{token.i=}")"""token.text='Sec.' token.pos_='NOUN' token.ent_type_='ProvisionNum' token.i=0token.text='36(b)(21)' token.pos_='NUM' token.ent_type_='ProvisionNum' token.i=1"""Token entities can be merged:fromlextokimportlextoklex=lextok(finalize_entities=True)doc2=lex("Sec. 36(b)(21)")fortokenindoc2:print(f"{token.text=}{token.pos_=}{token.ent_type_=}{token.i=}")"""token.text='Sec. 36(b)(21)' token.pos_='NUM' token.ent_type_='ProvisionNum' token.i=0"""Pattern creationA pattern consists of a list of tokens, e.g. space space between the word, a dot, and the number?[{"ORTH":{"IN":["Tit","Bk","Ch","Sub-Chap","Art","Sec","Par","Sub-Par"]}},{"ORTH":"."},# with dot{"POS":"NUM"},]This is another pattern where the dot is connected to the word:[{"ORTH":{"IN":["Tit.","Bk.","Ch.","Sub-Chap.","Art.","Sec.","Par.","Sub-Par.",]}},{"POS":"NUM"},]# no separate dotThere are many variations. It becomes possible to generate a list of patterns algorithmically and save them to a*.jsonlfile, e.g.:fromlextok.entity_rules_citeableimportstatutory_provisionsprint(statutory_provisions.patterns)# view patternsstatutory_provisions.create_file()# located in /lextok/rules/ if path not specifiedRules and LabelsEachRulemay consist of manypatterns, and this collection of patterns can be associated with aLabel.In spacy parlance, the label represents theENT_TYPEbut for this library's purpose, it's also adopted for non-entities to cater toSpanRulerpatterns.To distinguish labels strictly for entities from labels for non-entities, a collection of labels is defined inSPAN_RULER_LABELS. If not included in this list of labels, then the implication is that the Rule's patterns ought to be governed by theEntityRuler; otherwise, theSpanRuler.Considering the number ofRulesdeclared (or to be declared), instead of importing each instance individually, these can be extracted dynamically withRule.extract_from_files().Existing data structuresfromlextokimportLabel,ENTITY_RULES,SPAN_RULESforlabelinLabel:print(label.name)# pattern labelsforeinENTITY_RULES:print(e)forsinSPAN_RULES:print(s)Add more entity rulesCreate a list ofRuleobjects, e.g.:fromlextokimportlextok,Rule,ENTITY_RULES,Labeladded_rules=[Rule(id="ministry-labor",label=Label.GovtDivision,patterns=[[{"LOWER":"the","OP":"?"},{"LOWER":"ministry"},{"LOWER":"of"},{"LOWER":"labor"},]],),Rule(id="intermediate-scrutiny",label=Label.Doctrine,patterns=[[{"LOWER":"test","OP":"?"},{"LOWER":"of","OP":"?"},{"LOWER":"intermediate"},{"LOWER":"scrutiny"},{"LEMMA":{"IN":["test","approach"]},"OP":"?"},]],),]# Include new rules in lextok languagenlp=lextok(finalize_entities=True,entity_rules=ENTITY_RULES+added_rules)# Test detectiondoc=nlp("Lorem ipsum, sample text. The Ministry of Labor is a govt division. Hello world. The test of intermediate scrutiny is a constitutional law concept.")doc.ents# (The Ministry of Labor, test of intermediate scrutiny)