Package: pubtator-loader
PubTator Loaderpubtator_loaderis a python module that allows loading corpus from PubTator format and manipulate documents as Python object.
It can also be used in combination withspacyto tokenize the documents and convert them toBILUO Tagsto use for different
NLP tasks.PubTator FormatThe PubTator format uses the following format:<PMID>|t|<TITLE>
<PMID>|a|<ABSTRACT>
<PMID>	<START OFFSET 1>	<LAST OFFSET 1>	<MENTION 1>	<TYPE 1>	<IDENTIFIER 1>
<PMID>	<START OFFSET 2>	<LAST OFFSET 2>	<MENTION 2>	<TYPE 2>	<IDENTIFIER 2>

<PMID>|t|<TITLE>
<PMID>|a|<ABSTRACT>
<PMID>	<START OFFSET 1>	<LAST OFFSET 1>	<MENTION 1>	<TYPE 1>	<IDENTIFIER 1>
<PMID>	<START OFFSET 2>	<LAST OFFSET 2>	<MENTION 2>	<TYPE 2>	<IDENTIFIER 2>where:The first line contains the title of the paper.The second line contains the abstract of the paper.The subsequent lines contain the annotations for the entities in a tab separated format:PMIDStart OffsetEnd OffsetMention (entity text)Type of EntityIdentifier (normalized form)Usagefrompubtator_loaderimportPubTatorCorpusReaderdataset_reader=PubTatorCorpusReader('./sample_pubator_input.txt')corpus=dataset_reader.load_corpus()# corpus will be a List[PubtatorDocuments]fordocincorpus:print(doc)"""Console Output:{"id": 25763772,"title_text": "DCTN4 as a modifier of chronic ....","abstract_text": "Pseudomonas aeruginosa (Pa) infection in cystic fibrosis .....","entities": [{"document_id": 25763772,"start_index": 0,"end_index": 5,"text_segment": "DCTN4","semantic_type_id": "T103","entity_id": "UMLS:C4308010"},...{"document_id": 25763772,"start_index": 67,"end_index": 82,"text_segment": "cystic fibrosis","semantic_type_id": "T038","entity_id": "UMLS:C0010674"}]}"""importspacyimportscispacy# load the scispacy modelnlp=spacy.load('en_core_sci_lg')# Convert PubTator document to BILUO format.doc_in_BILUO=doc.tokenize_and_convert_to_bilou(nlp)foridx,(token,semantic_type_id,entity_id)inenumerate(doc_in_BILUO):print(f'{idx}\t{token}\t{semantic_type_id}\t{entity_id}')"""Console Output:0         <START>          <START>     <START>1           DCTN4      U-T116,T123  U-C43080102              as                O           O3               a                O           O4        modifier                O           O5              of                O           O6         chronic           B-T047  B-C08541357     Pseudomonas           I-T047  I-C08541358      aeruginosa           I-T047  I-C08541359       infection           L-T047  L-C085413510             in                O           O11         cystic           B-T047  B-C001067412       fibrosis           L-T047  L-C001067413    Pseudomonas           B-T047  B-C085413514     aeruginosa           I-T047  I-C085413515              (           I-T047  I-C085413516             Pa           I-T047  I-C085413517              )           I-T047  I-C085413518      infection           L-T047  L-C085413519             in                O           O20         cystic           B-T047  B-C001067421       fibrosis           L-T047  L-C0010674.               .                .           ..               .                .           ..               .                .           ..               .                .           ."""