Package: pubmedparser2
Read XML files and pull out selected values. Values to collect are
determined by paths found in astructure file. The
structure file also includes a key which associates the values with a
parent element and names, which determine which file to place the
elements in.Files can be passed as either gzipped or uncompressed XML files or from
standard in.For more info on Pubmed's XML files see:pubmed190101.dtd.Usage:importpubmedparserimportpubmedparser.ftp# Download datafiles=pubmedparser.ftp.download(range(1,6))# Read XML files using a YAML file to describe what data to collect.data_dir="file_example"structure_file="example/structure.yml"results=pubmedparser.read_xml(files,structure_file,data_dir)Seethe example
filefor more options.In python, the structure file can be replaced with a dictionary of
dictionaries as well.Or, as a CLI:xml_read--cache-dir=cache--structure-file=structure.yml\data/*.xml.gzInstalling with pippipinstallpubmedparser2Building python packageRequireszlib.Clone the repository and in the directory. Then usepoetryto install the dependencies.poetryinstallThen run the make command:makepythonStructure fileThe structure file is a YAML file containing key-value pairs for
different tags and paths. There are two required keys:rootandkey.Rootprovide the top-level tag, in the case of the pubmed files this
will bePubmedArticleSet.root:"/PubmedArticleSet"The/is not strictly required as the program will ignore them, but
they are used to conform to thexpathsyntax (although this
program does not handle all cases forxpath).Only tags below the root tag will be considered and the parsing will
terminate once the program has left the root of the tree.Keyis a reference tag. In the pubmed case, all data is with respect
to a publication, so the key should identify the publication the values
are linked to. ThePMIDtag is a suitable candidate.key:"/PubmedArticle/MedlineCitation/PMID"Afterroot, all paths are taken as relative to the root node.The other name-pairs in the file determine what other items to collect.
These can either be a simple name and path, like the key, such as:Language:"/PubmedArticle/MedlineCitation/Article/Language"Keywords:"/PubmedArticle/MedlineCitation/KeywordList/Keyword"Or they can use a hierarchical representation to get multiple values
below a child. This is mainly used to handle lists of items where there
is an indefinite number of items below the list.Author:{root:"/PubmedArticle/MedlineCitation/Article/AuthorList",key:"/Author/auto_index",LastName:"/Author/LastName",ForeName:"/Author/ForeName",Affiliation:"/Author/AffiliationInfo/Affiliation",Orcid:"/Author/Identifier/[@Source='ORCID']"}Here, all paths are relative to the sub-structuresrootpath, which is
in turn relative to the parent structure'sroot. This sub-structure
uses the same rules as the parent structure, so it needs both arootandkeyname-value pair. The results of searching each path are
written to separate files. Each file gets a column for the parent and
child key. So in this case, each element of the author is linked by an
author key and that is related to the publication they authored through
the parent key.The main parser is called recursively to parse this structure so it's
worth thinking about what the root should be under the context that the
parser will be called with that root. This means if, instead of stopping
at/AuthorList,/Authorwas added to the end of the root, the parser
would be called for each individual author, instead of once per author
list, leading to all author's getting the index 0.There are a number of additional syntax constructs to note in the above
example. The key uses the special nameauto_index, since there is no
author ID in the XML data, an index is used to count the authors in the
order they appear. This resets for each publication and starts at 0.
Treating theauto_indexas the tail of a path allows you to control
when the indexing occurs—the index is incremented whenever it hits a/Authortag.In addition to theauto_indexkey, there is a second special index
name,condensed.Reference:{root:"/PubmedArticle/PubmedData/ReferenceList/Reference/ArticleIdList"key:"/condensed"PMID:"/ArticleId/[@IdType='pubmed']"DOI:"/ArticleId/[@IdType='doi']"}In the case ofcondensed, instead of writing the results to separate
files, they will printed as columns in the same file, and therefore do
not need an additional key for the sub-structure. If any of the elements
are missing, they will be left blank, for example, if the parser does
not find a pubmed ID for a given reference, the row will look like"%s\t\t%s"where the first string will contain the parent key (thePMIDof the publication citing this reference) and the second string
will contain the reference'sDOI.The/[@attribute='value']syntax at the end of a path tells the parser
to only collect an element if it has an attribute and the attribute's
value matches the supplied value. Similarly the/@attributesyntax,
tells the parser to collect the value of the attributeattributealong
with the element's value. Then both values will be written to the output
file. Currently only a single attribute can be specified.Lastly, there is a special syntax for writing condensed sub-structures:Date:"/PubmedArticle/MedlineCitation/Article/Journal/JournalIssue/PubDate/{Year,Month,Day}"The{child,child,child}syntax allows you to select multiple children
at the same level to be printed to a single file. This is useful when
multiple children make up a single piece of information (i.e. the
publication date).A similar example structure file can be found in the example directory
of this project at:file:./example/structure.yml.Structure dictionaryThe structure of the xml data to read can also be described as a python
dictionary of dictionaries.The form is similar to the file:structure={"root":"//PubmedArticleSet","key":"/PubmedArticle/MedlineCitation/PMID","DOI":"/PubmedArticle/PubmedData/ArticleIdList/ArticleId/[@IdType='doi']","Date":"/PubmedArticle/MedlineCitation/Article/Journal/JournalIssue/PubDate/{Year,Month,Day}","Journal":"/PubmedArticle/MedlineCitation/Article/Journal/{Title,ISOAbbreviation}","Language":"/PubmedArticle/MedlineCitation/Article/Language","Author":{"root":"/PubmedArticle/MedlineCitation/Article/AuthorList","key":"/Author/auto_index","LastName":"/Author/LastName","ForName":"/Author/ForeName","Affiliation":"/Author/AffiliationInfo/Affiliation","Orcid":"/Author/Identifier/[@Source='ORCID']",},"Grant":{"root":"/PubmedArticle/MedlineCitation/Article/GrantList","key":"/Grant/auto_index","ID":"/Grant/GrantID","Agency":"/Grant/Agency",},"Chemical":"/PubmedArticle/MedlineCitation/ChemicalList/Chemical/NameOfSubstance/@UI","Qualifier":"/PubmedArticle/MedlineCitation/MeshHeadingList/MeshHeading/QualifierName/@UI","Descriptor":"/PubmedArticle/MedlineCitation/MeshHeadingList/MeshHeading/DescriptorName/@UI","Keywords":"/PubmedArticle/MedlineCitation/KeywordList/Keyword","Reference":{"root":("/PubmedArticle/PubmedData/ReferenceList/Reference/ArticleIdList"),"key":"/condensed","PMID":"/ArticleId/[@IdType='pubmed']","DOI":"/ArticleId/[@IdType='doi']",},}This can then be passed topubmedparser.read_xmlin place of the
structure file.Future goalsImprove printing logicCurrently, values are printed as they are read in. Since the results for
the different paths are written to separate files, this shouldn't
matter, except for the case of the key. The key is not printed to its
own results file, instead whatever the last seen key was is printed as
the key for the current value being printed. If the key is not the first
element to be read in the subtree, there will be a mismatch between
value and publication ID.In the case ofPMIDthis is consistently the first element, so there
should not be a problem, however, it could be in other scenarios.Error handlingAfter refactoring the code, I have started adding some error handling
code, however this has not been consistently applied. Ideally, the
default behavior will be for functions to return error codes. Then use
an error checking macro to test that the result was not an error. I
would also like to add a set error strings that would be printed
depending on the error code. Possibly use a structure to represent
errors so that the erroring function could supply an additional string
along with the error.Better error handling like this could also allow the python package to
write it's own error handling function in the C API to override the
default error mechanism to use python level errors. This would be done
by testing if an error handler function was defined, if so the error
checking macro would use that function, otherwise it would fallback to a
default function.