TITLE: Parsing of NLM metadata in OpenRefine from OJS articles
AUTHORS: Alessandra Moi, carlo.bianchini , Andrea   Marchitelli

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">Protocols for parsing metadata of articles harvested from Open Journal Systems (OJS) in order to create entities in WikiData</div></div>

[STEPS]
?. [Attività preliminari]
Import the OAI identifiers and generate the baseurl list:
?. [Attività preliminari]
Test the operation of one of the generated URLs by clicking on the link and verifying that it opens a page with NLM metadata in XML format.Es. https://aibstudi.aib.it/oai?verb=GetRecord&metadataPrefix=nlm&identifier=oai:ojs.riviste.aib.it:article/11366
OpenRefine will create a column containing XML text with the NLM metadata of the article
?. [Harvesting dei metadati]
Click on the metadataURL column , select Edit Column > Add Column by retrieving from URLs, calling the new column metadata_oai.Once you have the metadata value for all items, you can move on to parsing
?. [Parsing dei dati]
Aggiungere una colonna per ciascuno dei dati che si vogliono ottenere attraverso il parsing dei dati XML ottenuti, come negli esempi sotto (il codice tra parentesi si riferisce alla proprietà WikiData)
?. [Attività preliminari]
Import the identifiers following the instructions in the protocol
{"blocks":[{"key":"52lge","text":"Protocollo per importare in OpenRefine una lista di identificatori OAI-PMH per successive elaborazioni.","type":"unstyled","depth":0,"inlineStyleRanges":[],"entityRanges":[],"data":[]}],"entityMap":[]}
?. [Attività preliminari]
From the list of imported identifiers, generate the list of baseurl for harvesting the metadata of each item according to the indications given in the protocol "Creare  baseurl per harvesting di repository OAI con OpenRefine"
N.B. For NLM metadata collection, the metadataPrefix in the url must be =nlm
?. [Parsing dei dati]
Published in (P1433)value.parseXml().select('article')[0].select('journal-meta')[0].select('journal-title')[0].ownText()
?. [Parsing dei dati]
DOI (P356)value.parseXml().select('article')[0].select('article-meta')[0].select('article-id[pub-id-type="doi"]')[0].ownText()
?. [Parsing dei dati]
Title (P1476)value.parseXml().select('article')[0].select('article-meta')[0].select('article-title')[0].ownText()
?. [Parsing dei dati]
Athors name and related infoAutore#1 (P50)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[0].select('given-names')[0].ownText() +'  ' + value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[0].select('surname')[0].ownText()First nameAutore#1 (P735) value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[0].select('given-names')[0].ownText()Surname Autore#1  (P734)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[0].select('surname')[0].ownText()Affiliation Autore#1  (P1416)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[0].select('aff')[0].ownText()Autore#2 (P50)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[1].select('given-names')[0].ownText() +' ' + value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[1].select('surname')[0].ownText()First nameAutore#2  (P735)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[1].select('given-names')[0].ownText()Surname Autore# 2 (P734)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[1].select('surname')[0].ownText()Affiliation Autore#2 (P1416)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[1].select('aff')[0].ownText()Autore#3 (P50)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[2].select('given-names')[0].ownText() +' ' + value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[2].select('surname')[0].ownText()First nameAutore#3 (P735) value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[2].select('given-names')[0].ownText()Surname Autore# 3 (P734)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[2].select('surname')[0].ownText()Affiliation Autore# 3 (P1416)value.parseXml().select('article')[0].select('article-meta')[0].select('contrib[contrib-type="author"]')[2].select('aff')[0].ownText()
?. [Parsing dei dati]
Publication date (P577)value.parseXml().select('article')[0].select('article-meta')[0].select('pub-date[pub-type="epub"]')[0].select('day')[0].ownText() + '/' + value.parseXml().select('article')[0].select('article-meta')[0].select('pub-date[pub-type="epub"]')[0].select('month')[0].ownText() + '/' + value.parseXml().select('article')[0].select('article-meta')[0].select('pub-date[pub-type="epub"]')[0].select('year')[0].ownText()
?. [Parsing dei dati]
Volume & Issuevolume (P478)value.parseXml().select('article')[0].select('article-meta')[0].select('volume')[0].ownText()Fascicolo (P433)value.parseXml().select('article')[0].select('article-meta')[0].select('issue')[0].ownText()
?. [Parsing dei dati]
LinkURL (P2699)value.parseXml().select('article')[0].select('article-meta')[0].select('self-uri')[0].xmlAttr("xlink:href")PDF Full text (P953)forEach(value.parseXml().select('article')[0].select('article-meta')[0].select('self-uri[content-type="application/pdf"]'),v,v.xmlAttr("xlink:href")).join("|")HTML Full text (P953)forEach(value.parseXml().select('article')[0].select('article-meta')[0].select('self-uri[content-type="text/html"]'),v,v.xmlAttr("xlink:href")).join("|")
?. [Parsing dei dati]
Keyword, per language. Once you have the values in a single cell, divide using the command Edit Column > Split and choosing the separator |KW itaforEach(value.parseXml().select('article')[0].select('article-meta')[0].select('kwd-group[xml:lang="IT"]')[0].select('kwd'),v,v.ownText()).join("|")KW engargomento principale (P921)forEach(value.parseXml().select('article')[0].select('article-meta')[0].select('kwd-group[xml:lang="EN"]')[0].select('kwd'),v,v.ownText()).join("|")
?. [Parsing dei dati]
Metadata Parsing
?. [Parsing dei dati]
SectionsExtract name of Sections, e.g. to separate reviews from other articlesvalue.parseXml().select('article')[0].select('article-meta')[0].select('subject')[0].ownText()
?. [Parsing dei dati]
Pages from/to and Number of pages.Si devono rifare i passaggi 2 e 3, ma utilizzando come formato oai_dc. e creare una nuova colonna "Metadati_oai_dc".Per esempio, https://aibstudi.aib.it/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai:ojs.riviste.aib.it:article/11366Pages from/to  (P304)From the metadata_oai_dc column create a new DC_Source column (where source is the metadata that contains the value of the start and end page of the article. Command grel:value.parseXml().select('record')[0].select('metadata')[0].select('oai_dc|dc')[0].select('dc|source')[0].ownText()To extract only the page value from-to, create two columns with the split command and the appropriate separator (e.g.: ';')Remove columns that do not have page numbers and rename the column with the correct value "PagDaA (P304)" (e.g.: 3-21)Numero di pagine totale (P1104)From the column "PagDaA (P304)" derive two more columns (PagIniz and PagFin) with the value of the starting page and the value of the ending page.Create a new colonn "NumTotPag (P1104)" based on the PagFin column with the grel command:toNumber(value-cells['PagIniz'].value) + 1