TITLE: Investigating Invalid DOIs in COCI
AUTHORS: Nooshin Shahidzadeh, Alessia Cioffi, Arianna Moretti, Sara Coppini

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block"><span style = "font-weight:bold;">A preliminary note</span></div><div class = "text-block"><span>This protocol illustrates the workflow adopted within a scholarly research that operates within the OpenCitations environment, which is an independent infrastructure organization for open scholarship dedicated to the publication of open bibliographic and citation data by the use of </span><a href="https://en.wikipedia.org/wiki/Semantic_Web" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">Semantic Web</span></a><span> (</span><a href="https://en.wikipedia.org/wiki/Linked_data" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">Linked Data</span></a><span>) technologies. COCI is the OpenCitations Index of Crossref open DOI-to-DOI citations.</span></div><div class = "text-block"><span style = "font-weight:bold;">Purpose</span></div><div class = "text-block">The purpose of this research is to find the publishers responsible for the missing citations in COCI by sending incorrect metadata to Crossref, the publishers to whom such invalid citations point to and the number of previously invalid citations which are currently valid. The ultimate aim would be of contributing to the resolution of this type of problem in order to insert the citations now valid in COCI, and correct those still invalid always in order to increase the number of open citations available and indexed in the OpenCitations project.</div><div class = "text-block"><span style = "font-weight:bold;">Study design/methodology</span></div><div class = "text-block"><span>In the beginning, we use an already generated CSV file, containing the valid citing DOIs and the invalid cited DOIs, which is available from Peroni, S. (2021). Citations to invalid DOI-identified entities obtained from processing DOI-to-DOI citations to add in COCI (1.0). Zenodo. </span><a href="https://doi.org/10.5281/ZENODO.4625300" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">https://doi.org/10.5281/ZENODO.4625300</span></a><span>. These citations to invalid DOIs have been retrieved while processing Crossref data for adding open citations in COCI, but they have not been added in COCI since they point to a non-resolvable cited document.</span></div><div class = "text-block">Two REST API services can be of help: the DOI REST API to check if the invalid cited DOI is now valid; and the Crossref REST API to retrieve the publisher from the prefix of the DOI, both for the cited publications and the citing ones.</div><div class = "text-block"><span style = "font-weight:bold;">Findings</span></div><div class = "text-block">In addition to collecting the names of the publishers involved in these missing citations, either as the publisher of the citing article or as the publisher of the cited article, which was sufficient to answer our research questions, we have decided to collect additional information that can help us to get a better picture of the situation. As regards the JSON file, we found for each individual publisher 1) the number of incorrect given citations metadata sent, and 2) the number of invalid citations received. </div><div class = "text-block">On the other hand, as required by the initial research questions, we also extracted the total number of invalid citations that have since been corrected.</div><div class = "text-block"><span style = "font-weight:bold;">Originality/value</span></div><div class = "text-block">The results of this research may point us to publishers who generally send out incorrect citation metadata and, inversely, those who generally receive invalid citations. These findings can first of all raise awareness of the accuracy of certain publishing houses in managing their metadata (or lack thereof). Moreover, finding these trends and showcasing the labor of the corrections may lead to increasingly valid citations if the proper measures are taken.</div><div class = "text-block"><span style = "font-weight:bold;">Research limitations/implications</span></div><div class = "text-block">Based on the available data for the COCI, there may be a slight bias in our sample, causing some publishers to be incorrectly represented.</div><div class = "text-block"><span style = "font-weight:bold;">Minimal bibliography on related projects</span></div><div class = "text-block"><span>Heibi, I., Peroni, S. & Shotton, D. Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations.</span><span style = "font-style:italic;">Scientometrics </span><span>121, 1213–1228 (2019). https://doi.org/10.1007/s11192-019-03217-6.</span></div><div class = "text-block"><span>Silvio Peroni, David Shotton (2020). OpenCitations, an infrastructure organization for open scholarship. </span><a href="https://www.mitpressjournals.org/toc/qss/1/1" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">Quantitative Science Studies, 1(1)</span></a><span>: 428-444. </span><a href="https://doi.org/10.1162/qss_a_00023" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">https://doi.org/10.1162/qss_a_00023</span></a><span style = ":;">.</span></div><div class = "text-block"><span>Peroni, S. (2021). Citations to invalid DOI-identified entities obtained from processing DOI-to-DOI citations to add in COCI (1.0). Zenodo. </span><a href="https://doi.org/10.5281/ZENODO.4625300" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">https://doi.org/10.5281/ZENODO.4625300</span></a><span style = ":;">.</span></div></div>

[STEPS]
?. [Reading the CSV Data]
First we read the CSV of the invalid DOIs, containing all the invalid cited DOIs in one column and their valid citing counterparts in another.
?. [Creating the Output JSON File]
We create a JSON file for the output data that strictly answer the research questions and data for the visualizations of our findings. We opted for JSON file format to store all the data answering the research questions, because this file format allows us to store more heterogeneous information in a more complex and structured way. Indeed, we stored information not directly requested by the research questions, such as the number of missing citations for each single publisher, to provide a more precise answer. For what concerns the visualizations, the JSON file format allows us to separate different kinds of data and save also data that is relevant for the graphic representations of our findings but not strictly related to the research questions (e.g. which publishers were involved in citations that were originally invalid and then validated with the DOI API request).
?. [Creating the Output JSON File]
We add a key called "responsible_publishers" that will include a dictionary containing all publisher data for citing articles.
?. [Creating the Output JSON File]
We add a key called "receiving_publishers" that will include a dictionary containing all publisher data for cited articles.
?. [Creating the Output JSON File]
We add a key called "total_number‌_of‌_corrected‌_dois" that will hold only a number.
?. [Processing Each Line in the CSV File and Extracting the Needed Information]
We use the DOI REST API (https://doi.org/api/handles/[doi]) to search for each of the invalid cited DOIs in the CSV. If we receive the response code "1", we know that the citation data is now valid, we now have two cases.
?. [Extracting Publisher Data for Citing and Cited Articles]
We search through the REST API for Crossref for the id of the publisher, which is the prefix of the DOI (i.e., all the characters preceding the backslash) using: https://api.crossref.org/prefixes/The request returns a JSON file with information about the member code of the publisher, the name of the publisher, and the prefix.
?. [Extracting Publisher Data for Citing and Cited Articles]
We check if the publisher ID exists in the "responsible_publishers" dictionary as a key. If it does not, we create a new item in the dictionary: the key is the ID of the publisher, and the value is a dictionary containing the name of the publisher and the number of invalid citation data it has sent to Crossref (which at the time of creation would be 1). If, on the other hand, it does exist, we just increment the number inside the inner dictionary of that publisher by 1.
?. [Extracting Publisher Data for Citing and Cited Articles]
We assume the first 6 characters of the invalid cited DOI to be the ID of the publisher to which the invalid citation is pointing.
?. [Extracting Publisher Data for Citing and Cited Articles]
We check the validity of this ID through the Crossref REST API. If this ID exists and belongs to a publisher, we add it to the "receiving_publisher" dictionary as a key, the value of which would be a dictionary containing the number of the citations received and the name of the publisher. (Similar to step 4.1 above)
?. [Extracting Publisher Data for Citing and Cited Articles]
We search through the REST API of Crossref for the reference list of the citing article and see if there is any additional data (for example in the "unstructured" field) given in the XML that could lead us to the cited publisher.
?. [Extracting Publisher Data for Citing and Cited Articles]
If additional data was found, then we use the "query" attribute of the Crossref REST API to search for the newly found string. If any publication was found, we add the "publisher" field of that publication to the "receiving_publishers" dictionary, in the form explained above.
?. [Output]
We return the completed JSON file after processing all the lines. (CSV addition possible later.)
?. [Action Plan for Corrected DOI]
We add the citing and cited DOIs to a list as a tuple to later be put back into the COCI data.
?. [Action Plan for Corrected DOI]
Increment the "total_number‌_of‌_corrected‌_dois" by 1.
?. [Input material]
Given the academic background of the project, we got our input material from Peroni, S. (2021). Citations to invalid DOI-identified entities obtained from processing DOI-to-DOI citations to add in COCI (1.0). Zenodo. https://doi.org/10.5281/ZENODO.4625300. As explained in the description of the resource, this dataset contains a two-column CSV file, where the first column ("Valid_citing_DOI") contains the DOI of a citing entity retrieved in Crossref, while the second column ("Invalid_cited_DOI") contains the invalid DOI of a cited entity identified by looking at the field "reference" in the JSON document returned by querying the Crossref API with the citing DOI.These citations to invalid DOIs have been retrieved while processing Crossref data for adding open citations in COCI, but they have not been added in COCI since they point to a non-resolvable cited document.