TITLE: Protocol: Investigating DOIs classes of errors 
AUTHORS: Ricarda Boente, Arcangelo  Massari, Cristian Santini, Deniz Tural

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">The purpose of this protocol is to provide an automated process to repair invalid DOIs that have been collected by the OpenCitations Index Of Crossref Open DOI-To-DOI References (COCI) while processing data provided by Crossref.</div><div class = "text-block"><span>The data needed for this work is </span><a href="http://doi.org/10.5281/zenodo.4625300" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">provided by Silvio Peroni as a CSV</span></a><span> containing pairs of valid citing DOIs and invalid cited DOIs. With the goal to determine an automated process, we first classified the errors that characterize the wrong DOIs in the list. The starting hypothesis is that there are two main classes of errors: factual errors, such as wrong characters, and DOIs that are not yet valid at the time of processing. The first class can be furtherly divided into three classes: errors due to irrelevant strings added to the beginning (prefix-type errors) or at the end (suffix-type errors) of the correct DOI, and errors due to unwanted characters in the middle (other-type errors). Once the classes of errors are addressed, we propose automatic processes to obtain correct DOIs from wrong ones. These processes involve the use of the information returned from </span><a href="https://www.doi.org/factsheets/DOIProxy.html" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">DOI API</span></a><span>, the </span><a href="https://doi.org/10.13003/GU3DQMJVG4" style = "text-decoration:underline;color:blue;cursor:pointer;"><span style = ":;">January 2021 Public Data File from Crossref</span></a><span>, as well as rule-based methods, including regular expressions to correct invalid DOIs.</span></div><div class = "text-block">The application of this methodology produced a CSV dataset containing all the pairs of citing and cited DOIs in the original dataset, each one enriched by 5 fields: "Already_Valid", which tells if the cited DOI was already valid before cleaning, "New_DOI", which contain a clean, valid DOI (if our procedure was able to produce one), and "prefix_error", "suffix_error" and "other-type_error" fields, which contain, for each cleaned DOI the number of errors that were cleaned.</div></div>

[STEPS]
?. [DATA IMPORT]
We import a CSV containing invalid DOI-to-DOI citations. The invalid DOIs were collected by Silvio Peroni while processing data provided by Crossref, and are provided on public licence in Citations to invalid DOI-identified entities obtained from processing DOI-to-DOI citations to add in COCI (Peroni, 2021).The CSV is organized as follows: the first column contains valid citing DOIs and the second contains invalid cited DOIs.
10.14778/1920841.192095410.5555/646836.708343	10.5406/ethnomusicology.59.2.0202	10.2307/20184517	10.1161/01.cir.63.6.1391	10.1161/circ.37.4.509
?. To make the DOI validity check procedure dramatically more efficient, Crossref's public data file containing over 120 million metadata records can be used. The reason is that the DOI names reported by Crossref at the work level are assigned by Crossref itself and not by publishers and, therefore, they are necessarily correct. The records are provided on public licence in January 2021 Public Data File (Crossref, 2021). According to the statistics reported by Crossref at the indicated address, the compressed file weighs 102.51 GB. The download time at an average speed of 12.73 MB/s is 2 hours.
This sub-step and the following one are completely optional and it is recommended to perform them only having at least 64 GB of RAM available since over 120 million DOIs must be stored in a set and kept in RAM during the entire process shown at step 3.
?. Once downloaded, the file consists of a folder containing 40,228 JSON files. The function get_all_crossref_dois() is used to extract the DOI names from each work.The function cycles through all the JSON files in a folder, opens them and parses them looking for works' DOIs. For this purpose the ijson library was used, that instead of loading the whole file into memory and parsing everything at once, lazily streams the data only responding to specific events, similar to what SAX does for XML. Finally, the list thus created is saved in a CSV file. The CSV file is preferred over JSON for this purpose because, as there is no hierarchy in the data, CSV takes up less space as it uses fewer characters than JSON. The CSV thus obtained consists of a list of all the DOIs of all the works found in Crossref, as shown in the table below:
10.1001/.38910.1001/.39110.1001/.40510.1001/.411
?. [ERROR ANALYSIS]
The DOI error's taxonomy proposed in Errors in DOI indexing by bibliometric databases (Franceschini et al., 2015) divides the errors in Digital Object Identifiers into two main classes: author errors, made by authors when creating the list of cited articles for their publication, and database mapping errors, related to a data-entry error. Other types of errors may be, more in general, classified as human-made errors (e.g. those introduced by editors when managing the references across publications). As proposed in Types of DOI errors of cited references in Web of Science with a cleaning method (Xu et al., 2019), the errors in DOIs can be furtherly divided into prefix-type errors, suffix-type errors, and other-type errors. Our methodology uses the division proposed by Xu et al. (2019) to classify errors in DOIs, by identifying these classes of errors as belonging to the class of factual errors, distinguished from temporarily invalid DOIs, that should be excluded from the cleaning procedure.For further clarifications on this taxonomy consult the paper Cleaning different types of DOI errors found in cited references on Crossref using automated methods (Boente et al., 2021a).
?. [ERROR ANALYSIS]
For cleaning prefix-type errors, we took a modified version of the regular expression from Xu et al. (2019) in order to remove unwanted characters at the beginning of a DOI. The regular expression for matching prefix-type errors is the following:It is worth noting that in this as in the following regular expressions the character "O" and the number "0" both count as a match, since, as demonstrated by Zhu et al. (2019), the two characters are often confused.
"(.*?)(?:\.)?(?:HTTP:\/\/DX\.D[0|O]I\.[0|O]RG\/|HTTPS:\/\/D[0|O]I\.[0|O]RG\/)(.*)"
?. [ERROR ANALYSIS]
For capturing suffix-type errors, we enriched the approach provided by Xu et al. (2019) and we devised a set of 17 alternative suffix regular expressions. The regular expression are the following:All the suffix regular expressions can be combined in a single regular expression, by using alternatives within a non-capturing group. This is how the regular expression for suffix-type errors looks like:
1) "(.*?)(?:\/-\/DCSUPPLEMENTAL)$"2) "(.*?)(?:\/SUPPINF[0|O](\.)?)$"3) "(.*?)(?:[\.|\(|,|;]?PMID:\d+.*?)$"4) "(.*?)(?:[\.|\(|,|;]?PMCID:PMC\d+.*?)$"5) "(.*?)(?:[\(|\[]EPUBAHEADOFPRINT[\)\]])$"6) "(.*?)(?:[\.|\(|,|;]?ARTICLEPUBLISHEDONLINE.*?\d{4})$"7) "(.*?)(?:[\.|\(|,|;]*HTTP:\/\/.*?)$"8) "(.*?)(?:\/(META|ABSTRACT|FULL|EPDF|PDF|SUMMARY)([>|\)](LAST)?ACCESSED\d+)?)$"9) "(.*?)(?:[>|)](LAST)?ACCESSED\d+)$"10) "(.*?)(?:[\.|(|,|;]?[A-Z]*\.?SAGEPUB.*)$?"11) "(.*?)(?:\.{5}.*?)$"12) "(.*?)(?:[\.|,|13) "(.*?)(?:\[DOI\].*?)$"14) "(.*?)(?:\(\d{4}\)?)$"15) "(.*?)(?:\?.*?=.*?)$"16) "(.*?)(?:#.*?)$"
"(.*?)(?:\/-\/DCSUPPLEMENTAL|\/SUPPINF[0|O](\.)?|[\.|\(|,|;]?PMID:\d+.*?|[\.|\(|,|;]?PMCID:PMC\d+.*?|[\(|\[]EPUBAHEADOFPRINT[\)\]]|[\.|\(|,|;]?ARTICLEPUBLISHEDONLINE.*?\d{4}|[\.|\(|,|;]*HTTP:\/\/.*?|\/(META|ABSTRACT|FULL|EPDF|PDF|SUMMARY)([>|\)](LAST)?ACCESSED\d+)?|[>|\)](LAST)?ACCESSED\d+|[\.|\(|,|;]?[A-Z]*\.?SAGEPUB.*?|\.{5}.*?|[\.|,|
?. [DATA CLEANING PROCEDURE]
Here we describe the steps through which we carried out the cleaning procedure of factually invalid DOIs. The workflow of the procedure is organized as follows:We created Python software for handling the whole procedure. The source code is publicly available on ISC license (Massari et al., 2021). For our data cleaning system, we devised two class objects: one called class Clean_DOIs(), with all the methods required to carry the cleaning/validation procedure, and class Support(), with all the methods required to process and dump CSV as well as carry HTTP requests. In this protocol, we describe the use of the different methods that can be imported from the two classes. If you want to reproduce the procedure described in the next step, once you have verified to have Python 3.x installed and the required libraries, you can clone our Github repository and launch the tutorial.py file in a terminal inside the folder or you can execute in a Python console the commands listed in the following sub-sections.
?. [DATA CLEANING PROCEDURE]
First, with the function process_csv_input() from the class Support(), we import the data in CSV format and store it in a DictReader, an object that maps the information in each row to a dictionary whose keys are given by the optional fieldnames parameter.We perform this method on our input CSV and store it as a  variable. Optionally, it is also possible to import in the  variable the Crossref DOIs previously stored in a CSV file as shown in step 1.2. It is recommended to perform this step having at least 64GB of RAM, as over 120 million DOIs need to be stored in a set and kept on RAM throughout the entire process. You can reproduce the experiment by launching:After this procedure, the  variable (and, if included, the  variable) will consist of a list of dictionaries, as shown below:
[{"Valid_citing_DOI": "10.14778/1920841.1920954", "Invalid_cited_DOI": "10.5555/646836.708343"}, {"Valid_citing_DOI": "10.5406/ethnomusicology.59.2.0202", "Invalid_cited_DOI": "10.2307/20184517"}, ...]
?. [DATA CLEANING PROCEDURE]
The procedure first checks for each cited DOI if it is factually invalid or it has become valid in the meanwhile. For this purpose, the function check_dois_validity() from the class Clean_DOIs() is used, which takes in input a list of dictionaries, like the one described above, and does a GET request to the DOI Proxy with the cited DOI (https://www.doi.org/factsheets/DOIProxy.html) of each row of the CSV: if the status code corresponding to that specific DOI is different from 1, it means that it is not valid; otherwise, that DOI has become valid in the meanwhile. The function outputs an enriched version of the list of dictionaries in input, with two additional fields: "Valid_DOI", which stores a validated DOI, and "Already_valid", which stores 1 if it has become valid in the meanwhile, 0 otherwise.In case the optional  argument has been passed, a set containing all DOIs in the corresponding dictionaries list is generated. The set was chosen and not the list because, having all the values hashed, makes the content check instantaneous, at the expense of more RAM. Therefore, the request to the DOI Proxy Server is made only if the current DOI is not found in the set, in which case it is registered as valid.We can now instantiate the Clean_DOIs() class, passing none or more of the optional parameters ,  and ,. Then, we run the check_dois_validity() function passing the  and storing the results in a  variable:Depending on the dataset size, on the bandwidth and whether or not the  parameter has been used, this step can take a long time, up to a week in the case of the over one million DOIs of the input dataset mentioned at step 1 at an average bandwidth of 100 Mb/s.Once the execution is complete, this is how the  variable appears:
[{"Valid_citing_DOI": "10.1007/s11771-020-4410-2", "Invalid_cited_DOI": "10.13745/j.esf.2016.02.011", "Valid_DOI": "10.13745/j.esf.2016.02.011", Already_valid": 1},...]
?. [DATA CLEANING PROCEDURE]
Afterwards, we pass the  variable obtained in the previous step to another function called procedure(), aimed to clean the DOIs with the regular expressions described above and to verify the validity of the cleaned DOIs.In our data cleaning procedure we store the output of the procedure() function in the  variable:For each cited DOI, the function procedure() performs five actions:it removes white spaces in the string;it applies the prefix regular expression to catch unwanted characters that precede a DOI. If there is a match, the regular expression removes the unwanted characters and the DOI is replaced with the cleaned one;it applies the suffix regular expressions as a list of alternative unwanted patterns that may occur at the end of the string. If there is a match, the regular expression removes the unwanted characters and the DOI is replaced with the cleaned one;it removes unwanted characters, like double slashes, double points, XML tags and backward slashes;finally, after the cleaning procedure, the algorithm checks if the cited DOI is different from the one in , if it is contained in the  set or returned as valid by the DOI Handle API. If it is different and is validated, the algorithm stores the cleaned DOI in the "Valid_DOI"  field of the dictionary; if it is not different, the dictionary will store no new DOI.By applying procedure() to our  we obtain a list of dictionaries containing 7 fields: the four already present in the input list ("Valid_citing_DOI", "Invalid_cited_DOI", "Valid_DOI", "Already_Valid") and, in addition, "prefix_error", "suffix_error", "other-type" fields, which contains, for each cleaned DOI, the number of errors that were cleaned. This is how the  variable looks like:In order to handle the various exceptions that could occur during a request to a server, and to save the results already obtained and not have to repeat them in the event of a crash, blackout or other incidents, two different strategies have been employed. First, the handle_request () function handles HTTP 500 errors, sets a 10-second timeout, repeats the request a second time on an error, and caches the responses. On the other hand, if a cache file has been created, the read_cache () function reads the data processed up to that moment, in order to restart from the last line read and not from the beginning. This function has been integrated into both check_dois_validity() and procedures(), that create or update a cache file every n DOIs. The number of DOIs after which to update the cache is customizable, as is the location of the cache file.
[{"Valid_citing_DOI": "10.1007/s40617-018-00299-1", "Invalid_cited_DOI": "10.1901/jaba.2012.45-657", "Valid_DOI": "10.1901/JABA.2012.45-657", Already_valid": 1, "prefix_error": 0, "suffix_error" 0:, "other-type": 0 }, {"Valid_citing_DOI": "10.1074/jbc.m508416200", "Invalid_cited_DOI": "10.1059/0003-4819-100-4-483", "Valid_DOI": "", Already_valid": 0, "prefix_error": 0, "suffix_error" 0:, "other-type": 0 }, {"Valid_citing_DOI": "10.1177/2054358119836124", "Invalid_cited_DOI": "10.1016/j.amepre.2015.07.017.", "Valid_DOI": "10.1016/J.AMEPRE.2015.07.017", Already_valid": 0, "prefix_error": 0, "suffix_error" 1:, "other-type": 0 },...]
?. [DATA CLEANING PROCEDURE]
At the end of our procedure we dump our  in a csv file through the dump_csv() method of the class Support():This is how the method is used in our data cleaning procedure:The output of the function is then stored in a CSV file, organized in 7 fields as the dictionary mentioned above:10.1007/s40617-018-00299-110.1901/jaba.2012.45-65710.1901/JABA.2012.45-657100010.1074/jbc.m50841620010.1059/0003-4819-100-4-483000010.1177/205435811983612410.1016/j.amepre.2015.07.017.10.1016/J.AMEPRE.2015.07.0170010The output dataset is available on Zenodo (Boente et al., 2021b).Further information about the possible uses of research data generated by this project, and the support provided by the authors for reuse, are provided in our Data Management Plan, available on Zenodo (Boente et al., 2021c).
10.1007/s40617-018-00299-110.1901/jaba.2012.45-65710.1901/JABA.2012.45-657100010.1074/jbc.m50841620010.1059/0003-4819-100-4-483000010.1177/205435811983612410.1016/j.amepre.2015.07.017.10.1016/J.AMEPRE.2015.07.0170010
?. [STATISTICS AND VISUALIZATION]
Finally, two visualizations were produced from the output file. For this purpose, d3.js v6.7.0, an open-source Javascript library for manipulating data-driven documents, was used (Bostock, 2021). In particular, a bar chart and a treemap have been created, which compare the number of DOIs still not valid after the cleaning procedure with those that have been corrected, and the number of those belonging to the four classes of errors considered, i.e. DOIs already valid and DOIs with prefix, suffix, or other-type errors. In addition, the bar chart allows comparing the results obtained through this study with those that would have been obtained using the regular expressions suggested by Xu et al. (2019). Finally, the bar chart can be sorted in ascending or descending order and it is possible to hover the mouse over both visualizations to highlight the represented value and the percentage of the total. It is possible to interact with the visualization at the following address: https://open-sci.github.io/2020-2021-grasshoppers-code/.
?. [ERROR ANALYSIS]
To find recurrent errors to automatically clean, we manually isolated from the dataset a subset of strings at the beginning and at the end of factually invalid DOIs, which contained prefix or suffix errors. In addition, we found other types of errors, like double underscores, double periods, XML tags, spaces, and forward slashes that could be removed at the end of the data cleaning process. A sample of this subset of corrupting string is provided in Table 3 of the article Cleaning different types of DOI errors found in cited references on Crossref using automated methods (Boente et al., 2021a).
?. [REFERENCES]
Boente, R., Massari, A., Santini, C., & Tural, D. (2021a). Cleaning different types of DOI errors found in cited references on Crossref using automated methods (Version 2). Zenodo. http://doi.org/10.5281/zenodo.4734513Boente, R., Massari, A., Santini, C., & Tural, D. (2021b). Classes of errors in DOI names: output dataset (Version v1.0.0-alpha) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.4733647Boente, Ricarda, Massari, Arcangelo, Santini, Cristian, & Tural, Deniz. (2021c). Classes of errors in DOI names (Data Management Plan) (Version 5). Zenodo. 10.5281/zenodo.4733919Bostock, M. (2021). D3: Data-Driven Documents. Software Heritage. https://archive.softwareheritage.org/swh:1:dir:35fe697ae5a21e96d9fc01d890b30010e23c16ddCrossref. (2021). January 2021 Public Data File. https://doi.org/10.13003/GU3DQMJVG4Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2015). Errors in DOI indexing by bibliometric databases. Scientometrics, 102(3), 2181–2186. https://doi.org/10.1007/s11192-014-1503-4Massari, A., Santini, C., & Boente, R.. (2021). open-sci/2020-2021-grasshoppers-code: Classes of errors in DOI names (Version v1.1.0). Zenodo. http://doi.org/10.5281/zenodo.4734013Peroni, S. (2021). Citations to invalid DOI-identified entities obtained from processing DOI-to-DOI citations to add in COCI (Version 1.0) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.4625300Xu, S., Hao, L., An, X., Zhai, D., & Pang, H. (2019). Types of DOI errors of cited references in Web of Science with a cleaning method. Scientometrics 120, 1427–1437. https://doi.org/10.1007/s11192-019-03162-4Zhu, J., Hu, G. & Liu, W. (2019). DOI errors and possible solutions for Web of Science. Scientometrics 118, 709–718. https://doi.org/10.1007/s11192-018-2980-7