DOI: 10.5281/zenodo.4279978
title: Complete Rxivist dataset of scraped bioRxiv data

description: <strong>October 2020: Please note—though this dataset does include Twitter activity for 2018 and 2019, it appears our source for social media data is not going to be coming back online.</strong> rxivist.org allows readers to sort and filter the tens of thousands of preprints posted to bioRxiv. Rxivist uses a custom web crawler to index all papers on biorxiv.org; this is a snapshot of Rxivist the production database. The version number indicates the date on which the snapshot was taken. See the included "README.md" file for instructions on how to use the "rxivist.backup" file to import data into a PostgreSQL database server. Please note this is a different repository than the one used for the Rxivist manuscript—that is in a separate Zenodo repository. You're welcome (and encouraged!) to use this data in your research, but <strong>please cite our paper, now published in <em>eLife</em>.</strong> Going forward, this information will also be available pre-loaded into Docker images, available at blekhmanlab/rxivist_data. <strong>Version notes:</strong> <strong>2020-11-18</strong> Publication checks should be back on schedule. <strong>2020-10-26</strong> This snapshot fixes most of the data issues found in the previous version. Indexed papers are now up to date, and download metrics are back on schedule. <em>The check for publication status remains behind schedule</em>, however, and the database may not include published DOIs for papers that have been flagged on bioRxiv as "published" over the last two months. Another snapshot will be posted in the next few weeks with updated publication information. <strong>2020-09-15</strong> A crawler error caused this snapshot to exclude all papers posted after about August 29, with some papers having download metrics that were more out of date than usual. The "last_crawled" field is accurate. <strong>2020-09-08</strong> This snapshot is misconfigured and will not work without modification; it has been replaced with version 2020-09-15. <strong>2019-12-27</strong> Several dozen papers did not have dates associated with them; that has been fixed. Some authors have had two entries in the "authors" table for portions of 2019, one profile that was linked to their ORCID and one that was not, occasionally with almost identical "name" strings. This happened after bioRxiv began changing author names to reflect the names in the PDFs, rather than the ones manually entered into their system. These database records are mostly consolidated now, but some may remain. <strong>2019-11-29</strong> The Crossref Event Data API remains down; Twitter data is unavailable for dates after early October. <strong>2019-10-31</strong> The Crossref Event Data API is still experiencing problems; the Twitter data for October is incomplete in this snapshot. The README file has been modified to reflect changes in the process for creating your own DB snapshots if using the newly released PostgreSQL 12. <strong>2019-10-01</strong> The Crossref API is back online, and the "crossref_daily" table should now include up-to-date tweet information for July through September. About 40,000 authors were removed from the author table because the name had been removed from all preprints they had previously been associated with, likely because their name changed slightly on the bioRxiv website ("John Smith" to "J Smith" or "John M Smith"). The "author_emails" table was also modified to remove entries referring to the deleted authors. The web crawler is being updated to clean these orphaned entries more frequently. <strong>2019-08-30</strong> The Crossref Event Data API, which provides the data used to populate the table of tweet counts, has not been fully functional since early July. While we are optimistic that accurate tweet counts will be available at some point, the sparse values currently in the "crossref_daily" table for July and August should not be considered reliable. <strong>2019-07-01</strong> A new "institution" field has been added to the "article_authors" table that stores each author's institutional affiliation <em>as listed on that paper</em>. The "authors" table still has each author's most recently observed institution. We began collecting this data in the middle of May, but it has not been applied to older papers yet. <strong>2019-05-11</strong> The README was updated to correct a link to the Docker repository used for the pre-built images. <strong>2019-03-21</strong> The license for this dataset has been changed to CC-BY, which allows use for any purpose and requires only attribution. A new table, "publication_dates," has been added and will be continually updated. This table will include an entry for each preprint that has been published externally for which we can determine a date of publication, based on data from Crossref. (This table was previously included in the "paper" schema but was not updated after early December 2018.) Foreign key constraints have been added to almost every table in the database. This should not impact any read behavior, but anyone writing to these tables will encounter constraints on existing fields that refer to other tables. Most frequently, this means the "article" field in a table will need to refer to an ID that actually exists in the "articles" table. The "author_translations" table has been removed. This was used to redirect incoming requests for outdated author profile pages and was likely not of any functional use to others. The "README.md" file has been renamed "1README.md" because Zenodo only displays a preview for the file that appears first in the list alphabetically. The "article_ranks" and "article_ranks_working" tables have been removed as well; they were unused. <strong>2019-02-13.1</strong> After consultation with bioRxiv, the "fulltext" table will not be included in further snapshots until (and if) concerns about licensing and copyright can be resolved. The "docker-compose.yml" file was added, with corresponding instructions in the README to streamline deployment of a local copy of this database. <strong>2019-02-13</strong> The redundant "paper" schema has been removed. BioRxiv has begun making the full text of preprints available online. Beginning with this version, a new table ("fulltext") is available that contains the text of preprints that have been processed already. <strong>The format in which this information is stored may change in the future</strong>; any digression will be noted here. This is the first version that has a corresponding Docker image.