Package: bio2csv
bio2csv 🐧bio2csvis a Python package that allows you to easily scrape all research papers that match a search query (such aspenguins) on BioRxiv. It retrieves metadata (title, authors, link to each paper), and it can also fetch the abstract and the full text if specified.
You can also scrape all research papers that fall under a specific Biorxiv subject area, such as Genetics or Paleontology. To encourage responsible use of biorxiv, short random delays are implemented into the code to prevent overload/spam.Easy InstallationYou can install thebio2csvpackagewith pip:pipinstallbio2csvYou probably already have most of these dependencies:from bs4 import BeautifulSoup(This is less common. Run pip install beautifulsoup4)from tqdm import tqdmimport requestsimport reimport timeimport pandas as pdimport randomUsageThere are two functions available: scrape_biorxiv() and fetch_paper_details().
scrape_biorxiv() repeatedly calls fetch_paper_details().scrape_biorxivParameters:base_url(str):READ THIS FULLYThe base URL to scrape the papers from. Default is "https://www.biorxiv.org/collection/genetics?page=". You can choose from any of the subject areas here:https://www.biorxiv.org/. Or choose a search result URL such ashttps://www.biorxiv.org/search/penguins.You MUST APPEND ?page= to the end of the URL!pages(int, optional): The number of pages to scrape. Default is 5.get_abstract(bool, optional): Whether to fetch the abstract of each paper. Default isTrue. If you don't want to fetch the abstracts, set this toFalse.get_full_text(bool, optional): Whether to fetch the full text of each paper. Default isTrue. If you don't want to fetch the full texts, set this toFalse. Images will not be fetched.Returns:pandas.DataFrame: A DataFrame containing the details of the scraped papers.fetch_paper_detailsFunctionfetch_paper_detailsfetches the abstract and the full text of a single paper.Parameters:paper_url(str): The URL of the paper to fetch the details from.session(requests.Session): An activerequests.Session()to fetch the details.Returns:tuple: A tuple containing the abstract and the full text of the paper.Note: If the function encounters any error while fetching the details, it will return "Not found" for the abstract and/or the full text.QuickstartParametersHere's a simple usage example 🐧:!pipinstallbio2csvfrombio2csvimportscrape_biorxiv# 🐧Scrape the first 2 pages of the search results for "penguin" and get the abstract and full texts. 🐧df=scrape_biorxiv(pages=2,base_url='https://www.biorxiv.org/search/penguin?page=',get_abstract=True,get_full_text=True)# Print the resulting DataFrameprint(df)# Save to CSVdf.to_csv("PenguinPapers.csv")Fetching text for a single paperYou can also use thefetch_paper_detailsfunction to fetch the abstract and full text of asinglepaper:frombio2csvimportfetch_paper_detailsimportrequests# Initialize a sessionsession=requests.Session()# URL of a paper about penguin conservation 🐧paper_url="https://www.biorxiv.org/content/10.1101/2021.04.06.438390v1"# Fetch detailsabstract,full_text=fetch_paper_details(paper_url,session)# Print detailsprint(f"Abstract:{abstract}")print(f"Full Text:{full_text}")Please note that thefetch_paper_detailsfunction needs an activerequests.Session()to work.Only Scraping Abstractsfrombio_scraperimportscrape_biorxiv# Scrape only the abstracts from the first 5 pages of the Genetics collection (remember, the default base_url is for the Genetics collection)df_abstracts=scrape_biorxiv(pages=5,get_abstract=True,get_full_text=False)print(df_abstracts)Only Scraping Full Textfrombio_scraperimportscrape_biorxiv# Scrape only the full text from the first 5 pages of the genetics collectiondf_full_texts=scrape_biorxiv(pages=5,get_abstract=False,get_full_text=True)print(df_full_texts)Remember, it's important to use web scraping responsibly and respect terms of service!
This code sends about one request every 10 seconds so it will not overload the biorxiv servers.
I intentionally did not implement multithreading in order to prevent abuse of biorxiv.
Also, you don't want to get IP banned.ContributingContributions tobio2csvare welcome! If you have a feature request, bug report, or proposal, please open an issue on this repository. If you wish to contribute code, please fork the repository, make your changes, and submit a pull request.
The penguin examples were inspired by my CS161 class at Stanford which features Plucky the Pedantic Penguin.
If you find this repository useful, consider donating to theGlobal Penguin Society🐧🐧🐧Licensebio2csvis released under the MIT License. For more details, see theLICENSEfile in this repository.
You are responsible for how you use this package. I am not liable for any losses, harms, damages, or other consequences incurred by this package.