Package: paperscraper
paperscraperOverviewpaperscraperis apythonpackage that ships viapypiand facilitates scraping
publication metadata as well as full PDF files fromPubMedor from preprint servers such asarXiv,medRxiv,bioRxivandchemRxiv. It provides a streamlined interface to scrape metadata and comes
with simple postprocessing functions and plotting routines for meta-analysis.Since v0.2.4paperscraperalso supports scraping PDF files directly! Thanks to@daenuprobstfor suggestions!Getting startedpip install paperscraperThis is enough to queryPubMed,arXivor Google Scholar.Download X-rxiv DumpsHowever, to scrape publication data from the preprint serversbiorxiv,medrxivandchemrxiv, the setup is different. The entire dump is downloaded and stored in theserver_dumpsfolder in a.jsonlformat (one paper per line).frompaperscraper.get_dumpsimportbiorxiv,medrxiv,chemrxivmedrxiv()#  Takes ~30min and should result in ~35 MB filebiorxiv()# Takes ~1h and should result in ~350 MB filechemrxiv()#  Takes ~45min and should result in ~20 MB fileNOTE: Once the dumps are stored, please make sure to restart the python interpreter
so that the changes take effect.Since v0.2.5paperscraperalso allows to scrape {med/bio/chem}rxiv for specific dates! Thanks to@achouhan93for contributions!medrxiv(begin_date="2023-04-01",end_date="2023-04-08")But watch out. The resulting.jsonlfile will be labelled according to the current date and all your subsequent searches will be based on this fileonly. If you use this option you might want to keep an eye on the source files (paperscraper/server_dumps/*jsonl) to ensure they contain the paper metadata for all papers you're interested in.Examplespaperscraperis build on top of the packagespymed,arxivandscholarly.Publication keyword searchConsider you want to perform a publication keyword search with the query:COVID-19ANDArtificial IntelligenceANDMedical Imaging.Scrape papers from PubMed:frompaperscraper.pubmedimportget_and_dump_pubmed_paperscovid19=['COVID-19','SARS-CoV-2']ai=['Artificial intelligence','Deep learning','Machine learning']mi=['Medical imaging']query=[covid19,ai,mi]get_and_dump_pubmed_papers(query,output_filepath='covid19_ai_imaging.jsonl')Scrape papers from arXiv:frompaperscraper.arxivimportget_and_dump_arxiv_papersget_and_dump_arxiv_papers(query,output_filepath='covid19_ai_imaging.jsonl')Scrape papers from bioRiv, medRxiv or chemRxiv:frompaperscraper.xrxiv.xrxiv_queryimportXRXivQueryquerier=XRXivQuery('server_dumps/chemrxiv_2020-11-10.jsonl')querier.search_keywords(query,output_filepath='covid19_ai_imaging.jsonl')You can also usedump_queriesto iterate over a bunch of queries for all available databases.frompaperscraperimportdump_queriesqueries=[[covid19,ai,mi],[covid19,ai],[ai]]dump_queries(queries,'.')Or use the harmonized interface ofQUERY_FN_DICTto query multiple databases of your choice:frompaperscraper.load_dumpsimportQUERY_FN_DICTprint(QUERY_FN_DICT.keys())QUERY_FN_DICT['biorxiv'](query,output_filepath='biorxiv_covid_ai_imaging.jsonl')QUERY_FN_DICT['medrxiv'](query,output_filepath='medrxiv_covid_ai_imaging.jsonl')Scrape papers from Google Scholar:Thanks toscholarly, there is an endpoint for Google Scholar too.
It does not understand Boolean expressions like the others, but should be used just like
theGoogle Scholar search fields.frompaperscraper.scholarimportget_and_dump_scholar_paperstopic='Machine Learning'get_and_dump_scholar_papers(topic)Scrape PDFspaperscraperalso allows you to download the PDF files.frompaperscraper.pdfimportsave_pdfpaper_data={'doi':"10.48550/arXiv.2207.03928"}save_pdf(paper_data,filepath='gt4sd_paper.pdf')If you want to batch download all PDFs for your previous metadata search, use the wrapper.
Here we scrape the PDFs for the metadata obtained in the previous example.frompaperscraper.pdfimportsave_pdf_from_dump# Save PDFs in current folder and name the files by their DOIsave_pdf_from_dump('medrxiv_covid_ai_imaging.jsonl',pdf_path='.',key_to_save='doi')NOTE: This works robustly for preprint servers, but if you use it on a PubMed dump, dont expect to obtain all PDFs.
Many publishers detect and block scraping and many publications are simply behind paywalls.Citation searchA plus of the Scholar endpoint is that the number of citations of a paper can be fetched:frompaperscraper.scholarimportget_citations_from_titletitle='Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I.'get_citations_from_title(title)NOTE: The scholar endpoint does not require authentification but since it regularly
prompts with captchas, it's difficult to apply large scale.Journal impact factorYou can also retrieve the impact factor for all journals:>>>frompaperscraper.impactimportImpactor>>>i=Impactor()>>>i.search("Nat Comms",threshold=85,sort_by='impact')[{'journal':'Nature Communications','factor':17.694,'score':94},{'journal':'Natural Computing','factor':1.504,'score':88}]This performs a fuzzy search with a threshold of 85.thresholddefaults to 100 in which case an exact search
is performed. You can also search by journal abbreviation,E-ISSNorNLM ID.i.search("Nat Rev Earth Environ")# [{'journal': 'Nature Reviews Earth & Environment', 'factor': 37.214, 'score': 100}]i.search("101771060")# [{'journal': 'Nature Reviews Earth & Environment', 'factor': 37.214, 'score': 100}]i.search('2662-138X')# [{'journal': 'Nature Reviews Earth & Environment', 'factor': 37.214, 'score': 100}]# Filter results by impact factori.search("Neural network",threshold=85,min_impact=1.5,max_impact=20)# [#   {'journal': 'IEEE Transactions on Neural Networks and Learning Systems', 'factor': 14.255, 'score': 93},#   {'journal': 'NEURAL NETWORKS', 'factor': 9.657, 'score': 91},#   {'journal': 'WORK-A Journal of Prevention Assessment & Rehabilitation', 'factor': 1.803, 'score': 86},#   {'journal': 'NETWORK-COMPUTATION IN NEURAL SYSTEMS', 'factor': 1.5, 'score': 92}# ]# Show all fieldsi.search("quantum information",threshold=90,return_all=True)# [#   {'factor': 10.758, 'jcr': 'Q1', 'journal_abbr': 'npj Quantum Inf', 'eissn': '2056-6387', 'journal': 'npj Quantum Information', 'nlm_id': '101722857', 'issn': '', 'score': 92},#   {'factor': 1.577, 'jcr': 'Q3', 'journal_abbr': 'Nation', 'eissn': '0027-8378', 'journal': 'NATION', 'nlm_id': '9877123', 'issn': '0027-8378', 'score': 91}# ]PlottingWhen multiple query searches are performed, two types of plots can be generated
automatically: Venn diagrams and bar plots.BarplotsCompare the temporal evolution of different queries across different servers.frompaperscraperimportQUERY_FN_DICTfrompaperscraper.postprocessingimportaggregate_paperfrompaperscraper.utilsimportget_filename_from_query,load_jsonl# Define search terms and their synonymsml=['Deep learning','Neural Network','Machine learning']mol=['molecule','molecular','drug','ligand','compound']gnn=['gcn','gnn','graph neural','graph convolutional','molecular graph']smiles=['SMILES','Simplified molecular']fp=['fingerprint','molecular fingerprint','fingerprints']# Define queriesqueries=[[ml,mol,smiles],[ml,mol,fp],[ml,mol,gnn]]root='../keyword_dumps'data_dict=dict()forqueryinqueries:filename=get_filename_from_query(query)data_dict[filename]=dict()fordb,_inQUERY_FN_DICT.items():# Assuming the keyword search has been performed alreadydata=load_jsonl(os.path.join(root,db,filename))# Unstructured matches are aggregated into 6 bins, 1 per year# from 2015 to 2020. Sanity check is performed by having# `filtering=True`, removing papers that don't contain all of# the keywords in query.data_dict[filename][db],filtered=aggregate_paper(data,2015,bins_per_year=1,filtering=True,filter_keys=query,return_filtered=True)# Plotting is now very simplefrompaperscraper.plottingimportplot_comparisondata_keys=['deeplearning_molecule_fingerprint.jsonl','deeplearning_molecule_smiles.jsonl','deeplearning_molecule_gcn.jsonl']plot_comparison(data_dict,data_keys,title_text="'Deep Learning' AND 'Molecule' AND X",keyword_text=['Fingerprint','SMILES','Graph'],figpath='mol_representation')Venn Diagramsfrompaperscraper.plottingimport(plot_venn_two,plot_venn_three,plot_multiple_venn)sizes_2020=(30842,14474,2292,35476,1904,1408,376)sizes_2019=(55402,11899,2563)labels_2020=('Medical\nImaging','Artificial\nIntelligence','COVID-19')labels_2019=['Medical Imaging','Artificial\nIntelligence']plot_venn_two(sizes_2019,labels_2019,title='2019',figname='ai_imaging')plot_venn_three(sizes_2020,labels_2020,title='2020',figname='ai_imaging_covid'))Or plot both together:plot_multiple_venn([sizes_2019,sizes_2020],[labels_2019,labels_2020],titles=['2019','2020'],suptitle='Keyword search comparison',gridspec_kw={'width_ratios':[1,2]},figsize=(10,6),figname='both')CitationIf you usepaperscraper, please cite the papers that motivated our development of this tool.@article{born2021trends,title={Trends in Deep Learning for Property-driven Drug Design},author={Born, Jannis and Manica, Matteo},journal={Current Medicinal Chemistry},volume={28},number={38},pages={7862--7886},year={2021},publisher={Bentham Science Publishers}}@article{born2021on,title={On the role of artificial intelligence in medical imaging of COVID-19},journal={Patterns},volume={2},number={6},pages={100269},year={2021},issn={2666-3899},url={https://doi.org/10.1016/j.patter.2021.100269},author={Jannis Born and David Beymer and Deepta Rajan and Adam Coy and Vandana V. Mukherjee and Matteo Manica and Prasanth Prasanna and Deddeh Ballah and Michal Guindy and Dorith Shaham and Pallav L. Shah and Emmanouil Karteris and Jan L. Robertus and Maria Gabrani and Michal Rosen-Zvi}}