Package: pubmed-api
Problem StatementDue to recent changes in PubMed API policies, they don't allow users get more than 10k results, resulting
in results capping against search strings.Solution #1edirect- edirect is a command-line utility. It provides capabilities of fetching PMIDs against a
search string whenesearchandefetchcommands are piped together.Solution #2Running pubmed API on search strings with some date-ranges. But sometimes even a single day has more than 10k PMIDsSolution #3Instead of usingedirector concept of date-ranges, let's divide the search strings results in ranges of PMIDs.AlgorithmTake a usual string and run over PubMed API usingsort=pub_date. It will get some PMIDs and Count of actual PMIDs from the pubmed db.The returned results are sorted on the basis of most recent articles as we have usedsort=pub_date. In essence, the most recent PMID is
returned in the very first API call. So, take the maximum PMID from the batch of 10K PMIDs as this PMID is the most is most recent.If the returned PMIDs count (counted PMIDs from response) <= actual counts<Counts>47851</Counts>mentioned in the API response. Then our first API call is the required result.If notStep 3and current call is first API call, then divide the maximum PMID fromStep 2by2. It will give you two halves.
First from1tomax_pmid / 2and second frommax_pmid / 2tomax_pmidIf notStep 3and current call is 2nd, 3rd or so on. API call then divideendby 2 and make two halves
First from1toend / 2and second fromend / 2toendNow, change the original search string as follows:Supposemax_pmid = 15000orend = 15000(endisnullduring first API call. It is initialised withmax_pmid / 2during 1st API call
and used in all subsequent recursive API calls)NOTE: we are only usingmax_pmidin first API call. In all subsequent call we're just making half ofmax_pmidagain and again.Suppose our search string is"human immunodeficiency virus (hiv)", so, two new search strings will be"human immunodeficiency virus (hiv)" AND 1:7500[UID]and"human immunodeficiency virus (hiv)" AND 7500:15000[UID]For 1st half
makestart=1end=7500For 2nd half
makestart=7500end=15000Now, Run these two string over the API and repeatStep 3for each new string.This way we can grab all PMIDs using API, without shifting to any other new tool.If you see we are dividing each sets into two halves recursively. That means PMIDs from range1toxcan be fetched
in an API call and the minimumxcan go is1. We are using same concept of date ranges but dividing sets into halves
using PMID itself. This way the issue of more than 10k PMIDs in a day can be corrected, and it's far simpler than
usingedirecton the terminal.The above process takes only41 Secondsfor106KPMIDs.The edirect actually uses the PubMed API to get the results, but how it is able to get the results from PubMed API if it is using the same API? 🤔Actually it uses the same above concept to get the results from API. 🎉UsageTo get PMIDs for a search stringfrompubmed_api.pubmedimportPubMedAPIpa=PubMedAPI()# instantiate once and use it for n number of search terms# pa = PubMedAPI(10)  # pass any number. This number tells API to fetch only last 10 years of PMIDs# pa.__PUBMED_LIMIT__ = 100_000  # update with new pubmed limitsearch_terms=['"parkinson\'s disease"','"human immunodeficiency virus (hiv)"']forterminsearch_terms:result=pa.extract(term)# result will the object of ResultSet()print(result.pmids)# will return list of PMIDs (list)print(result.record_count)# will return number if PMIDs (int)