How To Programmatically Obtain Chemical Data From PubChem | by Hien Vo, M.S. | Towards Data Science

Have you ever wondered how scientists used to communicate back when the internet wasn’t around? I hadn’t thought about it either until I heard this story from a professor I worked with as an undergraduate. Apparently, organic chemists used to make new molecules and then didn’t make much of an effort to tell anybody about it. Back then, scientists were much less interdisciplinary than now. Therefore, it was just by chance that he met a biochemist who was looking to make a particular molecule. And it was also by chance that this professor had made this type of molecule.
Now, pharmaceutical companies are intentionally expending billions of dollars [1] to make and test chemicals searching for new drugs. Naturally, machine learning is being used in every step of the process [2]. And, of course, machine learning methods are useless without lots of good data. That’s where PubChem, the world’s largest collection of freely available chemical data, comes in. In this tutorial, I want to go over ways to programmatically obtain data for PubChem in case you wanted to dabble.
There are two methods to obtaining data from PubChem programmatically: PUG-REST [3] and PUG-View [4]. Which one you choose depends on the type of data you want. Below is a short introduction on how to use these two services. Please refer to the sources at the bottom for a more exhaustive list of what these services can do.
PUG-REST
PUG-View
What’s the difference?
Usage Policy
Requests & Beautiful Soup
PUG-REST gives you a way to access data via a URL, which looks something like this (<prolog> is “https://pubchem.ncbi.nlm.nih.gov/rest/pug”):
<prolog>/<input >/<operation>/<output>
You use <input> to tell the service which compound(s) or substance(s) to look for. For example, if I want to look up aspirin, I can construct the <prolog>/<input spec> like so:
<prolog>/compound/name/aspirin
Here, the <input> is broken down into 3 parts. First, I tell the service the type of record, which is ‘compound.’ Then, I want to search by ‘name.’ And that name is ‘aspirin.’ This link gives me an XML file that contains all the information PUG-REST has on aspirin. Suppose I want only the molecular weight of aspirin. I can do this:
<prolog>/compound/name/aspirin/property/MolecularWeight
where the <operation> is to look into the ‘property’ of aspirin and retrieve ‘molecular weight.’ Then, suppose I don’t like XML, and I want it in JSON instead:
<prolog>/compound/name/aspirin/property/MolecularWeight/json
How data is retrieved via PUG-View is very similar to PUG-REST. In this case, the prolog is “https://pubchem.ncbi.nlm.nih.gov/rest/pug_view”. Suppose I want to see what PUG-View has on aspirin:
<prolog>/data/compound/2244
Here, 2244 is aspirin’s CID, which stands for compound ID. This link gives me everything PUG-View has on aspirin. To narrow it down, I do this
<prolog>/data/compound/2244?heading=Melting-Point
to ask for aspirin’s melting point. If you look at this link, you’ll see that it’s not just one number, but multiple along with annotations as well. This brings me to my next point.
First, PUG-REST retrieves information computed by PubChem, while PUG-View gives you information collected from other sources along with annotations. This means if you want experimental data, you should first look in PUG-View. For a list of allowed headings, see [5]. If you click on ‘Chemical & Physical Properties’ then ‘Experimental Properties,’ you’ll see ‘Melting Point.’ So, using the template above, you can get other types of information in this list on aspirin (and other compounds), such as boiling point or density.
Second, I didn’t use the name ‘aspirin’ to look up its information in PUG-View. This is because, unlike PUG-REST, PUG-View only tasks CID as its input. For a full list of identifiers, you can use with PUG-REST, see [3].
Third, using PUG-REST, you can retrieve information on multiple identifiers. For example, I can get information on compounds with CIDs 1 and 2 like so
https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/cid/1,2
On the other hand, PUG-View only allows one identifier at a time.
Neither PUG-REST nor PUG-View is designed for large volumes (millions) of requests. PubChem has the following limits:
No more than 5 requests per second.
No more than 400 requests per minute.
No longer than 300 seconds running time per minute.
If your demands exceed these limits, you can contact them to sort it out.
Here’s a Google Colab notebook on how to interact with PubChem. I can’t guarantee that it’ll work for all chemicals. I only tried it on aspirin. Hope this helps!
import requests
from bs4 import BeautifulSoup
import re

Let's try to get all data on aspirin
data = requests.get("https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/aspirin")
data

<Response [400]>
Hmmm... A 400 response code means error. It looks like we have to specify the output file type. Otherwise, requests won't work.
data = requests.get("https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/aspirin/xml")
data

<Response [200]>
Ok. Let's parse this information using Beautiful Soup.
html = BeautifulSoup(data.content, "xml")

Let's find out what kind of information we can get.
html.find_all("PC-Urn_label")

[<PC-Urn_label>Compound</PC-Urn_label>,
 <PC-Urn_label>Compound Complexity</PC-Urn_label>,
 <PC-Urn_label>Count</PC-Urn_label>,
 <PC-Urn_label>Count</PC-Urn_label>,
 <PC-Urn_label>Count</PC-Urn_label>,
 <PC-Urn_label>Fingerprint</PC-Urn_label>,
 <PC-Urn_label>IUPAC Name</PC-Urn_label>,
 <PC-Urn_label>IUPAC Name</PC-Urn_label>,
 <PC-Urn_label>IUPAC Name</PC-Urn_label>,
 <PC-Urn_label>IUPAC Name</PC-Urn_label>,
 <PC-Urn_label>IUPAC Name</PC-Urn_label>,
 <PC-Urn_label>IUPAC Name</PC-Urn_label>,
 <PC-Urn_label>InChI</PC-Urn_label>,
 <PC-Urn_label>InChIKey</PC-Urn_label>,
 <PC-Urn_label>Log P</PC-Urn_label>,
 <PC-Urn_label>Mass</PC-Urn_label>,
 <PC-Urn_label>Molecular Formula</PC-Urn_label>,
 <PC-Urn_label>Molecular Weight</PC-Urn_label>,
 <PC-Urn_label>SMILES</PC-Urn_label>,
 <PC-Urn_label>SMILES</PC-Urn_label>,
 <PC-Urn_label>Topological</PC-Urn_label>,
 <PC-Urn_label>Weight</PC-Urn_label>]
Let's get the Molecular Weight. First, we locate the tag.
mw_tag = html.find(name="PC-Urn_label", string="Molecular Weight")
mw_tag

<PC-Urn_label>Molecular Weight</PC-Urn_label>
Then, let's take a look at the parent of this tag.
mw_parents = mw_tag.find_parent("PC-InfoData")
mw_parents

<PC-InfoData>
<PC-InfoData_urn>
<PC-Urn>
<PC-Urn_label>Molecular Weight</PC-Urn_label>
<PC-Urn_datatype>
<PC-UrnDataType value="string">1</PC-UrnDataType>
</PC-Urn_datatype>
<PC-Urn_version>2.1</PC-Urn_version>
<PC-Urn_software>PubChem</PC-Urn_software>
<PC-Urn_source>ncbi.nlm.nih.gov</PC-Urn_source>
<PC-Urn_release>2021.05.07</PC-Urn_release>
</PC-Urn>
</PC-InfoData_urn>
<PC-InfoData_value>
<PC-InfoData_value_sval>180.16</PC-InfoData_value_sval>
</PC-InfoData_value>
</PC-InfoData>
The info we want is contained between 'PC-InfoData_value_sval' tag. So let's get to it.
mw = mw_parents.find('PC-InfoData_value_sval').string
mw

'180.16'
There you go. It takes some playing around with. You can always go to the URL, which will render the data on your browser. That's what I did to help navigate this tree.
Let's do the same for PUG-View and see what we can get from it for aspirin.
data = requests.get("https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/2244/xml")
data

<Response [200]>
html = BeautifulSoup(data.content, "xml")

To get an idea of what information is available, we can look up TOCHeadings with html.find_all('TOCHeading'). I'm not gonna run the code here, because it's a really long list. Let's say we want to look up melting points.
mp_tag = html.find(name='TOCHeading', string='Melting Point')
mp_tag

<TOCHeading>Melting Point</TOCHeading>
If you look at the xml structure, you'll see that this TOCHeading has siblings with tags 'Information'. The information we want is the childrean of these 'Information' tag under the tag 'String.' So, to find the first value, we can do this:
mp_tag.find_next_sibling('Information').find(name='String').string

'275 °F (NTP, 1992)'
As you can see, these numbers come with not just the units, but annotations on where the data comes from. Not only that, there are multiple values for melting point, since it's experimentally measured. This was quite a long way to find the melting point. An eaiser way is to include what we want in the URL.
data = requests.get("https://pubchem.ncbi.nlm.nih.gov/rest/pug_view/data/compound/2244/xml?heading=Melting+Point")
data

<Response [200]>
html = BeautifulSoup(data.content, "xml")

html.find_all('String')

[<String>275 °F (NTP, 1992)</String>,
 <String>138-140</String>,
 <String>135.0 °C</String>,
 <String>135 °C (rapid heating)</String>,