Airtable & Python Made Possible II: Uploading Data into Airtable from Python using Airtable’s API | by Kaleb Nyquist | Towards Data Science

—Understanding Relational Data & Record IDs—Matching Field Values to Record IDs — Creating New Records in Airtable — Matching (& Uploading) Multiple Records—Updating Records in Airtable — From pandas to Airtable — 🎉 Copy-and-Paste Airtable Functions — ⏮️ Part I: Downloading Airtable Data into Python
Python is a mature programming language that, despite its age, has been experiencing something of a renaissance as of late — emerging recently as the second most widely-used general programming language. This boom likely has been driven by rapid growth in the field of data science, a field Python dominates due to Python being relatively easy to learn, yet useful for both data engineering and data analysis.
Because certain sectors of the economy have relatively low barriers to obtaining rich data, their adoption rate of Python-driven data science has been relatively swift. For example, the finance sector is fundamentally about a particular data point — “price” —and so it’s not surprising that organizations like Citigroup teach Python to its incoming analysts to manage their price data. Empirical science, both applied and academic, is fundamentally about data-driven observation and accordingly many scientists of all disciplines learn Python as part of their education to make sense of all their observed data points. And then of course there is the software development sector, which includes all of those apps we spend an increasing percentage of our day engaging with, apps like Netflix which log every click, view, or other interaction we have on their platform as data points. And — guess what — Netflix uses Python “through the full content lifecycle” to optimize their product.
However, not every organization has it quite so easy when it comes to obtaining rich data. For example, my specialization is in the non-profit and social change sectors. For those of us who work in “the real world” (i.e., rather than behind computer screens or stock tickers), getting useful data on our operations can take some real effort and planning. The good news is that once we have obtained this data, the power of Python becomes readily available. In other words, I believe that the initial hurdle of obtaining data does not disqualify “real world” organizations from taking advantage of what Python has to offer.
Enter Airtable. Airtable combines a relational database structure with a user-friendly graphical interface, lowering the barrier to entry (particularly for those “real world” organizational contexts characterized by low data literacy). Ideally, we want to be able not only to collect data in Airtable that can be used in Python, but also produce datasets in Python that can then be viewed in Airtable. Having this functionality can enable data transparency and audits, while also empowering the non-programmers within the organization to do their own analysis of the data.
In a previous tutorial, I went step-by-step through how to download Airtable data into Python using an Airtable base’s API. Here, I bring the tutorial full-circle: a step-by-step guide on how to upload Python data into Airtable using the same techniques and the requests library. However, to take full advantage of Airtable’s relational data structure requires some careful pre-processing. This tutorial will cover both: first, that pre-processing step, and then the actual uploading step—whether that upload step is creating a new record in Airtable or updating an old record.
🎉 Special Bonus: if you do not have time for the full tutorial, click here to access my foolproof Airtable upload function that I generalized for use in multiple projects. You can run it as-is and it will guide you through any missing information (i.e. where to get an API key) and highlight possible errors. You are welcome to copy-and-paste-and-modify this code free of charge, but if you would like to share your gratitude you can click here to buy me a coffee.
Compared to Google Sheets or Microsoft Excel, the “magic” of Airtable is the integration of relational data. In addition to more disciplined data management, Airtable’s relational data structure means that data in one table can be accessed in another table for easy analysis. This is done through “Link to another record” field type.
For example, as you can see in the figure below, there are four entries in the “People” table who worked in the Rogers Park location. These people are named Bruce Buck, Gerard Greer, Lula Lozano, and Malcom McIntonsh and worked 9, 3, 4, & 8 shifts respectively. In the “Location” table, these people are summarized in a single row with a total of 24 shifts.
In the graphical web interface, relational data is represented by light cerulean blue boxes with human-readable text that corresponds with the “Primary Field”. When we access Airtable data through the API rather than through the graphical web interface, this relational data looks quite different and maybe a little intimidating.
Consider this observation, from the “People” table:
{'id': 'recrN5WVNHl1pGxET', 'fields': {'Name': 'Bruce Buck',             'Shifts': 9,             'Location': ['rec887G00dQIujFel'], 'createdTime': '2021-02-17T22:59:38.000Z'}
And, from the “Location” table:
{'id': 'rec887G00dQIujFel', 'fields': {'Name': 'Rogers Park',            'People': ['recrN5WVNHl1pGxET',                       'recMQieZAnYIu860r',                       'recXyTN7Mo9LNtxhe',                       'recqYx6e3BrvxFv2s'],            'Total Shifts': 24, 'createdTime': '2021-02-17T22:59:56.000Z'}
If you look carefully, you will see that the data visually represented as “Bruce Buck” has an id of recrN5WVNHl1pGxET and the data visually represented as “Rogers Park” has an id of rec887G00dQIujFel. By using record ids to refer to relational data like this, Airtable has the flexibility to handle complexities that might arise (such as: what if there are two different Bruce Bucks? What if the user changes how locations are referred to, going not by name but by street address?).
This relational data structure has implications when it comes to uploading data from Python into Airtable via the API. For example, let us say we have a coded a webscraper looking for mentions of Bruce Buck on a website about Rogers Park. The webscraper is not looking for a string match with "recrN5WVNHl1pGxET" but "Bruce Buck" . But if we upload "Bruce Buck" into a relational data field instead of an array of record IDs, Airtable’s API will return an invalid value for column error.
Fortunately, this a simple problem with an obvious solution that (with a couple of exceptions) will work reliability, as we will now see.
In Python, dictionaries are the fundamental “mapping” type, used to translate one value from another value, or “key”. For our purposes, we will need to create a dictionary of record ids and field values, something that looks like this:
Using the list of Airtable records such as those generated from Part I (“downloading”) of this tutorial as a starting point, all it takes is some simple Python code to create this dictionary.
1 | record_dict = {}2 | match_field = "Name"3 | for record in airtable_records:4 |     if match_field in record['fields']:5 |         record_value = record['fields'][match_field]6 |         record_dict.update({record_value : record['id']})
Line (1) creates an empty dictionary. Line (2) defines the Airtable column whose values we will be matching record ids to. Line (3) begins an iteration over each Airtable Record, and to prevent errors Line (4) checks whether or not a value exists in the field we are pulling data from. Lines (5) and (6) take this value, match it with the record id, and updates the dictionary accordingly.
⚠️ Note that with this solution that there will conflicts if there are duplicate values (e.g., two “Bruce Buck”s). This can be resolved with either in Airtable (“Bruce Buck 1” and “Bruck Buck 2”) or Python (with additional code). The best resolution to this issue will be context dependent.
With this dictionary created, we can begin the mapping process, as visualized below:
There are a number of ways to achieve this result. For a single value, either record_dict["Dawn Douglas"] or record_dict.get("Dawn Douglas") will work fine; for a Pandas data series pd.Series(data).map(record_dict) will convert all values that it can find a match for.
But, what if we can’t find a match? This is where the process of uploading data to Airtable gets quite a bit more complicated, but still entirely doable if we are willing to undertake a few extra steps. But first we need the basic mechanism for creating a new record in Airtable through the API.
In Part I of this tutorial, we used the requests library to download data from Airtable using the GET method. Now, using similar code, we are going to use the POST method to create a record (and later, we will use the PATCH method to update a record).
Compare, and notice the differences marked in italics:
# Downloadresponse = requests.get(url, params=params, headers=headers)# Upload Newresponse = requests.post(url, data=upload_json, headers=headers)
We can use the same URL ("https://api.airtable.com/v0/" + base_id + "/" + table_name) we used for download and uploading. We have to expand the headers, however to:
{"Authorization" : "Bearer " + api_key, "Content-Type" : "application/json"}
The presence of this new Content-Type header is a hint for us that when uploading data to Airtable we now send data in the form of a JSON (you might think of this as the mirror image of passing parameters when downloading).
To make this concrete, let’s run through an example.
# Step 1upload_data = {"Name" : "Otis Osborn",               "Shifts" : 9,               "Location" : ["rec887G00dQIujFel"]}# Step 2upload_dict = {"records" : [{"fields" : upload_data}],                "typecast" : False}# Step 3upload_json = json.dumps(upload_dict)# Step 4response = requests.post(url, data=upload_json, headers=headers)
Step (1) creates a dictionary of data that we want to upload to Airtable. Notice the use of a string, integer, and list data type. It is important here to make sure the field names and data types match — refer to the API documentation for your base if you are getting an error.
Step (2) subsets that data into another dictionary that Airtable’s API can read. You can set "typecast" to be either True or False. If for some reason you can’t get data types to match in Step 1, setting this to True will convert strings to Airtable’s best guess at the corresponding data type.
Step (3) converts this dictionary into a JSON, and Step (4) uploads the final product to Airtable. If successful, Airtable will send back a response that, when response.json() is called will look something like this:
{'records': [  {'id': 'recoiMuxqVvbNMRlG',   'fields': {'Name': 'Otis Osborn',              'Location': ['rec887G00dQIujFel'],              'Shifts': 9},   'createdTime': '2021-02-19T14:41:41.000Z'}]}
Note that in addition to a “created time” we now have another key piece of new data, an id specifically for this record.
We can verify that Otis Osborn has been added by checking the graphical interface:
Simple enough. This will work for adding records to Airtable just one at a time. However, when we start matching and creating multiple Airtable records in batches of any size, we start running into a matching problem.
Earlier, we matched field values with record ids using a Python dictionary based off a download of Airtable data. However, the moment we upload new data into Airtable, that Python dictionary is outdated.
Imagine, for example, if we come across “Otis Osborn” in a dataset twice, and we know this “Otis Osborn” is the same person (i.e., there is no Otis 1 and Otis 2). If we do not keep our Python dictionary up-to-date when matching multiple records, two separate records will be created for the same Otis Osborn.
One solution would be to re-download the Airtable data after each new upload, but this is would be bandwidth-intensive especially for larger datasets. A more optimal solution is to update the Python dictionary using the response data we receive when we upload a new relational data entry to Airtable.
Below, this process is represented visually.
After you have compared this figure with Figure 3 above, let’s consider how each of these steps could be coded.
Step 1: attempt to match field value to record ID
match_value = "Otis Osborn"record_id = record_dict.get(match_value)
Here, we are matching an Airtable field values to a Record IDs. In this specific case, we know that Otis Osborn is *not* in the dictionary of field values and record IDs, so record_dict.get("Otis Osborn") will return a Python NoneType object.
Step 2: create a new record in Airtable
match_field = "Name"if record_id is None:    upload_data = {match_field : match_value}    upload_dict = {"records" : [{"fields" : upload_data}]}    response = requests.post(url, data=upload_json, headers=headers)
Because the record id is None, we instruct the computer to upload Otis Osborn to Airtable. We create a simple dictionary of data, add this simple dictionary to the “upload dictionary” formatted for Airtable uploads, and then send a POST request.
Step 3: parse Airtable response
airtable_record = response.json()['records'][0]
The code above helps us navigate the Airtable response. Since we successfully uploaded one record, we get one record back, indexed at [0].
Step 4: get Airtable record ID
record_id = airtable_record['id']
The record ID should look something like: rec3QjKqz8RiCz26r. This has gotten us to same point as if we had this field value and record ID pair already in the dictionary, but we have one more step if we want to avoid duplicate uploads.
Step 5: Update dictionary of field values and record IDs
record_dict.update({match_value : record_id})
Congrats! At this point, we have done all the work necessary to undertake another iteration of the matching code. Keeping this Python dictionary synchronized with Airtable is easily the most difficult part of uploading data through the API.
Now that we have wrapped out head around this complicated process of matching and/or creating records, we are ready to move onto the final part of the upload process: updating Airtable records using Python.
Updating records in Airtable is a very similar process to creating a record, with only a couple of key differences, detailed below.
The first difference is the URL used in the API request: it needs to point towards the specific record we are updating. Fortunately, all we need to do is append the record ID to the URL we used for creating a new record, like so:
url + "https://api.airtable.com/v0/" + base_id + "/" + table_namerecord_url = url + "/" + record_id 
The second difference is the dictionary of data we sent through the API. Since we are only uploading a single record at a time when updating, the dictionary is much simpler (see Step 2, above).
upload_dict = {“fields” : upload_data, “typecast” : typecast}
The third difference is the RESTful method we are employing. For creating, we POSTed; for updating, we are going to be PATCHing, like so:
# Upload Updateresponse = requests.patch(record_url, data=upload_json, headers)
This might not seem like much compared to the rest of the tutorial, but the ability to update Airtable records allows you to create scalable data solutions that marry the analytical power of Python with the user-friendly data storage capabilities of Airtable.
pandas is a popular package for data analysis and manipulation in Python. When working with Airtable data, it is possible that we might have a pandas dataframe that looks something like this, where the record ids make up the index values:
In this case, our first step is to convert the dataframe (“df”) to a dictionary of dictionaries. We can do so with the with the code df.to_dict(orient="index"), which will generate something that looks like this:
{'rec6vFIUEH4Fk9Z4a': {'Shifts': 4,  'Name': 'Alva Arnold',  'Location': ['recyz79aQzGl3LXPT']}, 'recrN5WVNHl1pGxET': {'Shifts': 9,  'Name': 'Bruce Buck',  'Location': ['rec887G00dQIujFel']}, 'recUzv7g6PZi4FGbB': {'Shifts': 8,  'Name': 'Charlotte Chaney',  'Location': ['recSZBITRlt2juAJ0']}, ...}
We can now iterate over each key-value pair in the dictionary, like so:
dict_of_dicts = df.to_dict(orient="index")for record_id, upload_data in dict_of_dicts.items():   record_url = url + "/" + record_id   upload_dict = {"records" : [{"fields" : upload_data}]}   upload_json = json.dumps(upload_dict)   requests.post(url, data=upload_json, headers=headers)
Now, we can seamlessly move from pandas to Airtable, and vice versa (see Part I of this tutorial for downloading Airtable data into a pandas dataframe)!
First off, if you have followed each of the steps above and are still have problems, you are welcome to copy-and-paste my foolproof Airtable helper functions into your project. This Python code snippet not only integrates all of the ideas above in a single function, but it also will print troubleshooting messages to help you figure out what is not working.
Once you have gotten Airtable uploading to work, the sky is the limit. I have used Python-to-Airtable uploads to facilitate collaboration on data audits, to add transparency to data analyses, and to seed databases with generated data. Drop me a line to share your creative Python-Airtable integrations!
And there’s exciting news for Python-Airtable developers, more functionality is reportedly coming our way. In particular, there is a metadata API in the works, that gives developers the ability to list bases, tables, fields types, and views. Among other things, this will make it easier to locate data for downloading and uploading data without it breaking because of a key error or data type mismatch. Drop a comment or some applause on this article to let me know if you’d be interested in a Part III of this tutorial when the metadata API is released!
The original draft of this article was workshopped as part of a fellowship through the Flatiron School’s Washington DC Campus. For those interested in learning Python-driven data analytics, I highly recommend Flatiron School’s data science bootcamp.
Thanks to David T., a reader who spotted an error in the original code sample. Can you spot the mistake?{"Authorization" : "Bearer" + api_key, "Content-Type" : "application/json"}
Also, thanks to my supporters on ko-fi who’ve made it possible for me to put together these tutorials. If this tutorial was valuable to you, I would welcome your support! ☕