RESTful APIs to the Rescue!
By: Eric Weiner
Tags: Customer Success, API, SaaS, Integration

PROBLEM

How can we reduce the perceived pain our customers have when it comes to loading data with the following characteristics: • The data must be loaded frequently and consistently.

• The data being loaded often requires a high degree of customizable transformations.

• The process should not rely on the customer’s internal IT department for (what would likely be) a custom integration efforts.

Let me break down the integration equation: ETL

Extract | Transform | Load. This formula starts with first taking data, massaging it, and finally placing it somewhere. For our work stream, we decided to approach the steps in reverse order to determine a technical solution of loading into our system. If we were unsuccessful with getting data into our system then it didn’t matter how we acquired and worked with the customer’s data.

Load (/lōd/): Starting at the end and working backwards, we knew that we could leverage our tool’s APIs to access the same methods used by our UI to load data for our customers. Looking at a previous implementation where we assisted a customer in implementing some very complicated data calculations utilizing R code, we took the appropriate APIs and ported them into R, building a package and placing it on GitHub (check it out here!). After setting that up, we were quickly able to implement the correct calls into an R script and send data into our customer’s instance.

Transform (/tran(t)sˈfôrm/): Each of our customers utilizes specific data and wants the data represented in a manner which tells a story when it comes to strategic planning for their agency. This often requires data transformations (as mentioned above) beyond the basic add, subtract, multiply and divide. And many times we are compiling and extrapolating data from a multitude of sources and tying them back together. Once we had access to [updated] customer data we could transform it in whatever way our customers needed and load it for them. But how to get that data…

Extract (/ikˈstrakt/): We knew that establishing a traditional data extraction (connecting to a customer’s source database(s)) would almost certainly require their agency’s internal IT resources and that would open a Pandoras box of challenges. We also knew that our customers were receiving the necessary data from regular reports and extracts, from other departments, on a regular basis and in a consistent format (always in Excel files). The lowest barrier to entry for getting data was to have the customer provide their data from their source. Therefore we set up a web folder repository (Dropbox), thereby allowing the customer to upload files when received.

Solution

With all individual components outlined we could solve for the ETL equation: • [E]xtract, which typically denotes an automatic process, was morphed to become a document repository per customer to enable the automatic dropping and updating of documents as needed for their unique task.

• [T]ransform the data in their file(s) using R.

• [L]oad only the necessary data into the respective customer’s database using our REST APIs, as called from the R script.

Visual ETL process shared with customers

With all of the building blocks in place we set up a Linux 2, AWS server to host Dropbox with “server-use-only” credentials, R and git (with a copy of our integrations repository). We created a process by which our customers could tell the integration to run when they were ready. We did this by including an indication trigger in each Dropbox web folder and instructed customers on how to indicate, using said trigger, when they were ready to process their files. Because our customers LOVE Excel, we set up our scripts to handle those file types in spades. Dropbox worked for this use, but Google Docs, Microsoft Office Online or Tableau can perform similarly.

This process will greatly reduce any repetitive updates of data that a customer would otherwise be required to perform manually (either by direct upload into the solution or by copying and pasting in-app). And all without requiring a large, resource-intensive, systems integration.

Looking Ahead

To enable easy repeatability of this solution, we architected it in a manner to allow it to scale from the initial 1 customer to N+1. There are a handful of Customer Success Managers with the aptitude to develop custom R scripts to process customer data and we look to tap into these talents as additional customer use cases are stood up.

Because of the use of RESTful APIs from our web application we are already discussing the inclusion of additional web calls into our data processing repertoire in support of additional automation for customer data in the future.