API Usability at Scale
Venue: PPIG

Designing and maintaining useful and usable APIs remains challenging. At Google, we manage hundreds of externally visible web APIs. Here, we report on our experiences and describe six on going challenges: resource allocation, empirically grounded guidelines, communicating issues, supporting API evolution over time, usable auth, and usable client libraries at scale.  Application Program Interfaces (APIs) are the way in which programmers access functionality created by others. Every substantive program written in every language today makes extensive use of APIs. It's no surprise that their usability often dominates the experience of programmers. APIs have been used for decades to connect software modules on a single machine. With the rise of web services and microservice architectures, web APIs have become the primary user interface for many types of services being offered online. While the paper is not centered only on web APIs, we do focus on them (as opposed to standard libraries) a few times. When we do, we are generally referring to representational state transfer (REST) APIs over HTTP. However, the structure of the API (REST vs. say, Simple Object Access Protocol [SOAP] and the communication protocol (e.g. HTTP vs. local connection) are orthogonal for the discussion. Despite their importance, there is comparatively little work on understanding API usability. A special interest group was held at the 2009 ACM CHI conference to discuss this problem, the result of which was the founding of apiusability.org (Daughtry, Farooq, Myers, Stylos, 2009) As of the time of writing, some 50 publications have been listed on this site. A recent paper by Myers and Stylos (2016) provides a summary of the existing work and outlines the various techniques in use for understanding and improving API usability. In this experience report, we describe some of the major challenges that we have faced building usable APIs at Google and discuss some of our approaches to those challenges. We aim to show the kinds of problems that Google faces as a major API provider and to start a broader conversation as to how we might go about addressing these challenges. 1. Challenge 1: UX Resource Allocation Traditional usability evaluation techniques can be applied to APIs in order to measure, understand, and improve usability (Clarke, 2004) (Stylos et al. 2008) One of the most cited techniques is running API usability lab studies. This is effective, but not scalable, as participants with professional software experience and researchers with a blend of human computer interaction, domain and technical knowledge are limited resources (Farooq Zirkler, 2010) Replacing lab studies with design reviews has been explored by both Microsoft (Farooq Zirkler, 2010) and Google (Macvean, Maly, Daughtry, 2016) our experience so far suggests this can augment lab studies, but not replace them. We are beginning to explore having software engineers run their own studies for APIs, which Google has employed successfully in the past for other domains (Baki et al. 2013) This reduces the need to find qualified researchers but doesn't address the time commitment or the burden of recruiting professional programmers as participants. Stylos and Myers (2007) described heuristics for how to prioritize research of API design decisions; Metric Data for a single API Opened API Explorer 6,067 sessions Clicked Execute 1,687 sessions Experienced 4XX error 47% (795 sessions) Achieved 2XX response 65% (1,099 sessions) Table 1 Explorer usage over one week of data for a single Google API.  Percentage of Explorer sessions that encountered a 4XX error across 26 Google APIs. namely, prioritizing for decisions that developers make often and that severely affect API use. We are using two sources to find the APIs which create the most friction for our users: (1) API exploration usage data, and (2) API Customer Satisfaction surveys. We can identify invalid API requests by looking for responses with a 4XX status code (i.e. an error in the client request, as opposed to, say, an issue caused by the server of the API. For example, a 404 or 429 HTTP response code) and we can compute the percentage of these responses across APIs. The problem with this approach is that API log data doesn't distinguish between calls that are placed when the developer is learning the API and calls that are run in production, making it difficult to distinguish between a hard to use API and existing buggy code. However, nearly all modern web APIs have a 'try it out' feature available online, such as the Google API Explorer1 or Swagger UI2. When we examine log data coming from such a tool, we know that a human is on the other side. Our hypothesis is that by exposing the Explorer data as usability metrics to the producers of those APIs, we can shine a light on problems and elicit positive change. As a first step, we explored a set of metrics for a small set of APIs. We can, for example, look at a success funnel on a per API basis, see Table 1. In isolation, these metrics are not actionable. While we want success rate to be higher, what constitutes a good success rate? As a next step, we took a look at the 26 most used APIs in Explorer in a one week period, see Figures 1, 2 and 3. We now have actionable information. Each graph exposes APIs that are outliers and need significant attention, while also showing us which APIs have further room for improvement.  We now know which APIs to focus on improving and can set a realistic numeric goal based on the less errorprone APIs (e.g. 40% When we look at Figure 2, we see that many of our APIs average two 4XX errors per session. If you make an error, then the response should include enough information for you to fix the mistake. So, we know that we should look at the APIs in the 3 range and try to bring them down close to 2. Figure 3 shows us the percentage of sessions that experienced success, where success is defined as achieving a syntactically correct (2XX) request. Again, we see a few APIs that are underperforming and room for improvement on many others. There are three downsides to these metrics, which must be remembered during interpretation. First, 1https:/developers.google.com/apis explorer/ 2http:/swagger.io/swagger ui/  Mean number of 4XX errors encountered per Explorer session over 26 Google APIs.  Percentage of sessions that experienced a 2XX response over 26 Google APIs. while we know errors are a signal for usability problems, we don't know exactly what usability issue(s) led to the erroneous request. Second, the problems they are experiencing could be usability problems with Explorer itself (the tool) as opposed to API usability issues. Third, we know from other research we have conducted that not every use of Explorer has a goal of achieving a 2XX response; for example, you may be trying to intentionally replicate a bug, or exploring the error messages that occur when you include invalid parameters. In practice, these three problems do not detract from the utility of the method. As long as the metrics guide follow up evaluation, human investigation can eliminate falsepositives while also better understanding the specific usability problems. While we don't include representative data here, we can also change the granularity of our metrics to expose problems on a per method basis as opposed to a per API basis. This can expose potentially confusing methods within a single API as well as providing UX researchers and software engineers more specific targets for lab studies or design reviews. Our next step is to broaden the scope of the data we include in the analysis to include all of the Google APIs exposed via Explorer and to expose the metrics to API producers. Having access to so many APIs will allow us to set benchmarks that the API producers can compare their API against (e.g. more than one standard deviation above or below the mean) We are also conducting exploratory analysis to establish whether we can reliably distinguish between the different types of usage we see in Explorer. Splitting off the data from those, for example, intentionally getting errors to replicate a bug, would help with the validity of the data, as well as opening up avenues for other interesting analysis. In parallel to our API log file analysis, we are also exploring the application of customer satisfaction surveys to our APIs. Using a simple Likert scale question (Likert, 1932) which asks users to rate their overall satisfaction with an API on a 5 point scale between 'Very dissatisfied' and 'Very satisfied' we can begin to quantify user sentiment towards a particular API. By distributing surveys across our suite of APIs, we also begin to get a picture of how users perceive our services, and, as with the Explorer data, the ways in which one API may stack up against others in order to set benchmarks for improvement. Furthermore, this lightweight approach to surveying allows tracking against time, allowing us to, for example, gauge the impact of a new version of an API with the old. Surveying API consumers is difficult, given the fact that much of their interaction with the API will be spent within their IDE or text editor, writing code. We are exploring distribution of our surveys via an unobtrusive intercept survey method (Muller Sedley, 2014) letting us target API consumers reading documentation. This approach may introduce a selection bias, as using the documentation is not a requirement for using an API, and users viewing the documentation may not have any experience with the API. None the less, when distributed equally and consistently across all of our APIs, we can uncover interesting trends. Much like with our API Explorer data, comparing sentiment across APIs provides a reference point for improvement.