Every Second Counts When It Comes to Getting Taxes Done
By: Elangovan Shanmugam
Tags: Cloud Native, Cloud Computing, Infrastructure, Digital Transformation, Cloud

Every Second Counts When It Comes to Getting Taxes Done

5X Performance Gains for TurboTax Tax Form Import

Executive Summary

A key feature of TurboTax is the import capability of tax documents such as W2 or 1099 forms and of the associated data directly in the tax return, helping taxpayers prepare their taxes faster and with less manual data entry.

To provide the best experience for Intuit customers, Intuit partners closely with financial institutions (FIs) or banks to streamline the tax filing process. Recently, a user experience widget was introduced into Intuit’s existing Financial Data Platform (FDP) architecture to better streamline the flow of tax documents (W2, 109x forms) from various financial institutions for tax filers using Intuit software products.

During the initial integration and testing phases, this widget experienced significant performance issues with response times over ten seconds. To address these latency challenges, a data-driven, methodical approach was applied over the course of eight weeks.

Fig 1: Connecting to FI

Prioritizing Performance Goals

To kick off the performance improvement project, the team decided to focus on and solve only for performance goals. Any technical issue, unrelated to latency, would intentionally be excluded (for example, choosing not to introduce NIO HTTP clients to increase concurrency).

Additionally, two project goals were defined and prioritized in this order:

1) Make no compromises on quality and security.

2) Focus on shared widgets consumption in tax offerings for tax flows (109x, W2, document upload).

Using a Data-Driven Approach

During this discovery phase, a widget (UI) perspective was used to discover how tax applications were consuming services, how they were talking to the service, and what the overall performance was.

In assessing the sources of latency, the following questions were posed:

Was the network slowing the widget down?

Was it the processing time?

Was it somewhere else?

Analysis and deep testing* of the entire flow was performed to understand and assess time (seconds or milliseconds) spent on each service in the 109x flow. A hypothesis emerged: to be successful, latency would need to be reduced at each layer. Data was gathered, areas needing improvement were identified, and then steps were taken to reduce latency at each layer.

* A baseline measuring legacy flows of tax traffic was applied and defining a target end-to-end SLA at different traffic loads that would deliver a great user experience.

Based on this study, the team identified five improvement areas.

Lengthy polling intervals: Async interaction was used between UI and the backend to fetch documents and process them from a FI. The UI layer polled the backend to check the status every three seconds. Too many hops: The applications backend had to hop through two data centers to apply the fetched tax documents in the tax return. This took two and one half seconds. Animation: An animation in the UI layer took ~0.5 seconds. Avoidable pre-flight calls: The offerings and UI components were served from different domains and the browsers introduced Cross Origin Resource Sharing (CORS using OPTIONS calls). Latency across data centers: Multiple hops occurred between multiple data centers.

Read more below on the details on how these areas were improved and how many seconds we saved for our end customers in each of these areas.

Optimizing Polling Saves Three Seconds

When a taxpayer submitted a request from the application, the backend had to send the request to the FI’s to acquire the tax data. This was a long running async job. Because of this, the client would then have to poll the server to check if the response was available. This polling had a static configuration such that it would poll the server at a constant frequency of 3 seconds.

This static polling resulted in three seconds of wait time, before it sent another request. Yet during this three second span, the team theorized that a job might have already been completed. Not having received an answer, the UI would ask for it again, setting up a cycle of inefficiency.

Fig 2: Asynchronous & Polling to get the status of connections

The team hypothesized that shortening the wait time could be achieved via a progressive polling feature, used widely in the software industry. Before applying progressive polling to the widget, the existing polling behavior of the widget was observed to identify criteria used. The team decided to experiment and poll again after 750ms, instead of waiting for the current three seconds. If no answer was received after the initial request, the second poll could occur in the next 200ms, and so on.

Optimizing polling based on the response time was the key to improving performance internally, shaving three seconds off of the workflow. Inspired by this dramatic performance improvement, the team continued to uncover additional areas of latency at other layers. We felt comfortable in our systemic analysis approach across all services and interactions between layers optimizing in all areas, and were motivated to continue our work in other areas that needed improvement.

Reducing Hops Saves Three Seconds

When taxpayers made a request to get the tax documents from their FIs, the Intuit platform would send the requests to the FI. When the Intuit platform received the response from the FI, the response was parsed and transferred to canonical format. The response was getting stored temporarily in the distributed in-memory data grid. This triggered an event to the offerings to then send a request to get the response from FDP. This reduced several complexities for multiple parties in the system, as well, helped to save around three seconds.

Fig 3: On Successful connection get the responses

Disabling Fade in/Out Animations Saves 450 Milliseconds

Our Experience Design (XD) team created beautiful animations in the UI Layer to depict a user’s request connecting to a financial institution to retrieve data. However, this animation ran 450 ms before an initial request was made to the service.

Fig 4: animation to simple image

After much internal debate on the benefits of showing animation to the end user, the team agreed to disable the animation. Instead, a simple (non-animated) acknowledgment would be displayed to the user that a request was being made. This saved another 450ms.

Eliminating Unnecessary Calls Saves 200 Milliseconds

Typically when mixed content was served from multiple domains, the browser would check with the app servers using OPTIONS, a preflight request from the browser. This added latency to the overall performance. We made configuration changes so that the OPTIONS calls would be avoided; this saved about 200ms.

Fig 5: Pre-flight OPTIONS calls add to the response time

Migrating to AWS Eliminates Cross Data Center Latency and Saves 600 Milliseconds

Intuit platform services had been deployed into the public cloud; however, the API gateway remained configured in the on-premise data center. Every service call had to be bounced back and forth between the on-premise data center and the public-cloud. This caused several wasted milliseconds traversing the network. During this project, we moved everything to be colocated in the public cloud. This saved 600ms.

Fig 6: Colocated Gateway & Service reduces the response time

Summary: Systematic Performance Analysis and Improvements End to End Improved Performance by 5X

By reducing ten seconds of latency, the team improved performance over the given baseline or functionality for tax flows in the Intuit financial data platform. It was important to keep things at par to meet the SLA. We chose to focus on the customer benefit. We knew every second did count when it comes to helping our customers get their taxes done as quickly as possible. In doing so, we carved out three distinct goals and hit all three. Over the course of eight weeks, this data-driven, methodical approach addressed and solved for the latency challenge at various layers.