CASE STUDY

Walmart Embraces Test Automation
and Open Source to Increase
Coverage and Deploy More OftenIndustry:
RetailCompany Size:
2 million+SUMMARY• As part of an initiative to improve customer e-commerce experiences and“Thanks to Sauce Labs,

developer productivity, the Walmart team adopted Continuous Testing best

we are able to ensure

practices using Selenium and their proprietary open source qualitythat our apps work on

automation platform.

over 700 browser/• Walmart now executes more than 50,000 automated tests per day and accelerated
their deploy cycles from twice per month to multiple times daily.

OS combinations,
172 device emulatorsand over 300 uniquereal devices.”

THE CHALLENGE
Walmart has long been a disruptor in the retail industry by revolutionizing the supply
chain to offer lower prices. They also continue to make big investments in their digitalstrategy to help customers save time.
Through this transformation, Walmart Labs (Walmart’s technology arm) was finding it
difficult to seamlessly transition away from legacy practices. There was a misalignment

Pranav Vermaon goals across the organization and teams were using their own tools and technologies.

Sr. Eng Manager,

They found themselves relying on time-consuming manual testing, which only allowed

Walmart Labs

updates to deploy twice per month on a three-week development cycle, leaving them
unable to keep up with the pace of app improvement they wanted. Furthermore,because their tests were completely manual, teams weren’t able to scale testing toensure coverage across the multitude of browsers, operating systems and mobile devices
that their customers would be accessing their apps from.
They knew they needed to increase application quality and increase deployment velocity
with zero friction for the dev teams, which led to the adoption of automated testing.

LEARN MORE AT SAUCELABS.COMPAGE 1

THE SOLUTION
In their move to automated testing, Walmart realized there were no test management
tools that worked at the scale that they required. So in 2015, they started developing
a proprietary open source quality automation platform designed to support native,
web and backend testing services. This now is a central tool that supports functional
testing, performance testing, mocking and data insights across all Walmart.com,
Grocery and backed store systems. Walmart also understood that in order to besuccessful they needed to change the entire culture of how their teams developed
and tested their apps. This led to the creation of their Center of Excellence - theDXT team. Their focus was to choose the best testing frameworks and processes tocomplement their automation platform. Once selected, they standardized and rolledthese out to various teams by having everyone working similarly.
Walmart realized that open source software gave them the latest tools, flexibility andsupport they needed. So they chose the open source frameworks Selenium, Appium,Espresso and XCUITest for automated testing. These allowed them to develop
automated tests with widely adopted frameworks used by hundreds of thousandsof organizations worldwide. This was important as they could build and scale anautomated testing organization knowing that they could hire engineers that had
experience in these technologies who could be productive immediately.
Walmart also leveled up their testing by implementing the Sauce Labs Continuous
Testing platform, the world’s largest cloud-based solution for web and mobile app
testing. Sauce Labs gave them access to hundreds of browser and operating system
combinations, increasing test coverage and saving time. Walmart’s automation
platform imports data from their own runners and Sauce Labs test details pages todisplay a quality dashboard so QA and engineers can quickly view the quality status of
their builds.
As Walmart standardized their testing frameworks and began managing them with
their own automation platform, Sauce Labs provided the tools that allowed them toscale their testing, enhance productivity, improve code quality and speed delivery. Andwith these systems in place, each team at Walmart knows that they are using the best
tools, and they don’t have to constantly reinvent the wheel when they add a new app
to their suite.

THE RESULTS
Today, Walmart has over 40 different projects around the world using their proprietary
open source automation platform to manage all of their tests. Used in conjunction
with the Sauce Labs Continuous Testing Cloud, these teams have run over 14 million
tests in the past 7 years - about 50,000 tests per day currently. These tools have given

LEARN MORE AT SAUCELABS.COMPAGE 2

teams the time to build out more sophisticated automation, with an increase from
200 test cases in 2016 to 500 test cases with every build to production in the case ofone team.
Thanks to Sauce Labs, Walmart is also able to ensure that their apps work on over 700
browser/ OS combinations, 172 device emulators and over 300 unique real devices,
and all without having to manage any infrastructure internally.
However, this increase in quality and coverage has not come at the expense of speed.
With Continuous Testing practices in place, teams at Walmart are deploying updates
to their apps daily on every front end build. This velocity is only possible with theconfidence knowing their code is well-tested and covered across every browser/OS/
device combination.
This change in culture and process left not only the the technical teams happy andmore productive, but it has also got the executive team excited. Within a year’s time,the efforts of the DXT team, along with the implementation of Sauce Labs, saved
750,000+ man hours that would have been spent on manual testing/updating support
for new browsers and operating systems.

LEARN MORE AT SAUCELABS.COM

PAGE 3