Case Study

BIGCOMMERCE

How BigCommerce Upleveled
Kafka Management for Digital
Retail Innovation
When it comes to smooth e-commerce capability in the modern era, BigCommerce provides
the technology behind the technology—the platform that’s ultimately the backbone of buying
for tens of thousands of merchants, including Solo Stove, Skullcandy, Yeti, and a lot of otherincredibly popular brands and products.

Headquarters
Austin, TX

Industry

For a company that enables varying levels of demand for merchants in the digital age,timeliness and accuracy of data are crucial. Inventory has to be accurate to the millisecond,
and when data is delayed by a day, or even a minute, it results in lost revenue for merchantsand damages to the organization’s reputation.

Retail technology

Challenge
Difficulty scaling architecture with open
source Apache Kafka® to meet real-time
demands of merchants. Resulting in over
20 hours a week managing low-level Kafka
infrastructure and not enough time spent
innovating new products and features.

Solution
Switching from self-managed Kafka toConfluent Cloud gave BigCommerce a
cloud-native, fully managed data streaming
platform that is cloud agnostic with prebuilt enterprise features.

Results• Time and cost savings, industry-leading
reliability with a 99.99% uptime SLA,
and seamless integration with thirdparty systems—in short, more innovation
with far less effort.
• In just five months, BigCommerce
migrated 1.6 billion events a day toConfluent, and saved 20 hours a weekin Kafka management, along with thescalability savings gained from no longer
having to overprovision clusters toaccommodate seasonal traffic spikes.

© 2023 Confluent, Inc. | Confluent.io

That’s only one of the reasons BigCommerce pursued Apache Kafka for their data streaming
use case. The modern, open SaaS, e-commerce platform works with tens of thousands ofmerchants who serve millions of customers around the world. Its digital commerce solutions
are available 24/7, all year long, for B2B, B2C, multi-storefront, international, and omnichannel
customers.
As Mahendra Kumar, VP of data and software engineering at BigCommerce, puts it, “Realtime data is critical for current business. The value of data drops almost exponentially as timegoes by. The sooner you can act on data, the more valuable it is.”Recognizing the need for real-time data while understanding the burden of self-managing
Kafka on their own led BigCommerce to choose Confluent—allowing them to tap into data
streaming without having to manage and maintain the data infrastructure.

“We started wondering, can we offload all of the management ofKafka—and still get all of the benefits of Kafka? That’s when
Confluent came into the picture.”— Mahendra Kumar, VP of data and software engineering, BigCommerce

The challenges of self-managed Kafka when instantaccess to data insights is critical to business
Before Confluent, BigCommerce was managing its own Kafka cluster that was growing andrequired increasing maintenance. BigCommerce started out with six broker nodes, but as theirtraffic and use cases increased, they ended up trying to manage 22 broker nodes and five
ZooKeeper nodes. This required over 20 hours a week dedicated to maintaining and managing
Kafka infrastructure and about half a full-time engineer’s time.

1

Case Study | BigCommerce

They chose Kafka because it is a robust, resilient system designed
for high throughput, which would allow the company’s data teams
to retain data for a period of time, as well as offset and reprocessdata as needed. However, open source (OSS) Kafka presented
significant limitations. “It was fine initially,” says Kumar, “but as thevolume of data started getting bigger and bigger, we got to a point
where managing Kafka became almost like a full-time job.”
Rather than being able to focus on delivering new services that
improve user experiences, team members were bogged down bysoftware patches, blind spots in data-related infrastructure, andsystem updates.

Challenge #1:
Maintaining and managing Kafka“We had to constantly patch software, and fromtime to time we would see traffic surging and haveto constantly manage the scaling of resources. It”The next evolution of our use case in Kafka: taking
each critical type of e-commerce event and building
a data model which could load data in real timeusing a stream processor.”— Mahendra Kumar, VP of data and software engineering,
BigCommerce

Kafka was originally instituted to help BigCommerce capture
critical retail events such as orders and cart updates. These eventswere stored in a NoSQL database as they came in. But with thatmodel, the company was not able to tap into streaming data, andthat meant they couldn’t exactly provide real-time data analysis
for merchants who wanted to be able to extract quick insightsfrom the data.
From time to time, the system would go down, and that would
cause a spike of events to backlog, which degraded the merchantand end-user experience. BigCommerce had an ETL, batch-based

was a painful journey.”system for analytics and insights, resulting in a wait time of eight— Mahendra Kumar, VP of data and software engineering,
BigCommerce

system consisted of 30+ MapReduce jobs. On occasion, jobs would

hours before merchants could get their analytics reports. This
fail and need manual intervention, causing even further delays.

Having the latest and greatest improvements to Kafka isimportant, and BigCommerce wanted to get the most out of
the infrastructure that was running Kafka. However, managing
upgrade priorities had to be balanced with other roadmap featuresthat the team needed to deliver. “If we have to spend time on
Kafka upgrades and patching, then it will take time away from ourother roadmap priorities,” said Kumar.

”Overall, the merchant experience has definitelyimproved. It’s been seamless, and the spikes
and delays are no longer there.”— Mahendra Kumar, VP of data and software engineering,
BigCommerce

Once the company got up to 22 Kafka nodes, management became
unwieldy. The engineering team was composed of nine people tomanage the data platform, data applications, data APIs, analytics,
and data infrastructure. However, they had three engineers spending
half their time on Kafka-related maintenance and upgrades.
Before big shopping events like CyberWeek, the entire teamneeded a month to get the technology ready to scale and had tooverprovision their clusters by 10-15% to account for the uncertain
traffic volume. This exercise was expensive and time-consuming, andvaluable resources were being squandered because of the unknowns.

Challenge #2:
Inability to tap into real-time data createsburden for customers© 2023 Confluent, Inc. | Confluent.io

The decision to adopt a fully managed
Kafka solutionAt the time of migration, BigCommerce processed around 1.6B
events a day. These were high-traffic, high-value events such asorder events, cart events, and page view events. The team knewthat with Kafka, they already had the ability to consume data inreal time. They just needed the support of a fully managed data
streaming platform to help them tap into the potential of Kafka.
As Kumar puts it, “The people who built Kafka are the people at
Confluent, so we were already familiar with the company.”
In addition, the company’s storefront architecture was already on
Google Cloud, so the fact that Confluent Cloud integrates so easily
with GCP was a key lever. Using Confluent on Google Cloud allowed
BigCommerce to significantly cut down on data transfer costs.

2

Case Study | BigCommerce

But Kumar’s team wanted the platform they chose to ultimately
be cloud agnostic, in case they ever decided to change vendors
or add another cloud. Confluent met that criteria, too. “Thecombination of Confluent on Google Cloud was the perfect choice,”says Kumar. “It provided the best of both worlds.”
Last but not least, BigCommerce needed the platform to enable
them to stream data to their own customers—the merchants—aspart of their valuable open SaaS strategy. Kumar explains, “Wewanted to enable our merchants to get their commerce data
into their own data warehouses so they could create reportingand combine it with their other kinds of data.” This would enablemerchants to integrate with third-party systems such as Google
Analytics and other APIs to, for instance, run ad campaigns forcustom audiences.

Migrating from self-managed Kafka
to Confluent Cloud
When BigCommerce wanted to migrate from their existing open
source Kafka to Confluent Cloud, they embarked upon a phased
approach to ensure a smooth transition without any impact totheir critical workloads.
Kumar’s team meticulously planned the migration with the helpof Confluent’s expertise to design the partition strategy andmigration solution. This was done to prepare for future needs andtraffic variations for production use cases.
The first phase of the migration saw the piped production data
being sent to both OSS Kafka and Confluent to monitor latency
and avoid any possible downtime. Once the systems were fully insync, the switch to Confluent was seamless.
In the second phase, BigCommerce integrated consumer lag andsink connector lag reporting into a single unified Google Cloud
monitoring dashboard with the help of Confluent Metrics API. This
helped them keep track of the lagging issues and provided a way toaddress them in a timely manner.
The third phase involved the migration of insights and data lake
workloads, loading data into Google BigQuery using Confluent SinkConnector. Finally, in the fourth phase, BigCommerce optimized
Kafka stream performance and determined the optimal number ofconsumers for future workloads.
In five months, BigCommerce had migrated 1.6 billion events perday, 22 broker nodes, three clusters, and 15 TB of storage data toConfluent Cloud. They experienced zero downtime, zero data loss,
and zero disruption to the merchant experience.

© 2023 Confluent, Inc. | Confluent.io”Confluent is designed for scale and resilience. We
get all of these benefits, too, which makes it evenmore appealing.”— Mahendra Kumar, VP of data and software engineering,BigCommerceWith Confluent Cloud and fully
managed Kafka, attention turns toinnovation
The resulting benefits of migration for BigCommerce’s engineering
team have had a strong impact on the business, the merchant, andthe end-user experience.
The biggest benefit is the simplification of using Kafka. WithKafka fully managed in Confluent Cloud, the engineering team nolonger has to spend 20 hours a week managing low-level Kafka
infrastructure—upgrades, traffic spikes, compliance, security,nodes, and other tasks. They also haven’t had to overprovision
clusters by 10-15% “just in case,” because Confluent is elasticallyscalable. Instead, they have been able to turn their full attention to
innovating products in a highly competitive industry.
“That’s been huge on the business side,” says Kumar, “because it
frees up engineers from worrying about operations so they canfocus on functionality, like, for instance, our new Meta integrations,”
which allow merchants to use BigCommerce Kafka streams to feeddata right into Meta’s Conversion API and create ad campaigns tohighly customized audiences in real time.
In addition, Confluent enables the team to design products andfeatures around asynchronous communication—critical because,
as Kumar says, “The world is moving toward a more disconnected
one where every system can publish and consume messages. Somesystems consume immediately, and some consume at other times.
The ability to publish and consume messages asynchronously will
be central to any good systems we build.”
For merchants, the switch to Confluent means they can get
real-time analytics on BigCommerce’s open platform and tap into
precise insights for their specific use cases. The data coming out ofBigCommerce is now trustworthy, timely, accurate, and actionable.
Because Confluent is cloud native and has the ability to “connecteverywhere” via integrations, BigCommerce is able to take realadvantage of the data pipeline.
They’re now pulling in data via the flexibility and ease ofConfluent’s Google Cloud Storage Sink Connector, and merchants
can easily combine this real-time data with the data they alreadyhave in their ecosystems so they can get to the ideal state of a360-degree view.

3Case Study | BigCommerce

The challenges of building a real-time solution on Kafka are notalways easy to solve, particularly for an in-house engineering team
that’s better off spending time on product innovation versus techstack management. As Kumar explains, “If you’re a company looking
at building a real-time solution and managing your own open source
Kafka, there is going to be a lot of overhead. Some costs are visible;
some are hidden.” Kafka self-management requires people, time,and effort—and can be a process of trial and error.

Technical Results:
Elastic scalability
Confluent has the ability for high throughput, so it handles traffic
seamlessly, even during seasonal superspikes like CyberWeek.
There’s no need to overprovision clusters by 10-15% “just in case”
like they used to.
No downtime“In our case,” says Kumar, “Confluent makes so much sense. We
no longer worry about Kafka clusters or scaling. We just focus on
product development.”

zero data loss and zero downtime.

”The system we decided to go with is a real-time

The ability to connect streaming data from Kafka with Confluent’s

messaging system (Kafka) backed by a managed
provider (Confluent) on a reliable, robust,

Since completing the migration to Confluent Cloud, there has been

Integration with third-party systems
many connectors provides great engineering flexibility andopportunity.

performance cloud platform (Google Cloud).”— Mahendra Kumar, VP of data and software engineering,
BigCommerce

Business Results:
Reduced operational burden

Learn More About BigCommerce
Read BigCommerce’s Product and Engineering blog
Follow BigCommerce Product and Engineering onTwitterThe team no longer worries about managing Kafka and instead
focuses on feature functionality. Specifically, there’s no longer aneed for the data team to spend 20 hours a week managing lowlevel Kafka infrastructure.
Faster time to market
As artificial intelligence (AI) and other technologies becomecompetitive levers for retail companies, BigCommerce can now get
things like a product-recommendation proof of concept built faster.
The ability to design for asynchronicity
Confluent enables BigCommerce to build products and features thatenable every system to consume messages in real time or not.
Cost savings
The simple fact that BigCommerce no longer has to assign engineersto manage Kafka creates cost savings, specifically around theoperational overhead that had previously been required to scale,patch, upgrade, monitor, rebalance, and optimize the platform.

© 2023 Confluent, Inc. | Confluent.io4