5 Database technologies used by 2000 Wix microservices
By: Natan Silnitsky
Tags: Elasticsearch, Dynamodb, MySQL, Database, Mongodb

Photo by Gabriel Sollmann on Unsplash

Unlike with UI frameworks, or software industry buzzwords, data storage technologies, i.e. databases are famously slow to change, and new innovation takes longer to be adopted.

For a large organization, switching to a different OLTP database as your main workhorse happens very rarely. In recent years, with the advent of microservices, more flexibility is given for each service to choose the way it persists it’s data. Still, the risk of using unproven technology sometimes outweighs the benefits, as losing crucial business related data is unacceptable. While the technology may be proven elsewhere, it may not be proven and battle-tested in your organization yet.

At Wix, the situation is not different, as MySQL is still the main go-to database, more than 10 years after it was first used. Albeit, new and modern MySQL APIs (e.g. JSON Documents) are being used more and more.

But with 2000 microservices in production, there are of course several storage technologies other than MySQL that are used by a significant portion of services due to various reasons. The reasons range from different API use-cases (e.g. support CRUD API, allow free-text search) to different tech stacks (JVM vs. Node.js) to more demanding technical requirements (infinitely scaling storage).

5 technologies are described below, each with its own use cases at Wix, unique characteristics, strengths and weaknesses. The technologies are MySQL, MongoDB, DynamoDB, Elasticsearch and S3.

Read on to get some insights that may help you choose your next microservice storage technology.

1. MySQL

Other DBs in this category

PostgreSQL, MariaDB, SQL Server, Oracle, Db2, SQLite

When to use MySQL

When you already widely use it in your organization

When you want both relational tables (when you know the schema upfront) and JSON collections (Schemaless)

Relational / normalized — when you need to optimize on writes instead of reads, to have strong read consistency

MySQL Advantages

Maturity & Reliability — MySQL is highly used, battle tested and mature

Fast read performance

Improved JSON/Document support (MySQL 8)

Cross-DC write consistency (when ProxySQL is used)

MySQL Disadvantages

Scalability — Does not scale horizontally. Limited by amount of disk space

Consistency and Replication Issues (when not using ProxySQL)

Example MySQL use cases at Wix

A. Pre MySQL8 JSON support — relational tables with BLOB

Almost all JVM (Scala/Java) services at Wix used to store their data in MySQL relational tables. Data was usually denormalized across multiple tables (SQL joins were rarely used). For added flexibility most of the content was kept in BLOB type in order to avoid frequent schema changes that add a lot of risk for production stability.

B. JSON support and Simple Data Layer

Simple Data Layer (SDL) is an internal infrastructure built at Wix in order to help speed-up the development time and reduce cognitive load when persisting a document data structure.

This is a very common pattern as persistence exists in CRUD methods as well as business operations. The value spans from having sensible defaults like page size through support for WixQL which is part of our API and up to wider cross cutting concerns like GDPR support, emitting domain-changed events, and more.

It supports the very common need of microservices for a non relational embedded document data structure (to have more operations done by DB instead of in-memory).

Its engine is MySQL8 with JSON datatype. MySQL was chosen mainly because of Wix DBAs long time experience with MySQL, a lot of infrastructure already in place for backups, migrations, indexing, and for the strong consistency and availability support provided by ProxySQL (see next paragraph)

C. Multi DC & replication support with ProxySQL

ProxySQL is used in order to seamlessly route write traffic to the currently active main data center’s MySQL clusters without any data loss. It also manages read distribution among replicas. It provides high availability and high performance out of the box.

2. MongoDB

Other DBs in this category

CouchDB (many relational/NoSQL DBs now offer JSON support)

When to use MongoDB

When data schema is predicted to keep changing and evolving

When working with dynamic JSON content

When keeping data denormalized is not a problem. i.e. to have eventual consistency

MongoDB Advantages

Flexibility — with Schemaless documents, the number of fields, content and size of the document can differ from one document to another in the same collection.

with Schemaless documents, the number of fields, content and size of the document can differ from one document to another in the same collection. Easy to scale with sharding

MongoDB Disadvantages

High Memory Usage — a lot of denormalized data is kept in memory

Document size limit — 16MB

Non optimal replication solution (data cannot be re-replicated after recovery from failure). Consistency issues on traffic switch to another data center (No automatic remaster)

Example MongoDB use cases at Wix

A. Wix Data

Wix Data is a service that allows anyone with a Wix website to own a document database — “Database as a Service”). It uses MongoDB behind the scenes to store the data.

Mongo was chosen in order to be able to store any schemaless (JSON) doc. When the team compared MongoDB performance to MySql 5.7.6, the former had 2X write and read performance over the latter (albeit, this older version of MySql didn’t support JSON natively yet)

Recently the Wix Data team published a post on how to host 1 million sites per 1 MongoDB cluster

B. Wix Blog

Wix Blog is Wix’s blog publishing site-app. The backend Node.js microservices use MongoDB to store several TBs worth of blog posts.

When the project was young the schema was still evolving, and MongoDB’s schema-less nature helped ease the pain of adding new fields to a fast growing database. Data is mostly kept denormalized as MongoDB joins ($lookup) are not convenient to use.

3. Amazon DynamoDB

Other DBs in this category

Cassandra, HBase, Redis (Key-value Store)

When to use DynamoDB

When you need a simple key value store without complex querying patterns

When you need to store expirable data

Low-medium throughput apps as writes are expensive and consistent reads are twice the cost of eventually consistent reads.

DynamoDB Advantages

Fast performance in any scale (as long as enough capacity is provisioned)

No storage limit

Schemaless — it’s possible to define a schema for each item, rather than for the whole table

Multi-master replication (update data in multiple regions)

Supports TTL per item

Built-in CDC events (DynamoDB streams)

DynamoDB Disadvantages

Size limit — item can only reach 400KB in size

Limited querying options (limited number of indices)

Throttling on burst throughput (and hot keys in certain situations)

Example DynamoDB use cases at Wix

A. KVStore + Cache

DynamoDB is the back-bone for Wix KV-store cache abstraction. You can find more details in this article (search for (Dynamo)DB+CDC based Cache)

B. Wix Chat

Wix Chat is a service that lets site owners instantly communicate with visitors.

They recently migrated their database from Cassandra to DynamoDB, in order to be able to handle growing scale without needing to increase the DBA & operations teams.

In DynamoDB the Wix Chat team stores chat messages and chat message indications (sent / seen / failed). In addition, data is stored about online site visitors (page, location, entrance time, etc.) which also takes advantage of DynamoDB TTL feature.

C. Wix Contacts

Wix Contacts is a service that manages all contacts of a Wix site owner (Part of Ascend business tools suite)

DynamoDB is used to help orchestrate event-driven bulk jobs (they ​​save the job current status and current offset/page). Each of the job stages is advanced by sending a Kafka event to the appropriate topic. Then the Relevant data is fetched from DynamoDB.

4. Elasticsearch

Other DBs in this category

Apache Solr, Splunk, Amazon CloudSearch

When to use Elasticsearch

When you need to perform fuzzy search or have results with ranking

When you have another data store as source of truth (populate Elasticsearch as a materialized view)

Elasticsearch Advantages

Easy to horizontally scale with index sharding

Rich search API

Query for analytical data using aggregations

Elasticsearch Disadvantages

Indexes are created with a predefined number of shards. More shards requires migration to a new index. Usually done with ReIndex API

Performance issues when indices hit very large scale (> 1TB)

Example Elasticsearch use cases at Wix

Wix Search

Wix search allows searching inside any Wix site (out of the box) by having a generic API for indexing per site and installed apps.

It also provides custom search services for creating and managing self-owned indices by other Wix teams (basically serves as a proxy to Elasticsearch).

5. Amazon Simple Storage Service (S3)

Other DBs in this category

Google Cloud Storage, Azure Blob Storage

S3 Stored file URL

When to use S3

When you need to store large binary objects/files (up to 5TB each)

When the amount of data you need to store is large (>10TB), continues to grow daily, and may need to be retrieved (can’t be deleted)

S3 Advantages

Supports very high throughput

Infinite scalability — No limit on amount of storage

S3 Disadvantages

No Query support, only key-based retrieval

Latency is 100–200 ms for small objects. Caching can alleviate this

Example S3 use cases at Wix

A. Wix Media

Wix Media manages all the media files that are uploaded by Wix site owners. These files are stored, and then accessed when the user’s site gets a visitor.

In order to have highest possible availability and lowest possible retrieval latency, the media files are stored and replicated between both Amazon S3 and Google GCS and cross-region within the same cloud provider.

A dedicated process moves files to cheaper storage classes (e.g. from S3 Standard to S3 Standard-IA) once it finds out they haven’t been accessed for a long time. for more information you can read — Storage Lifecycle: How We Cut Cost on 55% of Our Storage Without Deleting a Single File

B. Wix Editor

The Wix Editor backend team is in charge of (among other responsibilities) storage and fast retrieval of all Wix site pages during site editing.

A site page is actually a JSON file. Every change made to a site page means a new immutable JSON file is inserted to the database.

A couple of years ago they migrated the site pages from MySQL to S3. The main reason was that S3 guaranteed infinite scale (MySQL was unable to handle a scale >10TB well). The S3 latency they measured for read/write ranged from 30 ms on median case to 400ms on p99. In order to improve this they added a write-through caching layer on top of S3.