Building The Analytics Team At Wish Part 2 — Scaling Data Engineering
By: Samson Hu
Tags: Big Data, Data Science, Ecommerce, Engineering, Analytics

Now that the infrastructure is rebuilt, in part 2 we’ll talk about how to build and manage a high performance data engineering team.

Setting a Vision for the Team

Data engineering exists on a spectrum between two poles.

On one end is the traditional data engineering team, where the goal is to build and own the data pipelines that data analysts and data scientists use to output data. On these teams, the majority of work is spent helping people access data, and producing new pipelines if needed.

While this works, the main drawback is that it’s easy for this type of team to become a cog in the system. The work can become monotonous, and visibility into wins are low. Generally, it will be hard to recruit and retain strong talent.

On the opposite end of the spectrum, data engineering teams build and maintain platforms and frameworks where anyone can write and run their own data pipelines, and they themselves do no pipeline work. This allows analysts and data scientists to own their work end to end. You can read more about this philosophy here.

This works best if you want to optimize for recruiting top talent. The best engineers want to work on scaling systems and platforms, and this setup offers plenty of technical challenges. This also greatly increases the productivity of top analysts, where they no longer have any external dependencies to produce their work.

But there’s two drawbacks:

There’s an explosion of tables since it’s often easier writing new pipelines than digging up and auditing old ones. This leads to inconsistent metrics across reports & analysis. This also minimizes the re-usability of pipelines. The infrastructure costs required to support this is much higher. ETL jobs written by analysts aren’t always optimal based on query performance. So capacity needs to be upped significantly to buffer this.

Unless there’s existing factors that call for choosing one of the two extremes, the optimal solution is somewhere in the middle.

Data engineering should build a system that allows analysts to produce their own pipelines. But they should also be the ones building core event and dimension tables on top of each system in the company.

These system tables should have a high degree of quality and trust, and should form the foundation of ETL tasks built out by the rest of the company. Data engineering should set SLAs for these tables, and cover them with tests and monitoring.

Data engineering should also handle the operational tasks needed to maintain an open data platform.

As non system tables grow, data engineering should also be on the lookup for refactoring opportunities. Unless there’s unlimited compute and storage resources for data, inefficient data models can be detrimental to the system.

And lastly, data engineering should always be on the lookup for systems improvements. The goal should be to increase analyst productivity. For example, if analysts are repeatedly calculating week over week metrics for every metric they product, then data engineering can build out a library or process to make this automatic and save time per ETL build task.

Building The ETL Framework