What is Sharding, Partitioning and Why Sharding Mangers
By: Sampath Kumar
Tags: MySQL, Database, Partitioning

Firstly, what is Sharding?A database shard, or simply a shard, is a horizontal partition of data in a database or search engine. Each shard is held on a separate database server instance, to spread load.

Now, some of you who are into System Design or not so much into Databases Performance might confuse this with Partitioning. So, how is Sharding different from Partitioning? For performance, maintainability and access the data in data bases are usually separated into logical sets to be stored in same or different machine other than host machine. This is Partitioning.

Say in an E- commerce, we have a table named “Product” which has details like title, desc, cost, specification, image links. Now say this table is also having details like product dimensions, weight, storage space required, isGlassware which are required for transportations but not so often a user looks for.

Product Table [ID, Title, Desc, Cost, Specs, Image link1,…..Length, Width, Height, Weight, isGlassWare, isFood, hasExpiryDate] This table can be partitioned into two, Product Table1 [ID, Title, Desc, Cost, Specs, Image link1] Here Most queried information can be stored in on separate table, while less frequently information can be stored into another table Product Table2 [ID, Length, Width, Height, Weight, isGlassWare, isFood, hasExpiryDate].

This is called Vertical Partitioning.

Another ways to partition this information is to split the data based on the rows. Say we have another columns in Product Table called “Type”, which can tell you what type of products we have like electronic, food, clothes and so on.

Product Table [ID, Title, Desc, Cost, Specs, Image link1,…..Length, Width, Height, Weight, isGlassWare, isFood, hasExpiryDate] (partition1-electronics) Product Table [ID, Title, Desc, Cost, Specs, Image link1,…..Length, Width, Height, Weight, isGlassWare, isFood, hasExpiryDate] (partition1-food) Product Table [ID, Title, Desc, Cost, Specs, Image link1,…..Length, Width, Height, Weight, isGlassWare, isFood, hasExpiryDate] (partition1-clothes)

This is called Horizontal Partitioning or Sharding.

Although the both the logics improve the query performances these comes with following challenges. Here are two simple challenges

Say one these partition is not updated/failed to update a record. Say we have query that required all information in all the partitions. Data is skew, only one partition’s data has 80% data

If we consider the CAP theorem too, we have a whole set of challenges. So we circle back to our first question — How to improve Database query performances? A simple answer is go for data base clustering system like Vitess.io and go for eventual consistency.

(This eventual consistency, might sound like drawback, but in real world this could be as low as 0.0005 second. Compare this to table which is never partitioned, this is absolutely a big performance win.)

Cluster Mangers Vitess.io is used in Youtube, Slack, Github and other for managing the data into MySql.

Source: https://vitess.io/

Above is a design diagram of how Vitess works. “vtctld” acts like a metadata repository, “vtgate” acts as loadbalencer and “vttablet” acts as actual mysql data node. All the sql queries are optimised according the data distributions at “vtgates” and queries are run accordingly.

Best part is from a developer point of view, the data might look like is in a single table.

To learn more about vitess.io