Challenges when Scaling: Adventures in Swift's Sharding
Channel: Linux.conf.au 2016 -- Geelong, Australia
Date: 2016-02-06

Matthew Oliver
https://linux.conf.au/schedule/30123/view_talk
Swift is an open-source, highly available, distributed, eventually-consistent object storage system.  As the project has matured, so has the need to store more and more data. Objects in swift are tracked in the containers they are placed and a container in essence is a SQLite database which is distributed throughout the cluster. As the number of objects in a container increases, so does latency meaning larger sites have moved to using expensive solid state drives (SSDs) for container storage, but there has to be a better way we can solve this in software, right?

For this presentation we'll cover the design decisions that have enabled Swift's success, and then show how these same decisions need to be revisited as we cater for the ever increasing size of containers.  We'll consider a proof of concept container sharding implementation that attempts to address a few of these limitations, allowing for larger datasets to be deployed on cheaper commodity hardware, splitting a container database over many databases and nodes. 
We won't just present the final solution, but we'll iterate over the different approaches tried, so that you can learn from our mistakes and lessons learnt.