Seagull: A Distributed, Fault Tolerant, Concurrent Task Runner - Sagar Patwardhan, Yelp Inc.
Channel: The Linux Foundation
Date: 2017-10-31

Seagull: A Distributed, Fault Tolerant, Concurrent Task Runner - Sagar Patwardhan, Yelp Inc.

At Yelp, we all strive to increase developer productivity by decreasing the time to test, deploy, and monitor changes. To enable developers to push code safely, we run more than 20 million tests every day. Yelp has a monolith web application which has 100,000 tests; running them sequentially takes approximately 2 days to finish. We built an in-house distributed system called Seagull, which splits these 100,000 tests into smaller chunks(bundles) using our bespoke algorithm and runs these chunks in parallel on the compute cluster to finish all the tests in less than 10 mins. Seagull uses Apache Mesos to schedule these run test bundles on AWS spotfleet. We have written a custom autoscaler for AWS spotfleet which dynamically adjusts the cluster capacity based on different utilization metrics for optimal use of resources. Seagull runs more than 2 million ephemeral docker containers every day.

About Sagar Patwardhan
My name is Sagar Patwardhan and I am a software engineer at Yelp Inc. I am part of the distributed systems(Infrastructure) team. I have been with Yelp for a little over two years; I am currently working on building mesos infrastructure and next-gen batch processing infrastructure at Yelp. My previous projects include an asynchronous processing system called kew, which is used for processing all of Yelp's emails, push notifications and online transactions in real time. I also worked on our distributed, fault tolerant, concurrent task executor called seagull, which is used for running millions of tests every day in a cost effective manner. I got my M.S. in computer science(specialized in distributed systems and networks) from USC. I was part of the networked system lab where I did research in TCP. I got my B.E. in computers from University of Pune, India.