Simulating Our Apache Mesos Framework for Fun and Profit and...
Channel: The Linux Foundation
Date: 21/09/2017

Simulating Our Apache Mesos Framework for Fun and Profit and... - Wil Yegelwel, TwoSigma

At Two Sigma, we use Cook, our open source batch scheduling mesos framework, to run millions of compute-hours of work for hundreds of users every day. This year, we are working on changing the scheduler optimization algorithm, but we want some confidence that it will actually improve utilization and the user experience (latency). Therefore, we built a mesos simulator which lets us test algorithm changes, without running the whole distributed system, and in doing so, found some subtle bugs and performance bottlenecks, and highlighted areas needing better test coverage.

In this talk, Wil will describe how we built the simulator for cook, some of the challenges in building it, and share the insights we learned by running our historical job traces through it. We will highlight the value and practicality of simulation testing for real, production systems.

Wil Yegelwel
Software Developer, TwoSigma
I'm a developer at Two Sigma Investments working on Distributed System and Analysis tools. Previously, I was a student at Brown University studying Computer Science and Applied Math.