Alexander Lourenco - Realtime Distributed Computing At Scale: Storm And Streamparse
Channel: EuroPython Conference
Date: 2017-10-13

"Realtime Distributed Computing At Scale (in pure Python!): Storm And Streamparse
[EuroPython 2017 - Talk - 2017-07-12 - Arengo]
[Rimini, Italy]

Realtime distributed computing is tough, especially at scale: managing a large data pipeline is tough, and it’s even tougher to keep latency low and availability high when processing tens of thousands of items per second. Many people turn in despair to Java or Scala when it comes time to scale up, but we can do it in Python: Apache Storm is a distributed realtime computation system that can let you scale up- and no need to reach for a new language!

This talk will walk the audience through the basics of Apache Storm and how it’s an elegant, useful solution to realtime distributed computing,  as well as how streamparse can let you write your storm components in Python by writing some code and a basic storm topology in Python. We’ll also look at how Parsely uses Storm in production to handle billions of realtime events a month. If we have time, we’ll go a bit into how Storm has several advantages over other common Python computing data streaming solutions, like Spark’s microbatching.

Goals:

At the end of the talk, ideally you should be able to understand:
What Apache Storm is, how it works generally, and what scenarios it’s useful for
How streamparse can be used to write your Storm topologies
How Storm + streamparse is used in an actual high-availability, low-latency production environment

License: This video is licensed under the CC BY-NC-SA 3.0 license: https://creativecommons.org/licenses/by-nc-sa/3.0/
Please see our speaker release agreement for details: https://ep2017.europython.eu/en/speaker-release-agreement/