apache/incubator-spark — pull request #332 by tgravescs (2014-01-05 20:53:06+00:00)
Title: Add Security to Spark - Akka, Http, ConnectionManager, UI use servlets

I need to finish some3 testing and convert the setting to use SparkConf still. I thought I would post to let people start looking at it.

We use a shared secret between all the components. On Yarn this secret is generated and distributed via the UserGroupInformation. In standalone mode its expected that you export the env variable SPARK_SECRET.

Akka - uses the remoting cookie
Http - uses Digest MD5
ConnectionManager - uses SASL
UI - the UI was changed to use servlets so that filters could be used. Yarn requires a specific AmIpFilter be installed for security to work properly. The filters can be used for many different purposes also. Someone could write one to filter through a companies normal login service for example. You can do compression through them, etc..

I've made some fairly major changes since the last posting of this. The SecurityManager was changed to be in the SparkEnv and used from their where possible. There are places this isn't possible since SparkEnv isn't created yet so it instantiates its own. I did this based on previous review comments and also it made adding security to yarn-client much easier.