Shopify/spark — pull request #2 by dklassen (2014-01-23 02:02:49+00:00)
Title: Config hdfs

pyspark looks in spark/conf for core-site.xml when setting up HDFS access. When running spark locally this causes an issue with spark trying to look for an input file based on the fs.defaultFS property set in core-site.xml. The PR sets HADOOP_CONF_DIR to point to the configs on dn05.chi.shopify.com when MASTER is not set.

@hornairs @tyro89 a small change.