There might be a performance problem here.

Spark actually uses broadcast variable to broadcast the JobConf in HadoopRDD to avoid having that in the task closure (10KB). If every worker has to send a message back to the master, it might slow things down since every HadoopRDD we create will need to do that ... 

Ideally, we should track the memory usage of broadcast variables, but I am not sure what the best way to do this is.