Wait, so are you sure it's a System.setProperty from Spark that's causing this? I didn't see that. Now that I look at the code, it seems we start task/dagScheduler.start() and ui.start() before this block -- that could be a problem. Would creating this hadoopConf variable at the start of SparkContext help? Or are you still worried that other code in totally different places will set properties?

It's just weird to have catch(ConcurrentModificationException) as a common case, that's why I'm trying to avoid this.