Unfortunately the SparkHadoopUtil.newConfiguration gives you a YarnConfiguration not a jobConf. The jobConf is used for mapreduce specific things like FileInputFormat where as the YarnConfiguration is generic for Yarn. Both are based on Configuration though.    I'll look at exactly how that is used to see if it really needs to be YarnConfiguration. 

otherwise as you mentioned I was looking at added a function to SparkHadoopUtil to add in the credentials only for the hadoop2-yarn profile.