Excellent question @rahulsinghaliitd is all related with the _"instability"_ of the YARN API and the different versions people are using. See https://issues.apache.org/jira/browse/SPARK-1233