It needs to be included whenever a JobConf that is going to be passed to something that would need the tokens (ie tries to access HDFS from a hadoop container - like the AM or workers). In some of the cases it creates a new JobConf right there, in other cases like the hadoopRDD ithe jobConf is passed in by the user so we just need to make sure its added. I'll see if I can move to a common place. 

 It doesn't hurt being there for hadoop1 or hadoop2 and the deployment shouldn't matter, generally in those cases the credentials will just be empty, but since some versions don't have the api I'll conditionalize that out and I'll see if I can make this cleaner.