[AE-727] Support multiple version of HADOOP
[AE-727] remove work folder in /opt/spark/, it looks like this is no longer necessary for spark 1.1