Port Spark plugin to latest plugin API

Change hdfs user to hadoop

Change-Id: I58ba4541dbae0a583315fc59452687676430c35a