[SPARK-1860] Fixing broken unit test.

FileUtils.listFiles from apache commons does not list directories. Use
FileUtils.listFilesAndDirs instead.

Also reorganizing a few imports and style changes from pull request.