use best practices when using subprocess.Popen.  Also fix a bug in hdfs.listdir.

- whenever using subprocess.Popen, use close_fds=True
- make sure to catch errors in hdfs listdir
  (and make some minor improvements there)
- add one FIXME question

Most of the changes are simply adding "close_fds=True" to calls to subprocess.Popen.
However, I found a bug in hdfs.listdir wherein if the hadoop process failed, luigi did not notice (because it didn't check the return code of the process).  A cursory examination led me to believe that this was the only place where the returncode was not checked explicitly, or the nice wrapper (call_check) was not used.

I also added a FIXME, mostly as a reminder to see if using the .communicate() method might not be better in that particular case. That issue should probably be resolved before merging.