Repository: piskvorky/smart_open (python)

Issue:
Bug (maybe feature) in HdfsOpenRead
On line 249 I have noticed a minor bug in reading from HDFS

There is used command `hadoop fs -cat` which based on the documentation http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html reads any file mounted to Hadoop (HDFS, FS, HFTP FS, S3 FS, and others)

Correctly to meet what smart_open claims to do there should be used `hdfs dfs -cat`.
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfs

I can do a PR with fix if this is really a bug.

If this is expected behaviour then I would change the documentation + I would not limit files schema to only HDFS.

Pull request description:
Hdfs __iter__ implementation fix
Fixed HDFS read to be able to read both `hdfs://` and `hdfs:///` (both should be equivalent and it depends on hdfs distribution)

Also fixed to use `hdfs dfs -cat` which is recommended.

Closes #24 

Patch:
diff --git a/smart_open/smart_open_lib.py b/smart_open/smart_open_lib.py
index f2a944b..b21704d 100644
--- a/smart_open/smart_open_lib.py
+++ b/smart_open/smart_open_lib.py
@@ -246,7 +246,7 @@ class HdfsOpenRead(object):
         self.parsed_uri = parsed_uri
 
     def __iter__(self):
-        hdfs = subprocess.Popen(["hadoop", "fs", "-cat", self.parsed_uri.uri_path], stdout=subprocess.PIPE)
+        hdfs = subprocess.Popen(["hdfs", "dfs", "-cat", os.path.join("/", self.parsed_uri.uri_path)], stdout=subprocess.PIPE)
         return hdfs.stdout
 
     def read(self, size=None):
@@ -387,7 +387,7 @@ class S3OpenWrite(object):
         if type is not None:
             self._termination_error()
             return False
-        
+
         try:
             self.close()
         except:


Test patch:
diff --git a/smart_open/tests/test_smart_open.py b/smart_open/tests/test_smart_open.py
index 8297027..58eb2c8 100644
--- a/smart_open/tests/test_smart_open.py
+++ b/smart_open/tests/test_smart_open.py
@@ -102,8 +102,12 @@ class SmartOpenReadTest(unittest.TestCase):
         smart_open_object = smart_open.HdfsOpenRead(smart_open.ParseUri("hdfs:///tmp/test.txt"))
         smart_open_object.__iter__()
         # called with the correct params?
-        mock_subprocess.Popen.assert_called_with(["hadoop", "fs", "-cat", "/tmp/test.txt"], stdout=mock_subprocess.PIPE)
+        mock_subprocess.Popen.assert_called_with(["hdfs", "dfs", "-cat", "/tmp/test.txt"], stdout=mock_subprocess.PIPE)
 
+        # second possibility of schema
+        smart_open_object = smart_open.HdfsOpenRead(smart_open.ParseUri("hdfs://tmp/test.txt"))
+        smart_open_object.__iter__()
+        mock_subprocess.Popen.assert_called_with(["hdfs", "dfs", "-cat", "/tmp/test.txt"], stdout=mock_subprocess.PIPE)
 
     @mock.patch('smart_open.smart_open_lib.boto')
     @mock.patch('smart_open.smart_open_lib.s3_iter_lines')