Repository: spotify/luigi

Issue:
Exception when a command-line parametter contains a non-ascii character
Hey,

We've just upgraded to Luigi 1.1.2, and now we get an error when launching Tasks that have utf-8 characters on the parameters.  (using python 2.7.3)

Here's the stacktrace we get : 

```
Traceback (most recent call last):
  File "examples/top_artists.py", line 283, in <module>
    luigi.run()
  File "/home/jonas/.virtualenvs/website/local/lib/python2.7/site-packages/luigi/interface.py", line 434, in run
    return interface.run(tasks, worker_scheduler_factory, override_defaults=override_defaults)
  File "/home/jonas/.virtualenvs/website/local/lib/python2.7/site-packages/luigi/interface.py", line 165, in run
    not(lock.acquire_for(env_params.lock_pid_dir, env_params.lock_size))):
  File "/home/jonas/.virtualenvs/website/local/lib/python2.7/site-packages/luigi/lock.py", line 59, in acquire_for
    my_pid, my_cmd, pid_file = get_info(pid_dir)
  File "/home/jonas/.virtualenvs/website/local/lib/python2.7/site-packages/luigi/lock.py", line 44, in get_info
    pid_file = os.path.join(pid_dir, hashlib.md5(my_cmd.encode('utf8')).hexdigest()) + '.pid'
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 97: ordinal not in range(128)
```

Here I've just added a `plop = luigi.Parameter()` to `AggregateArtists`, and called it with 
`python examples/top_artists.py AggregateArtists --local-scheduler --date-interval 2012-06 --plop à`

The error seems to come from the [Python3 refactoring](https://github.com/spotify/luigi/pull/745). Removing `.encode('utf8')` on [this line](https://github.com/spotify/luigi/blob/master/luigi/lock.py#L44) seems to solve the problem, but I guess it was added for a reason ^^.

Any ideas of how we could solve this on our side, other than removing the encode?  (I'll try to add a test for it later)



Discussion:
@gpoulin, do you know the why of this? :)

i would argue utf-8 support on the command line is one of those things we can live without. i don't see why it's needed

but it's probably an easy fix.

@Tarrasch On python3, `hashlib.md5` need `bytes` and not `str` (which make sense). I'll fix that.


Resolving patch:
diff --git a/luigi/lock.py b/luigi/lock.py
--- a/luigi/lock.py
+++ b/luigi/lock.py
@@ -24,6 +24,8 @@
 import hashlib
 import os
 
+from luigi import six
+
 
 def getpcmd(pid):
     """
@@ -32,16 +34,23 @@ def getpcmd(pid):
     :param pid:
     """
     cmd = 'ps -p %s -o command=' % (pid,)
-    p = os.popen(cmd, 'r')
-    return p.readline().strip()
+    with os.popen(cmd, 'r') as p:
+        return p.readline().strip()
 
 
-def get_info(pid_dir):
+def get_info(pid_dir, my_pid=None):
     # Check the name and pid of this process
-    my_pid = os.getpid()
+    if my_pid is None:
+        my_pid = os.getpid()
+
     my_cmd = getpcmd(my_pid)
 
-    pid_file = os.path.join(pid_dir, hashlib.md5(my_cmd.encode('utf8')).hexdigest()) + '.pid'
+    if six.PY3:
+        cmd_hash = my_cmd.encode('utf8')
+    else:
+        cmd_hash = my_cmd
+
+    pid_file = os.path.join(pid_dir, hashlib.md5(cmd_hash).hexdigest()) + '.pid'
 
     return my_pid, my_cmd, pid_file
 


Test patch:
diff --git a/test/lock_test.py b/test/lock_test.py
--- a/test/lock_test.py
+++ b/test/lock_test.py
@@ -45,9 +45,16 @@ def setUp(self):
         self.pid, self.cmd, self.pid_file = luigi.lock.get_info(self.pid_dir)
 
     def tearDown(self):
-        os.remove(self.pid_file)
+        if os.path.exists(self.pid_file):
+            os.remove(self.pid_file)
         os.rmdir(self.pid_dir)
 
+    def test_get_info(self):
+        p = subprocess.Popen(["yes", "à我ф"], stdout=subprocess.PIPE)
+        pid, cmd, pid_file = luigi.lock.get_info(self.pid_dir, p.pid)
+        p.kill()
+        self.assertEqual(cmd, 'yes à我ф')
+
     def test_acquiring_free_lock(self):
         acquired = luigi.lock.acquire_for(self.pid_dir)
         self.assertTrue(acquired)