Repository: spotify/luigi

Issue:
Task finished yet the luigi failed.
I have py script contains the following task

class UpdateUserParameter(luigi.Task):
    udt = luigi.DateParameter()
    subject=luigi.Parameter()

```
def output(self):
    return luigi.LocalTarget(local_proj_dir+'/data/crontab/{sub}/{dts}/is_user_parameter_updated.txt'.format(sub=self.subject, dts=self.udt.strftime('%Y-%m-%d')))
def requires(self):
    return [users.UpdateUserParameterChain(subject=self.subject, udt=self.udt),
            users.UpdateUserKnowledgePointMasteryBuffer(subject=self.subject, udt=self.udt)
        ]
```

When the task runs, all task output is produced (including the master task) but the master task returns the following error:

Traceback (most recent call last):
  File "run_script.py", line 91, in <module>
    luigi.run()
  File "/usr/local/lib/python2.7/site-packages/luigi/interface.py", line 210, in run
    return _run(_args, *_kwargs)['success']
  File "/usr/local/lib/python2.7/site-packages/luigi/interface.py", line 238, in _run
    return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)
  File "/usr/local/lib/python2.7/site-packages/luigi/interface.py", line 198, in _schedule_and_run
    logger.info(execution_summary.summary(worker))
  File "/usr/local/lib/python2.7/site-packages/luigi/execution_summary.py", line 384, in summary
    return _summary_wrap(_summary_format(_summary_dict(worker), worker))
  File "/usr/local/lib/python2.7/site-packages/luigi/execution_summary.py", line 333, in _summary_format
    str_output += '{0}\n'.format(_get_str(group_tasks[status], status in _PENDING_SUB_STATUSES))
  File "/usr/local/lib/python2.7/site-packages/luigi/execution_summary.py", line 132, in _get_str
    params = _get_set_of_params(tasks)
  File "/usr/local/lib/python2.7/site-packages/luigi/execution_summary.py", line 173, in _get_set_of_params
    params[param] = {getattr(task, param[0]) for task in tasks}
  File "/usr/local/lib/python2.7/site-packages/luigi/execution_summary.py", line 173, in <setcomp>
    params[param] = {getattr(task, param[0]) for task in tasks}
TypeError: unhashable type: 'dict'

Because this error, other task depends on this cluster cannot be started. I cannot reliably reproduce this error and wonder how to fix it.



Discussion:
I am seeing this same TypeError from execution_summary.py. (I have a bunch of Tracebacks much like @junchenfeng's OP) I'm not sure when this first appeared, but based on the commit history for that file, my best guess is it was something introduced here: https://github.com/spotify/luigi/commit/3b249492a4cf203a988b47545b72425bbbb2a68f - the next commit previous was back in March and I think I would have noticed it before a few days ago.

For my purposes, this is just annoying, but doesn't really break anything. I believe this code only prints out a human-readable summary status at the end of a workflow. All tasks still get marked as DONE by the scheduler in appropriate ways.

I'll see if there's anything I might be able to contribute here by way of a PR to fix, but at the moment I must admit I don't quite understand the code here or what might be going wrong exactly, so this may well be a job for someone more skilled than myself. I am checking it out, though.

@tym-oao Cool. if you can find a minimal self-contained example (and post an easier-to read error report than this issue). I might at least guide you to how to fix the issue.

@Tarrasch This took a few tries, but at this point I believe the issue with `execution_summary.py` occurs whenever a workflow invokes a Task that uses a `DictParameter`. This would also explain the timing better than the commit I referred to in my previous comment — I simply wasn't using the `DictParameter` type in my code until I made some changes recently. Here's a very simple case:

```
import luigi


class JobA(luigi.Task):
    fdict = luigi.DictParameter(default={'foo': 'bar'})
    def run(self):
        with self.output().open('w') as out:
            out.write(self.fdict['foo'])

    def output(self):
        return luigi.LocalTarget('/tmp/outfile')
```

Which results in:

```
Traceback (most recent call last):
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/retcodes.py", line 61, in run_with_retcodes
    worker = luigi.interface._run(argv)['worker']
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/interface.py", line 238, in _run
    return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/interface.py", line 198, in _schedule_and_run
    logger.info(execution_summary.summary(worker))
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/execution_summary.py", line 388, in summary
    return _summary_wrap(_summary_format(_summary_dict(worker), worker))
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/execution_summary.py", line 337, in _summary_format
    str_output += '{0}\n'.format(_get_str(group_tasks[status], status in _PENDING_SUB_STATUSES))
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/execution_summary.py", line 123, in _get_str
    attributes = {getattr(task, tasks[0].get_params()[0][0]) for task in tasks}
  File "/Users/thomas/.virtualenvs/tmp-3d4605015530c32/lib/python3.5/site-packages/luigi/execution_summary.py", line 123, in <setcomp>
    attributes = {getattr(task, tasks[0].get_params()[0][0]) for task in tasks}
TypeError: unhashable type: 'dict'
```

As I mentioned before, the actual task runs as expected, output is written correctly, any upstream dependencies would presumably fire as expected, etc. Only the Execution Summary at the end of the log output fails to print, replaced with the Traceback immediately above.

Hope that helps.

I just encountered exactly same problem... 
Here are my codes

```
paras = {}

outfile = {'outfile': 'word.txt'}

paras['SimpleTask'] = outfile


class SimpleTask(luigi.Task):

    my_paras = luigi.DictParameter()

    def output(self):
        st_para = self.my_paras['SimpleTask']
        tar_file = st_para['outfile']
        return luigi.LocalTarget(tar_file)

    def run(self):
        words = [
            'apple',
            'banana',
            'grapefruit'
        ]
        with self.output().open('w') as f:
            for word in words:
                f.write('{word}\n'.format(word=word))


class DecoratedTask(luigi.Task):

    def requires(self):
        return SimpleTask(my_paras=paras)

    def output(self):
        return luigi.LocalTarget('letter_counts.txt')

    def run(self):

        # read in file as list
        with self.input().open('r') as infile:
            words = infile.read().splitlines()

        # write each word to output file with its corresponding letter count
        with self.output().open('w') as outfile:
            for word in words:
                outfile.write(
                        '{word} | {letter_count}\n'.format(
                            word=word,
                            letter_count=len(word)
                            )
                        )
if __name__ == '__main__':
    luigi.run(["--local-scheduler"], main_task_cls=DecoratedTask)
```

And the other question is how I can put 'paras' in luigi.run, rather than put it in "class DecoratedTask", Thanks!

@tym-oao. Ok I see the problem and solution. We use a FrozenOrderedDict which thankfully _is_ hashable. But the problem is when we have default values that are normal dict. Like `DictParameter(default=dict(blah="foo"))`. One way to fix this:
1. Add test case in execution summary code
2. Run test and see it fails as expected
3. (the fix): Go to this code https://github.com/spotify/luigi/blob/31d9d81291ac697721594856613a653b5c2f6162/luigi/parameter.py#L747-L803, To `DictParameter` class add a `normalize` method. Ensuring that it's converts it's input to a value of type `FrozenOrderedDict`. Then the parameter-values will actually be hashable.
4. Run tests and hopefully it passes.

@tym-oao. Do you want to send a PR? :)

I am a bit confused since my task does not (explicitly) contain DictParameter, unless DateParameter is a subclass of DictParameter.

@junchenfeng, oh that's interesting. Maybe it's a separate bug from @tym-oao. Or perhaps the dependencies somewhere down the dependency tree has tasks with DictParameters?

The culprit is ExternalHiveTask() whose partition parameter is a dict. Curiously, when I resort to a hand written ExternalTask, it is OK. 

I guess the problem is specific to the require method.

Thanks for digging in to this @junchenfeng. Do we have any volunteer to fix this (relatively serious) issue?

Hey @Tarrasch I'll see if I can handle this over the weekend.  Do you think what @tym-oao posted would serve as a good basis for a failing test case?

@nugend, kindof. It's even simpler, just copy one of the tests from `execution_summary_test.py` and use a DictParameter somewhere. I think that's it.


Resolving patch:
diff --git a/luigi/parameter.py b/luigi/parameter.py
--- a/luigi/parameter.py
+++ b/luigi/parameter.py
@@ -786,6 +786,12 @@ def default(self, obj):
                 return obj.get_wrapped()
             return json.JSONEncoder.default(self, obj)
 
+    def normalize(self, value):
+        """
+        Ensure that dictionary parameter is converted to a FrozenOrderedDict so it can be hashed.
+        """
+        return FrozenOrderedDict(value)
+
     def parse(self, s):
         """
         Parses an immutable and ordered ``dict`` from a JSON string using standard JSON library.


Test patch:
diff --git a/test/execution_summary_test.py b/test/execution_summary_test.py
--- a/test/execution_summary_test.py
+++ b/test/execution_summary_test.py
@@ -932,3 +932,56 @@ def requires(self):
         self.run_task(Foo())
         s = self.summary()
         self.assertIn('yellow', s)
+
+    def test_with_dict_dependency(self):
+        """ Just test that it doesn't crash with dict params in dependencies """
+
+        args = dict(start=datetime.date(1998, 3, 23), num=3)
+
+        class Bar(RunOnceTask):
+            args = luigi.DictParameter()
+
+        class Foo(luigi.Task):
+            def requires(self):
+                for i in range(10):
+                    new_dict = args.copy()
+                    new_dict['start'] = str(new_dict['start'] + datetime.timedelta(days=i))
+                    yield Bar(args=new_dict)
+
+        self.run_task(Foo())
+        d = self.summary_dict()
+        exp_set = set()
+        for i in range(10):
+            new_dict = args.copy()
+            new_dict['start'] = str(new_dict['start'] + datetime.timedelta(days=i))
+            exp_set.add(Bar(new_dict))
+        exp_set.add(Foo())
+        self.assertEqual(exp_set, d['completed'])
+        s = self.summary()
+        self.assertIn('"num": 3', s)
+        self.assertIn('"start": "1998-0', s)
+        self.assertIn('Scheduled 11 tasks', s)
+        self.assertIn('Luigi Execution Summary', s)
+        self.assertNotIn('00:00:00', s)
+        self.assertNotIn('\n\n\n', s)
+
+    def test_with_dict_argument(self):
+        """ Just test that it doesn't crash with dict params """
+
+        args = dict(start=str(datetime.date(1998, 3, 23)), num=3)
+
+        class Bar(RunOnceTask):
+            args = luigi.DictParameter()
+
+        self.run_task(Bar(args=args))
+        d = self.summary_dict()
+        exp_set = set()
+        exp_set.add(Bar(args=args))
+        self.assertEqual(exp_set, d['completed'])
+        s = self.summary()
+        self.assertIn('"num": 3', s)
+        self.assertIn('"start": "1998-0', s)
+        self.assertIn('Scheduled 1 task', s)
+        self.assertIn('Luigi Execution Summary', s)
+        self.assertNotIn('00:00:00', s)
+        self.assertNotIn('\n\n\n', s)