Repository: spotify/luigi

Issue:
PR  #1137 yields SIGUSR1 AttributeError on Windows
PR #1137 put in place `signal.SIGUSR1` which breaks luigi on Windows machines. Replacing all instances of `SIGUSR1` with `SIGTERM` appears to fix the issue.

Specifically, the error generated:

```
(env) PS C:\Users\...> python .\hello_world.py
python : 'ps' is not recognized as an internal or external command,
At line:1 char:1
+ python .\hello_world.py
+ ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ('ps' is not rec...ternal command,:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

operable program or batch file.
Traceback (most recent call last):
  File ".\hello_world.py", line 12, in <module>
    luigi.run(['examples.HelloWorldTask', '--workers', '1', '--local-scheduler'])
  File "C:\Users\...\env\lib\site-packages\luigi\interface.py", line 207, in run
    return _run(*args, **kwargs)['success']
  File "C:\Users\...\env\lib\site-packages\luigi\interface.py", line 235, in _run
    return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)
  File "C:\Users\...\env\lib\site-packages\luigi\interface.py", line 185, in _schedule_and_run
    scheduler=sch, worker_processes=env_params.workers, assistant=env_params.assistant)
  File "C:\Users\...\env\lib\site-packages\luigi\interface.py", line 138, in create_worker
    scheduler=scheduler, worker_processes=worker_processes, assistant=assistant)
  File "C:\Users\...\env\lib\site-packages\luigi\worker.py", line 370, in __init__
    signal.signal(signal.SIGUSR1, self.handle_interrupt)
AttributeError: 'module' object has no attribute 'SIGUSR1'
```



Discussion:
can you send a PR

daniel do you want to do this? I'm happy to send a PR. Let me know.

@erikbern @danimcgoo - happy to; will submit in a min or two

@daveFNbuck you added the SIGUSR1 signal in 3800a986 – any issues changing it to SIGTERM?

Just submitted PR #1361  

If you use SIGTERM, then a vanilla kill command won't kill a worker anymore. Maybe this should be configurable and have a different default on windows? We'd also need to make sure to store the replaced SIGTERM handler and call it at the end of the handler in the worker, as various mapreduce-related contrib libraries also handle SIGTERM.

A quicker solution might be to just catch the exception and do nothing when registering the signal handler fails. I can put out a quick PR for that, as I'm not sure how to make this feature work in an acceptable way on windows.

yeah that sounds reasonable. or just throw SIGTERM on windows and SIGUSR1 on other platforms


Resolving patch:
diff --git a/luigi/worker.py b/luigi/worker.py
--- a/luigi/worker.py
+++ b/luigi/worker.py
@@ -367,7 +367,10 @@ def __init__(self, scheduler=None, worker_id=None, worker_processes=1, assistant
         self.run_succeeded = True
         self.unfulfilled_counts = collections.defaultdict(int)
 
-        signal.signal(signal.SIGUSR1, self.handle_interrupt)
+        try:
+            signal.signal(signal.SIGUSR1, self.handle_interrupt)
+        except AttributeError:
+            pass
 
         self._keep_alive_thread = KeepAliveThread(self._scheduler, self._id, self._config.ping_interval)
         self._keep_alive_thread.daemon = True


Test patch:
diff --git a/test/worker_test.py b/test/worker_test.py
--- a/test/worker_test.py
+++ b/test/worker_test.py
@@ -158,7 +158,10 @@ def test_stop_getting_new_work(self):
         self.w.add(d)
 
         self.assertFalse(d.complete())
-        self.w.handle_interrupt(signal.SIGUSR1, None)
+        try:
+            self.w.handle_interrupt(signal.SIGUSR1, None)
+        except AttributeError:
+            raise unittest.SkipTest('signal.SIGUSR1 not found on this system')
         self.w.run()
         self.assertFalse(d.complete())
 
@@ -355,6 +358,11 @@ def requires(self):
         self.assertTrue(self.w.add(B()))
         self.assertFalse(self.w.run())
 
+    def test_fails_registering_signal(self):
+        with mock.patch('luigi.worker.signal', spec=['signal']):
+            # mock will raise an attribute error getting signal.SIGUSR1
+            Worker()
+
     def test_allow_reschedule_with_many_missing_deps(self):
         class A(Task):