Repository: getredash/redash

Issue:
Cancel doesnt actually cancel a query in redshift.
I was using redash and wrote a pretty bad query. Soon after I hit cancel and proceeded to run other queries but they wouldn't go through. I suspected redshift to be busy running my previous erroneous query so logged into the aws console and sure enough it was running. I had to terminate the query manually there.



Discussion:
Cancel is implemented by terminating the process that is running the query. I've noticed that it sometimes takes time for Celery (the background workers framework) to process the cancel request. How much time did it pass between when you cancelled in re:dash and when you checked the console?

Hmm it had to be around 10 to 15 min since I let it run for a while as I worked on other things.

We've had a similar issue with Redshift while using redash. I think the problem is that terminating the process doesn't actually kill the query in Redshift. You need to send a cancel request to Redshift by sending the INT signal to the process.

To test this, I fired off a query that I knew would take a long time in psql. In another window, I killed it with TERM. When I checked running queries, that query was still running, even though the psql process had been killed. I then tried the same thing but sent an INT signal. This caused a cancel request to be sent and the query was cancelled.

The easy way to do this would be to add `signal=INT` to the [revoke method](https://github.com/EverythingMe/redash/blob/5d1c75df1cc22978b4fb6e9e56ef5492a76e6447/redash/tasks.py#L135). However, I don't know enough about how that interacts with non-Redshift tasks. I'm happy to help implement this if needed.

@alexdebrie is correct, the Postgres/Redshift adapter sends cancel request when it receives the `INT` signal. I was under the impression that when revoking a task it will send `INT` signal, but from looking at Celery documentation it looks like the default is indeed `TERM`.

I'll double check this again tomorrow, and fix (and test with other adapters) accordingly.

Thanks for the debugging, @alexdebrie !

No problem, thanks for the awesome project!

No, thank You!

Regards,
Anthony

On 20 October 2015 at 22:30, Alex DeBrie notifications@github.com wrote:

> No problem, thanks for the awesome project!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/EverythingMe/redash/issues/598#issuecomment-149693063
> .

## 

Anthony Kalinde
Big Data Engineer
_DharmicData.com_

skype:       anthonykalinde

mobile:     +47 90 22 5857

linkedin:   no.linkedin.com/in/anthonykalinde

@arikfr -- Just checking to see if you've had a chance to test this with other adapters. We like to stay in sync with official redash repo as much as possible, but we'll just make this fix in our repo if changing it will break other runners. 

Started testing it now, and even when I use the INT signal (`self._async_result.revoke(terminate=True, signal="SIGINT"`), I see that it just kills the process but doesn't call my code that tries to catch the INT signal.

Did you have the chance to try it out?

Ahh darn. I haven't tried it yet. I should have some time later this week, and I'll let you know what I find.


Resolving patch:
diff --git a/redash/query_runner/__init__.py b/redash/query_runner/__init__.py
--- a/redash/query_runner/__init__.py
+++ b/redash/query_runner/__init__.py
@@ -9,6 +9,7 @@
 __all__ = [
     'ValidationError',
     'BaseQueryRunner',
+    'InterruptException',
     'TYPE_DATETIME',
     'TYPE_BOOLEAN',
     'TYPE_INTEGER',
@@ -38,6 +39,9 @@
     TYPE_DATE
 ])
 
+class InterruptException(Exception):
+    pass
+
 class BaseQueryRunner(object):
     def __init__(self, configuration):
         jsonschema.validate(configuration, self.configuration_schema())
diff --git a/redash/query_runner/pg.py b/redash/query_runner/pg.py
--- a/redash/query_runner/pg.py
+++ b/redash/query_runner/pg.py
@@ -142,7 +142,7 @@ def run_query(self, query):
             logging.exception(e)
             error = e.message
             json_data = None
-        except KeyboardInterrupt:
+        except (KeyboardInterrupt, InterruptException):
             connection.cancel()
             error = "Query cancelled by user."
             json_data = None
diff --git a/redash/tasks.py b/redash/tasks.py
--- a/redash/tasks.py
+++ b/redash/tasks.py
@@ -1,5 +1,6 @@
 import time
 import logging
+import signal
 from flask.ext.mail import Message
 import redis
 from celery import Task
@@ -8,7 +9,7 @@
 from redash import redis_connection, models, statsd_client, settings, utils, mail
 from redash.utils import gen_query_hash
 from redash.worker import celery
-from redash.query_runner import get_query_runner
+from redash.query_runner import get_query_runner, InterruptException
 
 logger = get_task_logger(__name__)
 
@@ -132,7 +133,7 @@ def ready(self):
         return self._async_result.ready()
 
     def cancel(self):
-        return self._async_result.revoke(terminate=True)
+        return self._async_result.revoke(terminate=True, signal='SIGINT')
 
     @staticmethod
     def _job_lock_id(query_hash, data_source_id):
@@ -263,9 +264,12 @@ def check_alerts_for_query(self, query_id):
 
                 mail.send(message)
 
+def signal_handler(*args):
+    raise InterruptException
 
 @celery.task(bind=True, base=BaseTask, track_started=True)
 def execute_query(self, query, data_source_id, metadata):
+    signal.signal(signal.SIGINT, signal_handler)
     start_time = time.time()
 
     logger.info("Loading data source (%d)...", data_source_id)


Test patch: