Repository: getsentry/sentry-python

Issue:
Support upcoming version of RQ
### Problem Statement

I'm [RQ](https://github.com/rq/rq)'s maintainer and I use Sentry in production. I plan on releasing a major improvement to RQ sometime in the next few weeks, but this upcoming change breaks Sentry's integration.


### Solution Brainstorm

This [pull request](https://github.com/rq/rq/pull/1964) introduces a change that causes Sentry's RQ integration to fail. From what I can tell, the failure is caused by [this change](https://github.com/rq/rq/pull/1964/commits/fadbb233d81c5c1a53bfcae7149b80888c1e7bdd).

Can we get Sentry to support this change prior to the next major release of RQ? I plan on releasing this sometime in the next few weeks.



Discussion:
Hey @selwin, thanks a lot for the heads up! We'll have to see if someone has time to squeeze this in this week, because if not, realistically speaking it's not likely that we will be able to look at this until after next week.
Thanks @sentrivana !
Hey @selwin !

Comming week we have our Sentry interal hackweek. (Where we stop our day to day work, and hack on other stuff)
So we will pick up the RQ issue again in the week of August 28th. Don't know when exactly we can work on this, because we have still other things in our pipeline that need finishing first.

When exactly is your release planned?
Hi @antonpirker, no worries. My schedule is currently quite packed so the next release will probably be a month or two from now. Plenty of time still :)
Related: https://github.com/getsentry/sentry-python/issues/2363
Hey @selwin !
The `executions` branch of RQ is the one I should test with, right? Just double checking. All the other branches do not seem to be as new as this one.
> The executions branch of RQ is the one I should test with, right? Just double checking. All the other branches do not seem to be as new as this one.

That's correct, thanks :).
Hey I was taking a look at this issue and working for a fix. 

@selwin  my understanding of the problem -  because of reversing the order of calling _handle_exception_ and _handle_job_failure_, this will raise an exception without setting the job as failed using  _job.set_status(JobStatus.FAILED,pipeline)_. does that make sense ?

@vagi8 thanks for investigating. If all it takes to make it work again is simply setting `job._status` to `FAILED`, this is something I can fix on my end. Can you point me to the block of code where `job.status` is cheked?
I think I know now what the problem is.

The function `handle_exception` is the one that Sentry patches to capture errors. In there we check `job.is_failed`: https://github.com/getsentry/sentry-python/blob/master/sentry_sdk/integrations/rq.py#L98

And only if `job.is_failed=True` then we capture an exception (and set `last_event_id` that you check in your tests)

But `is_failed` is set in `handle_job_failure` that you want to call after `handle_exception`.
So for Sentry the job looks perfectly fine and it does not capture any exception.

Why do you want to call `handle_exception` now before `handle_job_failure`?
> Why do you want to call handle_exception now before handle_job_failure?

One of the operations performed in `worker.handle_job_failure()` is removing the job from `StartedJobRegistry`, which is used to monitor which jobs are still running. `worker.handle_exception` runs user supplied exception handlers, which could take a while to run.

With the current (master branch) sequence of operations , if `worker.handle_exception()` runs for a long time, we won't be able to see that the job is actually still running.

If all you need is a flag indicating that a job has failed, I can set `job._status` to Failed so you know that an error has happened. How does that sound?

FYI, a major feature introduced in this branch is job `Execution`, which allows a single job to have multiple concurrent executions so the way to detect errors may also change.
> If all you need is a flag indicating that a job has failed, I can set job._status to Failed so you know that an error has happened. How does that sound?

If this sounds good, I can push an update on this branch so you can test your code against the updated branch.
> > If all you need is a flag indicating that a job has failed, I can set job._status to Failed so you know that an error has happened. How does that sound?
> 
> If this sounds good, I can push an update on this branch so you can test your code against the updated branch.

yes, please!

@antonpirker done! 

https://github.com/rq/rq/pull/1964/commits/dccfd979c2192828213241ab5d92050a35f15ff7

Resolving patch:
diff --git a/sentry_sdk/integrations/rq.py b/sentry_sdk/integrations/rq.py
--- a/sentry_sdk/integrations/rq.py
+++ b/sentry_sdk/integrations/rq.py
@@ -20,6 +20,7 @@
     from rq.timeouts import JobTimeoutException
     from rq.version import VERSION as RQ_VERSION
     from rq.worker import Worker
+    from rq.job import JobStatus
 except ImportError:
     raise DidNotEnable("RQ not installed")
 
@@ -95,7 +96,9 @@ def sentry_patched_perform_job(self, job, *args, **kwargs):
 
         def sentry_patched_handle_exception(self, job, *exc_info, **kwargs):
             # type: (Worker, Any, *Any, **Any) -> Any
-            if job.is_failed:
+            # Note, the order of the `or` here is important,
+            # because calling `job.is_failed` will change `_status`.
+            if job._status == JobStatus.FAILED or job.is_failed:
                 _capture_exception(exc_info)  # type: ignore
 
             return old_handle_exception(self, job, *exc_info, **kwargs)


Test patch: