Repository: HangfireIO/Hangfire

Issue #628: Deadlock (??) caused by Hangfire Query consumes all SQL Azure DTUs!
Created: 2016-07-27T09:56:58Z
Hi!
We are using the latest Hangfire.SqlServer bits (1.6.0) in our Azure environment which consists of

a virtual machine hosting an ASP.NET Web Application using Hangfire on IIS
a SQL Azure Database with 20 DTUs (test environment)

Now the problem is, easily understandable viewing the following screenshot of the Azure management portal.
screenshot
As you can see, the following query consumes almost all dtus, as it takes ages to execute (9 to 22 MINUTES!!)
(@now datetime,@count int)delete top (@count) from [HangFire].[Job] with (readpast) where ExpireAt < @now
So either the SQL Azure Query Performance insight is not trustworthy at all, or there seems to be a serious problem in Hangfire, right?
Some more infos:

I checked the [Hangfire].[Job] table and there is an unclustered index on ExpireAt.
We currently have 22.000 completed jobs, no recurring jobs, and only one server (see here)
The issue was already present with the last version (Hangfire.SqlServer 1.5.0), but as I came across the following issue #429 I thought it would have been resolved with version 1.6.0.

Currently we are using the following packages:
  <package id="Hangfire.Core" version="1.6.0" targetFramework="net452" />
  <package id="Hangfire.Dashboard.Authorization" version="2.1.0" targetFramework="net452" />
  <package id="Hangfire.SqlServer" version="1.6.0" targetFramework="net452" />
  <package id="Hangfire.Windsor" version="1.0.5981.19913" targetFramework="net452" />
Is there anything that we can do to workaround this issue? Disabling the deletion of jobs, or specifying that this should only happen once a day? - currently it seems this is done every 5 minutes, which is why our DTUs are all gone very fast.
see this DTU graph of SQL Azure Management Portal

Duplicate issue #489: Large amount of data in State table causes connection timeout and application lock exceptions
Created: 2015-12-02T18:56:59Z
I have spent the whole day today trying to find out why one of our environments (UAT) was slow to the point of being unusable.
I did notice that our logs were full of instances of both the exceptions mentioned in #471 and #273.
I had noticed the connection timeout in another environment (production) as well. However, that environment was still usable and did not seem to be affected beyond having a large number of entries in the logs.
After a long and winding road of discussions with our host, looking at the code and doubting my own existence I ended up at the following:
I looked at the State table and noticed that it had reached the 2GB mark!
I decided to truncate the table.
To my surprise and disappointment at the time I wasted before I tried the truncate statement, the environment was rejuvenated and everything started working normally again!
I'm not quite sure what to suggest as I'm a bit brain dead after a day of trying to resolve this issue, but I'm happy to wait for your questions @odinserj and start from there 😄

Closing comment (2016-08-04T16:21:32Z): Problem with huge amount of records in the State table is related to problem with unhandled OperationCanceledException and its derived classes that caused background job to be re-queued again and again, leading to duplicate Processing state entries. This issue was fixed 1.5.0 and 1.5.9.
Problem with timeouts is related to #628, where we investigated that ExpirationManager's queries is using INDEX SCAN operators, that blocks records in State and JobParameter tables. This issue will be fixed in upcoming 1.6.2.
Problems with application locks were fixed in 1.5.3 and 1.5.6.