Repository: getsentry/sentry

Issue:
Duplicate entry errors in version 4 -- like #382
New event from client 'raven-python/1.7.6' (id=4bdd0304fb784c22b7d314754[2/1957$
Unable to process log entry: (1062, "Duplicate entry '5-705768fdb0f8fc3db8c79ccf
6f99b140-amqp_cloader_account_indexer.' for key 'sentry_groupedmessage_project_id_23269c7de401f075_uniq'")
Traceback (most recent call last):
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 446, in from_kwargs
    group, is_new, is_sample = self._create_group(event, **group_kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 503, in _create_group
    defaults=kwargs
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 219, in get_or_create
    instance, created = super(BaseManager, self).get_or_create(defaults=defaults, **kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/manager.py", line 135, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/query.py", line 393, in get_or_create
    raise e

I'm receiving a lot of these errors--so many that it's causing the load to go up on my system.

If I drop/recreate the database, these entries go away until I update my code (the code that I'm tracking). Then the errors start up again after a code restart.

Chris



Discussion:
Are you using any of the caching or queueing? 

I need to add a good guide to the docs for configuring an optimal Sentry install 

On Sunday, June 3, 2012 at 9:29 PM, MrCerulean wrote: 

> New event from client 'raven-python/1.7.6' (id=4bdd0304fb784c22b7d314754[2/1957$
> Unable to process log entry: (1062, "Duplicate entry '5-705768fdb0f8fc3db8c79ccf
> 6f99b140-amqp_cloader_account_indexer.' for key 'sentry_groupedmessage_project_id_23269c7de401f075_uniq'")
> Traceback (most recent call last):
> File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 446, in from_kwargs
> group, is_new, is_sample = self._create_group(event, **group_kwargs)
> File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 503, in _create_group
> defaults=kwargs
> File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 219, in get_or_create
> instance, created = super(BaseManager, self).get_or_create(defaults=defaults, **kwargs)
> File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/manager.py", line 135, in get_or_create
> return self.get_query_set().get_or_create(**kwargs)
> File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/query.py", line 393, in get_or_create
> raise e
> 
> I'm receiving a lot of these errors--so many that it's causing the load to go up on my system.
> 
> If I drop/recreate the database, these entries go away until I update my code (the code that I'm tracking). Then the errors start up again after a code restart.
> 
> Chris
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510

Currently only have the following config:

SENTRY_WEB_HOST = '0.0.0.0'
SENTRY_WEB_PORT = 9000
SENTRY_WEB_OPTIONS = {
    'workers': 3,  # the number of gunicorn workers
    'worker_class': 'gevent',
}

No queues being used, but I will try that, too.

Chris

On Mon, Jun 4, 2012 at 11:38 AM, David Cramer
reply@reply.github.com
wrote:

> Are you using any of the caching or queueing?
> 
> I need to add a good guide to the docs for configuring an optimal Sentry install
> 
> On Sunday, June 3, 2012 at 9:29 PM, MrCerulean wrote:
> 
> > New event from client 'raven-python/1.7.6' (id=4bdd0304fb784c22b7d314754[2/1957$
> > Unable to process log entry: (1062, "Duplicate entry '5-705768fdb0f8fc3db8c79ccf
> > 6f99b140-amqp_cloader_account_indexer.' for key 'sentry_groupedmessage_project_id_23269c7de401f075_uniq'")
> > Traceback (most recent call last):
> > File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 446, in from_kwargs
> > group, is_new, is_sample = self._create_group(event, **group_kwargs)
> > File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 503, in _create_group
> > defaults=kwargs
> > File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 219, in get_or_create
> > instance, created = super(BaseManager, self).get_or_create(defaults=defaults, **kwargs)
> > File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/manager.py", line 135, in get_or_create
> > return self.get_query_set().get_or_create(**kwargs)
> > File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/query.py", line 393, in get_or_create
> > raise e
> > 
> > I'm receiving a lot of these errors--so many that it's causing the load to go up on my system.
> > 
> > If I drop/recreate the database, these entries go away until I update my code (the code that I'm tracking). Then the errors start up again after a code restart.
> > 
> > Chris
> > 
> > ---
> > 
> > Reply to this email directly or view it on GitHub:
> > https://github.com/dcramer/sentry/issues/510
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092244

The cache stuff is undocumented, but here's what getsentry.com runs for optimal setup. 

Start with memcache, do queueing only if that doesn't suffice. Memcache will stop the race condition error you're seeing

CACHES = {
    'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
        'LOCATION': '%s:%s' % (MEMCACHED_HOST, MEMCACHED_PORT),
    }
}

SENTRY_CACHE_BACKEND = 'default'

Queue specific settings:

BROKER_URL = 'redis://redis:%s@%s:%s' % (REDIS_PASSWORD, REDIS_HOST, REDIS_PORT)

SENTRY_USE_QUEUE = (
    # 'sentry.tasks.store.store_event',
    # 'sentry.tasks.index.index_event',
    'sentry.tasks.cleanup.cleanup',
    'sentry.tasks.post_process.post_process_group',
    'sentry.tasks.process_buffer.process_incr',
)

The buffer is only useful if you use queueing:

SENTRY_BUFFER = 'sentry.buffer.redis.RedisBuffer'
SENTRY_BUFFER_OPTIONS = {
    'hosts': {
        0: {
            'password': REDIS_PASSWORD,
            'host': REDIS_HOST,
            'port': REDIS_PORT,
        }
    }
}

## 

David Cramer

twitter.com/zeeg
disqus.com/zeeg

On Sunday, June 3, 2012 at 9:48 PM, MrCerulean wrote:

> Currently only have the following config:
> 
> SENTRY_WEB_HOST = '0.0.0.0'
> SENTRY_WEB_PORT = 9000
> SENTRY_WEB_OPTIONS = {
> 'workers': 3, # the number of gunicorn workers
> 'worker_class': 'gevent',
> }
> 
> No queues being used, but I will try that, too.
> 
> Chris
> 
> On Mon, Jun 4, 2012 at 11:38 AM, David Cramer
> <reply@reply.github.com (mailto:reply@reply.github.com)>
> wrote:
> 
> > Are you using any of the caching or queueing?
> > 
> > I need to add a good guide to the docs for configuring an optimal Sentry install
> > 
> > On Sunday, June 3, 2012 at 9:29 PM, MrCerulean wrote:
> > 
> > > New event from client 'raven-python/1.7.6' (id=4bdd0304fb784c22b7d314754[2/1957$
> > > Unable to process log entry: (1062, "Duplicate entry '5-705768fdb0f8fc3db8c79ccf
> > > 6f99b140-amqp_cloader_account_indexer.' for key 'sentry_groupedmessage_project_id_23269c7de401f075_uniq'")
> > > Traceback (most recent call last):
> > > File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 446, in from_kwargs
> > > group, is_new, is_sample = self._create_group(event, **group_kwargs)
> > > File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 503, in _create_group
> > > defaults=kwargs
> > > File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py", line 219, in get_or_create
> > > instance, created = super(BaseManager, self).get_or_create(defaults=defaults, **kwargs)
> > > File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/manager.py", line 135, in get_or_create
> > > return self.get_query_set().get_or_create(**kwargs)
> > > File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/query.py", line 393, in get_or_create
> > > raise e
> > > 
> > > I'm receiving a lot of these errors--so many that it's causing the load to go up on my system.
> > > 
> > > If I drop/recreate the database, these entries go away until I update my code (the code that I'm tracking). Then the errors start up again after a code restart.
> > > 
> > > Chris
> > > 
> > > ---
> > > 
> > > Reply to this email directly or view it on GitHub:
> > > https://github.com/dcramer/sentry/issues/510
> > 
> > ---
> > 
> > Reply to this email directly or view it on GitHub:
> > https://github.com/dcramer/sentry/issues/510#issuecomment-6092244
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092316

Does the cache section apply per project? It looks as though I'm
reading 'default' as being the project name. Do I need a separate
cache definition for each project?

Chris

Actually default is just the cache backend alias

Thanks. That clarifies...

Stupid question: Which module do I need to install to proivde
django_pylibmc.memcached.PyLibMCCache?

django.core.cache.backends.base.InvalidCacheBackendError: Could not
find backend 'django_pylibmc.memcached.PyLibMCCache': 'module' object
has no attribute 'PyLibMCCache'

Chris

On Mon, Jun 4, 2012 at 12:03 PM, David Cramer
reply@reply.github.com
wrote:

> Actually default is just the cache backend alias
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092428

Ah that might only be Django 1.3 

If you're on Django 1.2, here's the configuration setting: https://docs.djangoproject.com/en/1.2/topics/cache/#memcached 

## 

David Cramer

twitter.com/zeeg
disqus.com/zeeg

On Sunday, June 3, 2012 at 10:15 PM, MrCerulean wrote:

> e

I'm running on Django 1.3.1...

Chris

Oops. Sorry. 

That's django-pylibmc :)

It's an extra package you'd need to install. It's a better memcached client. 

## 

David Cramer

twitter.com/zeeg
disqus.com/zeeg

On Sunday, June 3, 2012 at 10:21 PM, MrCerulean wrote:

> I'm running on Django 1.3.1...
> 
> Chris
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092546

When I change the syntax per the referenced doc, Sentry starts and
works, but I'm back to the original problem. :)

Chris

I think the 1.2 strings should be compatible. It's the exact same error? It should at least stop the concurrent writes to the same key assuming it can connect to the cache.

Memcache is actually setup, right? :)

Unable to process log entry: (1062, "Duplicate entry '2-9ed604078761e78343f2d772
37eb7896-amqp_cloader_account_indexer.' for key 'sentry_groupedmessage_project_i
d_23269c7de401f075_uniq'")
Traceback (most recent call last):
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py",
line 446, in from_kwargs
    group, is_new, is_sample = self._create_group(event, **group_kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py",
line 503, in _create_group
    defaults=kwargs
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py",
line 219, in get_or_create
    instance, created = super(BaseManager,
self).get_or_create(defaults=defaults, **kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 135, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/query.py",
line 393, in get_or_create
    raise e

Is the current error.

At the time:

Trying 127.0.0.1...                                                    [31/1841]
Connected to localhost.
Escape character is '^]'.
stats
STAT pid 26641
STAT uptime 2690
STAT time 1338788565
STAT version 1.4.13
STAT libevent 2.0.16-stable
STAT pointer_size 64
STAT rusage_user 0.192012
STAT rusage_system 0.368023
STAT curr_connections 9
STAT total_connections 82

Chris

On Mon, Jun 4, 2012 at 12:28 PM, David Cramer
reply@reply.github.com
wrote:

> I think the 1.2 strings should be compatible. It's the exact same error? It should at least stop the concurrent writes to the same key assuming it can connect to the cache.
> 
> Memcache is actually setup, right? :)
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092600

And did you end up using CACHES or CACHE_BACKEND? If the latter, can you try switching back to CACHES and changing the backend engine specification

Unable to process log entry: (1062, "Duplicate entry '2-9ed604078761e78343f2d772
37eb7896-amqp_cloader_account_indexer.' for key 'sentry_groupedmessage_project_i
d_23269c7de401f075_uniq'")
Traceback (most recent call last):
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py",
line 446, in from_kwargs
    group, is_new, is_sample = self._create_group(event, **group_kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py",
line 503, in _create_group
    defaults=kwargs
  File "/srv/sentry/local/lib/python2.7/site-packages/sentry/manager.py",
line 219, in get_or_create
    instance, created = super(BaseManager,
self).get_or_create(defaults=defaults, **kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/manager.py",
line 135, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "/srv/sentry/local/lib/python2.7/site-packages/django/db/models/query.py",
line 393, in get_or_create
    raise e

I changed to CACHES:

CACHES = {
   'default': {
       'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
       'LOCATION': '127.0.0.1:11211',
   }
}

If I use:

```
   'BACKEND': 'django_pylibmc.memcached.PyLibMCCache',
```

then I get:

django.core.cache.backends.base.InvalidCacheBackendError: Could not
find backend 'django_pylibmc.memcached.PyLibMCCache': 'module' object
has no attribute 'PyLibMCCache'

although:

pylibmc==1.2.3

Chris

On Mon, Jun 4, 2012 at 12:47 PM, David Cramer
reply@reply.github.com
wrote:

> And did you end up using CACHES or CACHE_BACKEND? If the latter, can you try switching back to CACHES and changing the backend engine specification
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092750

OK. PylibMCCache is now working properly, but the error persists.

Chris

On Mon, Jun 4, 2012 at 12:57 PM, chris@dollmont.net chris@dollmont.net wrote:

> OK. PylibMCCache is now working properly, but the error persists.
> 
> Chris

And, possibly unrelated, attempting to access the web interface now gives me:

DatabaseError: (1064, "You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for the right
syntax to use near 'GROUP BY a.group_id) as z\n        ON z.group_id =
sentry_messagecountbyminute.gr' at line 8")

Chris

You get that error on all pages? What version of Sentry is it?

That's on the main page. I can go to the error page (details). I've
not tried jumping around, but the short answer is: not on all pages.

Chris

On Mon, Jun 4, 2012 at 1:15 PM, David Cramer
reply@reply.github.com
wrote:

> You get that error on all pages? What version of Sentry is it?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6092975

I have a feeling that's a MySQL-specific bug that I may have introduced today. I'm working on getting testing (automated) for all of the databases now, and I'll see if I can figure out what's going on with that. 

Regarding the other issue. the lock really should have fixed it (it did in our cases), and I honestly don't know what else to check. If you have the resources I'd definitely recommend running the queue and buffer, though using Redis as a queue adds up in memory usage pretty quickly. 

## 

David Cramer

twitter.com/zeeg
disqus.com/zeeg

On Sunday, June 3, 2012 at 11:55 PM, MrCerulean wrote:

> That's on the main page. I can go to the error page (details). I've
> not tried jumping around, but the short answer is: not on all pages.
> 
> Chris
> 
> On Mon, Jun 4, 2012 at 1:15 PM, David Cramer
> <reply@reply.github.com (mailto:reply@reply.github.com)>
> wrote:
> 
> > You get that error on all pages? What version of Sentry is it?
> > 
> > ---
> > 
> > Reply to this email directly or view it on GitHub:
> > https://github.com/dcramer/sentry/issues/510#issuecomment-6092975
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6093370

I've solved the MySQL issues in 4.5.5

I'm not sure this will help with the duplicate error situation you were seeing, but the dashboard is now fixed (and that endpoint tested).

We also now run mysql and postgres tests automatically along with the fast sqlite tests (yay! http://travis-ci.org/#!/dcramer/sentry)

You are correct--it fixed the dashboard, but not the duplicate
problem. I will try cache/queue combo to see what happens.

Chris

On Mon, Jun 4, 2012 at 3:10 PM, David Cramer
reply@reply.github.com
wrote:

> I've solved the MySQL issues in 4.5.5
> 
> I'm not sure this will help with the duplicate error situation you were seeing, but the dashboard is now fixed (and that endpoint tested).
> 
> We also now run mysql and postgres tests automatically along with the fast sqlite tests (yay! http://travis-ci.org/#!/dcramer/sentry)
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/dcramer/sentry/issues/510#issuecomment-6094223

I'm actually still seeing those so I think we still arent handling at least one race condition


Resolving patch:
diff --git a/sentry/manager.py b/sentry/manager.py
--- a/sentry/manager.py
+++ b/sentry/manager.py
@@ -633,8 +633,8 @@ def add_tags(self, group, tags):
                 'last_seen': date,
             })
 
-    def get_by_natural_key(self, logger, culprit, checksum):
-        return self.get(logger=logger, view=culprit, checksum=checksum)
+    def get_by_natural_key(self, project, logger, culprit, checksum):
+        return self.get(project=project, logger=logger, view=culprit, checksum=checksum)
 
     def get_accelerated(self, queryset=None, minutes=15):
         # mintues should
diff --git a/sentry/migrations/0062_correct_del_index_sentry_groupedmessage_logger__view__checksum.py b/sentry/migrations/0062_correct_del_index_sentry_groupedmessage_logger__view__checksum.py
new file mode 100644
--- /dev/null
+++ b/sentry/migrations/0062_correct_del_index_sentry_groupedmessage_logger__view__checksum.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+import datetime
+from south.db import db
+from south.v2 import SchemaMigration
+from django.db import models
+
+
+class Migration(SchemaMigration):
+
+    def forwards(self, orm):
+        # Removing unique constraint on 'GroupedMessage', fields ['logger', 'view', 'checksum']
+        # FIXES 0015
+        db.delete_unique('sentry_groupedmessage', ['logger', 'view', 'checksum'])
+
+    def backwards(self, orm):
+        # Adding unique constraint on 'GroupedMessage', fields ['checksum', 'logger', 'view']
+        #FIXES 0015
+        db.create_unique('sentry_groupedmessage', ['checksum', 'logger', 'view'])
+
+    complete_apps = ['sentry']
diff --git a/sentry/models.py b/sentry/models.py
--- a/sentry/models.py
+++ b/sentry/models.py
@@ -429,7 +429,7 @@ def get_absolute_url(self):
         return '#'
 
     def natural_key(self):
-        return (self.logger, self.culprit, self.checksum)
+        return (self.project, self.logger, self.culprit, self.checksum)
 
     def get_score(self):
         return int(math.log(self.times_seen) * 600 + float(time.mktime(self.last_seen.timetuple())))


Test patch: