Repository: getsentry/sentry
Pull request: fix(migrations): Fix event migration
Add missing method to calculate primary hash to event migration.
This fixes a bug affecting onpremise users upgrading from 9.x to 10.x

Fixes https://github.com/getsentry/onpremise/issues/366

Problem statement:
Upgrade 9.1.2 to 10. Migration error
Hello!
I've started to use sentry from version 9.0.0. On prod running sentry 9.1.2 now.
I decided install stage with onpremise. Also migrated database from prod. Version 9.1.2 works fine.
When upgrading to 10 I've got an error during migration 0024_auto_20191230_2052:
```
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-29 11:37:43.582226) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-27 11:37:43.582226) while time zone support is active.
  RuntimeWarning)
Events to process: 131482

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 70, in backfill_eventstream
    primary_hash = event.get_primary_hash()
AttributeError: 'Event' object has no attribute 'get_primary_hash'
```

How can I fix it?

Hints from issue discussion:
Heya, thanks for reporting - we are looking into this.

Commit message: fix(migrations): Fix event migration

Add missing method to calculate primary hash to event migration.

Fixes https://github.com/getsentry/onpremise/issues/366

Patch to review:
diff --git a/src/sentry/migrations/0024_auto_20191230_2052.py b/src/sentry/migrations/0024_auto_20191230_2052.py
index b62a3b80af5410..0a2b758b411bb8 100644
--- a/src/sentry/migrations/0024_auto_20191230_2052.py
+++ b/src/sentry/migrations/0024_auto_20191230_2052.py
@@ -52,6 +52,26 @@ def _attach_related(_events):
             event.group = groups.get(event.group_id)
         eventstore.bind_nodes(_events, "data")
 
+    def _get_grouping_variants(event):
+        from sentry.grouping.api import get_grouping_variants_for_event, load_grouping_config
+        from sentry.stacktraces.processing import normalize_stacktraces_for_grouping
+
+        config = event.data.get("grouping_config")
+        config = load_grouping_config(config)
+        return get_grouping_variants_for_event(event, config)
+
+    def _get_hashes(event):
+        hashes = event.data.get("hashes")
+        if hashes is not None:
+            return hashes
+
+        return filter(
+            None, [x.get_hash() for x in _get_grouping_variants(event).values()]
+        )
+
+    def _get_primary_hash(event):
+        return  _get_hashes(event)[0]
+
     if skip_backfill:
         print("Skipping backfill.\n")
         return
@@ -67,7 +87,7 @@ def _attach_related(_events):
 
     processed = 0
     for event in RangeQuerySetWrapper(events, step=100, callbacks=(_attach_related,)):
-        primary_hash = event.get_primary_hash()
+        primary_hash = _get_primary_hash(event)
         if event.project is None or event.group is None:
             print("Skipped {} as group or project information is invalid.\n".format(event))
             continue


Code review comments:
diff_hunk: @@ -52,6 +52,26 @@ def _attach_related(_events):
             event.group = groups.get(event.group_id)
         eventstore.bind_nodes(_events, "data")
 
+    def _get_grouping_variants(event):
+        from sentry.grouping.api import get_grouping_variants_for_event, load_grouping_config
+        from sentry.stacktraces.processing import normalize_stacktraces_for_grouping
+
+        config = event.data.get("grouping_config")
+        config = load_grouping_config(config)
+        return get_grouping_variants_for_event(event, config)
+
+    def _get_hashes(event):
+        hashes = event.data.get("hashes")
+        if hashes is not None:
+            return hashes
+
+        return filter(
+            None, [x.get_hash() for x in _get_grouping_variants(event).values()]
path: src/sentry/migrations/0024_auto_20191230_2052.py
text: @user1:
```suggestion
            None, (x.get_hash() for x in _get_grouping_variants(event).values())
```

@user1:
Since we're never gonna use anything except the first not-None value, we may wanna use `next` and a clever generator here too (only if the grouping variants can be long):

```py
return next(hash for hash in (x.get_hash() for x in _get_grouping_variants(event).itervalues()) if hash is not None)
```

This should give us the best memory efficiency.