Repository: getsentry/sentry
Pull request #86737: migration(shared-views): Create backfill migration for desynced views
Author: login: MichaelSun48
Labels: name: Scope: Backend
description: Automatically applied to PRs that change backend components
Created: 2025-03-10T21:24:44Z | Merged: 2025-03-11T20:35:45Z

I forgot to double write to the groupsearchviewstarred table when a non-EA user creates a pinned search (syncing three systems is hard), which caused the `groupsearchview` and `groupsearchviewstarred` tables to fall out of sync (they should have the same number of entries). 

That hole has been plugged by [this PR](https://github.com/getsentry/sentry/pull/86718), which this migration depends on.  

This PR just creates new entries for any missed groupsearchviews to resync the two tables. 


Diff:
diff --git a/migrations_lockfile.txt b/migrations_lockfile.txt
index 55eb971f4bb4c4..1b313207d10354 100644
--- a/migrations_lockfile.txt
+++ b/migrations_lockfile.txt
@@ -15,7 +15,7 @@ remote_subscriptions: 0003_drop_remote_subscription
 
 replays: 0004_index_together
 
-sentry: 0840_savedsearch_type_non_null
+sentry: 0841_backfill_desynced_starred_views
 
 social_auth: 0002_default_auto_field
 
diff --git a/src/sentry/migrations/0841_backfill_desynced_starred_views.py b/src/sentry/migrations/0841_backfill_desynced_starred_views.py
new file mode 100644
index 00000000000000..b25d5952360018
--- /dev/null
+++ b/src/sentry/migrations/0841_backfill_desynced_starred_views.py
@@ -0,0 +1,57 @@
+# Generated by Django 5.1.5 on 2025-03-10 18:54
+
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from sentry.new_migrations.migrations import CheckedMigration
+from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
+
+
+def backfill_desynced_groupsearchview_positions_to_gsvstarred(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    GroupSearchView = apps.get_model("sentry", "GroupSearchView")
+    GroupSearchViewStarred = apps.get_model("sentry", "GroupSearchViewStarred")
+
+    # List of all GroupSearchView IDs that already have corresponding entries in GroupSearchViewStarred
+    existing_gsv_ids = set(
+        GroupSearchViewStarred.objects.values_list("group_search_view_id", flat=True)
+    )
+
+    for gsv in RangeQuerySetWrapperWithProgressBar(GroupSearchView.objects.all()):
+        if gsv.id not in existing_gsv_ids:
+            GroupSearchViewStarred.objects.update_or_create(
+                group_search_view=gsv,
+                user_id=gsv.user_id,
+                organization_id=gsv.organization_id,
+                defaults={"position": gsv.position},
+            )
+
+
+class Migration(CheckedMigration):
+    # This flag is used to mark that a migration shouldn't be automatically run in production.
+    # This should only be used for operations where it's safe to run the migration after your
+    # code has deployed. So this should not be used for most operations that alter the schema
+    # of a table.
+    # Here are some things that make sense to mark as post deployment:
+    # - Large data migrations. Typically we want these to be run manually so that they can be
+    #   monitored and not block the deploy for a long period of time while they run.
+    # - Adding indexes to large tables. Since this can take a long time, we'd generally prefer to
+    #   run this outside deployments so that we don't block them. Note that while adding an index
+    #   is a schema change, it's completely safe to run the operation after the code has deployed.
+    # Once deployed, run these manually via: https://develop.sentry.dev/database-migrations/#migration-deployment
+
+    is_post_deployment = True
+
+    dependencies = [
+        ("sentry", "0840_savedsearch_type_non_null"),
+    ]
+
+    operations = [
+        migrations.RunPython(
+            backfill_desynced_groupsearchview_positions_to_gsvstarred,
+            reverse_code=migrations.RunPython.noop,
+            hints={"tables": ["sentry_groupsearchview", "sentry_groupsearchviewstarred"]},
+        ),
+    ]
diff --git a/tests/sentry/migrations/test_0841_backfill_desynced_starred_views.py b/tests/sentry/migrations/test_0841_backfill_desynced_starred_views.py
new file mode 100644
index 00000000000000..b09399c3913ef3
--- /dev/null
+++ b/tests/sentry/migrations/test_0841_backfill_desynced_starred_views.py
@@ -0,0 +1,42 @@
+from sentry.models.groupsearchview import GroupSearchView
+from sentry.models.groupsearchviewstarred import GroupSearchViewStarred
+from sentry.testutils.cases import TestMigrations
+
+
+class BackfillDesyncedStarredViewsTest(TestMigrations):
+    migrate_from = "0840_savedsearch_type_non_null"
+    migrate_to = "0841_backfill_desynced_starred_views"
+
+    def setup_initial_state(self):
+        self.user = self.create_user()
+        self.org = self.create_organization(owner=self.user)
+
+        self.project = self.create_project(organization=self.organization)
+
+        self.gsv1 = GroupSearchView.objects.create(
+            user_id=self.user.id,
+            organization_id=self.organization.id,
+            name="Test View 1",
+            query="is:unresolved",
+            position=1,
+        )
+
+        GroupSearchViewStarred.objects.create(
+            group_search_view=self.gsv1,
+            user_id=self.user.id,
+            organization_id=self.organization.id,
+            position=1,
+        )
+
+        self.gsv2 = GroupSearchView.objects.create(
+            user_id=self.user.id,
+            organization_id=self.organization.id,
+            name="Test View 2",
+            query="is:resolved",
+            position=2,
+        )
+
+    def test(self):
+        assert GroupSearchViewStarred.objects.count() == 2
+        assert GroupSearchViewStarred.objects.get(group_search_view=self.gsv1).position == 1
+        assert GroupSearchViewStarred.objects.get(group_search_view=self.gsv2).position == 2


Reviews:
user: login: JoshFerge
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-11T16:09:25Z
user: login: JoshFerge
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T16:11:08Z
user: login: MichaelSun48
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T16:20:50Z
user: login: JoshFerge
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-11T16:32:23Z
user: login: wedamija
body: Ideally this would have a migration test
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-11T18:17:24Z
user: login: MichaelSun48
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T18:18:08Z

Review comments:
diff_hunk: @@ -0,0 +1,56 @@
+# Generated by Django 5.1.5 on 2025-03-10 18:54
+
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from sentry.new_migrations.migrations import CheckedMigration
+from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
+
+
+def backfill_desynced_groupsearchview_positions_to_gsvstarred(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    GroupSearchView = apps.get_model("sentry", "GroupSearchView")
+    GroupSearchViewStarred = apps.get_model("sentry", "GroupSearchViewStarred")
+
+    # List of all GroupSearchView IDs that already have corresponding entries in GroupSearchViewStarred
+    existing_gsv_ids = GroupSearchViewStarred.objects.values_list("group_search_view_id", flat=True)
+
+    for gsv in RangeQuerySetWrapperWithProgressBar(
+        GroupSearchView.objects.exclude(id__in=existing_gsv_ids)
path: src/sentry/migrations/0841_backfill_desynced_starred_views.py
user: login: JoshFerge
body: in general placing a query with a filter in a range queryset wrapper is not ideal. the idea of the RangeQuerySetWrapper is that it iterates over all rows in the database using the primary key index, so each chunk query runs extremely fast. when you add a filter to it, it is no longer fast and defeats the purpose of RangeQuerySetWrapper, because each chunk has to do a full table scan to do the filtering. you can simply move the filtering to be on each GroupSearchView object after it gets pulled from the db.

example: https://github.com/getsentry/sentry/blob/3e40d1c55ce747dc4cc81f10cb703b3c00636803/src/sentry/migrations/0514_migrate_priority_saved_searches.py#L14
created_at: 2025-03-11T16:09:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+# Generated by Django 5.1.5 on 2025-03-10 18:54
+
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from sentry.new_migrations.migrations import CheckedMigration
+from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
+
+
+def backfill_desynced_groupsearchview_positions_to_gsvstarred(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    GroupSearchView = apps.get_model("sentry", "GroupSearchView")
+    GroupSearchViewStarred = apps.get_model("sentry", "GroupSearchViewStarred")
+
+    # List of all GroupSearchView IDs that already have corresponding entries in GroupSearchViewStarred
+    existing_gsv_ids = GroupSearchViewStarred.objects.values_list("group_search_view_id", flat=True)
+
+    for gsv in RangeQuerySetWrapperWithProgressBar(
+        GroupSearchView.objects.exclude(id__in=existing_gsv_ids)
path: src/sentry/migrations/0841_backfill_desynced_starred_views.py
user: login: JoshFerge
body: in your case you'd convert the above query to a set, and then check for existence in the set
created_at: 2025-03-11T16:11:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+# Generated by Django 5.1.5 on 2025-03-10 18:54
+
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from sentry.new_migrations.migrations import CheckedMigration
+from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
+
+
+def backfill_desynced_groupsearchview_positions_to_gsvstarred(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    GroupSearchView = apps.get_model("sentry", "GroupSearchView")
+    GroupSearchViewStarred = apps.get_model("sentry", "GroupSearchViewStarred")
+
+    # List of all GroupSearchView IDs that already have corresponding entries in GroupSearchViewStarred
+    existing_gsv_ids = GroupSearchViewStarred.objects.values_list("group_search_view_id", flat=True)
+
+    for gsv in RangeQuerySetWrapperWithProgressBar(
+        GroupSearchView.objects.exclude(id__in=existing_gsv_ids)
path: src/sentry/migrations/0841_backfill_desynced_starred_views.py
user: login: MichaelSun48
body: That makes total sense, thanks for the explanation! Will update this now 

created_at: 2025-03-11T16:20:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+# Generated by Django 5.1.5 on 2025-03-10 18:54
+
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from sentry.new_migrations.migrations import CheckedMigration
+from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
+
+
+def backfill_desynced_groupsearchview_positions_to_gsvstarred(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    GroupSearchView = apps.get_model("sentry", "GroupSearchView")
+    GroupSearchViewStarred = apps.get_model("sentry", "GroupSearchViewStarred")
+
+    # List of all GroupSearchView IDs that already have corresponding entries in GroupSearchViewStarred
+    existing_gsv_ids = set(
+        GroupSearchViewStarred.objects.values_list("group_search_view_id", flat=True)
+    )
path: src/sentry/migrations/0841_backfill_desynced_starred_views.py
user: login: wedamija
body: I assume there aren't too many of these?  
created_at: 2025-03-11T18:16:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,57 @@
+# Generated by Django 5.1.5 on 2025-03-10 18:54
+
+from django.db import migrations
+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from sentry.new_migrations.migrations import CheckedMigration
+from sentry.utils.query import RangeQuerySetWrapperWithProgressBar
+
+
+def backfill_desynced_groupsearchview_positions_to_gsvstarred(
+    apps: StateApps, schema_editor: BaseDatabaseSchemaEditor
+) -> None:
+    GroupSearchView = apps.get_model("sentry", "GroupSearchView")
+    GroupSearchViewStarred = apps.get_model("sentry", "GroupSearchViewStarred")
+
+    # List of all GroupSearchView IDs that already have corresponding entries in GroupSearchViewStarred
+    existing_gsv_ids = set(
+        GroupSearchViewStarred.objects.values_list("group_search_view_id", flat=True)
+    )
path: src/sentry/migrations/0841_backfill_desynced_starred_views.py
user: login: MichaelSun48
body: a little over 100k, i dont think memory should be an issue?
created_at: 2025-03-11T18:18:08Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-10T21:30:29Z
body: This PR has a migration; here is the generated SQL for  `src/sentry/migrations/0841_backfill_desynced_starred_views.py` ()

```sql
--
-- Raw Python operation
--
-- THIS OPERATION CANNOT BE WRITTEN AS SQL
```
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-10T21:33:28Z
body: ## [Codecov](https://app.codecov.io/gh/getsentry/sentry/pull/86737?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getsentry) Report
All modified and coverable lines are covered by tests :white_check_mark:

:white_check_mark: All tests successful. No failed tests found.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master   #86737      +/-   ##
==========================================
+ Coverage   87.78%   87.79%   +0.01%     
==========================================
  Files        9778     9781       +3     
  Lines      553920   554511     +591     
  Branches    21713    21713              
==========================================
+ Hits       486238   486851     +613     
+ Misses      67288    67266      -22     
  Partials      394      394              
```

</details>
author_association: NONE