Repository: PostHog/posthog
Pull request #29379: fix(flags): Ensure activity log reflects removing variants
Author: login: haacked
Created: 2025-02-28T23:29:50Z | Merged: 2025-03-04T04:33:29Z

Updates the activity log for feature flags to describe two types of variant deletions:

1. When individual variants are deleted, but there are still variants left.
2. When all variants are removed (via changing release toggle).

## Problem

Fixes #26115

Turns out activity log doesn't mention when a variant is removed. It also didn't describe when all variants are removed (as would happen if you changed the release toggle to boolean or remote config)

## Changes

We'll start with a feature flag with four variants.

![Screenshot 2025-02-28 at 3 14 35 PM](https://github.com/user-attachments/assets/5988e227-60cd-4db8-b2f9-cd53c4fe6290)

And remove two of them, adjusting the rollout percentages to sum to 100%.

![Screenshot 2025-02-28 at 3 23 02 PM](https://github.com/user-attachments/assets/eade626c-8196-407d-b846-8f23e1a363fe)

Activity log shows:

![Screenshot 2025-02-28 at 3 23 15 PM](https://github.com/user-attachments/assets/41745fed-343f-4041-a278-ec72c0be97b2)

Ok, let's remove all variants by changing this to a remote config setting:

![Screenshot 2025-02-28 at 3 25 08 PM](https://github.com/user-attachments/assets/d26fbe79-def6-468b-aa02-7bb79b850fb5)

![Screenshot 2025-02-28 at 3 25 17 PM](https://github.com/user-attachments/assets/89cd6c92-d2d3-4741-b440-bb82b1ffe44a)

Finally, let's go back to a feature flag with variants:

![Screenshot 2025-02-28 at 3 27 33 PM](https://github.com/user-attachments/assets/f3ae864f-3229-486e-a99b-65e36ca459dc)

And we'll change its release toggle to boolean with no other changes.

![Screenshot 2025-02-28 at 3 27 44 PM](https://github.com/user-attachments/assets/152b07d6-6546-4845-989f-d2e3b781928e)

Just to make sure we get a standalone log.

![Screenshot 2025-02-28 at 3 27 55 PM](https://github.com/user-attachments/assets/1b63325b-7603-4379-9c56-f974ab331973)


## Does this work well for both Cloud and self-hosted?

Yes

## How did you test this code?

Manual tests and some unit tests.

Linked issue #26115: Add feature flag variant deletion to the feature flag history
### Feature request

## Is your feature request related to a problem?

We had a ticket where an issue described variants for their feature flags disappearing ([ticket](https://posthoghelp.zendesk.com/agent/tickets/20071)).  One of the things that made this ticket difficult to investigate was that our flag history doesn't flag deletion events for variants, only that they were modified.  It wouldn't hurt to re-visit the feature flag history in general, too – feels like the UI is a bit clunky and it could be slightly better.

## Describe the solution you'd like

- Clean up the feature flag history
- Make sure variant key deletions are a specific type of event.  Right now they're just changes to variant distributions, but that's not the same as a deletion and we should split those out.


Diff:
diff --git a/frontend/__snapshots__/components-activitylog--feature-flag-activity--dark.png b/frontend/__snapshots__/components-activitylog--feature-flag-activity--dark.png
index bac05fc6d8cb..b7f2d579de2d 100644
Binary files a/frontend/__snapshots__/components-activitylog--feature-flag-activity--dark.png and b/frontend/__snapshots__/components-activitylog--feature-flag-activity--dark.png differ
diff --git a/frontend/__snapshots__/components-activitylog--feature-flag-activity--light.png b/frontend/__snapshots__/components-activitylog--feature-flag-activity--light.png
index a02071f51de4..ae2e37c7fac7 100644
Binary files a/frontend/__snapshots__/components-activitylog--feature-flag-activity--light.png and b/frontend/__snapshots__/components-activitylog--feature-flag-activity--light.png differ
diff --git a/frontend/__snapshots__/components-activitylog--with-caption--dark.png b/frontend/__snapshots__/components-activitylog--with-caption--dark.png
index 8479bb4f3501..1a4116d95661 100644
Binary files a/frontend/__snapshots__/components-activitylog--with-caption--dark.png and b/frontend/__snapshots__/components-activitylog--with-caption--dark.png differ
diff --git a/frontend/__snapshots__/components-activitylog--with-caption--light.png b/frontend/__snapshots__/components-activitylog--with-caption--light.png
index 23cf521c98f1..42d2537482fd 100644
Binary files a/frontend/__snapshots__/components-activitylog--with-caption--light.png and b/frontend/__snapshots__/components-activitylog--with-caption--light.png differ
diff --git a/frontend/src/lib/components/ActivityLog/activityLogLogic.feature-flag.test.tsx b/frontend/src/lib/components/ActivityLog/activityLogLogic.feature-flag.test.tsx
index a3541e78e302..59e5744553e3 100644
--- a/frontend/src/lib/components/ActivityLog/activityLogLogic.feature-flag.test.tsx
+++ b/frontend/src/lib/components/ActivityLog/activityLogLogic.feature-flag.test.tsx
@@ -528,5 +528,223 @@ describe('the activity log logic', () => {
                 'peter changed the filter conditions to apply to 76% of Initial Browser = Chrome , and 99% of Initial Browser Version = 100 on with two changes'
             )
         })
+
+        it('can handle changing variants from a multivariate flag', async () => {
+            const logic = await featureFlagsTestSetup('test flag', 'updated', [
+                {
+                    type: ActivityScope.FEATURE_FLAG,
+                    action: 'changed',
+                    field: 'filters',
+                    before: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 50 },
+                                { key: 'test-1', rollout_percentage: 50 },
+                            ],
+                        },
+                    },
+                    after: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 60 },
+                                { key: 'test-1', rollout_percentage: 40 },
+                            ],
+                        },
+                    },
+                },
+            ])
+
+            const actual = logic.values.humanizedActivity
+            expect(render(<>{actual[0].description}</>).container).toHaveTextContent(
+                'peter changed the rollout percentage for the variants to control: 60%, and test-1: 40% on test flag'
+            )
+        })
+
+        it('can handle removing variant from a multivariate flag', async () => {
+            const logic = await featureFlagsTestSetup('test flag', 'updated', [
+                {
+                    type: ActivityScope.FEATURE_FLAG,
+                    action: 'changed',
+                    field: 'filters',
+                    before: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 33 },
+                                { key: 'test-1', rollout_percentage: 33 },
+                                { key: 'test-2', rollout_percentage: 34 },
+                            ],
+                        },
+                    },
+                    after: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 50 },
+                                { key: 'test-1', rollout_percentage: 50 },
+                            ],
+                        },
+                    },
+                },
+            ])
+
+            const actual = logic.values.humanizedActivity
+            expect(render(<>{actual[0].description}</>).container).toHaveTextContent(
+                'peter changed the rollout percentage for the variants to control: 50%, and test-1: 50%, and removed variant test-2 on test flag'
+            )
+        })
+
+        it('can handle removing more than one variant from a multivariate flag', async () => {
+            const logic = await featureFlagsTestSetup('test flag', 'updated', [
+                {
+                    type: ActivityScope.FEATURE_FLAG,
+                    action: 'changed',
+                    field: 'filters',
+                    before: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 33 },
+                                { key: 'test-1', rollout_percentage: 33 },
+                                { key: 'test-2', rollout_percentage: 34 },
+                                { key: 'test-3', rollout_percentage: 34 },
+                            ],
+                        },
+                    },
+                    after: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 50 },
+                                { key: 'test-1', rollout_percentage: 50 },
+                            ],
+                        },
+                    },
+                },
+            ])
+
+            const actual = logic.values.humanizedActivity
+            expect(render(<>{actual[0].description}</>).container).toHaveTextContent(
+                'peter changed the rollout percentage for the variants to control: 50%, and test-1: 50%, and removed variants test-2, and test-3 on test flag'
+            )
+        })
+
+        it.each([
+            {
+                name: 'null multivariate',
+                after: { multivariate: null },
+            },
+            {
+                name: 'empty variants array',
+                after: { multivariate: { variants: [] } },
+            },
+            {
+                name: 'undefined multivariate',
+                after: { multivariate: undefined },
+            },
+        ])('can handle removing all variants when $name', async ({ after }) => {
+            const logic = await featureFlagsTestSetup('test flag', 'updated', [
+                {
+                    type: ActivityScope.FEATURE_FLAG,
+                    action: 'changed',
+                    field: 'filters',
+                    before: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [
+                                { key: 'control', rollout_percentage: 33 },
+                                { key: 'test-1', rollout_percentage: 33 },
+                            ],
+                        },
+                    },
+                    after: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        ...after,
+                    },
+                },
+            ])
+
+            const actual = logic.values.humanizedActivity
+            expect(render(<>{actual[0].description}</>).container).toHaveTextContent(
+                'peter removed all variants on test flag'
+            )
+        })
+
+        it('can handle removing the last variant from a multivariate flag', async () => {
+            const logic = await featureFlagsTestSetup('test flag', 'updated', [
+                {
+                    type: ActivityScope.FEATURE_FLAG,
+                    action: 'changed',
+                    field: 'filters',
+                    before: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: {
+                            variants: [{ key: 'control', rollout_percentage: 100 }],
+                        },
+                    },
+                    after: {
+                        groups: [
+                            {
+                                properties: [],
+                                rollout_percentage: 75,
+                            },
+                        ],
+                        multivariate: null,
+                    },
+                },
+            ])
+
+            const actual = logic.values.humanizedActivity
+            expect(render(<>{actual[0].description}</>).container).toHaveTextContent(
+                'peter removed the last variant on test flag'
+            )
+        })
     })
 })
diff --git a/frontend/src/scenes/feature-flags/activityDescriptions.tsx b/frontend/src/scenes/feature-flags/activityDescriptions.tsx
index 20d2bdf5cd76..d3caa8ebe2cd 100644
--- a/frontend/src/scenes/feature-flags/activityDescriptions.tsx
+++ b/frontend/src/scenes/feature-flags/activityDescriptions.tsx
@@ -142,7 +142,19 @@ const featureFlagActionsMapping: Record<
             }
         }
 
-        if (isMultivariateFlag) {
+        if (filtersBefore?.multivariate?.variants?.length && !filtersAfter?.multivariate?.variants?.length) {
+            changes.push(
+                <SentenceList
+                    key="remove-variants-list"
+                    listParts={[
+                        <span key="remove-variants">
+                            removed{' '}
+                            {filtersBefore.multivariate.variants.length === 1 ? 'the last variant' : 'all variants'}
+                        </span>,
+                    ]}
+                />
+            )
+        } else if (isMultivariateFlag) {
             filtersAfter.payloads &&
                 Object.keys(filtersAfter.payloads).forEach((key: string) => {
                     const changedPayload = filtersAfter.payloads?.[key]?.toString() || null
@@ -162,6 +174,12 @@ const featureFlagActionsMapping: Record<
                     )
                 })
 
+            // Identify removed variants
+            const beforeVariants = new Set((filtersBefore?.multivariate?.variants || []).map((v) => v.key))
+            const afterVariants = new Set((filtersAfter?.multivariate?.variants || []).map((v) => v.key))
+            const removedVariants = [...beforeVariants].filter((key) => !afterVariants.has(key))
+
+            // First add the rollout percentage changes
             changes.push(
                 <SentenceList
                     listParts={(filtersAfter.multivariate?.variants || []).map((v) => (
@@ -172,6 +190,25 @@ const featureFlagActionsMapping: Record<
                     prefix="changed the rollout percentage for the variants to"
                 />
             )
+
+            // Then add removed variants if any
+            if (removedVariants.length > 0) {
+                changes.push(
+                    <SentenceList
+                        listParts={removedVariants.map((key) => (
+                            <span key={key} className="highlighted-activity">
+                                <strong>{key}</strong>
+                            </span>
+                        ))}
+                        prefix={`removed ${pluralize(
+                            removedVariants.length,
+                            'variant',
+                            undefined,
+                            /* includeNumber: */ false
+                        )}`}
+                    />
+                )
+            }
         }
 
         if (changes.length > 0) {


Reviews:
user: login: greptile-apps[bot]
body: 
## PR Summary

This PR enhances the feature flag activity log to properly track and display variant deletions, addressing a gap in the previous implementation.

- Added test cases in `frontend/src/lib/components/ActivityLog/activityLogLogic.feature-flag.test.tsx` to verify proper logging of individual variant deletions and complete variant removal
- Implemented logic in `frontend/src/scenes/feature-flags/activityDescriptions.tsx` to detect when variants are removed by comparing before/after states using Sets
- Added specific handling for the case when all variants are removed (when changing from multivariate to boolean/remote config)
- Created human-readable activity log messages that clearly indicate which variants were removed
- Ensured the implementation works for both individual variant removals and bulk removals

<sub>2 file(s) reviewed, no comment(s)</sub>
<sub>[Edit PR Review Bot Settings](https://app.greptile.com/review/github) | [Greptile](https://greptile.com?utm_source=greptile_expert&utm_medium=github&utm_campaign=code_reviews&utm_content=posthog_29379)</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T23:30:32Z
user: login: dmarticus
body: generally solid (great test), a few grammar nitpicks on the messaging.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T18:51:17Z
user: login: haacked
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T03:54:50Z
user: login: haacked
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T04:02:16Z
user: login: haacked
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T04:02:23Z

Review comments:
diff_hunk: @@ -142,7 +142,14 @@ const featureFlagActionsMapping: Record<
             }
         }
 
-        if (isMultivariateFlag) {
+        if (filtersBefore?.multivariate?.variants?.length && filtersAfter.multivariate === null) {
+            changes.push(
+                <SentenceList
+                    key="remove-variants-list"
+                    listParts={[<span key="remove-variants">removed all variants</span>]}
path: frontend/src/scenes/feature-flags/activityDescriptions.tsx
user: login: dmarticus
body: grammar nitpick – do you think it's worth making this text smarter (e.g. say "removed variant" if the `filtersBefore` list only had one entry, and only say "removed all variants" if `filtersBefore` had more than one variant?
created_at: 2025-03-03T18:49:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +185,20 @@ const featureFlagActionsMapping: Record<
                     prefix="changed the rollout percentage for the variants to"
                 />
             )
+
+            // Then add removed variants if any
+            if (removedVariants.length > 0) {
+                changes.push(
+                    <SentenceList
+                        listParts={removedVariants.map((key) => (
+                            <span key={key} className="highlighted-activity">
+                                <strong>{key}</strong>
+                            </span>
+                        ))}
+                        prefix="removed variant(s)"
path: frontend/src/scenes/feature-flags/activityDescriptions.tsx
user: login: dmarticus
body: could we be smarter with the variants and either say "removed variant" if we have one and "removed variants" if we have > 1?  It's a little thing but makes it feel smoother.
created_at: 2025-03-03T18:49:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -142,7 +142,14 @@ const featureFlagActionsMapping: Record<
             }
         }
 
-        if (isMultivariateFlag) {
+        if (filtersBefore?.multivariate?.variants?.length && filtersAfter.multivariate === null) {
+            changes.push(
+                <SentenceList
+                    key="remove-variants-list"
+                    listParts={[<span key="remove-variants">removed all variants</span>]}
path: frontend/src/scenes/feature-flags/activityDescriptions.tsx
user: login: haacked
body: Not in this particular case. I think the "all variants" is important so it's clear none are left. I suppose I could make it say "removed the last variant".
created_at: 2025-03-04T03:54:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -172,6 +185,20 @@ const featureFlagActionsMapping: Record<
                     prefix="changed the rollout percentage for the variants to"
                 />
             )
+
+            // Then add removed variants if any
+            if (removedVariants.length > 0) {
+                changes.push(
+                    <SentenceList
+                        listParts={removedVariants.map((key) => (
+                            <span key={key} className="highlighted-activity">
+                                <strong>{key}</strong>
+                            </span>
+                        ))}
+                        prefix="removed variant(s)"
path: frontend/src/scenes/feature-flags/activityDescriptions.tsx
user: login: haacked
body: Done
created_at: 2025-03-04T04:02:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -142,7 +142,14 @@ const featureFlagActionsMapping: Record<
             }
         }
 
-        if (isMultivariateFlag) {
+        if (filtersBefore?.multivariate?.variants?.length && filtersAfter.multivariate === null) {
+            changes.push(
+                <SentenceList
+                    key="remove-variants-list"
+                    listParts={[<span key="remove-variants">removed all variants</span>]}
path: frontend/src/scenes/feature-flags/activityDescriptions.tsx
user: login: haacked
body: Done
created_at: 2025-03-04T04:02:23Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-02-28T23:40:57Z
body: **Size Change:** 0 B 

**Total Size:** 9.73 MB



<details><summary>ℹ️ <strong>View Unchanged</strong></summary>

| Filename | Size |
| :--- | :---: |
| `frontend/dist/toolbar.js` | 9.73 MB |

</details>



<a href="https://github.com/preactjs/compressed-size-action"><sub>compressed-size-action</sub></a>
author_association: CONTRIBUTOR
user: login: haacked
created_at: 2025-02-28T23:43:54Z
body: @dmarticus in the original issue you mentioned "Clean up the feature flag history" but it wasn't clear to me what you meant by that. My suggestion is we get this PR in first so that we start logging when variants are removed. We can always follow up with a new issue for improvements to the presentation of the activity log.
author_association: CONTRIBUTOR
user: login: posthog-bot
created_at: 2025-02-28T23:50:59Z
body: ## 📸 UI snapshots have been updated

**6** snapshot changes in total. **0** added, **6** modified, **0** deleted:

- **`chromium`**: **0** added, **6** modified, **0** deleted ([diff for shard 2](https://github.com/PostHog/posthog/pull/29379/commits/4c4d3a0736587c6b3209108af5b767c29db0ec13), [diff for shard 3](https://github.com/PostHog/posthog/pull/29379/commits/6d283f827b351f63ae7b3a34a4bf0f92cf80cafa))
- `webkit`: **0** added, **0** modified, **0** deleted

Triggered by [this commit](https://github.com/PostHog/posthog/pull/29379/commits/7c5a9222d55385f1b110ce74d496386991b73b08).

👉 **[Review this PR's diff of snapshots.](https://github.com/PostHog/posthog/pull/29379/files#:~:text=frontend/__snapshots__/)**
author_association: CONTRIBUTOR
user: login: posthog-bot
created_at: 2025-03-01T00:10:53Z
body: ## 📸 UI snapshots have been updated

**2** snapshot changes in total. **0** added, **2** modified, **0** deleted:

- **`chromium`**: **0** added, **2** modified, **0** deleted ([diff for shard 2](https://github.com/PostHog/posthog/pull/29379/commits/011cf21f656e78155e595d163d6ecd7be49f8852))
- `webkit`: **0** added, **0** modified, **0** deleted

Triggered by [this commit](https://github.com/PostHog/posthog/pull/29379/commits/ebbe8d398f4175e45cdf4a9954936ec9ea0475a1).

👉 **[Review this PR's diff of snapshots.](https://github.com/PostHog/posthog/pull/29379/files#:~:text=frontend/__snapshots__/)**
author_association: CONTRIBUTOR
user: login: dmarticus
created_at: 2025-03-03T18:46:41Z
body: > @dmarticus in the original issue you mentioned "Clean up the feature flag history" but it wasn't clear to me what you meant by that. My suggestion is we get this PR in first so that we start logging when variants are removed. We can always follow up with a new issue for improvements to the presentation of the activity log.

We talked about this in standup this morning, but here's the other things we I'd love to add to the activity log (I'm fine keeping them out of scope for this PR and doing them in a follow-up)

- Move activity log writer to `post_write` hook instead of `PATCH` endpoint
- Show expandable JSON diff on activity log
- would be cool to show the flag version for each activity log entry

author_association: CONTRIBUTOR
user: login: haacked
created_at: 2025-03-04T03:56:58Z
body: The other thing I'd like to fix, but maybe later, is the sentence list functionality. It uses the Oxford comma on a list with two items which even hard-core Oxford comma proponents can agree is incorrect. 😆 
author_association: CONTRIBUTOR
user: login: posthog-bot
created_at: 2025-03-04T04:22:47Z
body: ## 📸 UI snapshots have been updated

**4** snapshot changes in total. **0** added, **4** modified, **0** deleted:

- **`chromium`**: **0** added, **4** modified, **0** deleted ([diff for shard 3](https://github.com/PostHog/posthog/pull/29379/commits/333a937b60218f0d081845ecf6d0b66512e37863))
- `webkit`: **0** added, **0** modified, **0** deleted

Triggered by [this commit](https://github.com/PostHog/posthog/pull/29379/commits/6731ebda571f472dd5722f5fbc16e383de731a97).

👉 **[Review this PR's diff of snapshots.](https://github.com/PostHog/posthog/pull/29379/files#:~:text=frontend/__snapshots__/)**
author_association: CONTRIBUTOR
user: login: dmarticus
created_at: 2025-03-04T06:32:43Z
body: > The other thing I'd like to fix, but maybe later, is the sentence list functionality. It uses the Oxford comma on a list with two items which even hard-core Oxford comma proponents can agree is incorrect. 😆

I saw this too!!
author_association: CONTRIBUTOR