Repository: PostHog/posthog
Pull request: feat(apps): Make `setupPlugin` errors `RetryError`-only and send "fatal app error" email
 ## Changes

Resolves #6850 by sending an email whenever a ~~plugin~~ app is disabled by the system due to a VM initialization error (usually in `setupPlugin`). Also makes it so that only `RetryError` result in `setupPlugin` being retried. Both of these are part of the https://github.com/PostHog/posthog/issues/8829#issuecomment-1147363646 action plan.

<img width="734" alt="err" src="https://user-images.githubusercontent.com/4550621/172465447-4916a1c1-9104-4aa2-b5d4-ef4502a85f5b.png">

We only send emails from Python and Celery is the best way to queue emails, but at the same time I really wanted to avoid reverting #9936, so instead I read up on the Celery task message protocol + did some reverse engineering, and ended up adding a small method to `DB` called `celeryApplyAsync()`. It works pretty much like `Task.apply_async()` in Python, just with no bells and whistles.

## How did you test this code?

~~Working on some unit tests.~~ Added tests to `vm.lazy.test.ts`.

Problem statement:
Send an alert when a plugin gets autodisabled
Currently when a plugin errors, it gets autodisabled. This is however completely silent and easy to miss. The result can be degraded data quality or a hole in data in case of export plugins – something to be fixed ASAP.

A PagerDuty integration could come in useful here, allowing an alert to be sent in case of a plugin getting disabled. PagerDuty is tailored for this kind of thing, so this would potentially be better than e.g. an email – that could be easily missed in a sea of emails.

This problem was brought up by a current user of our Snowflake Export Plugin.

Hints from issue discussion:

Commit message: Update comment to be accurate

Patch to review:
diff --git a/plugin-server/src/config/constants.ts b/plugin-server/src/config/constants.ts
index 18cf4e66e2339..6b932bf3cdd3a 100644
--- a/plugin-server/src/config/constants.ts
+++ b/plugin-server/src/config/constants.ts
@@ -1,7 +1,7 @@
 import { logLevel } from 'kafkajs'
 
 export const ONE_HOUR = 60 * 60 * 1000
-
+export const CELERY_DEFAULT_QUEUE = 'celery'
 export const KAFKAJS_LOG_LEVEL_MAPPING = {
     NOTHING: logLevel.NOTHING,
     DEBUG: logLevel.INFO,
diff --git a/plugin-server/src/utils/db/db.ts b/plugin-server/src/utils/db/db.ts
index 06faf2f3ad8f8..7b8c34ee1b6d1 100644
--- a/plugin-server/src/utils/db/db.ts
+++ b/plugin-server/src/utils/db/db.ts
@@ -8,6 +8,7 @@ import { ProducerRecord } from 'kafkajs'
 import { DateTime } from 'luxon'
 import { Pool, PoolClient, QueryResult, QueryResultRow } from 'pg'
 
+import { CELERY_DEFAULT_QUEUE } from '../../config/constants'
 import {
     KAFKA_GROUPS,
     KAFKA_PERSON_DISTINCT_ID,
@@ -432,6 +433,37 @@ export class DB {
         })
     }
 
+    /** Calls Celery task. Works similarly to Task.apply_async in Python. */
+    async celeryApplyAsync(taskName: string, args: any[] = [], kwargs: Record<string, any> = {}): Promise<void> {
+        const taskId = new UUIDT().toString()
+        const deliveryTag = new UUIDT().toString()
+        const body = [args, kwargs, { callbacks: null, errbacks: null, chain: null, chord: null }]
+        /** A base64-encoded JSON representation of the body tuple. */
+        const bodySerialized = Buffer.from(JSON.stringify(body)).toString('base64')
+        await this.redisLPush(CELERY_DEFAULT_QUEUE, {
+            body: bodySerialized,
+            'content-encoding': 'utf-8',
+            'content-type': 'application/json',
+            headers: {
+                lang: 'js',
+                task: taskName,
+                id: taskId,
+                retries: 0,
+                root_id: taskId,
+                parent_id: null,
+                group: null,
+            },
+            properties: {
+                correlation_id: taskId,
+                delivery_mode: 2,
+                delivery_tag: deliveryTag,
+                delivery_info: { exchange: '', routing_key: CELERY_DEFAULT_QUEUE },
+                priority: 0,
+                body_encoding: 'base64',
+            },
+        })
+    }
+
     REDIS_PERSON_ID_PREFIX = 'person_id'
     REDIS_PERSON_UUID_PREFIX = 'person_uuid'
     REDIS_PERSON_CREATED_AT_PREFIX = 'person_created_at'
diff --git a/plugin-server/src/worker/vm/lazy.ts b/plugin-server/src/worker/vm/lazy.ts
index e2721014a5600..4738dc8a6b5ae 100644
--- a/plugin-server/src/worker/vm/lazy.ts
+++ b/plugin-server/src/worker/vm/lazy.ts
@@ -1,3 +1,4 @@
+import { RetryError } from '@posthog/plugin-scaffold'
 import equal from 'fast-deep-equal'
 import { VM } from 'vm2'
 
@@ -34,7 +35,6 @@ export class LazyPluginVM {
     vmResponseVariable: string | null
     pluginConfig: PluginConfig
     hub: Hub
-    lastError: Error | null
     inErroredState: boolean
 
     constructor(hub: Hub, pluginConfig: PluginConfig) {
@@ -44,7 +44,6 @@ export class LazyPluginVM {
         this.vmResponseVariable = null
         this.pluginConfig = pluginConfig
         this.hub = hub
-        this.lastError = null
         this.inErroredState = false
         this.initVm()
     }
@@ -97,7 +96,7 @@ export class LazyPluginVM {
                 // KLUDGE: setupPlugin is retried, meaning methods may fail initially but work after a retry
                 // Schedules on the other hand need to be loaded in advance, so retries cannot turn on scheduled tasks after the fact.
                 await this.createLogEntry(
-                    'Cannot load scheduled tasks because the plugin errored during setup.',
+                    'Cannot load scheduled tasks because the app errored during setup.',
                     PluginLogEntryType.Error
                 )
             }
@@ -155,13 +154,13 @@ export class LazyPluginVM {
                         await this._setupPlugin(vm.vm)
                         this.ready = true
                     }
+                    status.info('🔌', `Loaded ${logInfo}.`)
                     await this.createLogEntry(`Plugin loaded (instance ID ${this.hub.instanceId}).`)
-                    status.info('🔌', `Loaded ${logInfo}`)
                     resolve(vm)
                 } catch (error) {
-                    status.warn('⚠️', error.message)
+                    status.warn('⚠️', `Failed to load ${logInfo}. ${error}`)
                     if (!(error instanceof SetupPluginError)) {
-                        await this.processVmSetupError(error)
+                        await this.processFatalVmSetupError(error, true)
                     }
                     resolve(null)
                 }
@@ -192,39 +191,46 @@ export class LazyPluginVM {
     public async _setupPlugin(vm?: VM): Promise<void> {
         const logInfo = this.pluginConfig.plugin
             ? pluginDigest(this.pluginConfig.plugin)
-            : `pluginConfig with ID '${this.pluginConfig.id}'`
-
-        if (++this.totalInitAttemptsCounter > VM_INIT_MAX_RETRIES) {
-            this.inErroredState = true
-            const failureContextMessage = `Disabling it due to too many retries – tried to load it ${
-                this.totalInitAttemptsCounter
-            } time${this.totalInitAttemptsCounter > 1 ? 's' : ''} before giving up.`
-            await this.processVmSetupError(this.lastError || new Error('unknown error'), failureContextMessage)
-            throw new SetupPluginError(`setupPlugin failed for ${logInfo}. ${failureContextMessage}`)
-        }
-
+            : `plugin config ID '${this.pluginConfig.id}'`
+        this.totalInitAttemptsCounter++
         try {
             await vm?.run(`${this.vmResponseVariable}.methods.setupPlugin?.()`)
             this.ready = true
-            await this.createLogEntry(`setupPlugin completed successfully (instance ID ${this.hub.instanceId}).`)
-            status.info('🔌', `setupPlugin completed successfully for ${logInfo}`)
-            this.lastError = null
+            status.info('🔌', `setupPlugin succeeded for ${logInfo}.`)
+            await this.createLogEntry(`setupPlugin succeeded (instance ID ${this.hub.instanceId}).`)
             void clearError(this.hub, this.pluginConfig)
         } catch (error) {
-            const nextRetryMs =
-                INITIALIZATION_RETRY_MULTIPLIER ** (this.totalInitAttemptsCounter - 1) * INITIALIZATION_RETRY_BASE_MS
-            const nextRetrySeconds = `${nextRetryMs / 1000} s`
-            status.warn('⚠️', `setupPlugin failed for ${logInfo}. Retrying in ${nextRetrySeconds}.`)
-            await this.createLogEntry(
-                `setupPlugin failed (instance ID ${this.hub.instanceId}). Retrying in ${nextRetrySeconds}.`,
-                PluginLogEntryType.Error
-            )
-            await this.createLogEntry(error.message, PluginLogEntryType.Error)
             this.clearRetryTimeoutIfExists()
-            this.lastError = error
-            this.initRetryTimeout = setTimeout(async () => {
-                await this._setupPlugin(vm)
-            }, nextRetryMs)
+            if (error instanceof RetryError && this.totalInitAttemptsCounter < VM_INIT_MAX_RETRIES) {
+                const nextRetryMs =
+                    INITIALIZATION_RETRY_MULTIPLIER ** (this.totalInitAttemptsCounter - 1) *
+                    INITIALIZATION_RETRY_BASE_MS
+                const nextRetrySeconds = `${nextRetryMs / 1000} s`
+                const errorName = `RetryError (attempt ${this.totalInitAttemptsCounter}/${VM_INIT_MAX_RETRIES})`
+                status.warn(
+                    '⚠️',
+                    `setupPlugin failed with ${errorName} for ${logInfo}. Retrying in ${nextRetrySeconds}...`
+                )
+                await this.createLogEntry(
+                    `setupPlugin failed with ${errorName} (instance ID ${this.hub.instanceId}). Retrying in ${nextRetrySeconds}...`,
+                    PluginLogEntryType.Error
+                )
+                this.initRetryTimeout = setTimeout(async () => {
+                    await this._setupPlugin(vm)
+                }, nextRetryMs)
+            } else {
+                this.inErroredState = true
+                const errorInfo =
+                    error instanceof RetryError
+                        ? `RetryError (attempt ${VM_INIT_MAX_RETRIES}/${VM_INIT_MAX_RETRIES})`
+                        : error.toString()
+                await this.processFatalVmSetupError(error instanceof RetryError ? errorInfo : error, false)
+                await this.createLogEntry(
+                    `setupPlugin failed with ${errorInfo} (instance ID ${this.hub.instanceId}). Disabled the app!`,
+                    PluginLogEntryType.Error
+                )
+                throw new SetupPluginError(`setupPlugin failed with ${errorInfo} for ${logInfo}. Disabled the app!`)
+            }
         }
     }
 
@@ -238,14 +244,17 @@ export class LazyPluginVM {
         })
     }
 
-    private async processVmSetupError(error: Error, additionalContext?: string): Promise<void> {
-        void processError(this.hub, this.pluginConfig, error)
-        additionalContext = additionalContext ?? `Error: ${error.message}`
-        await this.createLogEntry(
-            `Plugin failed to load (instance ID ${this.hub.instanceId}). ${additionalContext}`,
-            PluginLogEntryType.Error
-        )
-        void disablePlugin(this.hub, this.pluginConfig.id)
+    private async processFatalVmSetupError(error: Error | string, isSystemError: boolean): Promise<void> {
+        await processError(this.hub, this.pluginConfig, error)
+        await disablePlugin(this.hub, this.pluginConfig.id)
+        await this.hub.db.celeryApplyAsync('posthog.tasks.email.send_fatal_plugin_error', [
+            this.pluginConfig.id,
+            // Using the `updated_at` field for email campaign idempotency. It's safer to provide it to the task
+            // from here, because the value DB may change in the DB while the task is queued.
+            this.pluginConfig.updated_at || null,
+            error.toString(),
+            isSystemError,
+        ])
     }
 
     private async updatePluginCapabilitiesIfNeeded(vm: PluginConfigVMResponse): Promise<void> {
diff --git a/plugin-server/tests/worker/vm.lazy.test.ts b/plugin-server/tests/worker/vm.lazy.test.ts
index ea9bfbf5ecf9e..2e21205b8c2f8 100644
--- a/plugin-server/tests/worker/vm.lazy.test.ts
+++ b/plugin-server/tests/worker/vm.lazy.test.ts
@@ -1,8 +1,10 @@
+import { RetryError } from '@posthog/plugin-scaffold'
+
 import { PluginLogEntrySource, PluginLogEntryType, PluginTaskType } from '../../src/types'
 import { status } from '../../src/utils/status'
 import { LazyPluginVM } from '../../src/worker/vm/lazy'
 import { createPluginConfigVM } from '../../src/worker/vm/vm'
-import { plugin60 } from '../helpers/plugins'
+import { plugin60, pluginConfig39 } from '../helpers/plugins'
 import { disablePlugin } from '../helpers/sqlMock'
 
 jest.mock('../../src/utils/db/error')
@@ -20,6 +22,7 @@ const mockConfig = {
 describe('LazyPluginVM', () => {
     const db = {
         queuePluginLogEntry: jest.fn(),
+        celeryApplyAsync: jest.fn(),
     }
 
     const mockServer: any = {
@@ -68,7 +71,7 @@ describe('LazyPluginVM', () => {
             void initializeVm(vm)
             await vm.resolveInternalVm
 
-            expect(status.info).toHaveBeenCalledWith('🔌', 'Loaded some plugin')
+            expect(status.info).toHaveBeenCalledWith('🔌', 'Loaded some plugin.')
             expect(mockServer.db.queuePluginLogEntry).toHaveBeenCalledWith(
                 expect.objectContaining({
                     instanceId: undefined,
@@ -114,20 +117,22 @@ describe('LazyPluginVM', () => {
             await vm.initialize!('some log info', 'failure plugin')
             await vm.resolveInternalVm
 
-            expect((status.warn as any).mock.calls).toEqual([['⚠️', 'VM creation failed before setupPlugin']])
+            expect((status.warn as any).mock.calls).toEqual([
+                ['⚠️', 'Failed to load failure plugin. Error: VM creation failed before setupPlugin'],
+            ])
 
             // plugin gets disabled
             expect(disablePlugin).toHaveBeenCalledTimes(1)
             expect(disablePlugin).toHaveBeenCalledWith(mockServer, 39)
         })
 
-        it('_setupPlugin handles retries correctly', async () => {
+        it('_setupPlugin handles RetryError succeeding at last', async () => {
             const mockedRun = jest.fn()
             const mockVm = {
                 run: mockedRun,
             }
             mockedRun.mockImplementation(() => {
-                throw new Error('oh no')
+                throw new RetryError()
             })
 
             const lazyVm = createVM()
@@ -137,7 +142,7 @@ describe('LazyPluginVM', () => {
             expect(mockServer.db.queuePluginLogEntry).toHaveBeenLastCalledWith(
                 expect.objectContaining({
                     instanceId: undefined,
-                    message: expect.stringContaining('oh no'),
+                    message: expect.stringContaining('setupPlugin failed with RetryError (attempt 1/5)'),
                 })
             )
             await lazyVm._setupPlugin(mockVm as any)
@@ -145,24 +150,111 @@ describe('LazyPluginVM', () => {
             await lazyVm._setupPlugin(mockVm as any)
 
             expect((status.warn as any).mock.calls).toEqual([
-                ['⚠️', expect.stringContaining('setupPlugin failed for plugin test-maxmind-plugin')],
-                ['⚠️', expect.stringContaining('setupPlugin failed for plugin test-maxmind-plugin')],
-                ['⚠️', expect.stringContaining('setupPlugin failed for plugin test-maxmind-plugin')],
-                ['⚠️', expect.stringContaining('setupPlugin failed for plugin test-maxmind-plugin')],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 1/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 2/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 3/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 4/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
             ])
 
             expect((status.info as any).mock.calls).toEqual([])
 
+            // The 5th, final attempt succeeds because we re-mock the implementation to succeed. Yay!
             mockedRun.mockImplementation(() => 1)
 
-            await lazyVm._setupPlugin(mockVm as any)
+            await expect(lazyVm._setupPlugin(mockVm as any)).resolves.toBeUndefined()
 
             expect((status.info as any).mock.calls).toEqual([
-                ['🔌', expect.stringContaining('setupPlugin completed successfully for plugin test-maxmind-plugin')],
+                ['🔌', expect.stringContaining('setupPlugin succeeded for plugin test-maxmind-plugin')],
             ])
 
-            // plugin never gets disabled
+            // Plugin does not get disabled
             expect(disablePlugin).toHaveBeenCalledTimes(0)
         })
+
+        it('_setupPlugin handles RetryError never succeeding', async () => {
+            const mockedRun = jest.fn()
+            const mockVm = {
+                run: mockedRun,
+            }
+            mockedRun.mockImplementation(() => {
+                throw new RetryError()
+            })
+
+            const lazyVm = createVM()
+
+            await lazyVm._setupPlugin(mockVm as any)
+
+            expect(mockServer.db.queuePluginLogEntry).toHaveBeenLastCalledWith(
+                expect.objectContaining({
+                    instanceId: undefined,
+                    message: expect.stringContaining('setupPlugin failed with RetryError (attempt 1/5)'),
+                })
+            )
+            await lazyVm._setupPlugin(mockVm as any)
+            await lazyVm._setupPlugin(mockVm as any)
+            await lazyVm._setupPlugin(mockVm as any)
+
+            expect(jest.mocked(status.warn).mock.calls).toEqual([
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 1/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 2/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 3/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+                [
+                    '⚠️',
+                    expect.stringContaining(
+                        'setupPlugin failed with RetryError (attempt 4/5) for plugin test-maxmind-plugin'
+                    ),
+                ],
+            ])
+
+            // Setup never succeeds
+            await expect(lazyVm._setupPlugin(mockVm as any)).rejects.toThrow(
+                'setupPlugin failed with RetryError (attempt 5/5) for plugin test-maxmind-plugin'
+            )
+
+            // Plugin gets disabled due to failure
+            expect(disablePlugin).toHaveBeenCalledTimes(1)
+            // An email to project members about the failure is queued
+            expect(mockServer.db.celeryApplyAsync).toHaveBeenCalledWith('posthog.tasks.email.send_fatal_plugin_error', [
+                pluginConfig39.id,
+                null,
+                'RetryError (attempt 5/5)',
+                false,
+            ])
+        })
     })
 })
diff --git a/posthog/models/plugin.py b/posthog/models/plugin.py
index 28d271d77cb50..cb10fd28e0d9f 100644
--- a/posthog/models/plugin.py
+++ b/posthog/models/plugin.py
@@ -175,6 +175,8 @@ def get_default_config(self) -> Dict[str, Any]:
         return config
 
     def __str__(self) -> str:
+        if not self.name:
+            return f"ID {self.id}"
         return self.name
 
     __repr__ = sane_repr("id", "name", "organization_id", "is_global")
diff --git a/posthog/models/team/team.py b/posthog/models/team/team.py
index 82d52f3732631..e34bb0bb38491 100644
--- a/posthog/models/team/team.py
+++ b/posthog/models/team/team.py
@@ -147,18 +147,9 @@ def aggregate_users_by_distinct_id(self) -> bool:
 
     objects: TeamManager = TeamManager()
 
-    def get_effective_membership_level(self, user_id: int) -> Optional["OrganizationMembership.Level"]:
-        """Return an effective membership level.
-        None returned if the user has no explicit membership and organization access is too low for implicit membership.
-        """
-        from posthog.models.organization import OrganizationMembership
-
-        try:
-            requesting_parent_membership: OrganizationMembership = OrganizationMembership.objects.select_related(
-                "organization"
-            ).get(organization_id=self.organization_id, user_id=user_id)
-        except OrganizationMembership.DoesNotExist:
-            return None
+    def get_effective_membership_level_for_parent_membership(
+        self, requesting_parent_membership: "OrganizationMembership"
+    ) -> Optional["OrganizationMembership.Level"]:
         if (
             not requesting_parent_membership.organization.is_feature_available(
                 AvailableFeature.PROJECT_BASED_PERMISSIONING
@@ -167,6 +158,7 @@ def get_effective_membership_level(self, user_id: int) -> Optional["Organization
         ):
             return requesting_parent_membership.level
         from ee.models import ExplicitTeamMembership
+        from posthog.models.organization import OrganizationMembership
 
         try:
             return (
@@ -180,6 +172,20 @@ def get_effective_membership_level(self, user_id: int) -> Optional["Organization
                 return None
             return requesting_parent_membership.level
 
+    def get_effective_membership_level(self, user_id: int) -> Optional["OrganizationMembership.Level"]:
+        """Return an effective membership level.
+        None returned if the user has no explicit membership and organization access is too low for implicit membership.
+        """
+        from posthog.models.organization import OrganizationMembership
+
+        try:
+            requesting_parent_membership: OrganizationMembership = OrganizationMembership.objects.select_related(
+                "organization"
+            ).get(organization_id=self.organization_id, user_id=user_id)
+        except OrganizationMembership.DoesNotExist:
+            return None
+        return self.get_effective_membership_level_for_parent_membership(requesting_parent_membership)
+
     @property
     def actor_on_events_querying_enabled(self) -> bool:
         enabled_teams = get_list(get_instance_setting("ENABLE_ACTOR_ON_EVENTS_TEAMS"))
diff --git a/posthog/tasks/email.py b/posthog/tasks/email.py
index 95fdc2e0499e3..6aa36087fcdf5 100644
--- a/posthog/tasks/email.py
+++ b/posthog/tasks/email.py
@@ -1,6 +1,6 @@
 import uuid
 from datetime import datetime
-from typing import List
+from typing import List, Optional
 
 import posthoganalytics
 import structlog
@@ -10,8 +10,7 @@
 from posthog.celery import app
 from posthog.email import EmailMessage, is_email_available
 from posthog.event_usage import report_first_ingestion_reminder_email_sent, report_second_ingestion_reminder_email_sent
-from posthog.models import Organization, OrganizationInvite, User
-from posthog.models.team import Team
+from posthog.models import Organization, OrganizationInvite, OrganizationMembership, Plugin, PluginConfig, Team, User
 from posthog.utils import absolute_uri
 
 logger = structlog.get_logger(__name__)
@@ -63,6 +62,35 @@ def send_member_join(invitee_uuid: str, organization_id: str) -> None:
         message.send()
 
 
+@app.task(max_retries=1)
+def send_fatal_plugin_error(
+    plugin_config_id: int, plugin_config_updated_at: Optional[str], error: str, is_system_error: bool
+) -> None:
+    plugin_config: PluginConfig = PluginConfig.objects.select_related("plugin", "team").get(id=plugin_config_id)
+    plugin: Plugin = plugin_config.plugin
+    team: Team = plugin_config.team
+    campaign_key: str = f"plugin_disabled_email_plugin_config_{plugin_config_id}_updated_at_{plugin_config_updated_at}"
+    message = EmailMessage(
+        campaign_key=campaign_key,
+        subject=f"[Alert] {plugin} has been disabled in project {team} due to a fatal error",
+        template_name="fatal_plugin_error",
+        template_context={"plugin": plugin, "team": team, "error": error, "is_system_error": is_system_error},
+    )
+    # Don't send this email to the new member themselves
+    memberships_to_email = list(
+        filter(
+            team.get_effective_membership_level_for_parent_membership,
+            OrganizationMembership.objects.select_related("user", "organization").filter(
+                organization_id=team.organization_id
+            ),
+        )
+    )
+    if memberships_to_email:
+        for membership in memberships_to_email:
+            message.add_recipient(email=membership.user.email, name=membership.user.first_name)
+        message.send(send_async=False)
+
+
 @app.task(max_retries=1)
 def send_canary_email(user_email: str) -> None:
     message = EmailMessage(
diff --git a/posthog/templates/email/fatal_plugin_error.html b/posthog/templates/email/fatal_plugin_error.html
new file mode 100644
index 0000000000000..785a70523f56e
--- /dev/null
+++ b/posthog/templates/email/fatal_plugin_error.html
@@ -0,0 +1,26 @@
+{% extends "email/base_transactional.html" %} {% load posthog_assets %} {% load posthog_filters %}
+{% block heading %}Fatal app error{% endblock %}
+{% block section %}
+<!-- prettier-ignore -->
+<p>
+    {% if is_system_error %}
+        A system error has prevented {{ plugin }}
+    {% else %}
+        An unhandled error in the <code>setupPlugin</code> function of {{ plugin }} has prevented it
+    {% endif %} from loading in project {{ team }}:
+</p>
+<pre><code>{{ error }}</code></pre>
+<p>The app has been automatically disabled, but you can re-enable it from the Apps page of PostHog.</p>
+<div class="mb mt text-center">
+    <a class="button" href="{% absolute_uri '/project/apps' %}">Go to Apps</a>
+</div>
+<p>
+    If you find that you're unable to resolve the issue, contact the author of the app or the PostHog team.
+</p>
+<div class="mt">
+    <p>
+        If the button above does not work, paste the following link into your browser:<br />
+        <a href="{% absolute_uri '/project/apps' %}">{% absolute_uri '/project/apps' %}</a>
+    </p>
+</div>
+{% endblock %}
diff --git a/posthog/templates/email/member_join.html b/posthog/templates/email/member_join.html
index 6500e95f06757..96e8bd2b10c77 100644
--- a/posthog/templates/email/member_join.html
+++ b/posthog/templates/email/member_join.html
@@ -1,4 +1,6 @@
-{% extends "email/base_transactional.html" %} {% load posthog_assets %} {% load posthog_filters %} {% block section %}
+{% extends "email/base_transactional.html" %} {% load posthog_assets %} {% load posthog_filters %}
+{% block heading %}{{ invitee.first_name }} accepted their PostHog invitation{% endblock %}
+{% block section %}
 <!-- prettier-ignore -->
 <p>
     {{ invitee.first_name }} (<a href="mailto:{{ invitee.email }}">{{ invitee.email }}</a>) just joined you at {{ organization.name }}.
@@ -12,4 +14,4 @@
         <a href="{% absolute_uri %}">{% absolute_uri %}</a>
     </p>
 </div>
-{% endblock %}{% block heading %}{{ invitee.first_name }} accepted their PostHog invitation{% endblock %}
+{% endblock %}


Code review comments:
diff_hunk: @@ -0,0 +1,26 @@
+{% extends "email/base_transactional.html" %} {% load posthog_assets %} {% load posthog_filters %}
+{% block heading %}Fatal app error{% endblock %}
+{% block section %}
+<!-- prettier-ignore -->
+<p>
+    {% if is_system_error %}
+        A system error has prevented {{ plugin }}
+    {% else %}
+        An unhandled error in the <code>setupPlugin</code> function of {{ plugin }} has prevented it
+    {% endif %} from loading in project {{ team }}:
+</p>
+<pre><code>{{ error }}</code></pre>
+<p>The app has been automatically disabled, but you can re-enable it from the Apps page of PostHog.</p>
path: posthog/templates/email/fatal_plugin_error.html
text: @user1:
maybe we can suggest checking the logs, resolving the error and then re-enabling it?

diff_hunk: @@ -63,6 +62,35 @@ def send_member_join(invitee_uuid: str, organization_id: str) -> None:
         message.send()
 
 
+@app.task(max_retries=1)
+def send_fatal_plugin_error(
+    plugin_config_id: int, plugin_config_updated_at: Optional[str], error: str, is_system_error: bool
+) -> None:
+    plugin_config: PluginConfig = PluginConfig.objects.select_related("plugin", "team").get(id=plugin_config_id)
+    plugin: Plugin = plugin_config.plugin
+    team: Team = plugin_config.team
+    campaign_key: str = f"plugin_disabled_email_plugin_config_{plugin_config_id}_updated_at_{plugin_config_updated_at}"
+    message = EmailMessage(
+        campaign_key=campaign_key,
+        subject=f"[Alert] {plugin} has been disabled in project {team} due to a fatal error",
+        template_name="fatal_plugin_error",
+        template_context={"plugin": plugin, "team": team, "error": error, "is_system_error": is_system_error},
+    )
+    # Don't send this email to the new member themselves
+    memberships_to_email = list(
+        filter(
+            team.get_effective_membership_level_for_parent_membership,
+            OrganizationMembership.objects.select_related("user", "organization").filter(
+                organization_id=team.organization_id
+            ),
+        )
+    )
path: posthog/tasks/email.py
text: @user1:
this was not easy to read, maybe we can instead of the comment make a clearly named small helper function to call `team.get_effective_membership_level_for_parent_membership,`

@author:
Rewrote this to use list comprehension, which should be more obvious

diff_hunk: @@ -192,39 +191,46 @@ export class LazyPluginVM {
     public async _setupPlugin(vm?: VM): Promise<void> {
         const logInfo = this.pluginConfig.plugin
             ? pluginDigest(this.pluginConfig.plugin)
-            : `pluginConfig with ID '${this.pluginConfig.id}'`
-
-        if (++this.totalInitAttemptsCounter > VM_INIT_MAX_RETRIES) {
-            this.inErroredState = true
-            const failureContextMessage = `Disabling it due to too many retries – tried to load it ${
-                this.totalInitAttemptsCounter
-            } time${this.totalInitAttemptsCounter > 1 ? 's' : ''} before giving up.`
-            await this.processVmSetupError(this.lastError || new Error('unknown error'), failureContextMessage)
-            throw new SetupPluginError(`setupPlugin failed for ${logInfo}. ${failureContextMessage}`)
-        }
-
+            : `plugin config ID '${this.pluginConfig.id}'`
+        this.totalInitAttemptsCounter++
         try {
             await vm?.run(`${this.vmResponseVariable}.methods.setupPlugin?.()`)
             this.ready = true
-            await this.createLogEntry(`setupPlugin completed successfully (instance ID ${this.hub.instanceId}).`)
-            status.info('🔌', `setupPlugin completed successfully for ${logInfo}`)
-            this.lastError = null
+            status.info('🔌', `setupPlugin succeeded for ${logInfo}.`)
+            await this.createLogEntry(`setupPlugin succeeded (instance ID ${this.hub.instanceId}).`)
             void clearError(this.hub, this.pluginConfig)
         } catch (error) {
-            const nextRetryMs =
-                INITIALIZATION_RETRY_MULTIPLIER ** (this.totalInitAttemptsCounter - 1) * INITIALIZATION_RETRY_BASE_MS
-            const nextRetrySeconds = `${nextRetryMs / 1000} s`
-            status.warn('⚠️', `setupPlugin failed for ${logInfo}. Retrying in ${nextRetrySeconds}.`)
-            await this.createLogEntry(
-                `setupPlugin failed (instance ID ${this.hub.instanceId}). Retrying in ${nextRetrySeconds}.`,
-                PluginLogEntryType.Error
-            )
-            await this.createLogEntry(error.message, PluginLogEntryType.Error)
             this.clearRetryTimeoutIfExists()
-            this.lastError = error
-            this.initRetryTimeout = setTimeout(async () => {
-                await this._setupPlugin(vm)
-            }, nextRetryMs)
+            if (error instanceof RetryError && this.totalInitAttemptsCounter < VM_INIT_MAX_RETRIES) {
+                const nextRetryMs =
+                    INITIALIZATION_RETRY_MULTIPLIER ** (this.totalInitAttemptsCounter - 1) *
+                    INITIALIZATION_RETRY_BASE_MS
+                const nextRetrySeconds = `${nextRetryMs / 1000} s`
+                const errorName = `RetryError (attempt ${this.totalInitAttemptsCounter}/${VM_INIT_MAX_RETRIES})`
+                status.warn(
+                    '⚠️',
+                    `setupPlugin failed with ${errorName} for ${logInfo}. Retrying in ${nextRetrySeconds}...`
+                )
+                await this.createLogEntry(
+                    `setupPlugin failed with ${errorName} (instance ID ${this.hub.instanceId}). Retrying in ${nextRetrySeconds}...`,
+                    PluginLogEntryType.Error
+                )
+                this.initRetryTimeout = setTimeout(async () => {
+                    await this._setupPlugin(vm)
+                }, nextRetryMs)
+            } else {
+                this.inErroredState = true
+                const errorInfo =
+                    error instanceof RetryError
+                        ? `RetryError (attempt ${VM_INIT_MAX_RETRIES}/${VM_INIT_MAX_RETRIES})`
path: plugin-server/src/worker/vm/lazy.ts
text: @user1:
Doesn't RetryError include a message, let's include it.
Furthermore what about just changing the object here to have the `toString` value be this instead, so we can use the `error` instead of `error | string` in the code below.


@author:
Good idea, updated `RetryError` in https://github.com/PostHog/plugin-scaffold to facilitate this.