Repository: getsentry/sentry

Issue:
The weekly reports mail is sent repeatedly 16 times, once every minute
### Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

### Version

23.1.1

### Steps to Reproduce

1、Run `Sentry cron` and `Sentry worker` services
2、`Sentry worker` has three instances

### Expected Result

Receive only one weekly newsletter per week

### Actual Result

Receive 16 Sentry weekly emails every Monday ，Received at one minute intervals，All users within the organization received 16 weekly report emails equally.

<img width="582" alt="image" src="https://user-images.githubusercontent.com/18591662/223436915-ab795659-3095-49f3-9aa6-73742706587b.png">

@Neo-Zhixing  Hi
I suspect it has something to do with this pr, https://github.com/getsentry/sentry/pull/39911, but it is not reproduced in my local development environment and the problem only exists in our production environment. What is the possible cause? Can you give any useful information? Thank you very much!


Discussion:
Assigning to @getsentry/support for [routing](https://open.sentry.io/triage/#2-route), due by **<time datetime=2023-03-08T01:00:00.000Z>Tuesday, March 7th at 5:00 pm</time> (sfo)**. ⏲️
Oh hi @klboke 
Thanks for pinging! It could be due to the rewrite but I can't be entirely sure, since I no longer work here.
My suspicion is that the weekly email task crashed for some unknown reason between the time it sends out the email to the end of the task. Or maybe the email was sent out but an error was reported. The task scheduler might then attempt to re-send the email over and over again. 16 sounds like one of those numbers you would put into the `max_retries` field.

It would be nice if you can take a look at the Sentry error logs, there might be something noteworthy. Self-hosted sentry instances report errors to itself afaik.

Pinging @armenzg

Btw 祝 TapTap 办的越来越好哈哈
Routing to @getsentry/open-source for [triage](https://develop.sentry.dev/processing-tickets/#3-triage), due by **<time datetime=2023-03-09T19:52:46.803Z>Thursday, March 9th at 11:52 am</time> (sea)**. ⏲️
@Neo-Zhixing  Thanks for your reply and best wishes.

- All users within the organization received 16 weekly report emails equally. 

- And this problem can be reliably reproduced in the production environment, although I was only able to reproduce it once (I tried to adjust the task that was scheduled to be sent every Monday to every Tuesday, so I received 16 weekly report emails on Tuesday as well), and the current max_retries = 5. It seems that there is not much significant relevance, but in combination with the 3 worker instances, it seems to be related after all.

- Based on the observed behavior and code analysis, the following method was triggered at least 16 times:
```python
        deliver_reports(
            ctx, dry_run=dry_run, target_user=target_user, email_override=email_override
        )
```
When triggering the weekly report schedule, there were two records in RabbitMQ queue "reports.prepare", indicating that there was no problem with the execution of the "schedule_organizations" method. Therefore, it is likely that the "prepare_organization_report" method was executed repeatedly, resulting in the issue.

- We have a scalable Sentry service deployed in Kubernetes without using Self-hosted, so there seems to be a problem with trace logging. I will look into this issue. Thank you very much for providing this critical information.

Resolving patch:
diff --git a/src/sentry/tasks/weekly_reports.py b/src/sentry/tasks/weekly_reports.py
--- a/src/sentry/tasks/weekly_reports.py
+++ b/src/sentry/tasks/weekly_reports.py
@@ -892,4 +892,4 @@ def send_email(ctx, user, dry_run=False, email_override=None):
         message.send(to=(email_override,))
     else:
         message.add_users((user.id,))
-        message.send()
+        message.send_async()


Test patch: