Repository: frappe/frappe
Pull request: perf: Re-use SMTP session while sending bulk emails
## Use case: 
Send newsletter emails to 20k recipients.

### Before: 
"Send Now" functionality of newsletter used to timeout from frontend. It used to take more than **15mins** just to queue emails when tried from backend console. Now to actually send all the 20k emails it used take around **5.7hrs** 🕔.
> Rough estimation of time that required to send 20k queued emails separately. Given that there are 5 workers available.
> 20000 (emails) * 5.2 (seconds) / 5 (worker) / 3600 (seconds per hour) = 5.77 hrs 

The main bottleneck due to which the email sending is this slow was SMTP session creation. It takes **~5secs** to establish SMTP session once. (We establish new session everytime we want to send an email)
![prun result for single `sendmail`](https://user-images.githubusercontent.com/13928957/178229802-4b2476b5-9090-4574-8dce-b495e244af18.png)
`%prun` result for single `sendmail`


### After:
- Immediately (almost) release the request by pushing queue creation to the background. 
- It takes **~5.15 mins** ⚡️ to send 20k emails now. How? by sending emails in batch & by re-using the SMTP session. 

There were also bad memory consumption issues. For example, we used to create Email Queue (~20k) without commit (RIP transaction size)🤦‍♂️ & `email_queue` variable was eventually holding 20k instances of "Email Queue" which was not necessary. This is no longer a concern due to batching.

### In Action


https://user-images.githubusercontent.com/13928957/178275670-6e9f1990-b844-43e1-a6a0-83afbacdbcd0.mp4



(Note: used a local instance of SMTP email server to catch all outgoing emails)

Resolves: https://github.com/frappe/frappe/issues/16749


Problem statement:
Newsletter issues
## Description of the issue

- [ ] Schedule feature on Newsletter is not working. Emails do not get sent at the scheduled time.
- [x] The unsubscribe feature is not working. If I click on unsubscribe in the newsletter, it asks me to select email groups to unsubscribe from but it doesn't really unsubscribe on the desk.
- [ ] On clicking send now for large numbers, the emails do not get sent. A timeout error pops up after some mins. A developer has been triggering it manually 3 times now.
- [ ] The speed at which emails get sent is too slow.
- [ ] This one might be an enhancement - If an email ID is present in multiple email groups and Newsletter is sent to all those email groups, the receiver gets spammed. Possible to limit email being received to only once to each unique email ID?

Scale: 27-35k recipients

Hints from issue discussion:
> 2. Unsubscribe feature is not working. If I click on unsubscribe in newsletter, it asks me to select email groups to unsubscribe from but it doesn't really unsubscribe in desk.

Fix: https://github.com/frappe/frappe/pull/16756
@palkanp The problems described are too vague and require a lot of maintainer time to interpret and guess what the possible causes for these would be. Each of the reported issues requires a lot more context in order for us to get something to work with, here. Perhaps, use the issue template if it makes it understand what's relevant to describe what you face.

Commit message: fix: Show "Queued" status on newsletter

Show "Queued" status on newsletter if emails are queued and "email sending" is not yet started.

Patch to review:
diff --git a/frappe/email/doctype/email_group_member/email_group_member.json b/frappe/email/doctype/email_group_member/email_group_member.json
index b3b87c6211b2..0e32135b72fe 100644
--- a/frappe/email/doctype/email_group_member/email_group_member.json
+++ b/frappe/email/doctype/email_group_member/email_group_member.json
@@ -1,151 +1,70 @@
 {
- "allow_copy": 0, 
- "allow_import": 1, 
- "allow_rename": 0, 
- "autoname": "hash", 
- "beta": 0, 
- "creation": "2015-03-18 06:15:59.321619", 
- "custom": 0, 
- "docstatus": 0, 
- "doctype": "DocType", 
- "document_type": "Document", 
- "editable_grid": 0, 
- "engine": "InnoDB", 
+ "actions": [],
+ "allow_import": 1,
+ "autoname": "hash",
+ "creation": "2015-03-18 06:15:59.321619",
+ "doctype": "DocType",
+ "document_type": "Document",
+ "engine": "InnoDB",
+ "field_order": [
+  "email_group",
+  "email",
+  "unsubscribed"
+ ],
  "fields": [
   {
-   "allow_on_submit": 0, 
-   "bold": 0, 
-   "collapsible": 0, 
-   "columns": 0, 
-   "fieldname": "email_group", 
-   "fieldtype": "Link", 
-   "hidden": 0, 
-   "ignore_user_permissions": 0, 
-   "ignore_xss_filter": 0, 
-   "in_filter": 0, 
-   "in_global_search": 0, 
-   "in_list_view": 1, 
-   "in_standard_filter": 1, 
-   "label": "Email Group", 
-   "length": 0, 
-   "no_copy": 0, 
-   "options": "Email Group", 
-   "permlevel": 0, 
-   "precision": "", 
-   "print_hide": 0, 
-   "print_hide_if_no_value": 0, 
-   "read_only": 0, 
-   "remember_last_selected_value": 0, 
-   "report_hide": 0, 
-   "reqd": 1, 
-   "search_index": 0, 
-   "set_only_once": 0, 
-   "unique": 0
-  }, 
+   "fieldname": "email_group",
+   "fieldtype": "Link",
+   "in_list_view": 1,
+   "in_standard_filter": 1,
+   "label": "Email Group",
+   "options": "Email Group",
+   "reqd": 1,
+   "search_index": 1
+  },
   {
-   "allow_on_submit": 0, 
-   "bold": 0, 
-   "collapsible": 0, 
-   "columns": 0, 
-   "fieldname": "email", 
-   "fieldtype": "Data", 
-   "hidden": 0, 
-   "ignore_user_permissions": 0, 
-   "ignore_xss_filter": 0, 
-   "in_filter": 0, 
-   "in_global_search": 1, 
-   "in_list_view": 1, 
-   "in_standard_filter": 0, 
-   "label": "Email", 
-   "length": 0, 
-   "no_copy": 0, 
-   "permlevel": 0, 
-   "precision": "", 
-   "print_hide": 0, 
-   "print_hide_if_no_value": 0, 
-   "read_only": 0, 
-   "remember_last_selected_value": 0, 
-   "report_hide": 0, 
-   "reqd": 1, 
-   "search_index": 0, 
-   "set_only_once": 0, 
-   "unique": 0
-  }, 
+   "fieldname": "email",
+   "fieldtype": "Data",
+   "in_global_search": 1,
+   "in_list_view": 1,
+   "label": "Email",
+   "reqd": 1
+  },
   {
-   "allow_on_submit": 0, 
-   "bold": 0, 
-   "collapsible": 0, 
-   "columns": 0, 
-   "fieldname": "unsubscribed", 
-   "fieldtype": "Check", 
-   "hidden": 0, 
-   "ignore_user_permissions": 0, 
-   "ignore_xss_filter": 0, 
-   "in_filter": 0, 
-   "in_global_search": 0, 
-   "in_list_view": 1, 
-   "in_standard_filter": 0, 
-   "label": "Unsubscribed", 
-   "length": 0, 
-   "no_copy": 0, 
-   "permlevel": 0, 
-   "precision": "", 
-   "print_hide": 0, 
-   "print_hide_if_no_value": 0, 
-   "read_only": 0, 
-   "remember_last_selected_value": 0, 
-   "report_hide": 0, 
-   "reqd": 0, 
-   "search_index": 0, 
-   "set_only_once": 0, 
-   "unique": 0
+   "default": "0",
+   "fieldname": "unsubscribed",
+   "fieldtype": "Check",
+   "in_list_view": 1,
+   "label": "Unsubscribed",
+   "search_index": 1
   }
- ], 
- "hide_heading": 0, 
- "hide_toolbar": 0, 
- "idx": 0, 
- "image_view": 0, 
- "in_create": 0, 
-
- "is_submittable": 0, 
- "issingle": 0, 
- "istable": 0, 
- "max_attachments": 0, 
- "modified": "2017-02-17 17:00:42.551806", 
- "modified_by": "Administrator", 
- "module": "Email", 
- "name": "Email Group Member", 
- "name_case": "", 
- "owner": "Administrator", 
+ ],
+ "links": [],
+ "modified": "2022-07-11 16:38:34.165271",
+ "modified_by": "Administrator",
+ "module": "Email",
+ "name": "Email Group Member",
+ "naming_rule": "Random",
+ "owner": "Administrator",
  "permissions": [
   {
-   "amend": 0, 
-   "apply_user_permissions": 0, 
-   "cancel": 0, 
-   "create": 1, 
-   "delete": 1, 
-   "email": 1, 
-   "export": 1, 
-   "if_owner": 0, 
-   "import": 1, 
-   "permlevel": 0, 
-   "print": 1, 
-   "read": 1, 
-   "report": 1, 
-   "role": "Newsletter Manager", 
-   "set_user_permissions": 0, 
-   "share": 1, 
-   "submit": 0, 
+   "create": 1,
+   "delete": 1,
+   "email": 1,
+   "export": 1,
+   "import": 1,
+   "print": 1,
+   "read": 1,
+   "report": 1,
+   "role": "Newsletter Manager",
+   "share": 1,
    "write": 1
   }
- ], 
- "quick_entry": 1, 
- "read_only": 0, 
- "read_only_onload": 0, 
- "show_name_in_global_search": 0, 
- "sort_field": "modified", 
- "sort_order": "DESC", 
- "title_field": "email", 
- "track_changes": 1, 
- "track_seen": 0
+ ],
+ "quick_entry": 1,
+ "sort_field": "modified",
+ "sort_order": "DESC",
+ "states": [],
+ "title_field": "email",
+ "track_changes": 1
 }
\ No newline at end of file
diff --git a/frappe/email/doctype/email_queue/email_queue.json b/frappe/email/doctype/email_queue/email_queue.json
index f251786c904f..7a4b8dec4542 100644
--- a/frappe/email/doctype/email_queue/email_queue.json
+++ b/frappe/email/doctype/email_queue/email_queue.json
@@ -58,7 +58,8 @@
    "in_list_view": 1,
    "in_standard_filter": 1,
    "label": "Status",
-   "options": "\nNot Sent\nSending\nSent\nError\nExpired"
+   "options": "\nNot Sent\nSending\nSent\nError\nExpired",
+   "search_index": 1
   },
   {
    "fieldname": "error",
@@ -77,13 +78,15 @@
    "fieldtype": "Link",
    "label": "Reference Document Type",
    "options": "DocType",
-   "read_only": 1
+   "read_only": 1,
+   "search_index": 1
   },
   {
    "fieldname": "reference_name",
    "fieldtype": "Data",
    "label": "Reference DocName",
-   "read_only": 1
+   "read_only": 1,
+   "search_index": 1
   },
   {
    "fieldname": "communication",
@@ -152,10 +155,11 @@
  "idx": 1,
  "in_create": 1,
  "links": [],
- "modified": "2021-04-29 06:33:25.191729",
+ "modified": "2022-07-12 12:52:26.290072",
  "modified_by": "Administrator",
  "module": "Email",
  "name": "Email Queue",
+ "naming_rule": "Random",
  "owner": "Administrator",
  "permissions": [
   {
@@ -169,5 +173,6 @@
  ],
  "sort_field": "modified",
  "sort_order": "DESC",
+ "states": [],
  "track_changes": 1
 }
\ No newline at end of file
diff --git a/frappe/email/doctype/email_queue/email_queue.py b/frappe/email/doctype/email_queue/email_queue.py
index 221f3fbb318b..eb07be0b38c2 100644
--- a/frappe/email/doctype/email_queue/email_queue.py
+++ b/frappe/email/doctype/email_queue/email_queue.py
@@ -16,6 +16,7 @@
 from frappe.email.doctype.email_account.email_account import EmailAccount
 from frappe.email.email_body import add_attachment, get_email, get_formatted_html
 from frappe.email.queue import get_unsubcribed_url, get_unsubscribe_message
+from frappe.email.smtp import SMTPServer
 from frappe.model.document import Document
 from frappe.query_builder import DocType, Interval
 from frappe.query_builder.functions import Now
@@ -99,7 +100,7 @@ def attachments_list(self):
 
 	def get_email_account(self):
 		if self.email_account:
-			return frappe.get_doc("Email Account", self.email_account)
+			return frappe.get_cached_doc("Email Account", self.email_account)
 
 		return EmailAccount.find_outgoing(
 			match_by_email=self.sender, match_by_doctype=self.reference_doctype
@@ -115,12 +116,12 @@ def can_send_now(self):
 
 		return True
 
-	def send(self, is_background_task=False):
+	def send(self, is_background_task: bool = False, smtp_server_instance: SMTPServer = None):
 		"""Send emails to recipients."""
 		if not self.can_send_now():
 			return
 
-		with SendMailContext(self, is_background_task) as ctx:
+		with SendMailContext(self, is_background_task, smtp_server_instance) as ctx:
 			message = None
 			for recipient in self.recipients:
 				if not recipient.is_mail_to_be_sent():
@@ -169,21 +170,32 @@ def clear_old_logs(days=30):
 
 
 @task(queue="short")
-def send_mail(email_queue_name, is_background_task=False):
-	"""This is equalent to EmqilQueue.send.
+def send_mail(email_queue_name, is_background_task=False, smtp_server_instance: SMTPServer = None):
+	"""This is equivalent to EmailQueue.send.
 
 	This provides a way to make sending mail as a background job.
 	"""
 	record = EmailQueue.find(email_queue_name)
-	record.send(is_background_task=is_background_task)
+	record.send(is_background_task=is_background_task, smtp_server_instance=smtp_server_instance)
 
 
 class SendMailContext:
-	def __init__(self, queue_doc: Document, is_background_task: bool = False):
+	def __init__(
+		self,
+		queue_doc: Document,
+		is_background_task: bool = False,
+		smtp_server_instance: SMTPServer = None,
+	):
 		self.queue_doc: EmailQueue = queue_doc
 		self.is_background_task = is_background_task
 		self.email_account_doc = queue_doc.get_email_account()
-		self.smtp_server = self.email_account_doc.get_smtp_server()
+
+		self.smtp_server = smtp_server_instance or self.email_account_doc.get_smtp_server()
+
+		# if smtp_server_instance is passed, then retain smtp session
+		# Note: smtp session will have to be manually closed
+		self.retain_smtp_session = bool(smtp_server_instance)
+
 		self.sent_to = [rec.recipient for rec in self.queue_doc.recipients if rec.is_main_sent()]
 
 	def __enter__(self):
@@ -200,11 +212,13 @@ def __exit__(self, exc_type, exc_val, exc_tb):
 			JobTimeoutException,
 		]
 
-		self.smtp_server.quit()
+		if not self.retain_smtp_session:
+			self.smtp_server.quit()
+
 		self.log_exception(exc_type, exc_val, exc_tb)
 
 		if exc_type in exceptions:
-			email_status = (self.sent_to and "Partially Sent") or "Not Sent"
+			email_status = "Partially Sent" if self.sent_to else "Not Sent"
 			self.queue_doc.update_status(status=email_status, commit=True)
 		elif exc_type:
 			if self.queue_doc.retry < get_email_retry_limit():
@@ -216,12 +230,12 @@ def __exit__(self, exc_type, exc_val, exc_tb):
 			email_status = self.is_mail_sent_to_all() and "Sent"
 			email_status = email_status or (self.sent_to and "Partially Sent") or "Not Sent"
 
-			update_fields = {"status": email_status}
-			if self.email_account_doc.is_exists_in_db():
-				update_fields["email_account"] = self.email_account_doc.name
-			else:
-				update_fields["email_account"] = None
-
+			update_fields = {
+				"status": email_status,
+				"email_account": self.email_account_doc.name
+				if self.email_account_doc.is_exists_in_db()
+				else None,
+			}
 			self.queue_doc.update_status(**update_fields, commit=True)
 
 	def log_exception(self, exc_type, exc_val, exc_tb):
@@ -249,6 +263,7 @@ def get_message_object(self, message):
 		return Parser(policy=SMTPUTF8).parsestr(message)
 
 	def message_placeholder(self, placeholder_key):
+		# sourcery skip: avoid-builtin-shadow
 		map = {
 			"tracker": "<!--email_open_check-->",
 			"unsubscribe_url": "<!--unsubscribe_url-->",
@@ -269,7 +284,7 @@ def build_message(self, recipient_email):
 		)
 		message = message.replace(self.message_placeholder("cc"), self.get_receivers_str())
 		message = message.replace(
-			self.message_placeholder("recipient"), self.get_receipient_str(recipient_email)
+			self.message_placeholder("recipient"), self.get_recipient_str(recipient_email)
 		)
 		message = self.include_attachments(message)
 		return message
@@ -304,14 +319,11 @@ def get_receivers_str(self):
 			to_str = ", ".join(self.queue_doc.to)
 			cc_str = ", ".join(self.queue_doc.cc)
 			message = f"This email was sent to {to_str}"
-			message = message + f" and copied to {cc_str}" if cc_str else message
+			message = f"{message} and copied to {cc_str}" if cc_str else message
 		return message
 
-	def get_receipient_str(self, recipient_email):
-		message = ""
-		if self.queue_doc.expose_recipients != "header":
-			message = recipient_email
-		return message
+	def get_recipient_str(self, recipient_email):
+		return recipient_email if self.queue_doc.expose_recipients != "header" else ""
 
 	def include_attachments(self, message):
 		message_obj = self.get_message_object(message)
@@ -628,7 +640,6 @@ def process(self, send_now=False):
 		if not (final_recipients + self.final_cc()):
 			return []
 
-		email_queues = []
 		queue_data = self.as_dict(include_recipients=False)
 		if not queue_data:
 			return []
@@ -636,17 +647,35 @@ def process(self, send_now=False):
 		if not queue_separately:
 			recipients = list(set(final_recipients + self.final_cc() + self.bcc))
 			q = EmailQueue.new({**queue_data, **{"recipients": recipients}}, ignore_permissions=True)
-			email_queues.append(q)
+			send_now and q.send()
 		else:
-			for r in final_recipients:
-				recipients = [r] if email_queues else list(set([r] + self.final_cc() + self.bcc))
-				q = EmailQueue.new({**queue_data, **{"recipients": recipients}}, ignore_permissions=True)
-				email_queues.append(q)
-
-		if send_now:
-			for doc in email_queues:
-				doc.send()
-		return email_queues
+			if send_now and len(final_recipients) >= 1000:
+				# force queueing if there are too many recipients to avoid timeouts
+				send_now = False
+			for recipients in frappe.utils.create_batch(final_recipients, 1000):
+				frappe.enqueue(
+					self.send_emails,
+					queue_data=queue_data,
+					final_recipients=recipients,
+					job_name=frappe.utils.get_job_name(
+						"send_bulk_emails_for", self.reference_doctype, self.reference_name
+					),
+					now=frappe.flags.in_test or send_now,
+					queue="long",
+				)
+
+	def send_emails(self, queue_data, final_recipients):
+		# This is used to bulk send emails from same sender to multiple recipients separately
+		# This re-uses smtp server instance to minimize the cost of new session creation
+		smtp_server_instance = None
+		for r in final_recipients:
+			recipients = list(set([r] + self.final_cc() + self.bcc))
+			q = EmailQueue.new({**queue_data, **{"recipients": recipients}}, ignore_permissions=True)
+			if not smtp_server_instance:
+				email_account = q.get_email_account()
+				smtp_server_instance = email_account.get_smtp_server()
+			q.send(smtp_server_instance=smtp_server_instance)
+		smtp_server_instance.quit()
 
 	def as_dict(self, include_recipients=True):
 		email_account = self.get_outgoing_email_account()
diff --git a/frappe/email/doctype/email_queue_recipient/email_queue_recipient.json b/frappe/email/doctype/email_queue_recipient/email_queue_recipient.json
index 8d1f77e818ef..c217886ce6af 100644
--- a/frappe/email/doctype/email_queue_recipient/email_queue_recipient.json
+++ b/frappe/email/doctype/email_queue_recipient/email_queue_recipient.json
@@ -1,122 +1,46 @@
 {
- "allow_copy": 0, 
- "allow_import": 0, 
- "allow_rename": 0, 
- "beta": 0, 
- "creation": "2016-12-08 12:01:07.993900", 
- "custom": 0, 
- "docstatus": 0, 
- "doctype": "DocType", 
- "document_type": "", 
- "editable_grid": 0, 
- "engine": "InnoDB", 
+ "actions": [],
+ "creation": "2016-12-08 12:01:07.993900",
+ "doctype": "DocType",
+ "engine": "InnoDB",
+ "field_order": [
+  "recipient",
+  "status",
+  "error"
+ ],
  "fields": [
   {
-   "allow_on_submit": 0, 
-   "bold": 0, 
-   "collapsible": 0, 
-   "columns": 0, 
-   "fieldname": "recipient", 
-   "fieldtype": "Data", 
-   "hidden": 0, 
-   "ignore_user_permissions": 0, 
-   "ignore_xss_filter": 0, 
-   "in_filter": 0,
-   "in_list_view": 1, 
-   "label": "Recipient", 
-   "length": 0, 
-   "no_copy": 0, 
-   "options": "Email", 
-   "permlevel": 0, 
-   "precision": "", 
-   "print_hide": 0, 
-   "print_hide_if_no_value": 0, 
-   "read_only": 0, 
-   "remember_last_selected_value": 0, 
-   "report_hide": 0, 
-   "reqd": 0, 
-   "search_index": 0, 
-   "set_only_once": 0, 
-   "unique": 0
-  }, 
+   "fieldname": "recipient",
+   "fieldtype": "Data",
+   "in_list_view": 1,
+   "label": "Recipient",
+   "options": "Email"
+  },
   {
-   "allow_on_submit": 0, 
-   "bold": 0, 
-   "collapsible": 0, 
-   "columns": 0, 
-   "default": "Not Sent", 
-   "fieldname": "status", 
-   "fieldtype": "Select", 
-   "hidden": 0, 
-   "ignore_user_permissions": 0, 
-   "ignore_xss_filter": 0, 
-   "in_filter": 0,
-   "in_list_view": 1, 
-   "label": "Status", 
-   "length": 0, 
-   "no_copy": 0, 
-   "options": "\nNot Sent\nSending\nSent\nError\nExpired", 
-   "permlevel": 0, 
-   "precision": "", 
-   "print_hide": 0, 
-   "print_hide_if_no_value": 0, 
-   "read_only": 0, 
-   "remember_last_selected_value": 0, 
-   "report_hide": 0, 
-   "reqd": 0, 
-   "search_index": 0, 
-   "set_only_once": 0, 
-   "unique": 0
-  }, 
+   "default": "Not Sent",
+   "fieldname": "status",
+   "fieldtype": "Select",
+   "in_list_view": 1,
+   "label": "Status",
+   "options": "\nNot Sent\nSending\nSent\nError\nExpired",
+   "search_index": 1
+  },
   {
-   "allow_on_submit": 0, 
-   "bold": 0, 
-   "collapsible": 0, 
-   "columns": 0, 
-   "fieldname": "error", 
-   "fieldtype": "Code", 
-   "hidden": 0, 
-   "ignore_user_permissions": 0, 
-   "ignore_xss_filter": 0, 
-   "in_filter": 0,
-   "in_list_view": 0, 
-   "label": "Error", 
-   "length": 0, 
-   "no_copy": 0, 
-   "permlevel": 0, 
-   "precision": "", 
-   "print_hide": 0, 
-   "print_hide_if_no_value": 0, 
-   "read_only": 0, 
-   "remember_last_selected_value": 0, 
-   "report_hide": 0, 
-   "reqd": 0, 
-   "search_index": 0, 
-   "set_only_once": 0, 
-   "unique": 0
+   "fieldname": "error",
+   "fieldtype": "Code",
+   "label": "Error"
   }
- ], 
- "hide_heading": 0, 
- "hide_toolbar": 0, 
- "idx": 0, 
- "image_view": 0, 
- "in_create": 0, 
-
- "is_submittable": 0, 
- "issingle": 0, 
- "istable": 1, 
- "max_attachments": 0, 
- "modified": "2016-12-08 14:05:33.578240", 
- "modified_by": "Administrator", 
- "module": "Email", 
- "name": "Email Queue Recipient", 
- "name_case": "", 
- "owner": "Administrator", 
- "permissions": [], 
- "quick_entry": 1, 
- "read_only": 0, 
- "read_only_onload": 0, 
- "sort_field": "modified", 
- "sort_order": "DESC", 
- "track_seen": 0
+ ],
+ "istable": 1,
+ "links": [],
+ "modified": "2022-07-11 16:38:10.644417",
+ "modified_by": "Administrator",
+ "module": "Email",
+ "name": "Email Queue Recipient",
+ "owner": "Administrator",
+ "permissions": [],
+ "quick_entry": 1,
+ "sort_field": "modified",
+ "sort_order": "DESC",
+ "states": []
 }
\ No newline at end of file
diff --git a/frappe/email/doctype/newsletter/newsletter.js b/frappe/email/doctype/newsletter/newsletter.js
index 55805ad48587..9cfdca1a68b8 100644
--- a/frappe/email/doctype/newsletter/newsletter.js
+++ b/frappe/email/doctype/newsletter/newsletter.js
@@ -30,12 +30,12 @@ frappe.ui.form.on('Newsletter', {
 			frm.add_custom_button(__('Send now'), () => {
 				if (frm.doc.schedule_send) {
 					frappe.confirm(__("This newsletter was scheduled to send on a later date. Are you sure you want to send it now?"), function () {
-						frm.call('send_emails').then(() => frm.refresh());
+						frm.events.send_emails(frm);
 					});
 					return;
 				}
-				frappe.confirm(__("Are you sure you want to send this newsletter now?"), function () {
-					frm.call('send_emails').then(() => frm.refresh());
+				frappe.confirm(__("Are you sure you want to send this newsletter now?"), () => {
+					frm.events.send_emails(frm);
 				});
 			}, __('Send'));
 
@@ -44,8 +44,7 @@ frappe.ui.form.on('Newsletter', {
 			}, __('Send'));
 		}
 
-		frm.events.setup_dashboard(frm);
-		frm.events.setup_sending_status(frm);
+		frm.events.update_sending_status(frm);
 
 		if (frm.is_new() && !doc.sender_email) {
 			let { fullname, email } = frappe.user_info(doc.owner);
@@ -56,6 +55,15 @@ frappe.ui.form.on('Newsletter', {
 		frm.trigger('update_schedule_message');
 	},
 
+	send_emails(frm) {
+		frappe.dom.freeze(__("Queuing emails..."));
+		frm.call('send_emails').then(() => {
+			frm.refresh();
+			frappe.dom.unfreeze();
+			frappe.show_alert(__("Queued {0} emails", [frappe.utils.shorten_number(frm.doc.total_recipients)]));
+		});
+	},
+
 	schedule_send_dialog(frm) {
 		let hours = frappe.utils.range(24);
 		let time_slots = hours.map(hour => {
@@ -128,77 +136,33 @@ frappe.ui.form.on('Newsletter', {
 		d.show();
 	},
 
-	setup_dashboard(frm) {
-		if (!frm.doc.__islocal && cint(frm.doc.email_sent)
-			&& frm.doc.__onload && frm.doc.__onload.status_count) {
-			var stat = frm.doc.__onload.status_count;
-			var total = frm.doc.scheduled_to_send;
-			if (total) {
-				$.each(stat, function (k, v) {
-					stat[k] = flt(v * 100 / total, 2) + '%';
-				});
-
-				frm.dashboard.add_progress("Status", [
-					{
-						title: stat["Not Sent"] + " Queued",
-						width: stat["Not Sent"],
-						progress_class: "progress-bar-info"
-					},
-					{
-						title: stat["Sent"] + " Sent",
-						width: stat["Sent"],
-						progress_class: "progress-bar-success"
-					},
-					{
-						title: stat["Sending"] + " Sending",
-						width: stat["Sending"],
-						progress_class: "progress-bar-warning"
-					},
-					{
-						title: stat["Error"] + "% Error",
-						width: stat["Error"],
-						progress_class: "progress-bar-danger"
-					}
-				]);
+	async update_sending_status(frm) {
+		if (frm.doc.email_sent && frm.$wrapper.is(':visible')) {
+			let res = await frm.call('get_sending_status');
+			let stats = res.message;
+			stats && frm.events.update_sending_progress(frm, stats);
+			if (stats.sent + stats.error >= frm.doc.total_recipients || (!stats.total && !stats.emails_queued)) {
+				frm.sending_status && clearInterval(frm.sending_status);
+				frm.sending_status = null;
 			}
 		}
-	},
 
-	setup_sending_status(frm) {
-		frm.call('get_sending_status').then(r => {
-			if (r.message) {
-				frm.events.update_sending_progress(frm, r.message.sent, r.message.total);
-			}
-			if (r.message.sent >= r.message.total) {
-				return;
-			}
-			if (frm.sending_status) return;
-
-			frm.sending_status = setInterval(() => {
-				if (frm.doc.email_sent && frm.$wrapper.is(':visible')) {
-					frm.call('get_sending_status').then(r => {
-						if (r.message) {
-							let { sent, total } = r.message;
-							frm.events.update_sending_progress(frm, sent, total);
-
-							if (sent >= total) {
-								clearInterval(frm.sending_status);
-								frm.sending_status = null;
-								return;
-							}
-						}
-					});
-				}
-			}, 5000);
-		});
+		if (frm.sending_status) return;
+		frm.sending_status = setInterval(() => frm.events.update_sending_status(frm), 5000);
 	},
 
-	update_sending_progress(frm, sent, total) {
-		if (sent >= total) {
+	update_sending_progress(frm, stats) {
+		if (stats.sent + stats.error >= frm.doc.total_recipients || !frm.doc.email_sent) {
+			frm.doc.email_sent && frm.page.set_indicator(__("Sent"), "green");
 			frm.dashboard.hide_progress();
 			return;
 		}
-		frm.dashboard.show_progress(__('Sending emails'), sent * 100 / total, __("{0} of {1} sent", [sent, total]));
+		if (stats.total) {
+			frm.page.set_indicator(__("Sending"), "blue");
+			frm.dashboard.show_progress(__('Sending emails'), stats.sent * 100 / frm.doc.total_recipients, __("{0} of {1} sent", [stats.sent, frm.doc.total_recipients]));
+		} else if (stats.emails_queued) {
+			frm.page.set_indicator(__("Queued"), "blue");
+		}
 	},
 
 	on_hide(frm) {
diff --git a/frappe/email/doctype/newsletter/newsletter.py b/frappe/email/doctype/newsletter/newsletter.py
index 757a8c875b31..b0cbb1993ddb 100644
--- a/frappe/email/doctype/newsletter/newsletter.py
+++ b/frappe/email/doctype/newsletter/newsletter.py
@@ -6,6 +6,7 @@
 import frappe.utils
 from frappe import _
 from frappe.email.doctype.email_group.email_group import add_subscribers
+from frappe.utils.safe_exec import is_job_queued
 from frappe.utils.verified_command import get_signed_params, verify_request
 from frappe.website.website_generator import WebsiteGenerator
 
@@ -35,13 +36,19 @@ def get_sending_status(self):
 			order_by="status",
 		)
 		sent = 0
+		error = 0
 		total = 0
 		for row in count_by_status:
 			if row.status == "Sent":
 				sent = row.count
+			elif row.status == "Error":
+				error = row.count
 			total += row.count
-
-		return {"sent": sent, "total": total}
+		emails_queued = is_job_queued(
+			job_name=frappe.utils.get_job_name("send_bulk_emails_for", self.doctype, self.name),
+			queue="long",
+		)
+		return {"sent": sent, "error": error, "total": total, "emails_queued": emails_queued}
 
 	@frappe.whitelist()
 	def send_test_email(self, email):
@@ -75,7 +82,6 @@ def send_emails(self):
 		self.schedule_sending = False
 		self.schedule_send = None
 		self.queue_all()
-		frappe.msgprint(_("Email queued to {0} recipients").format(self.total_recipients))
 
 	def validate_send(self):
 		"""Validate if Newsletter can be sent."""
@@ -140,7 +146,8 @@ def get_pending_recipients(self) -> list[str]:
 		"""Get list of pending recipients of the newsletter. These
 		recipients may not have receive the newsletter in the previous iteration.
 		"""
-		return [x for x in self.newsletter_recipients if x not in self.get_success_recipients()]
+		success_recipients = set(self.get_success_recipients())
+		return [x for x in self.newsletter_recipients if x not in success_recipients]
 
 	def queue_all(self):
 		"""Queue Newsletter to all the recipients generated from the `Email Group` table"""
diff --git a/frappe/email/doctype/newsletter/newsletter_list.js b/frappe/email/doctype/newsletter/newsletter_list.js
index 9ded6148e0e0..0b82f1c9e4b8 100644
--- a/frappe/email/doctype/newsletter/newsletter_list.js
+++ b/frappe/email/doctype/newsletter/newsletter_list.js
@@ -4,9 +4,9 @@ frappe.listview_settings['Newsletter'] = {
 		if (doc.email_sent) {
 			return [__("Sent"), "green", "email_sent,=,Yes"];
 		} else if (doc.schedule_sending) {
-			return [__("Scheduled"), "orange", "email_sent,=,No|schedule_sending,=,Yes"];
+			return [__("Scheduled"), "purple", "email_sent,=,No|schedule_sending,=,Yes"];
 		} else {
-			return [__("Not Sent"), "orange", "email_sent,=,No"];
+			return [__("Not Sent"), "gray", "email_sent,=,No"];
 		}
 	}
 };
diff --git a/frappe/email/test_email_body.py b/frappe/email/test_email_body.py
index 3ff4245924c4..d5b1013a73c6 100644
--- a/frappe/email/test_email_body.py
+++ b/frappe/email/test_email_body.py
@@ -5,6 +5,7 @@
 import os
 import unittest
 
+import frappe
 from frappe import safe_decode
 from frappe.email.doctype.email_queue.email_queue import QueueBuilder, SendMailContext
 from frappe.email.email_body import (
@@ -54,26 +55,27 @@ def test_prepare_message_returns_already_encoded_string(self):
 		uni_chr1 = chr(40960)
 		uni_chr2 = chr(1972)
 
-		queue_doc = QueueBuilder(
+		QueueBuilder(
 			recipients=["test@example.com"],
 			sender="me@example.com",
 			subject="Test Subject",
-			message="<h1>" + uni_chr1 + "abcd" + uni_chr2 + "</h1>",
+			message=f"<h1>{uni_chr1}abcd{uni_chr2}</h1>",
 			text_content="whatever",
-		).process()[0]
+		).process()
+		queue_doc = frappe.get_last_doc("Email Queue")
 		mail_ctx = SendMailContext(queue_doc=queue_doc)
 		result = mail_ctx.build_message(recipient_email="test@test.com")
 		self.assertTrue(b"<h1>=EA=80=80abcd=DE=B4</h1>" in result)
 
 	def test_prepare_message_returns_cr_lf(self):
-		queue_doc = QueueBuilder(
+		QueueBuilder(
 			recipients=["test@example.com"],
 			sender="me@example.com",
 			subject="Test Subject",
 			message="<h1>\n this is a test of newlines\n" + "</h1>",
 			text_content="whatever",
-		).process()[0]
-
+		).process()
+		queue_doc = frappe.get_last_doc("Email Queue")
 		mail_ctx = SendMailContext(queue_doc=queue_doc)
 		result = safe_decode(mail_ctx.build_message(recipient_email="test@test.com"))
 
diff --git a/frappe/utils/data.py b/frappe/utils/data.py
index 7a1f05220cfb..81ca14368259 100644
--- a/frappe/utils/data.py
+++ b/frappe/utils/data.py
@@ -2114,3 +2114,12 @@ def parse_timedelta(s: str) -> datetime.timedelta:
 		m = TIMEDELTA_BASE_PATTERN.match(s)
 
 	return datetime.timedelta(**{key: float(val) for key, val in m.groupdict().items()})
+
+
+def get_job_name(key: str, doctype: str = None, doc_name: str = None) -> str:
+	job_name = key
+	if doctype:
+		job_name += f"_{doctype}"
+	if doc_name:
+		job_name += f"_{doc_name}"
+	return job_name


Code review comments:
diff_hunk: @@ -58,7 +58,8 @@
    "in_list_view": 1,
    "in_standard_filter": 1,
    "label": "Status",
-   "options": "\nNot Sent\nSending\nSent\nError\nExpired"
+   "options": "\nNot Sent\nSending\nSent\nError\nExpired",
+   "search_index": 1
path: frappe/email/doctype/email_queue/email_queue.json
text: @user1:
There's already a multi-column index where status is the outermost column. 

https://github.com/frappe/frappe/blob/26d0e7a0f93439cec52263411b5c7db482001996/frappe/email/doctype/email_queue/email_queue.py#L370-L372