Repository: oppia/oppia

Issue:
Send email to selected recipients when ML training job fails.
In Oppia ML training jobs are trained separately on Oppia-ml. It may happen that some jobs will fail by exceeding TTL. To keep record of these failed jobs we need facility to send email to selected recipients whenever a training job fails. To implement this feature:

- [x] Add config property on admin page from where admin can add selected recipients' emails.
- [x] Implement feature for sending email to these selected recipients and Oppia admin.


Discussion:
Hi @mc-lovin, you're planning to look at the second part of this too, right? If so, feel free to assign this issue to yourself -- you can do so after accepting the invitation to collaborate at github.com/oppia. Thanks!
This hasn't been worked on for 3 months, so I'm deassigning @mc-lovin. 
Hi @seanlip , @anmolshkl , I would like to work on implementing this feature. Can you give me bit more details about it and how to proceed? Thanks!
@vibhor98, let's pause this for a bit. Some of the basic ML stuff is going to be worked on again in February; until then, I would suggest tackling some of the other issues which are more important.

Let me know if you need recommendations, I can help with that.
@seanlip yes, I need recommendations. Also, in the future I would like to
work on this once the ML stuff starts in February.

On Mon, Jan 22, 2018 at 1:51 AM, Sean Lip <notifications@github.com> wrote:

> @vibhor98 <https://github.com/vibhor98>, let's pause this for a bit. Some
> of the basic ML stuff is going to be worked on again in February; until
> then, I would suggest tackling some of the other issues which are more
> important.
>
> Let me know if you need recommendations, I can help with that.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/oppia/oppia/issues/3691#issuecomment-359277562>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AZZHvFUXAI1h2n6RmKtt4N2rR2OqhmMQks5tM5xXgaJpZM4Oqo36>
> .
>

@seanlip Hi, it seems I need a lot of Prasanna's help for the higher priority issue, so I thought while I wait for him to reply to my questions, I could work on this. Would this implementation be in the cron.py handler or in a different part?
I think in a different part -- you'll want to send the email as soon as the training job fails, rather than on a periodic basis.
(Probably make a call to email_services.py once you've found that a training job has failed.)
@seanlip I'm trying to understand what the first checkbox solved. Does this mean any email sent to the admin would also be sent to those selected recipients?
Log in as an admin, then go to /admin and go to the config tab. The admin should now be able to specify the email addresses of people who should get an email if a training job fails. The email should then be sent to these email addresses.

Resolving patch:
diff --git a/core/controllers/classifier.py b/core/controllers/classifier.py
--- a/core/controllers/classifier.py
+++ b/core/controllers/classifier.py
@@ -23,6 +23,7 @@
 from core.domain import acl_decorators
 from core.domain import classifier_services
 from core.domain import config_domain
+from core.domain import email_manager
 import feconf
 
 
@@ -123,6 +124,9 @@ def post(self):
             classifier_services.get_classifier_training_job_by_id(job_id))
         if classifier_training_job.status == (
                 feconf.TRAINING_JOB_STATUS_FAILED):
+            # Send email to admin and admin-specified email recipients.
+            # Other email recipients are specified on admin config page.
+            email_manager.send_job_failure_email(job_id)
             raise self.InternalErrorException(
                 'The current status of the job cannot transition to COMPLETE.')
         try:
diff --git a/core/domain/email_manager.py b/core/domain/email_manager.py
--- a/core/domain/email_manager.py
+++ b/core/domain/email_manager.py
@@ -328,6 +328,28 @@ def _send_bulk_mail_in_transaction(instance_id=None):
         _send_bulk_mail_in_transaction, instance_id)
 
 
+def send_job_failure_email(job_id):
+    """Sends an email to admin email as well as any email addresses
+    specificed on the admin config page.
+
+    Args:
+        job_id: str. The Job ID of the failing job.
+    """
+    mail_subject = 'Failed ML Job'
+    mail_body = ((
+        'ML job %s has failed. For more information,'
+        'please visit the admin page at:\n'
+        'https://www.oppia.org/admin#/jobs') % job_id)
+    send_mail_to_admin(mail_subject, mail_body)
+    other_recipients = (
+        NOTIFICATION_EMAILS_FOR_FAILED_TASKS.value)
+    if other_recipients:
+        email_services.send_bulk_mail(
+            feconf.SYSTEM_EMAIL_ADDRESS, other_recipients,
+            mail_subject, mail_body,
+            mail_body.replace('\n', '<br/>'))
+
+
 def send_mail_to_admin(email_subject, email_body):
     """Send an email to the admin email address.
 


Test patch:
diff --git a/core/controllers/classifier_test.py b/core/controllers/classifier_test.py
--- a/core/controllers/classifier_test.py
+++ b/core/controllers/classifier_test.py
@@ -23,6 +23,8 @@
 from constants import constants
 from core.controllers import classifier
 from core.domain import classifier_services
+from core.domain import config_domain
+from core.domain import email_manager
 from core.domain import exp_services
 from core.platform import models
 from core.tests import test_utils
@@ -44,6 +46,8 @@ def setUp(self):
             feconf.TESTS_DATA_DIR, 'string_classifier_test.yaml')
         with open(yaml_path, 'r') as yaml_file:
             self.yaml_content = yaml_file.read()
+        self.signup(self.ADMIN_EMAIL, self.ADMIN_USERNAME)
+        self.signup('moderator@example.com', 'mod')
 
         assets_list = []
         with self.swap(feconf, 'ENABLE_ML_CLASSIFIERS', True):
@@ -118,6 +122,68 @@ def test_trained_classifier_handler(self):
             classifier_training_jobs[0].status,
             feconf.TRAINING_JOB_STATUS_COMPLETE)
 
+    def test_email_sent_on_failed_job(self):
+
+        class FakeTrainingJob(object):
+            """Fake training class to invoke failed job functions."""
+            def __init__(self):
+                self.status = feconf.TRAINING_JOB_STATUS_FAILED
+
+        def return_fake_training_job(_):
+            return FakeTrainingJob()
+
+        can_send_emails_ctx = self.swap(
+            feconf, 'CAN_SEND_EMAILS', True)
+        can_send_feedback_email_ctx = self.swap(
+            feconf, 'CAN_SEND_FEEDBACK_MESSAGE_EMAILS', True)
+        fail_training_job = self.swap(
+            classifier_services,
+            'get_classifier_training_job_by_id',
+            return_fake_training_job)
+        config_property = config_domain.Registry.get_config_property(
+            'notification_emails_for_failed_tasks')
+        config_property.set_value(
+            'committer_id', ['moderator@example.com'])
+
+        with can_send_emails_ctx, can_send_feedback_email_ctx:
+            with fail_training_job:
+                # Adding moderator email to admin config page
+                # for sending emails for failed training jobs.
+                self.login(self.ADMIN_EMAIL, is_super_admin=True)
+                response_dict = self.get_json('/adminhandler')
+                response_config_properties = response_dict['config_properties']
+                expected_email_list = {
+                    'value': ['moderator@example.com']}
+                sys_config_list = response_config_properties[
+                    email_manager.NOTIFICATION_EMAILS_FOR_FAILED_TASKS.name]
+                self.assertDictContainsSubset(
+                    expected_email_list, sys_config_list)
+
+                # Check that there are no sent emails to either
+                # email address before posting json.
+                messages = self.mail_stub.get_sent_messages(
+                    to=feconf.ADMIN_EMAIL_ADDRESS)
+                self.assertEqual(len(messages), 0)
+                messages = self.mail_stub.get_sent_messages(
+                    to='moderator@example.com')
+                self.assertEqual(len(messages), 0)
+
+                # Post ML Job.
+                self.post_json(
+                    '/ml/trainedclassifierhandler', self.payload,
+                    expect_errors=True, expected_status_int=500)
+
+                # Check that there are now emails sent.
+                messages = self.mail_stub.get_sent_messages(
+                    to=feconf.ADMIN_EMAIL_ADDRESS)
+                expected_subject = 'Failed ML Job'
+                self.assertEqual(len(messages), 1)
+                self.assertEqual(messages[0].subject.decode(), expected_subject)
+                messages = self.mail_stub.get_sent_messages(
+                    to='moderator@example.com')
+                self.assertEqual(len(messages), 1)
+                self.assertEqual(messages[0].subject.decode(), expected_subject)
+
     def test_error_on_prod_mode_and_default_vm_id(self):
         # Turn off DEV_MODE.
         with self.swap(constants, 'DEV_MODE', False):
diff --git a/core/domain/email_manager_test.py b/core/domain/email_manager_test.py
--- a/core/domain/email_manager_test.py
+++ b/core/domain/email_manager_test.py
@@ -17,6 +17,7 @@
 import datetime
 import types
 
+from core.domain import config_domain
 from core.domain import config_services
 from core.domain import email_manager
 from core.domain import rights_manager
@@ -29,6 +30,48 @@
 (email_models,) = models.Registry.import_models([models.NAMES.email])
 
 
+class FailedMLTest(test_utils.GenericTestBase):
+    """Test that email functionality for sending failed ML Job emails
+    works.
+    """
+
+    def setUp(self):
+        super(FailedMLTest, self).setUp()
+        self.can_send_emails_ctx = self.swap(
+            feconf, 'CAN_SEND_EMAILS', True)
+        self.can_send_feedback_email_ctx = self.swap(
+            feconf, 'CAN_SEND_FEEDBACK_MESSAGE_EMAILS', True)
+        self.login(self.ADMIN_EMAIL, is_super_admin=True)
+        config_property = config_domain.Registry.get_config_property(
+            'notification_emails_for_failed_tasks')
+        config_property.set_value(
+            'committer_id', ['moderator@example.com'])
+
+    def test_send_failed_ml_email(self):
+        with self.can_send_emails_ctx, self.can_send_feedback_email_ctx:
+            # Make sure there are no emails already sent.
+            messages = self.mail_stub.get_sent_messages(
+                to=feconf.ADMIN_EMAIL_ADDRESS)
+            self.assertEqual(len(messages), 0)
+            messages = self.mail_stub.get_sent_messages(
+                to='moderator@example.com')
+            self.assertEqual(len(messages), 0)
+
+            # Send job failure email with mock Job ID.
+            email_manager.send_job_failure_email('123ABC')
+
+            # Make sure emails are sent.
+            messages = self.mail_stub.get_sent_messages(
+                to=feconf.ADMIN_EMAIL_ADDRESS)
+            expected_subject = 'Failed ML Job'
+            self.assertEqual(len(messages), 1)
+            self.assertEqual(messages[0].subject.decode(), expected_subject)
+            messages = self.mail_stub.get_sent_messages(
+                to='moderator@example.com')
+            self.assertEqual(len(messages), 1)
+            self.assertEqual(messages[0].subject.decode(), expected_subject)
+
+
 class EmailRightsTest(test_utils.GenericTestBase):
     """Test that only certain users can send certain types of emails."""