Repository: dmwm/wmcore
Issue: MicroServices throw UnboundLocalError when sending email alerts from kubernetes/docker containers

**Impact of the bug**
MSTransferor (but it will apply to any system trying to send alerts via SMTP)

**Describe the bug**
This issue is meant to report/address two issues:
1) some missing configuration on our kubernetes/docker setup which does not allow to create a connection on the host:port
2) and likely deal with this alert failure as a soft failure, still allowing the workflow to be moved to staging.

The second issue isn't really a problem at the moment, but it makes MSTransferor to have a "silent" behaviour, in the sense that: when the service fails to send that alert notification, the workflow in question is skipped from that cycle - even though the Rucio rule creation has already happened; then in the next cycle, MSTransferor finds an existent rule (likely in INJECTED/REPLICATING status), so the service assumes data is already available and simply moves that workflow to `staging` without persisting any rule id to be monitored by MSMonitor. Thus, MSMonitor will bypass this workflow right away because there are no rules to be monitored.

**How to reproduce it**
trigger email notification in a pod

**Expected behavior**
Microservices - or any other WMCore service - should be able to send email notifications.

Regarding the MSTransferor behaviour, I think we can log the exception, make a record with the content that was supposed to be sent via email (I think MSTransferor already does that), and move on with the workflow processing as if there were no problems sending the email.

**Additional context and error message**
Some sections of the MSTransferor log under
amaltaro@vocms0750:/cephfs/product/dmwm-logs $ less ms-transferor-20210121-ms-transferor-7744c99cd8-5p6m8.log 
```
2021-01-21 09:41:53,542:INFO:MSTransferor: Checking secondary data location for request: haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987, TrustPUSitelists: False, request white/black list PNNs: set([u'T2_CH_CE
RN'])
2021-01-21 09:41:53,543:INFO:MSTransferor: it has secondary: /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_realistic_v11_L1v1-v2/PREMIX, total size: 697625.14 GB, current disk locations: set([u'T1_FR_CCIN2P3_Disk', 
u'T1_US_FNAL_Disk'])
2021-01-21 09:41:53,543:INFO:MSTransferor: secondary: /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_realistic_v11_L1v1-v2/PREMIX will need data placement!!!
2021-01-21 09:41:53,543:INFO:MSTransferor: Finding final pileup destination for request: haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987
2021-01-21 09:41:53,543:INFO:MSTransferor:   found a PSN list: set([u'T2_CH_CERN']), which maps to a list of PNNs: set([u'T2_CH_CERN'])
2021-01-21 09:41:53,543:INFO:MSTransferor: Handling data subscriptions for request: haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987
2021-01-21 09:41:53,543:INFO:MSTransferor: Have whole PU dataset: /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_realistic_v11_L1v1-v2/PREMIX (697625.14 GB)
2021-01-21 09:41:53,558:DEBUG:connectionpool: http://cms-rucio.cern.ch:80 "GET /rses/?expression=T2_CH_CERN HTTP/1.1" 200 None
2021-01-21 09:41:53,612:DEBUG:connectionpool: http://cms-rucio.cern.ch:80 "GET /rses/T2_CH_CERN/attr/ HTTP/1.1" 200 None
2021-01-21 09:41:53,613:INFO:MSTransferor: Placing whole container, picked RSE: T2_CH_CERN out of an RSE list: T2_CH_CERN
2021-01-21 09:41:53,613:INFO:MSTransferor: Creating rule for workflow haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987 with 1 DIDs in container /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_
realistic_v11_L1v1-v2/PREMIX, RSEs: T2_CH_CERN, grouping: ALL
2021-01-21 09:41:53,664:DEBUG:connectionpool: http://cms-rucio.cern.ch:80 "POST /rules/ HTTP/1.1" 201 None
2021-01-21 09:41:53,664:INFO:MSTransferor: Rules successful created for /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_realistic_v11_L1v1-v2/PREMIX : [u'a383a3c0a23348b8a489b9296a09e454']
2021-01-21 09:41:53,665:ERROR:EmailAlert: Error sending alert email.
Details: [Errno 99] Cannot assign requested address
Traceback (most recent call last):
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.4.pre5/lib/python2.7/site-packages/Utils/EmailAlert.py", line 37, in send
    smtp = smtplib.SMTP(self.serverName)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/smtplib.py", line 256, in __init__
    (code, msg) = self.connect(host, port)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/smtplib.py", line 316, in connect
    self.sock = self._get_socket(host, port, self.timeout)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/smtplib.py", line 291, in _get_socket
    return socket.create_connection((host, port), timeout)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 99] Cannot assign requested address
2021-01-21 09:41:53,667:ERROR:MSTransferor: Unknown exception while making Transfer Request for haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987       Error: local variable 'smtp' referenced before assignment
Traceback (most recent call last):
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.4.pre5/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 207, in execute
    success, transfers = self.makeTransferRequest(wflow)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.4.pre5/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 625, in makeTransferRequest
    blocks, dataSize, nodes, idx)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.4.pre5/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 723, in makeTransferRucio
    self.notifyLargeData(aboveWarningThreshold, transferId, wflow.getName(), dataSize, dataIn)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.4.pre5/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 748, in notifyLargeData
    self.emailAlert.send(emailSubject, emailMsg)
  File "/data/srv/HG2101d/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.4.pre5/lib/python2.7/site-packages/Utils/EmailAlert.py", line 43, in send
    smtp.quit()
UnboundLocalError: local variable 'smtp' referenced before assignment
```

and in the subsequent MSTransferor cycle, this is what happened to that workflow:
```
2021-01-21 09:53:39,553:INFO:MSTransferor: Checking secondary data location for request: haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987, TrustPUSitelists: False, request white/black list PNNs: set([u'T2_CH_CERN'])
2021-01-21 09:53:39,553:INFO:MSTransferor: it has secondary: /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_realistic_v11_L1v1-v2/PREMIX, total size: 697625.14 GB, current disk locations: set([u'T1_FR_CCIN2P3_Disk', u'T2_CH_CERN', u'T1_US_FNAL_Disk'])
2021-01-21 09:53:39,553:INFO:MSTransferor: secondary dataset: /Neutrino_E-10_gun/RunIISummer19ULPrePremix-UL18_106X_upgrade2018_realistic_v11_L1v1-v2/PREMIX already in place. Common locations with site white/black list is: set([u'T2_CH_CERN'])
2021-01-21 09:53:39,553:INFO:MSTransferor: Request haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987 does not have any further data to transfer
2021-01-21 09:53:39,553:INFO:MSTransferor: Transfer requests successful for haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987. Summary: []
2021-01-21 09:53:39,656:INFO:ReqMgrAux: Update in-place: False for transfer doc: haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987 was successful.
2021-01-21 09:53:39,656:INFO:MSTransferor: Transfer document successfully created in CouchDB for: haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987
2021-01-21 09:53:39,656:INFO:MSCore: MSTransferor updating haozturk_task_PPS-RunIISummer19UL18pLHEGEN-00007__v1_T_210121_092904_8987 status to: staging
```


Changed files:
src/python/Utils/EmailAlert.py

diff --git a/src/python/Utils/EmailAlert.py b/src/python/Utils/EmailAlert.py
index e7e34783b..48c1de11c 100644
--- a/src/python/Utils/EmailAlert.py
+++ b/src/python/Utils/EmailAlert.py
@@ -25,10 +25,8 @@ class EmailAlert(object):
     def send(self, subject, message):
         """
         Send an email
-
         :param subject: Email subject
         :param message: Email body
-
         """
         msg = self.EMAIL_HEADER % (self.fromAddr, subject, ", ".join(self.toAddr))
         msg += message
@@ -38,7 +36,11 @@ class EmailAlert(object):
             smtp.sendmail(self.fromAddr, self.toAddr, msg)
         except Exception as ex:
             logging.exception("Error sending alert email.\\nDetails: %s", str(ex))
-        finally:
+
+        try:
             # clean up smtp connection
             smtp.quit()
+        except UnboundLocalError:
+            # it means our client failed connecting to the SMTP server
+            pass