Repository: dmwm/wmcore
Issue: MSTransferor fails to send alerts through Alertmanager

**Impact of the bug**
MStransferor

**Describe the bug**
Following the lead from this Jira ticket [1] we figured out that in some cases MSTransferor fails to send alerts through the newly established mechanism with the AlertManager as a mediator for the alerts.   

Based on the trace provided in the Jira ticket [3] the MSTransferror is just trying to generate an alert for a big dataset (nothing unusual so far), but the url to the Alertmanager itself has been lost somewhere on the way. As mentioned also in this stackoverflow reply [2] this type of badly expressed error is generated by `pycurl` when the url is `None`. I hope this is lost somewhere up in the chain but not in the `@portForward` decorator.

Another critical question that needs to be answered is whether MSTransferor keeps creating input rules over and over for the same input data, similarly to a previously observed cases. 

**How to reproduce it**
To be investigated what exactly has triggered the bug.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context and error message**
[1]
https://its.cern.ch/jira/browse/CMSCOMPPR-18772

[2]
https://stackoverflow.com/a/34402229

[3]
```
2021-04-21 06:14:09,570:ERROR:MSTransferor: Unknown exception while making Transfer Request for pdmvserv_task_TOP-RunIISummer20UL17MiniAODv2-00043__v1_T_210414_061607_9526     Error: unsetopt() is not supported for this option
Traceback (most recent call last):
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSTransferor/MSTransferor.py", line 208, in execute
    success, transfers = self.makeTransferRequest(wflow)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSTransferor/MSTransferor.py", line 628, in makeTransferRequest
    blocks, dataSize, nodes, idx)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSTransferor/MSTransferor.py", line 727, in makeTransferRucio
    self.notifyLargeData(aboveWarningThreshold, transferId, wflow.getName(), dataSize, dataIn)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/MicroService/MSTransferor/MSTransferor.py", line 755, in notifyLargeData
    self.alertManagerApi.sendAlert(alertName, alertSeverity, alertSummary, alertDescription, self.alertServiceName)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/Services/AlertManager/AlertManagerAPI.py", line 93, in sendAlert
    res = self.mgr.getdata(self.alertManagerUrl, params=params, headers=self.headers, verb='POST')
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/Services/pycurl_manager.py", line 316, in getdata
    encode=encode, decode=decode, cookie=cookie)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/Utils/PortForward.py", line 69, in portMangle
    return callFunc(callObj, url, *args, **kwargs)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/Services/pycurl_manager.py", line 280, in request
    verbose, verb, doseq, encode, cainfo, cookie)
  File "/data/srv/HG2104e/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.7.pre8/lib/python2.7/site-packages/WMCore/Services/pycurl_manager.py", line 221, in set_opts
    curl.setopt(pycurl.URL, encodeUnicodeToBytes(url))
TypeError: unsetopt() is not supported for this option
```

Changed files:
src/python/WMCore/MicroService/MSTransferor/MSTransferor.py

diff --git a/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py b/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
index bba92e954..3446205bf 100644
--- a/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
+++ b/src/python/WMCore/MicroService/MSTransferor/MSTransferor.py
@@ -100,7 +100,8 @@ class MSTransferor(MSCore):
         self.blockCounter = 0
         # service name used to route alerts via AlertManager
         self.alertServiceName = "ms-transferor"
-        self.alertManagerApi = AlertManagerAPI(self.msConfig.get("alertManagerUrl", None), logger=logger)
+        self.alertManagerUrl = self.msConfig.get("alertManagerUrl", None)
+        self.alertManagerApi = AlertManagerAPI(self.alertManagerUrl, logger=logger)
 
     @retry(tries=3, delay=2, jitter=2)
     def updateCaches(self):
@@ -752,7 +753,10 @@ class MSTransferor(MSCore):
             alertDescription += "data subscribed: {} TB,\\n".format(teraBytes(dataSize))
             alertDescription += "for {} data: {}.""".format(dataIn['type'], dataIn['name'])
 
-            self.alertManagerApi.sendAlert(alertName, alertSeverity, alertSummary, alertDescription, self.alertServiceName)
+            try:
+                self.alertManagerApi.sendAlert(alertName, alertSeverity, alertSummary, alertDescription, self.alertServiceName)
+            except Exception as ex:
+                self.logger.exception("Failed to send alert to %s. Error: %s", self.alertManagerUrl, str(ex))
             self.logger.info(alertDescription)
 
     def _getValidSites(self, wflow, dataIn):