Repository: dmwm/wmcore
Issue: Unhandled exception in condor Plugin when a job changes state before updateSiteInformation process it

When a site status is changed in resource control, condor plugin updates the DESIRED_Sites list ClassAd in the condor jobs. 
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/CondorPlugin.py#L707
ResourceControl passes a list of executing jobs to updateSiteInformation, but if a job in the list succeed or gets cleanout in the mean time, there is an unhandled exception (there is no more classAds to read from as the job changed the status):

2014-06-20 17:16:33,694:ERROR:BossAirAPI:Unhandled exception while calling update method for plugin CondorPlugin
'NoneType' object has no attribute 'get'
2014-06-20 17:16:33,696:ERROR:ResourceControlUpdater:Error occurred, will retry later:
2014-06-20 17:16:33,697:ERROR:ResourceControlUpdater:BossAirException
Message: Unhandled exception while calling update method for plugin CondorPlugin
'NoneType' object has no attribute 'get'
    ModuleName : WMCore.BossAir.BossAirAPI
    MethodName : updateSiteInformation
    ClassInstance : None
    FileName : /data/tier0/srv/wmagent/1.9.28/sw.pre/slc5_amd64_gcc461/cms/wmagent/0.9.49/lib/python2.6/site-packages/WMCore/BossAir/BossAirAPI.py
    ClassName : None
    LineNumber : 821
    ErrorNr : 0

Traceback: 
  File "/data/tier0/srv/wmagent/1.9.28/sw.pre/slc5_amd64_gcc461/cms/wmagent/0.9.49/lib/python2.6/site-packages/WMCore/BossAir/BossAirAPI.py", line 813, in updateSiteInformation
    tempjoblist = pluginInst.updateSiteInformation(jobs, siteName, excludeSite)

  File "/data/tier0/srv/wmagent/1.9.28/sw.pre/slc5_amd64_gcc461/cms/wmagent/0.9.49/lib/python2.6/site-packages/WMCore/BossAir/Plugins/CondorPlugin.py", line 738, in updateSiteInformation
    if siteName in jobAd.get('ExtDESIRED_Sites') and siteName not in jobAd.get('DESIRED_Sites') :


Changed files:
src/python/WMCore/BossAir/Plugins/CondorPlugin.py

diff --git a/src/python/WMCore/BossAir/Plugins/CondorPlugin.py b/src/python/WMCore/BossAir/Plugins/CondorPlugin.py
index cd9c9a045..06d5f7cd8 100644
--- a/src/python/WMCore/BossAir/Plugins/CondorPlugin.py
+++ b/src/python/WMCore/BossAir/Plugins/CondorPlugin.py
@@ -716,34 +716,38 @@ class CondorPlugin(BasePlugin):
         for job in jobs:
             jobID = job['id']
             jobAd = jobInfo.get(jobID)
-            if excludeSite :
-                if siteName in jobAd.get('DESIRED_Sites') and siteName in jobAd.get('ExtDESIRED_Sites') :
-                    usi = jobAd.get('DESIRED_Sites').split(', ')
-                    print len(usi)
-                    if len(usi) > 1 :
-                        usi.remove(siteName)
+
+            if not jobAd :
+                logging.error("No jobAd received for jobID %i"%jobID)
+            else :
+                if excludeSite :
+                    if siteName in jobAd.get('DESIRED_Sites') and siteName in jobAd.get('ExtDESIRED_Sites') :
+                        usi = jobAd.get('DESIRED_Sites').split(', ')
+                        print len(usi)
+                        if len(usi) > 1 :
+                            usi.remove(siteName)
+                            usi = usi.__str__().lstrip('[').rstrip(']')
+                            usi = filter(lambda c: c not in "\\'", usi)
+                            command = 'condor_qedit  -constraint \\'WMAgent_JobID==%i\\' DESIRED_Sites \\'"%s"\\'' %(jobID, usi)
+                            proc = subprocess.Popen(command, stderr = subprocess.PIPE,
+                                                    stdout = subprocess.PIPE, shell = True)
+                            out, err = proc.communicate()
+                        else:
+                            jobtokill.append(job)
+                    else :
+                        logging.error("Cannot find siteName %s in the sitelist" % siteName)
+                else :
+                    if siteName in jobAd.get('ExtDESIRED_Sites') and siteName not in jobAd.get('DESIRED_Sites') :
+                        usi = jobAd.get('DESIRED_Sites').split(', ')
+                        usi.append(siteName)
                         usi = usi.__str__().lstrip('[').rstrip(']')
                         usi = filter(lambda c: c not in "\\'", usi)
                         command = 'condor_qedit  -constraint \\'WMAgent_JobID==%i\\' DESIRED_Sites \\'"%s"\\'' %(jobID, usi)
                         proc = subprocess.Popen(command, stderr = subprocess.PIPE,
                                                 stdout = subprocess.PIPE, shell = True)
                         out, err = proc.communicate()
-                    else:
-                        jobtokill.append(job)
-                else :
-                    logging.error("Cannot find siteName %s in the sitelist" % siteName)
-            else :
-                if siteName in jobAd.get('ExtDESIRED_Sites') and siteName not in jobAd.get('DESIRED_Sites') :
-                    usi = jobAd.get('DESIRED_Sites').split(', ')
-                    usi.append(siteName)
-                    usi = usi.__str__().lstrip('[').rstrip(']')
-                    usi = filter(lambda c: c not in "\\'", usi)
-                    command = 'condor_qedit  -constraint \\'WMAgent_JobID==%i\\' DESIRED_Sites \\'"%s"\\'' %(jobID, usi)
-                    proc = subprocess.Popen(command, stderr = subprocess.PIPE,
-                                            stdout = subprocess.PIPE, shell = True)
-                    out, err = proc.communicate()
-                else :
-                    logging.error("Cannot find siteName %s in the sitelist" % siteName)
+                    else :
+                        logging.error("Cannot find siteName %s in the sitelist" % siteName)
         
         return jobtokill