Repository: dmwm/WMCore
Pull request #12270: Add support for microarchitecture
Author: login: khurtado
Created: 2025-02-21T19:18:01Z | Merged: 2025-03-10T21:01:29Z

Fixes #12168

#### Status
Ready

#### Description
Adds minimum microarchitecture classad based on Tag Collector information.
For x86 architecture, default to "2" when information not in Tag Collector.
When non-x86 arch in list of required architectures, return "0" instead

#### Is it backward compatible (if not, which system it affects?)
YES

#### Related PRs
None

#### External dependencies / deployment changes
Tag Collector URL


Linked issue #12168: CMSSW 15 won't support microarch-v2
**Impact of the new feature**
It was decided at the O&C week that starting from CMSSW 15 we won't build CMSSW for microarch v2. v1 has already been excluded and is not supported anymore. We need to make sure that jobs that will have a requirement for microarch x86_64-v[34] when they need to run on CMSSW15.

**Additional context**
Microarch can be found on the slots as:
```
[mmascher@vocms4100 ~]$ condor_status -af Microarch | sort | uniq -c
     15 x86_64-v1
  12810 x86_64-v2
 113836 x86_64-v3
  38921 x86_64-v4
```
Linked issue comments: user: login: mmascher
created_at: 2024-11-15T16:51:35Z
body: Adding an email from antonio about this:


> Hi Phat,
> 
> Thanks, we discussed this in the SI meeting today. Marco will open an issue, where we have to discuss basically 2 items:
> 
> - how (and who) should be doing the mapping between CMSSW versions and desired micro arch? In my opinion, this should happen in WM, which translates the request constraints into actual resource requirements. 
> 
> - where should we implement the matchmaking constraint for the micro arch, as general condition on the pilots, or rather as additional requirements for the jobs? We discussed it could be done in WM, if this is exclusive to production jobs, or rather insert it as a general condition in the slot matching requirements, if we expect this to somehow also affect analysis jobs in the future.    
> 
> Cheers,
> Antonio
> 
author_association: MEMBER
user: login: srimanob
created_at: 2024-11-15T17:12:44Z
body: Thanks @mmascher 

More discussion is ongoing as sites show concern if their people can't analyze Run 3 data with newer CMSSW. The current proposal is at the end of the comment. (2) is on WM+SI part

==========

(1) Building CMSW with Dual Version Compatibility (v2 + v3):
  + We continue to build CMSW with both v2 and v3, allowing cmssw/scram to select the version that best matches the available CPU architecture.
  + This approach enables analyzers using resources that only support v2 to still access and analyze data with the latest CMSSW versions.
  + All external dependencies are built with v2, maintaining compatibility for systems limited to this architecture. The situation will be the same as HLT at this moment.

(2) Central production:
  + By implementing a matching condition, jobs submitted with CMSSW_15_0_0 (and later) are directed to sites that support v3-only CPUs.
  + Sites restricted to v2 CPUs will continue to receive jobs for Run-2 and Run-3 campaigns (2022, 2023, 2024) as long as we maintain the current campaign setup. This ensures that all sites remain actively engaged in ongoing data processing without requiring immediate hardware upgrades.

(3) Validation:
  + Starting with CMSSW_15_0_0_pre1, we switch from v2 to v3 for validation and maintain validated v3 versions moving forward.
  + The High-Level Trigger (HLT) will benefit from this by not needing to perform separate validation for v3, simplifying the validation process and reducing redundancy.
author_association: NONE
user: login: smuzaffar
created_at: 2024-11-27T07:12:18Z
body: Hello, as @srimanob mentioned, we discussed this in detail during yesterday's [core sw meeting](https://indico.cern.ch/event/1482012/) and, as  @srimanob mentioned, CMSSW_15_0_X will support dual micro-arch for x86_64 with x86-64-v3 being the default micro-architecture.

How can we make sure that Central Production/RelVals use `x86-64-v3` compatible resources for 15.0.X and above release cycle? In future, SCRAM can set an environment variable to expose default micro-arch of a release so that Central Production/RelVals can use that in the job requirement.
author_association: NONE
user: login: belforte
created_at: 2024-11-27T09:18:27Z
body: @mmascher (there's no hook for Antonio here). Please link to the discussion issue in SI mentioned in the [mail](https://github.com/dmwm/WMCore/issues/12168#issuecomment-2479423087) from Antonio

Of course support is needed for analysis jobs as well.

Now we multiple threads on this same topic (this, the one in CRABClient mentioned above, the SI one, discussion in CMSSW https://indico.cern.ch/event/1482012/ ). I'd rather like a single document where people comment and which quickly converge to the specification for the solution.
Can SW-Core do that ? @smuzaffar 

author_association: MEMBER
user: login: smuzaffar
created_at: 2024-11-27T10:14:44Z
body: @belforte , why not just use this issue as a single doc then :-)

About "mapping between CMSSW versions and desired micro arch", I think it will be easier if SCRAM set this via an environment variable. This way you do not need to hardcode this mapping in WM. e.g in CMSSW_15_0_X scram can set `CMSSW_DEFAULT_MICRO_ARCH=x86-64-v3` and  Relvals/Central production use it to request resources. For Central production/Relvals we prefer to use default microarch.

For analysis jobs, where some users might want to use `v2` resources, SCRAM can set `CMSSW_MICRO_ARCHS` env to [comma separated list of micro-archs](https://github.com/dmwm/CRABClient/issues/5354#issue-2697558634)  and crab can make use of that to request the correct resources.

In case these env variable are not set ( e.g for CMSSW_14_2 and earlier) then just do what we are doing now.

FYI @makortel 
author_association: NONE
user: login: belforte
created_at: 2024-11-27T11:23:37Z
body: Assuming that  `CMSSW_MICRO_ARCHS` is set after `cmsenv`, adding a classAd  `MY.DESIRED_MICROARCH="x86-64-v2,x86-64-v3,..."` is easy.
And I presume that's what SI will give us. I do not want to make site lists  based on their hardware. Also some sites may have a mix, right ? @mmascher do you agree ?

I worry about the overall picture. Of course we should try to make it fully transparent for as many users as possible. Hopefully only those who develop for CMSSW_15 on v2 machines will have to do something. So I have to worry about what other changes to do in CRAB besides the above.

my questions are more like
- does this affect where DM (and CRAB tape recall) place datasets ?
- are those v2 machines grouped at specific sites or scattered all over ?
- are there sites which are mostly/only v2 ?
- what about jobs which run at sites other then where the data is ?
- will it matter which release was a dataset produced with ? Should we at least make sure that all data produced with CMSSW_15+ land at sites with e.g. >50% v3 machines ?


author_association: MEMBER
user: login: makortel
created_at: 2024-11-27T15:22:00Z
body: One question to WMCore (say @amaltaro @anpicci) came up in the core software meeting yesterday: does WMCore (production, not CRAB) run `cmsRun` from a release area, or does it create a developer area (`cmsrel CMSSW_X_Y_Z` / `scram project CMSSW_X_Y_Z`) first?
author_association: NONE
user: login: makortel
created_at: 2024-11-27T16:10:46Z
body: Another question to PdmV (@AdrianoDee @dickychant @miquork): on what sites are the RelVals being run nowadays?

In yesterday's core software meeting we had a feeling that the solution on WMCore side to limit production (including RelVal) jobs to x86-64-v3 might not be in place by CMSSW_15_0_0_pre1 (scheduled to be built on December 10), where core software is presently planning to switch the default from v2 to v3 (and where this change would be validated). We assumed PdmV would be able to add the necessary resource matchmaking criteria at the submission time, but then wondered if already the set of sites (e.g. FNAL?) used for RelVals would, in practice, guarantee x86-64-v3-only hardware.
author_association: NONE
user: login: AdrianoDee
created_at: 2024-11-27T16:14:51Z
body: @makortel currently only at FNAL. We could re-include CERN now that the data taking has ended. But we have no urgency or specific need to do so (FNAL resources are more than enough).
author_association: NONE
user: login: belforte
created_at: 2024-11-27T16:48:31Z
body: stupid Q. What about `v4` ? Will v3-built exe run there ?
author_association: MEMBER
user: login: smuzaffar
created_at: 2024-11-27T16:51:18Z
body: > stupid Q. What about `v4` ? Will v3-built exe run there ?

yes. Any exe build for vN should be able to run on vN+x ( where x>=0). So v2 exec should be able to run on v2, v3 and v4
author_association: NONE
user: login: makortel
created_at: 2024-11-27T17:00:38Z
body: > > stupid Q. What about `v4` ? Will v3-built exe run there ?
> 
> yes. Any exe build for vN should be able to run on vN+x ( where x>=0). So v2 exec should be able to run on v2, v3 and v4

Yeah, today we run v2 binaries on all v2, v3, and v4 hardware.
author_association: NONE
user: login: belforte
created_at: 2024-11-27T17:31:24Z
body: I know that what Matti [asked](https://github.com/dmwm/WMCore/issues/12168#issuecomment-2504147766) was not a question meant for CRAB. But since CRAB wrapper set ups the env. for cmsRun with same tools as WMA does, I  looked at the [code](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMRuntime/Tools/Scram.py) and confirmed my observation that a developer area is created and then the environment settings from `scramv1 run -sh` are copied into the wrapper memory and passed to subprocesses  as needed. Of course Alan can still correct me if I got it wrong.

author_association: MEMBER
user: login: makortel
created_at: 2024-11-27T18:57:13Z
body: Thanks @belforte. So both WMA and CRAB behave similarly in this regard.

@smuzaffar Reading your slide 5 again from yesterday, do I understand correctly that in the case that a job lands on a v2-only node and sets up the developer area, that developer area would set up the full multi-arch behavior including the selection of best microarchitecture (v2 in this case) by scram?
author_association: NONE
user: login: smuzaffar
created_at: 2024-11-28T10:09:29Z
body: Yes @makortel , (though not implemented yet but) idea is that if one creates a dev area on v2-only node then scram should automatically

- enable multi-microarch  support when `scram project` is called
- set env for best microarch and in this case it should be `v2`

All cmssw jobs should work fine as long as any part (shared libs/plugins) of cmssw was not (re)build in dev area before submitting the job. So if Central Production/Relval jobs just create cmssw dev area and run cmsDriver/cmsRun then it should work fine (regardless of the micro-arch of the node where it runs). 

Problem I see is for crab jobs where user do checkout part of cmssw and build and submit. If someone submits a job with `v3` only builds (default microarch) and it then lands on `v2` node then all the user libs will be ignored.
author_association: NONE
user: login: mapellidario
created_at: 2024-11-28T11:13:00Z
body: Stefano's conclusions are correct

> confirmed my observation that a developer area is created and then the environment settings from `scramv1 run -sh` are copied into the wrapper memory and passed to subprocesses as needed. 

However, the technical details are slightly different. WMCore runs cmssw from [WMCore/WMSpec/Steps/Executors/CMSSW.py](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py), and uses [Scram.py](https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMRuntime/Tools/Scram.py) for the pset tweak and a [bash](https://github.com/dmwm/WMCore/blob/83c08ffec544faa9c0ec62c7121dd0f5fda1632b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L410) script inside a [popen](https://github.com/dmwm/WMCore/blob/83c08ffec544faa9c0ec62c7121dd0f5fda1632b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L284) for cmsRun. An example of the arguments is [1]. 

Long story short, wmcore creates a developer area

https://github.com/dmwm/WMCore/blob/83c08ffec544faa9c0ec62c7121dd0f5fda1632b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L447

then loads into the env the output of `scramv1 runtime -sh`

https://github.com/dmwm/WMCore/blob/83c08ffec544faa9c0ec62c7121dd0f5fda1632b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L454

then executes cmsRun

https://github.com/dmwm/WMCore/blob/83c08ffec544faa9c0ec62c7121dd0f5fda1632b/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L461

---

[1] 

<details>

ref: https://cmsweb.cern.ch/reqmgr2/config?name=cmsunified_task_BTV-Run3Summer22NanoAODv12-00044__v1_T_241118_130440_5190

```plaintext
.steps.cmsRun1.application.section_('controls')
.steps.cmsRun1.application.section_('multicore')
.steps.cmsRun1.application.multicore.eventStreams = 0
.steps.cmsRun1.application.multicore.numberOfCores = 4
.steps.cmsRun1.application.section_('configuration')
.steps.cmsRun1.application.configuration.cacheName = 'reqmgr_config_cache'
.steps.cmsRun1.application.configuration.configId = 'fe55d6061c26d6e60fd635da09b033a0'
.steps.cmsRun1.application.configuration.retrieveConfigUrl = 'https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/fe55d6061c26d6e60fd635da09b033a0/configFile'
.steps.cmsRun1.application.configuration.pickledarguments = b'(dp0\nVglobalTag\np1\nV132X_mcRun3_2022_realistic_v3\np2\ns.'
.steps.cmsRun1.application.configuration.configCacheUrl = 'https://cmsweb.cern.ch/couchdb'
.steps.cmsRun1.application.configuration.section_('arguments')
.steps.cmsRun1.application.configuration.arguments.globalTag = '132X_mcRun3_2022_realistic_v3'
.steps.cmsRun1.application.configuration.configUrl = 'https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/fe55d6061c26d6e60fd635da09b033a0'
.steps.cmsRun1.application.section_('setup')
.steps.cmsRun1.application.setup.scramCommand = 'scramv1'
.steps.cmsRun1.application.setup.cmsswVersion = 'CMSSW_13_2_6_patch2'
.steps.cmsRun1.application.setup.scramProject = 'CMSSW'
.steps.cmsRun1.application.setup.softwareEnvironment = ''
.steps.cmsRun1.application.setup.buildArch = None
.steps.cmsRun1.application.setup.scramArch = ['el8_amd64_gcc11']
.steps.cmsRun1.application.section_('gpu')
.steps.cmsRun1.application.gpu.gpuRequirements = None
.steps.cmsRun1.application.gpu.gpuRequired = 'forbidden'
.steps.cmsRun1.application.section_('command')
.steps.cmsRun1.application.command.configurationHash = None
.steps.cmsRun1.application.command.executable = 'cmsRun'
.steps.cmsRun1.application.command.arguments = ''
.steps.cmsRun1.application.command.configuration = 'PSet.py'
.steps.cmsRun1.application.command.psetTweak = None
.steps.cmsRun1.application.command.configurationPickle = 'PSet.pkl'
```

</details> 


author_association: MEMBER
user: login: anpicci
created_at: 2024-11-28T11:18:35Z
body: @makortel FYI, Dario provided a reply for your question targeting WMCore
author_association: CONTRIBUTOR
user: login: anpicci
created_at: 2024-11-28T11:33:14Z
body: By the way, any development from WMCore on this topic will likely materialize in the next quarter, since IMU we still lack a common consensus about where to implement the binding to the specific microarchitecture, and our effort is committed to finalizing Q3 dev issues and urgent operation activities. According to the discussion here, I would support @belforte's proposal, and I am not convinced WM is the right place to introduce this binding.
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2024-11-28T13:37:51Z
body: Thanks @smuzaffar for your summary. Let me rephrase from our (WM) POV to make sure I understood:
* SI will be in charge of running jobs on machines which match the microarch version specified (or a later one)
* WMA will need to request v3 for CMSSW_15 and later.
* CRAB will need to figure out (from SCRAM_MICRO_ARCHITECTURE) where the user application can run and ask for the proper microarch.
* We still do not know where those v2-only resources are and if there are full v2-only sites etc.  so can't think how to deal with microarch-aware data placement or microarch-aware data access over WAN (a new twist to "overflow" ?)
author_association: MEMBER
user: login: mmascher
created_at: 2024-11-28T16:02:48Z
body: Hello all,

Just for my understanding, is this a use case that will happen very often, or is it something that, in general, is going to change rarely? I see WMAgent and CRAB are defining respecively:

```
CMSSW_Versions = "CMSSW_14_2_0_pre4"
CRAB_JobSW = "CMSSW_9_2_6"
```

It's very easy to add a job's requirement like (for example):

```
ifthenelse(regexp("$CMSSW_1[56789].*", CMSSW_Versions), Microarch=="x86_64-v3" || Microarch=="x86_64-v4", True)
```

We can possibly turn this into a "table" with CMSSW version and list of supported achitecture with some nested if.

I would still provision machines solely based on DESIRED_Sites, otherwise Factory ops will need to check all the worker nodes at all sites, and update the static factory xml configuration file with the information about sites.
author_association: MEMBER
user: login: smuzaffar
created_at: 2024-11-28T16:27:30Z
body: thanks @mmascher , your suggestion will work if we only allow to use the default micro-arch of a release. So yes for CMSSW_15_0 and above it can always request resources with `x86_64-v3`  (I think something like `Microarch>="x86_64-v3"`)` should also [work](https://indico.cern.ch/event/1274213/contributions/5571146/attachments/2715117/4723480/MicroArch.pdf). 

But we also need to allow running `CMSSW_15_0` on `Microarch>="x86_64-v2"` if explicitly requested by user.  As @belforte mentioned [above](https://github.com/dmwm/WMCore/issues/12168#issuecomment-2503621345) , support for something like `MY.DESIRED_MICROARCH=x86_64-v2` which then can be converted to `Microarch>="x86_64-v2"` should allow us to use `v2` resources too
author_association: NONE
user: login: belforte
created_at: 2024-11-28T16:41:24Z
body: yes @mmascher , as @smuzaffar said CRAB situation is trickier.
Maybe we can have a CMSSW-version dependent default but if MY.DESIRED_MICROARCH is present it needs to be respected. It *think* that I can avoid to specify a list, but only indicate the minimum required value for microarch.

I hope too that we can avoid using the microarch in the provisioning, but if some jobs have a requirement for v2 and v2 is a scarce resource, it may not work. See GPU e.g. We need to have some understanding of where those v2's are to define policies that will work for us. I'd rather not try to handle this  "a priori".
Does anybody have a clue about where those v2 machines are ? A small fraction here and there will not matter, whole sites are a different story.

author_association: MEMBER
user: login: smuzaffar
created_at: 2024-11-28T16:59:14Z
body: > I can avoid to specify a list, but only indicate the minimum required value for microarch

@belforte , I was thinking the same and I think scram env variable can help you there. Instead of setting a comma separated list , scram can just set the min require micro-arch
author_association: NONE
user: login: smuzaffar
created_at: 2024-12-02T15:02:19Z
body: @mmascher , I see this in the `Job.submit` file
```
# These attributes help gWMS decide what platforms this job can run on; see https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsMatchArchitecture
+REQUIRED_ARCH = "X86_64"
+DESIRED_CMSDataset = undefined
```

So how hard it is to add support for `+DESIRED_MICROARCH` which can then be converted to `Microarch>="${DESIRED_MICROARCH}"` ? 

We want to enable multi-arch with x86-64-v3 as default for 15.0.0.pre1 (which should be built on 12th Dec). If this can not be implemented before 15.0.0.pre1 then I am fine with the CMSSW version to microarch mapping
```
ifthenelse(regexp("^CMSSW_(2[0-9]+|1[5-9]+)_.*", CMSSW_Versions), Microarch>="x86_64-v3"", True)
```

At least this will allow us to validate 15.0.0.pre1 for `x86-64-v3`
author_association: NONE
user: login: mmascher
created_at: 2024-12-02T15:36:15Z
body: @belforte , we can do do something similar to ARCH, i.e., se in the job ad:

```
+REQUIRED_MICROARCH = "any" #default value if not set in the jobad
```

or

```
+REQUIRED_MICROARCH = "x86_64-v3,x86_64-v4" #job selects specific list of micro architectures
```

Then add the necessary matchmaking expressions in the machine start expression.
author_association: MEMBER
user: login: smuzaffar
created_at: 2024-12-02T15:44:55Z
body: thanks @mmascher , this will help.
By the way, do you really need comma separated list of microarch i.e. `+REQUIRED_MICROARCH = "x86_64-v3,x86_64-v4"` for matchmaking?  This means for cmssw `x86-64-v2` one has to set  `+REQUIRED_MICROARCH = "x86-64-v2,x86_64-v3,x86_64-v4"` as `v2` can run on any machines with v2 and above . 
author_association: NONE
user: login: belforte
created_at: 2024-12-02T16:04:59Z
body: the problem with that list is not just that it is long and hard to read (well...once coded it will be consumed by machines, when debugging is not needed :frowning_face:  ), but that as soon as a new, compatible microarch version is added, we do not match that hardware until we modify the code.  @mmascher how difficult would it be to parse  the `Microarch` machine classAd to extract the version number so that 
* job indicates e.g. `+REQUIRED_MINIMUM_MICROARCH_VERSION = 3`
* execution slots are matched on `Target.Microarch-version >= MY.REQUIRED_MINIMUM_MICROARCH_VERSION`

?
author_association: MEMBER
user: login: belforte
created_at: 2024-12-02T16:08:29Z
body: @mmascher 
is this
```quote
Then add the necessary matchmaking expressions in the machine start expression.
```
something that CRAB should do ? Or will SI do it ? I was thinking that matchmaking was on you, but could be wrong.

author_association: MEMBER
user: login: mmascher
created_at: 2024-12-02T16:10:49Z
body: SI will do it.

> how difficult would it be to parse the Microarch machine classAd to extract the version 

As long as the format is `x86-64-vN` and it doews not change it should be fairly easy.

FYI: this was just discussed at the DMWM development chat.
author_association: MEMBER
user: login: smuzaffar
created_at: 2024-12-02T16:32:36Z
body: For `REQUIRED_ARCH = "X86_64"` , microarch format should remain of the form `x86-64-vN` 
author_association: NONE
user: login: belforte
created_at: 2024-12-02T16:49:57Z
body: Is there a written output of the WMDM chat ?
author_association: MEMBER
user: login: makortel
created_at: 2024-12-02T19:16:47Z
body: > * We still do not know where those v2-only resources are and if there are full v2-only sites etc.  so can't think how to deal with microarch-aware data placement or microarch-aware data access over WAN (a new twist to "overflow" ?)

I'm aware that T2_FI_HIP has a "sizable" fraction of v2-only resources (1 of 2 clusters, but I don't know what that means in terms of CPU core count or HS06/23). I'm tagging @stlammel in case he'd have more details (or would SI's monitoring show these details? @mmascher).
author_association: NONE
user: login: stlammel
created_at: 2024-12-02T19:46:10Z
body: The v2 site information is of course not static and will change.
Right now T2_LB_HPC4L is the only site with exclusive v2 worker nodes and the majority of worker nodes at T2_BR_SPRACE, T2_KR_KISTI, T2_UA_KIPT, and T2_US_Vanderbilt are v2. For all other sites v2 is a small, tiny, or zero.
So, one could probably handle the work assignment with just a one-site blacklist.
 - Stephan
author_association: NONE
user: login: belforte
created_at: 2024-12-02T21:54:48Z
body: Hopefully HeavyIon community will not jump on using CMSSW_15 then. But the situation at Vanderbilt may become "a disturbance" with next year's HI run. 
author_association: MEMBER
user: login: stlammel
created_at: 2024-12-02T22:52:48Z
body: It could be that the v2 resources at Vanderbilt are in opportunistic clusters and not "CMS owned" resources. USCMS funds the Tier-2s quite well, so i don't think Vanderbilt will be an issue/bottleneck in the future.
 - Stephan
author_association: NONE
user: login: makortel
created_at: 2024-12-03T22:01:13Z
body: @smuzaffar By the way, should we switch the CMSSW IBs to the intended pre1 microarchitecture setup in the coming days to test it widely for a few rounds before the pre1 is built? (I'm not really concerned of problems, but just in case).
author_association: NONE
user: login: smuzaffar
created_at: 2024-12-03T22:16:55Z
body: yes @makortel , I am testing the changes locally and hopefully I will try to integrate those for DEVEL IBs tomorrow
author_association: NONE
user: login: amaltaro
created_at: 2024-12-03T22:36:11Z
body: Apologies for the delay in joining this discussion.

I share the same concerns that Stefano expressed in this comment:
https://github.com/dmwm/WMCore/issues/12168#issuecomment-2511939649

IMO, we should implement it such that new microarchitectures will be automatically considered by the overall machinery (SI, WM, CRAB).

Having said that, defining in WM/CRAB something like:
```
if cmssw < cmssw_15:
  min_micro_arch = 2
else
  min_micro_arch = 3
```

and having a job matchmaking expression in SI like
```
node.micro_arch >= job.min_micro_arch
```

looks the best commitment - and a sustainable solution -  to me.

Of course, if CRAB wants to enforce a specific version, then this expression needs to be revisited.

Dario, thank you for sharing these details in your comment https://github.com/dmwm/WMCore/issues/12168#issuecomment-2505866290, I agree with everything on that post.
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2024-12-04T08:50:57Z
body: thanks @amaltaro . I love concise recaps like that !
AFAIU CRAB needs a bit more of care since users could  e.g.  setup `CMSSW_15` and do `scram b` on a `v2` machine. Ending up with a sandbox which needs to be run on `v2`. 
About that: is the name of the new env.var. finalized ? Will it be present after `cmsenv` for all releases starting from .. Jan 1st  or shall I be prepared for it being not defined ?

BTW,  how do I tell the microarch of the machine I am logged on ?

author_association: MEMBER
user: login: stlammel
created_at: 2024-12-04T14:23:46Z
body: On newer Linux versions RHEL 8 and above you can just execute

/lib64/ld-linux-x86-64.so.2 --help

to get the microarchitecture information.
 - Stephan
author_association: NONE
user: login: smuzaffar
created_at: 2024-12-05T13:38:19Z
body: We now have first IB `CMSSW_15_0_DEVEL_X_2024-12-05-0900` for `el8_amd64_gcc12` which has 

- `x86-64-v3` as default micro arch
- `x86-64-v2` as additional micro-arch

SCRAM sets two env variables 

- `SCRAM_DEFAULT_MICROARCH`: Default micro-arch used for cmssw release
- `SCRAM_MIN_SUPPORTED_MICROARCH`: Least required micro-arch to run cmssw

This is what scram will be doing for this type of release

- On a build node which has `v3` or above
  - `scram project` will create a dev area **without multi microarchs**. Any locally checked out cmssw packages will be build only for default micro arch. 
  - `cmsenv` will set normal cmssw env and both above mentioned variables will be set to `x86-64-v3`
  - CRAB jobs submitted for this area should can run only on nodes with `v3` and above support
- On a build node with has `v2` support only:
  - `scram project` will create a dev area **with multi microarchs**. Any locally checked out cmssw packages will be build for both default micro arch and additional `x86-64-v2` micro arch
  - `cmsenv` will set env to use `x86-64-v2`  libs. env `SCRAM_MIN_SUPPORTED_MICROARCH=x86-64-v2` will be set too
  - CRAB jobs submitted for this area can run on any node with `v2` and above micro-arch
 

For production jobs (which I guess only create dev area without checkout/building cmssw packages) can run out of the box. If project jobs land on node with `x86-64-v3` and above then cmssw runtime for `x86-64-v3` will be used otherwise  `x86-64-v2` will be used.
author_association: NONE
user: login: smuzaffar
created_at: 2024-12-05T14:16:35Z
body: I am testing this IB with crab and here are the results of a job which was build on a node with `v3` support
- Submitted to site with `v3` resources i.e. CERN: As expacted it [ran successfully](http://vocms0107.cern.ch/mon/cmsbot/241205_133555:cmsbot_crab_Jenkins_CMSSW_15_0_DEVEL_X_2024-12-05-0900_el8_amd64_gcc12_6242/job_out.1.0.txt )
- Submitted to site with `v2` resources i.e. `T2_LB_HPC4L` : As expected it failed with `*** Break *** illegal instruction` [see log](http://vocms0107.cern.ch/mon/cmsbot/241205_134528:cmsbot_crab_Jenkins_CMSSW_15_0_DEVEL_X_2024-12-05-0900_el8_amd64_gcc12_6243/job_out.1.0.txt)

By the way, where can I see the env generated by SCRAM ? I see that https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMRuntime/Tools/Scram.py#L379-L380 directly send the env to the linux process, do we have any place where we log this? 
author_association: NONE
user: login: belforte
created_at: 2024-12-05T14:24:49Z
body: AFAIK there is no place where it is logged. While running CRAB you can use `script_exe` config. param to send a script like this:

```bash
env
cmsRun -j FrameworkJobReport.xml PSet.py
```

Or I can setup a dev CRAB for you which does it :-)  . YOu will still need to change the crab config. file to use that

author_association: MEMBER
user: login: smuzaffar
created_at: 2024-12-05T14:30:53Z
body: thanks @belforte , let me try `script_exe`
author_association: NONE
user: login: makortel
created_at: 2024-12-05T14:42:09Z
body: Thanks @smuzaffar for the detailed description in https://github.com/dmwm/WMCore/issues/12168#issuecomment-2520351777. I'd suggest to document this behavior somewhere more permanently (e.g. twiki or https://cms-sw.github.io/), and make it more clear what action exactly makes a developer area on `v3` node to switch from "multi microarch" behavior (that is used by production jobs) to "single microarch" (what a user building CMSSW packages will see), and what are the values of the two environment variables in each case.
author_association: NONE
user: login: belforte
created_at: 2024-12-06T11:23:46Z
body: @mmascher @amaltaro we need to finalize names :-)

I propose to use `MY.REQUIRED_MINIMUM_MICROARCH` as the new classAd in job submission
OK ? Or will you prefer to have something which starts with `DESIRED` as we do e.g. for sites ? (avoid using "require" and "request" works which are used in some standard HTC ads) 

author_association: MEMBER
user: login: amaltaro
created_at: 2024-12-06T12:04:41Z
body: Looking into https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py, we already have a mix of REQUIRED and DESIRED job classads.
And given that we are already using `My.REQUIRED_ARCH`, I think `MY.REQUIRED_MINIMUM_MICROARCH` would make the most sense. Agree?
author_association: CONTRIBUTOR
user: login: makortel
created_at: 2024-12-06T15:00:59Z
body: > `MY.REQUIRED_MINIMUM_MICROARCH`

Is this name something that in the future could work for ARM or RISC-V microarchitectures, that may have different nomenclature (e.g. not necessarily "versions"), or is this thought to be specific for the x86 case we have at hand now?


author_association: NONE
user: login: amaltaro
created_at: 2024-12-06T15:40:39Z
body: > Is this name something that in the future could work for ARM or RISC-V microarchitectures, that may have different nomenclature (e.g. not necessarily "versions"), or is this thought to be specific for the x86 case we have at hand now?

Ideally, yes. I think the only requirement is that we can use it with an integer, such that >, < comparisons are easily applied.
Anything more complex could potentially require further changes in glideinWMS (the requirements string), e.g. 1.2, etc.
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2024-12-06T15:41:54Z
body: @makortel you tell us ! In case you should also generalize  `SCRAM_MIN_SUPPORTED_MICROARCH`.

If a version # is not guaranteed, I do no see alternatives to a list. Or a "method" which creates the list on the fly given the "MIN" name.
It is also a matter of which "strings" HTCondor will expose for us to match with. Current proposal is indeed based on the list which Marco put in initial posting.

Hard to decide on anything w/o concrete examples.
author_association: MEMBER
user: login: makortel
created_at: 2024-12-06T16:03:49Z
body: A challenge is that we don't know how we would want to deal with multiple ARM or RISC-V microarchitectures (and we don't have any real experience on either, so deciding anything now would be premature). I think this implies that if you want to have a general solution that wouldn't have be changed ever, we'd need to spend more time to think what that would be (and there would be a risk of overengineering).

If we proceed with that we know for x86(which is perfectly fine for me), I'd suggest it's made clear the behavior is specific to x86, and it is understood that if we ever want/need to deal with multiple ARM/RISC-V microarchitectures the machinery will be reassessed.
author_association: NONE
user: login: belforte
created_at: 2024-12-06T17:03:27Z
body: I fully agree. Let's stick with what is simplest now. And when we will now better, reassess. In  optimistic spirit, let's avoid to put X86 in the name at this time.
author_association: MEMBER
user: login: belforte
created_at: 2024-12-10T16:55:56Z
body: answering my question about "which are v2-(almost-)only sites". I used `condor_status -af GLIDEIN_CMSSite Microarch` and a bit of python to get this table. Which suggests that T3_US_UMiss could be removed from the pool.
```
Site                 slots  v1%   v2%  v3%  v4%  
------------------------------------------------
T1_DE_KIT             4497   0%    0%  97%   2%  
T1_ES_PIC             1232   0%    0%  98%   1%  
T1_FR_CCIN2P3         2545   0%    0%  77%  22%  
T1_IT_CNAF            3249   0%    0%  71%  28%  
T1_RU_JINR            6416   0%    0%  86%  13%  
T1_UK_RAL             3610   0%    0%  76%  23%  
T1_US_FNAL           16839   0%    1%  63%  34%  
T2_AT_Vienna           221   0%    0%   0% 100%  
T2_BE_IIHE            4778   0%    0%  43%  56%  
T2_BE_UCL             1110   0%    0%  76%  23%  
T2_BR_SPRACE           993   0%   31%  68%   0%  
T2_BR_UERJ              99   0%    0% 100%   0%  
T2_CH_CSCS            3084   0%    0% 100%   0%  
T2_CN_Beijing          229   0%    0%   0% 100%  
T2_DE_DESY            4424   0%    0%  45%  54%  
T2_DE_RWTH            4859   0%   18%  63%  18%  
T2_EE_Estonia         2680   0%   37%  33%  29%  
T2_ES_CIEMAT          1544   0%    0%  19%  80%  
T2_ES_IFCA             930   0%   17%   0%  82%  
T2_FI_HIP              275   0%    0% 100%   0%  
T2_FR_GRIF            3070   0%    2%  77%  20%  
T2_FR_IPHC            1413   0%    6%  47%  45%  
T2_HU_Budapest        1423   0%    0% 100%   0%  
T2_IT_Bari            9728   0%    0%  96%   3%  
T2_IT_Legnaro         3042   0%    0%  88%  11%  
T2_IT_Pisa             342   0%    0%  12%  87%  
T2_IT_Rome             778   0%    0%  93%   6%  
T2_KR_KISTI            354   0%   90%   0%   9%  
T2_LB_HPC4L              7   0%  100%   0%   0%  
T2_PL_Cyfronet         752   0%    0% 100%   0%  
T2_PL_Swierk           151   0%    0% 100%   0%  
T2_PT_NCG_Lisbon       161   0%    0% 100%   0%  
T2_RU_IHEP             342   0%  100%   0%   0%  
T2_RU_INR               47   0%  100%   0%   0%  
T2_RU_ITEP              53   0%  100%   0%   0%  
T2_RU_JINR             734   0%   38%  41%  19%  
T2_TW_NCHC               9   0%    0% 100%   0%  
T2_UA_KIPT             186   0%   88%   5%   5%  
T2_UK_London_Brunel    725   0%   27%  70%   2%  
T2_UK_London_IC       5859   0%    8%  85%   5%  
T2_UK_SGrid_RALPP     2774   0%   14%  72%  13%  
T2_US_Caltech         2428   0%    3%  82%  14%  
T2_US_Florida         2186   0%    0% 100%   0%  
T2_US_MIT             8432   0%   12%  22%  65%  
T2_US_Nebraska        4756   0%   34%  37%  28%  
T2_US_Purdue          1350   0%    0%  71%  28%  
T2_US_UCSD            6258   0%    1%  96%   2%  
T2_US_Vanderbilt      6735   0%   55%  18%  26%  
T2_US_Wisconsin       8931   0%   16%  68%  15%  
T3_IT_Trieste         1185   0%   32%  59%   8%  
T3_KR_KNU              288   0%   52%  14%  33%  
T3_TW_TIDC               4   0%    0% 100%   0%  
T3_UK_London_QMUL       16   0%    0%   0% 100%  
T3_UK_London_RHUL      683   0%    0%   0% 100%  
T3_UK_SGrid_Oxford     495   0%    0%  97%   2%  
T3_UK_ScotGrid_GLA    5577   0%    0% 100%   0%  
T3_US_Baylor            37   0%    0% 100%   0%  
T3_US_Colorado         249   0%   50%  10%  39%  
T3_US_FNALLPC          606   0%  100%   0%   0%  
T3_US_OSG               38   0%   23%  76%   0%  
T3_US_PuertoRico        69   0%   14%  85%   0%  
T3_US_Rutgers           36   0%  100%   0%   0%  
T3_US_UMD                5   0%  100%   0%   0%  
T3_US_UMiss              8 100%    0%   0%   0%  
```
author_association: MEMBER
user: login: stlammel
created_at: 2024-12-10T17:06:01Z
body: Hallo Stefano,
   yes, T3_US_UMiss was hit hard when we phased out v1.
They do have one machine that is not v1. (According to
the site admin.)
   For the Tier-3s with 100% v2, this is likely due to the
condor_status being a snapshot in time and not all the
worker nodes there being used at the time.
   Thanks,
           cheers,  Stephan

author_association: NONE
user: login: belforte
created_at: 2024-12-10T17:12:02Z
body: @makortel @smuzaffar about "generalization" this is what drives current CRAB implementation which I am working on:

* ` SCRAM_MIN_SUPPORTED_MICROARCH` is a "free" string which carries enough information to use an algorithm to derive from it the correct requirements to be added in job submission to HTC
   * My initial algorithme will be:
   ```python
   if min_supported_microarch.startswith('x86-64-v'): MY.REQUIRED_MINIMUM_MICROARCH=min_supported_microarch
   ```
*  ` SCRAM_MIN_SUPPORTED_MICROARCH`  satisfies `(r"^(?=.{0,255}$)[a-zA-Z0-9\-\_\.]+$")` (i.e. max 255 chars allphanumeric plus dash, underscore and dot)

it will be easy for CRAB to change the algorithm, but I'd rather not change the above logic nor the string format. I.e. if you think that you may like to use `:`, let me know now. 

@mmascher please confirm if `MY.REQUIRED_MINIMUM_MICROARCH="'x86-64-v3"` (e.g.) is OK. I will let you know when I can have jobs which defined it so that we can test matching.
 
author_association: MEMBER
user: login: belforte
created_at: 2024-12-10T17:21:16Z
body: Hi @stlammel , sure. It is indeed curious that FNALLPC is v2-only ! Anyhow the only conclusions which I derive from that table are:
* it is OK to continue to provision resources based only on site name and leave microarch selection to the job matchmaking (as Marco stated)
* it is OK to submit jobs to site which have the data, w/o worrying about microarch
* it is OK not to worry about microarch when placing datasets at sites, as long as we avoid KISTI

In other words, we keep doing things "as now", simply jobs which require v3 will see some sites as "smaller" and may be queued longer

In the global view of things, T3's are almost only used for user MC and sort of "bundled together" 
author_association: MEMBER
user: login: stlammel
created_at: 2024-12-10T17:45:53Z
body: fully agree Stefano!   - Stephan
author_association: NONE
user: login: amaltaro
created_at: 2024-12-10T18:45:10Z
body: As mentioned in this comment:
https://indico.cern.ch/event/1487904/

I was considering to use it as a single integer, such that comparison/evaluation in glideinWMS can be simple.

However, if @mmascher and others think that a construction like `MY.REQUIRED_MINIMUM_MICROARCH="'x86-64-v3"` is going to be more solid, I see no reason why WMCore should do something different than CRAB. So we will just use the same convention.
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2024-12-10T19:52:57Z
body: Very easy for me to set a integer too. Waiting for Marco to say what he wants!

author_association: MEMBER
user: login: mmascher
created_at: 2024-12-12T15:11:05Z
body: Ok, let's go with the integer solution. Here is the constraint, I gave it a test.

```
MICROARCH_CONSTRAINT = (REQUIRED_MINIMUM_MICROARCH <= int(substr(split(Microarch,"-")[1],1)))
```

It could be either added as a machine requirements in the glideinWMS frontend, or as a job requirement by WMagent when it writed the JDL.

Should I add it to the frontend?
author_association: MEMBER
user: login: belforte
created_at: 2024-12-13T18:18:59Z
body: thanks @mmascher . My preference is that match making requirements are set in the gWms, like done e.g. for DESIRED_Sites. Having reuquirements in multiple places is confusing.
But maybe I miss something ... I see that CRAB jobs specify `Requirements = stringListMember(TARGET.Arch, REQUIRED_ARCH)` to which HTC adds requirements on OpSys and other things. In other words: I am confused :-)


author_association: MEMBER
user: login: belforte
created_at: 2024-12-13T18:28:32Z
body: @mmascher does this mean that the default is not `any` anymore as you [wrote](https://github.com/dmwm/WMCore/issues/12168#issuecomment-2511869840) earlier ?
what should it be ? `0` ?

author_association: MEMBER
user: login: mmascher
created_at: 2024-12-16T07:58:21Z
body: Hi @belforte ,

If we go with REQUIRED_MINIMUM_MICROARCH as an integer I would not set "any" as default. The final constraint I tested in ITB has a protection for REQUIRED_MINIMUM_MICROARCH being defined. So you can either let it undefined, or set it as 0 as a default behaviour. Either way all slots will be matched.
author_association: MEMBER
user: login: amaltaro
created_at: 2024-12-16T10:54:06Z
body: @mmascher one more question. Should we define this job classad only if the cpu architecture is `x86-64`? Or can we always have it defined, regardless of the architecture, and only update the minimum microarch according to the CMSSW version?

From what I can tell, production jobs could be requesting multiple architectures:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py#L641

will the settings of microarchitecture keep this functionality intact? In other words, a job setting for example:
```
REQUIRED_MINIMUM_MICROARCH=2
REQUIRED_ARC="X86_64,aarch64"
```
will it still manage to match against either resource type?
author_association: CONTRIBUTOR
user: login: srimanob
created_at: 2024-12-16T11:48:41Z
body: Hi @amaltaro 
I assume if we use ARM (aarch64), we should keep "any" for the moment. We don't have much experiance on the microarch for ARM. Thanks.
author_association: NONE
user: login: mmascher
created_at: 2024-12-16T13:12:32Z
body: I can add a protection for the machine microarch as well. As far as I know it is not defined for aarch64, so in this case we can allow jobs to run regardless of their REQUIRED_MINIMUM_MICROARCH. Does this sound ok?
author_association: MEMBER
user: login: amaltaro
created_at: 2024-12-16T13:23:23Z
body: Yes Marco, I think this would be ideal for the moment. In other words, the desired behavior I have in mind is:
```
if REQUIRED_ARCH != X86_64
  disregard REQUIRED_MINIMUM_MICROARCH
else
  MICROARCH >=  REQUIRED_MINIMUM_MICROARCH  # assuming that default is REQUIRED_MINIMUM_MICROARCH=0
```

Does it sound good to you? @mmascher @belforte 
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2024-12-16T13:52:01Z
body: that's implicit in the fact that we specify microarch with a number.  Should at some point we need sam afunctionality for Arm, or whatever else, even if still a number, will likely be  a different one.
I assumed that Marco is prepared to add the proper amount of `if REQUIRED_ARCH` at that point.
Of course we can always change on client side as well... as discussed already it is difficult to be future-proof against any possible future !
author_association: MEMBER
user: login: belforte
created_at: 2024-12-16T15:03:58Z
body: the agreed behavior has been implemented in CRAB, and deployed on production server, but it is only accessible using `crab-dev` for CRABClient.  `crab-prod` (i.e. `crab`) still ignores the microarchitecture.
I'd rather not touch `crab-prod` client until after the holidays.
author_association: MEMBER
user: login: khurtado
created_at: 2025-02-01T00:28:05Z
body: CC: @amaltaro 
Just to make sure I understand. We need something like this:

```
if 'x86_64' not in REQUIRED_ARCH.split(","):
  ad['My.REQUIRED_MINIMUM_MICROARCH'] = "0"
else
  ad['My.REQUIRED_MINIMUM_MICROARCH'] = str(getScramMicroArch()) 
```

where `getScramMicroArch()` gets the value of `SCRAM_MIN_SUPPORTED_MICROARCH` from the scram environment.
And in order to get the scram environment we need to use:
 scram project  &  eval `$SCRAM_COMMAND runtime -sh` 

Is this correct?
We do have this class:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMRuntime/Tools/Scram.py

But so far, we only use it at the worker node (runtime level) in the CMSSW executor.
Do I understand correctly that since we need to inject the value of `SCRAM_MIN_SUPPORTED_MICROARCH` as a classad, then we need to get the scram environment at the Agent-level? 
We do not have cvmfs at the agent-level, so we can't simply use the Scram() object.

We could alternatively use the value of cmsswVersions and define "3" for CMSSW>=15.x and "0" otherwise. But this is more like a workaround that may not age well.

https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py#L633-L634C53

We could also define a variable that is part of the request config, e.g.: `self.step.application.setup.scramMinMicroArch`, but setting the value has the same issue (no cvmfs to run scram) as above, it would need to be done manually (CMSSW 15.x->"3", etc)

What about StepChain, where we run multiple CMSSW versions?
Do we make it so that if there is:
CMSSW_12, CMSSW_15, CMSSW_14 -> Then we require v3, since there is 1 CMSSW15.x?

As a side note, the CRAB client has its own class, but this client runs on nodes with cvmfs access and that cmsenv a CMSSW release already, so the procedure is different.
https://github.com/dmwm/CRABClient/blob/8b8da2a40b2ee3bd8a9df03613ff291608262b56/src/python/CRABClient/JobType/ScramEnvironment.py#L107

So in short, I think the alternatives are 
1. To hardcode the conditions (CMSSW releases >=15 use "v3"), or 2
2. Request to get cvmfs access in the agent nodes so we can use cvmfs+scram using the CMSSW_release versions so we can get the output of `SCRAM_MIN_SUPPORTED_MICROARCH`. 

I'm trying to see if there is anything I'm missing here and there is a better approach.
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2025-02-01T13:57:52Z
body: A (hopefully not needed) warning:
I do not know how you further manipulates `ad[ ]` before this is handed to HTCondor, but make sure that eventually the ad is an integer, i.e.
`condor_q <somejob> -l |grep REQUIRED_MINIMUM_MICROARCH`
is e.g.
`REQUIRED_MINIMUM_MICROARCH = 2`
not
`REQUIRED_MINIMUM_MICROARCH = "2"`

It just happened to me to get lost in the variable manipulation in old CRAB code and end up with the latter so jobs were not matching for a day 😭  https://mattermost.web.cern.ch/cms-o-and-c/pl/z83bwi135in3jqgd6jd8zu545r

author_association: MEMBER
user: login: khurtado
created_at: 2025-02-03T21:37:16Z
body: @belforte Thank you, yes. When manipulating the `ad` dictionary, "2" should become an integer. Strings are created via `classad.quote()` 
author_association: CONTRIBUTOR
user: login: khurtado
created_at: 2025-02-03T22:22:58Z
body: Just had a chat with Alan via MM:
@smuzaffar Given that we do not have CVMFS access in the agents and hence cannot use `scram` at that level, can we get a web-based document like this:

https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML?anytype=1&anyarch=1

With the information of `SCRAM_MIN_SUPPORTED_MICROARCH` for each `CMSSW_version`: ?

Otherwise, we would likely be just hardwiring v3 for CMSSW 15 or greater
author_association: CONTRIBUTOR
user: login: belforte
created_at: 2025-02-03T23:09:04Z
body: why can't you have CVMFS ? It is very easy to mount it in the VM's where you run the agents (via puppet e.g.) and to bind it into the docker containers. And it may be useful for other things as well.

author_association: MEMBER
user: login: amaltaro
created_at: 2025-02-04T01:10:56Z
body: Mounting CVMFS is probably a small portion of this development. When it involves SCRAM, I feel like it always demands more effort than expected. So I am all in favor of having a clean and easy to read information provided from usptream, if possible.
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-07T14:59:23Z
body: Hi @amaltaro,
  `SCRAM_MIN_SUPPORTED_MICROARCH` is a dynamic thing. SCRAM changes its depending on the system where it is setting up cmssw env. Note that `CMSSW_15_0_X` and above supports two micro archs

- `x86-64-v3` which is default and should be used when system has `x86-64-v3` support
- `x86-64-v2` which is an additional microarch and scram falls back to it if system does not support `x86-64-v3`

So if you ask me what is the min supported micro-arch for `CMSSW_15_0_X` then my answer would be `x86-64-v2`. But we want to target `x86-64-v3` resources for `CMSSW_15_0_X` and remaining <5% `x86-64-v2` resources can be used by old cmssw releases.

I would suggest that I add an extra field in https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML?anytype=1 e.g.

```
<project label="CMSSW_15_0_0_pre3" type="Development" state="Announced" default_micro_arch="x86-64-v3"/>
```

I think you are using `ReleasesXML` already so you can extract the default micro-arch information for this. For older release where we do not have any micro-arch defined in the `ReleasesXML` you can always use the minimum `x86-64-v2` . Will that be enough you cover your use case?
author_association: NONE
user: login: smuzaffar
created_at: 2025-02-07T15:04:19Z
body: > For older release where we do not have any micro-arch defined in the ReleasesXML you can always use the minimum x86-64-v2

OR for such releases (where ReleasesXML do not provide micro-arch information) better to not add any requirement on micro-arch 
author_association: NONE
user: login: belforte
created_at: 2025-02-08T14:55:31Z
body: > not add any requirement on micro-arch

this is done by setting `REQUIRED_MINIMUM_MICROARCH=0`
author_association: MEMBER
user: login: khurtado
created_at: 2025-02-12T15:15:51Z
body: @smuzaffar 
So, basically, even if CMSSW 15 has support for v2 and `SCRAM_MIN_SUPPORTED_MICROARCH` can yield "v2", we want to match to resources with "v3" support, isn't it? 

If so, yes please, expanding that xml would work. I would need to update the first comment to make this clear.
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-12T15:34:59Z
body: @khurtado , for tools which do not have access to scram/cvmfs (e.g. the product system) should use the `ReleasesXML`  and if they find the `default_micro_arch` set for a release then they should use it and request resources for that micro-arch .

For tools, like crab, which submit jobs from a scram developer area should use the `SCRAM_MIN_SUPPORTED_MICROARCH` environment. 

Does this make sense? 
author_association: NONE
user: login: khurtado
created_at: 2025-02-12T16:06:58Z
body: @smuzaffar Yes, that makes sense.
Please let me know once `ReleasesXML` is expanded with the new value. Thank you!
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-19T15:28:38Z
body: @khurtado , https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML?anytype=1 now contains `default_micro_arch` information 
```
    <project label="CMSSW_14_2_1" type="Production" state="Announced"/>
    <project label="CMSSW_15_0_0_pre1" type="Development" state="Announced" default_micro_arch="x86-64-v3"/>
    <project label="CMSSW_14_1_7" type="Production" state="Announced"/>
    <project label="CMSSW_14_0_19_patch2" type="Production" state="Announced"/>
    <project label="CMSSW_14_0_20" type="Production" state="Announced"/>
    <project label="CMSSW_15_0_0_pre2" type="Development" state="Announced" default_micro_arch="x86-64-v3"/>
    <project label="CMSSW_14_2_2" type="Production" state="Announced"/>
    <project label="CMSSW_15_0_0_pre3" type="Development" state="Announced" default_micro_arch="x86-64-v3"/>
    <project label="CMSSW_14_0_21" type="Production" state="Announced"/>

```
author_association: NONE
user: login: khurtado
created_at: 2025-02-19T20:39:37Z
body: @smuzaffar Thank you! I just need to double check something:

I can see the new `default_micro_arch` is available for Development CMSSW releases only so far. 

Since [we use](https://github.com/dmwm/WMCore/blob/f4de587ea3d2c4ea43d90865e2333ab3026a14b2/src/python/WMCore/Services/TagCollector/TagCollector.py#L25C23-L25C69):
https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML

without `?anytype=1`, which only contains the Production CMSSW releases, should I expect the `default_micro_arch` to eventually get added in the link above (without `?anytype=1`) into production CMSSW 15.x releases once they become available?
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-19T20:58:09Z
body: > should I expect the default_micro_arch to eventually get added in the link above (without ?anytype=1) into production CMSSW 15.x releases once they become available?

yes once we have a production release e.g. CMSSW_15_0_0 then https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML should have `default_micro_arch` set for it. `default_micro_arch` is only going to be set for 15X and above
author_association: NONE
user: login: khurtado
created_at: 2025-02-19T21:19:04Z
body: Sounds good, thank you! 
@smuzaffar @amaltaro One more question:
Are we expecting any stepchain workflow (multiple cmsRuns per job) in a structure like:

Step 1: alma9, CMSSW 14 (cmsRun for e.g.: Digi work)
Step 2: alma9, CMSSW 15 (cmsRun for e.g.: Reco work)

and in that case, should we pick the higher default microarchitecture of all to match (in this case v3)?
Also (just to be 100% sure), the default microarchitecture depends on the CMSSW release version only, and is independent of the scram architecture, correct (that is a potential CMSSW15 version for `el8_amd64_gcc12` and `el8_amd64_gcc11` should both have the same default microarchitecture version, so I only would need to look into the CMSSW version itself)

author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-19T21:34:40Z
body: > and in that case, should we pick the higher default microarchitecture of all to match (in this case v3)?

@khurtado , yes in case of stepchain you should use the highest micro-arch

> Also (just to be 100% sure), the default microarchitecture depends on the CMSSW release version only, and is independent of the scram architecture, correct (that is a potential CMSSW15 version for el8_amd64_gcc12 and el8_amd64_gcc11 should both have the same default microarchitecture version, so I only would need to look into the CMSSW version itself)

yes for x86_64, the default micro-arch will remain same. Note that for now we only have one architecture as production arch, so if you are using https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML then you will only see one architecture for a release e.g. for CMSSW_15_0_0 you will only see el8_amd64_gcc12 with v3 as default micro-arch
author_association: NONE
user: login: amaltaro
created_at: 2025-02-19T21:36:19Z
body: I've been debating if having the explicit field `default_micro_arch` for every single release wouldn't be better than just not providing it for releases without a micro-architecture. Otherwise, we need to make sure our WMCore client sets that default properly (likely the lowest micro-arch - v2?)

> Step 1: alma9, CMSSW 14 (cmsRun for e.g.: Digi work)
> Step 2: alma9, CMSSW 15 (cmsRun for e.g.: Reco work)
> and in that case, should we pick the higher default microarchitecture of all to match (in this case v3)?

That is definitely a valid and common use case. In this case, I fear that we will have to set it to the lowest micro-arch, to make sure that we can successfully run all the cmsRun steps within a single job.
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-19T21:40:19Z
body: > That is definitely a valid and common use case. In this case, I fear that we will have to set it to the lowest micro-arch, to make sure that we can successfully run all the cmsRun steps within a single job.

@amaltaro , if you request for lowest micro-arch then there is a good chance that v3 executable will fail. So you need to request the highest micro-arch
author_association: NONE
user: login: amaltaro
created_at: 2025-02-19T21:51:52Z
body: Right! Thank you for the correction, Shahzad.
author_association: CONTRIBUTOR
user: login: khurtado
created_at: 2025-02-21T22:09:10Z
body: Okay, I started something in #12270 

The logic in short so far is that:

1. We can have many required architectures. If there is not a single `x86_64` architecture, we make `REQUIRED_MINIMUM_MICROARCH = 0` (which is the equivalent to no constraint at all)
2. If we have at least 1 `x86_64` architecture, then we check the tagCollector information. 
2a. In this case, for a list of CMSSW releases, we will return the highest default microarchitecture (the default is "2" when TagCollector does not specify a microarchitecture for a given release). So, if we have "CMSSW_14_0_19_patch2,CMSSW_15_0_0_pre3", this will yield "3"
If we have "CMSSW_14_1_0_pre6,CMSSW_13_3_3", this will yield "2".
3. `REQUIRED_MINIMUM_MICROARCH = str(number)` will be injected and `str(number)` will be treated as an integer in condor.

@amaltaro @smuzaffar  do you agree with the logic? Am I missing anything?
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-24T10:35:33Z
body: > Okay, I started something in https://github.com/dmwm/WMCore/pull/12270

logic seems correct
author_association: NONE

Diff:
diff --git a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
index 2605ba3fe5..41d6cf8ee0 100644
--- a/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
+++ b/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
@@ -20,7 +20,7 @@
 from WMCore.FwkJobReport.Report import Report
 from WMCore.WMInit import getWMBASE
 from WMCore.Lexicon import getIterMatchObjectOnRegexp, WMEXCEPTION_REGEXP, CONDOR_LOG_FILTER_REGEXP
-
+from WMCore.Services.TagCollector.TagCollector import TagCollector
 
 def activityToType(jobActivity):
     """
@@ -138,6 +138,8 @@ def __init__(self, config):
         # These are added now by the condor client
         #self.x509userproxysubject = proxy.getSubject()
         #self.x509userproxyfqan = proxy.getAttributeFromProxy(self.x509userproxy)
+        
+        self.tc = TagCollector()
 
         return
 
@@ -155,6 +157,7 @@ def submit(self, jobs, info=None):
             return successfulJobs, failedJobs
 
         schedd = htcondor.Schedd()
+        rel_microarchs = self.tc.defaultMicroArchVersionNumberByRelease()
 
         # Submit the jobs
         for jobsReady in grouper(jobs, self.jobsPerSubmit):
@@ -640,7 +643,14 @@ def getJobParameters(self, jobList):
                 ad['My.REQUIRED_ARCH'] = classad.quote(str(requiredArchs))
                 ad['Requirements'] = 'stringListMember(TARGET.Arch, REQUIRED_ARCH)'
 
-            jobParameters.append(ad)    
+            # Inject a microarchitecture classad. If x86_64 not on arch list, return 0
+            if 'X86_64' not in requiredArchs.split(","):
+                ad['My.REQUIRED_MINIMUM_MICROARCH'] = "0"
+            else:
+                minMicroArch = self.tc.getGreaterMicroarchVersionNumber(cmsswVersions, rel_microarchs=rel_microarchs)
+                ad['My.REQUIRED_MINIMUM_MICROARCH'] = str(minMicroArch) 
+
+            jobParameters.append(ad)
              
         return jobParameters
 
diff --git a/src/python/WMCore/Services/TagCollector/TagCollector.py b/src/python/WMCore/Services/TagCollector/TagCollector.py
index cc59f14452..4c167ccf00 100644
--- a/src/python/WMCore/Services/TagCollector/TagCollector.py
+++ b/src/python/WMCore/Services/TagCollector/TagCollector.py
@@ -112,3 +112,44 @@ def releases_by_architecture(self):
                 releases.add(item['label'])
             arch_dict[row['name']].extend(list(releases))
         return dict(arch_dict)
+
+
+    def defaultMicroArchVersionNumberByRelease(self, default_microarch=0):
+        """
+        Yield default microarchitecture by CMS release known in tag collector.
+
+        :param default_microarch: int, default microar when not found in TagCollector XML file.
+        :return: dictionary ("release": default_microarch (or default_microarch if not found)).
+        """
+        rel_microarchs = {}
+        for row in self.data():
+            for item in row['project']:
+                microarch = item.get('default_micro_arch', default_microarch)
+                if isinstance(microarch, str):
+                    # format of this value is "arch-vN"
+                    microarch = int((microarch.split("-")[-1]).lstrip("v"))
+                rel_microarchs[item['label']] = microarch
+        return rel_microarchs
+
+    def getGreaterMicroarchVersionNumber(self, releases=None, default_microarch=0, rel_microarchs=None):
+        """
+        Return the greated default microarchitecture, given a list of releases.
+
+        :param releases: str, comma-separated list of releases.
+        :param default_microarch: int, default microar when not found in TagCollector XML file.
+        :return: int, greater default microarchitecture version number.
+        """
+        if releases is None:
+            return default_microarch
+
+        if rel_microarchs is None:
+            rel_microarchs = self.defaultMicroArchVersionNumberByRelease()
+
+        releases = releases.split(",")
+        microarch = default_microarch
+        for r in releases:
+            m = rel_microarchs.get(r, default_microarch)
+            if m > microarch:
+                microarch = m
+
+        return microarch
diff --git a/test/python/WMCore_t/Services_t/TagCollector_t/TagCollector_t.py b/test/python/WMCore_t/Services_t/TagCollector_t/TagCollector_t.py
index 8350af8c52..555758f499 100644
--- a/test/python/WMCore_t/Services_t/TagCollector_t/TagCollector_t.py
+++ b/test/python/WMCore_t/Services_t/TagCollector_t/TagCollector_t.py
@@ -26,12 +26,21 @@ def testTagCollecorMethods(self):
         releases = self.tagCollecor.releases()
         architectures = self.tagCollecor.architectures()
         realsese_by_arch = self.tagCollecor.releases_by_architecture()
+        microarch_by_release = self.tagCollecor.defaultMicroArchVersionNumberByRelease()
+        microarch_testCMSSW_15 = self.tagCollecor.getGreaterMicroarchVersionNumber("CMSSW_15_0_0_pre3")
+        microarch_testCMSSW_12_15 = self.tagCollecor.getGreaterMicroarchVersionNumber("CMSSW_12_4_0_pre2,CMSSW_15_0_0_pre3")
+        microarch_testCMSSW_7_12 = self.tagCollecor.getGreaterMicroarchVersionNumber("CMSSW_7_1_10_patch2,CMSSW_12_4_0_pre2", rel_microarchs=microarch_by_release)
         self.assertIn('CMSSW_7_1_10_patch2', releases)
         self.assertIn('slc6_amd64_gcc493', architectures)
         self.assertIn('slc7_amd64_gcc620', architectures)
         self.assertEqual(len(architectures), len(realsese_by_arch))
         self.assertEqual(sorted(self.tagCollecor.releases('slc6_amd64_gcc493')),
                          sorted(realsese_by_arch.get('slc6_amd64_gcc493')))
+        self.assertEqual(0, microarch_by_release['CMSSW_7_1_10_patch2'])
+        self.assertEqual(3, microarch_by_release['CMSSW_15_0_0_pre3'])
+        self.assertEqual(3, microarch_testCMSSW_12_15)
+        self.assertEqual(3, microarch_testCMSSW_15)
+        self.assertEqual(0, microarch_testCMSSW_7_12)
 
         return
 


Reviews:
user: login: smuzaffar
state: COMMENTED
author_association: NONE
submitted_at: 2025-02-24T10:34:49Z
user: login: khurtado
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T22:39:47Z
user: login: amaltaro
body: @khurtado thank you for providing these changes. Please find some comments along the code.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T14:59:19Z
user: login: khurtado
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T17:46:52Z
user: login: amaltaro
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T19:43:50Z
user: login: amaltaro
body: Changes are looking good in general, however I am concerned about the code performance. AFAIU, we would be building the release x microarch dictionary for every single job that we need to find the x86 micro-architecture.
If it is just a couple of seconds for ~50k jobs, then we can probably disregard it.

Do you think we should evaluate the performance of `defaultMicroArchVersionNumberByRelease + getGreaterMicroarchVersionNumber` for a large loop? Or perhaps we should change the code to avoid this by-job overhead? @khurtado 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T14:15:06Z
user: login: amaltaro
body: That is great! Thank you for making the relevant changes.

I left one comment along the code that needs attention, but other than that, everything is looking good to me.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T04:18:30Z
user: login: khurtado
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T16:07:51Z
user: login: amaltaro
body: Thank you, Kenyi! LGTM.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T21:01:21Z

Review comments:
diff_hunk: @@ -640,7 +642,14 @@ def getJobParameters(self, jobList):
                 ad['My.REQUIRED_ARCH'] = classad.quote(str(requiredArchs))
                 ad['Requirements'] = 'stringListMember(TARGET.Arch, REQUIRED_ARCH)'
 
-            jobParameters.append(ad)    
+            # Inject a microarchitecture classad. If x86_64 not on arch list, return 0
+            if 'x86_64' not in requiredArchs.split(","):
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: smuzaffar
body: @khurtado , should this be `X86_64`  i.e. capital `X` ?
created_at: 2025-02-24T10:34:49Z
author_association: NONE
diff_hunk: @@ -640,7 +642,14 @@ def getJobParameters(self, jobList):
                 ad['My.REQUIRED_ARCH'] = classad.quote(str(requiredArchs))
                 ad['Requirements'] = 'stringListMember(TARGET.Arch, REQUIRED_ARCH)'
 
-            jobParameters.append(ad)    
+            # Inject a microarchitecture classad. If x86_64 not on arch list, return 0
+            if 'x86_64' not in requiredArchs.split(","):
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: khurtado
body: Good catch! It should be `X86_64`, thank you.
created_at: 2025-02-24T22:39:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -138,6 +138,8 @@ def __init__(self, config):
         # These are added now by the condor client
         #self.x509userproxysubject = proxy.getSubject()
         #self.x509userproxyfqan = proxy.getAttributeFromProxy(self.x509userproxy)
+        
+        self.tc = TagCollector()
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: amaltaro
body: I would suggest initializing this class with `cacheduration` of 10min, or something like this.
created_at: 2025-02-26T14:57:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,3 +112,38 @@ def releases_by_architecture(self):
                 releases.add(item['label'])
             arch_dict[row['name']].extend(list(releases))
         return dict(arch_dict)
+
+
+    def defaultMicroArchVersionNumberByRelease(self, default_microarch=2):
path: src/python/WMCore/Services/TagCollector/TagCollector.py
user: login: amaltaro
body: Kenyi, please create a unit test for this new method and also update the docstring according to the contribution guidelines: https://github.com/dmwm/WMCore/blob/master/CONTRIBUTING.rst#project-docstrings-best-practices
created_at: 2025-02-26T14:58:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,3 +112,38 @@ def releases_by_architecture(self):
                 releases.add(item['label'])
             arch_dict[row['name']].extend(list(releases))
         return dict(arch_dict)
+
+
+    def defaultMicroArchVersionNumberByRelease(self, default_microarch=2):
+        """
+        Yield default microarchitecture by CMS release known in tag collector.
+        :return: dictionary ("release": default_microarch (or default to "2" if not found)).
+        """
+        rel_microarchs = {}
+        for row in self.data():
+            for item in row['project']:
+                microarch = item.get('default_micro_arch', default_microarch)
+                if isinstance(microarch, str):
+                    # format of this value is "arch-vN"
+                    microarch = int((microarch.split("-")[-1]).lstrip("v"))
+                rel_microarchs[item['label']] = microarch
+        return rel_microarchs
+
+    def getGreaterMicroarchVersionNumber(self, releases=None, default_microarch=2):
path: src/python/WMCore/Services/TagCollector/TagCollector.py
user: login: amaltaro
body: Kenyi, please create a unit test for this new method and also update the docstring according to the contribution guidelines: https://github.com/dmwm/WMCore/blob/master/CONTRIBUTING.rst#project-docstrings-best-practices
created_at: 2025-02-26T14:58:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -138,6 +138,8 @@ def __init__(self, config):
         # These are added now by the condor client
         #self.x509userproxysubject = proxy.getSubject()
         #self.x509userproxyfqan = proxy.getAttributeFromProxy(self.x509userproxy)
+        
+        self.tc = TagCollector()
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: khurtado
body: @amaltaro The default right now is 1 hour, and is what ReqMgr uses. Do you think that is too long?
created_at: 2025-02-26T17:46:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -138,6 +138,8 @@ def __init__(self, config):
         # These are added now by the condor client
         #self.x509userproxysubject = proxy.getSubject()
         #self.x509userproxyfqan = proxy.getAttributeFromProxy(self.x509userproxy)
+        
+        self.tc = TagCollector()
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: amaltaro
body: Naah, I think 1h is pretty good! Honestly, I am always confused if that cacheduration is in seconds, minutes or hours (I thought it was set to 1 second :))
created_at: 2025-02-26T19:43:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -138,6 +138,9 @@ def __init__(self, config):
         # These are added now by the condor client
         #self.x509userproxysubject = proxy.getSubject()
         #self.x509userproxyfqan = proxy.getAttributeFromProxy(self.x509userproxy)
+        
+        self.tc = TagCollector()
+        self.rel_microarchs = self.tc.defaultMicroArchVersionNumberByRelease()
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: amaltaro
body: From what I see how this plugin is initialized, it looks like it would be initialized only once, upon JobSubmitter start up. 
If you agree with that assessment, then we need to evaluate this call to build the map of release vs micro-architecture as part of the component polling cycle, such that we keep updating that map. From what I can see, the `submit()` method would be a good place to keep it. Can you please investigate this further and fix it if needed?
created_at: 2025-03-08T04:17:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -138,6 +138,9 @@ def __init__(self, config):
         # These are added now by the condor client
         #self.x509userproxysubject = proxy.getSubject()
         #self.x509userproxyfqan = proxy.getAttributeFromProxy(self.x509userproxy)
+        
+        self.tc = TagCollector()
+        self.rel_microarchs = self.tc.defaultMicroArchVersionNumberByRelease()
path: src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py
user: login: khurtado
body: @amaltaro Done!
created_at: 2025-03-10T16:07:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: dmwm-bot
created_at: 2025-02-21T19:33:25Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 4 changes in unstable tests
 * Python3 Pylint check: failed
   * 5 warnings and errors that must be fixed
   * 4 warnings
   * 70 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/381/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-21T20:46:58Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 1 tests no longer failing
   * 3 changes in unstable tests
 * Python3 Pylint check: failed
   * 1 warnings and errors that must be fixed
   * 4 warnings
   * 70 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/383/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-21T20:50:51Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 2 changes in unstable tests
 * Python3 Pylint check: failed
   * 1 warnings and errors that must be fixed
   * 4 warnings
   * 70 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/384/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-21T21:17:07Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 3 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 70 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/385/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-21T21:56:57Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 3 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 69 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/386/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-21T23:16:20Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 1 tests no longer failing
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 69 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/387/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-24T22:54:38Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 2 new failures
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 69 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/389/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-25T16:29:24Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 69 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/393/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: khurtado
created_at: 2025-02-25T17:36:25Z
body: Tested with TaskCHain and StepChain:

TC:
```
$ condor_q -af:h REQUIRED_MINIMUM_MICROARCH CMSSW_Versions
REQUIRED_MINIMUM_MICROARCH CMSSW_Versions
3                          CMSSW_15_0_0_pre3
```

SC:

```
$ condor_q -af:h REQUIRED_MINIMUM_MICROARCH CMSSW_Versions
REQUIRED_MINIMUM_MICROARCH CMSSW_Versions
3                          CMSSW_12_4_0_pre2,CMSSW_12_4_0_pre2,CMSSW_15_0_0_pre3
```
author_association: CONTRIBUTOR
user: login: khurtado
created_at: 2025-02-25T17:37:57Z
body: @amaltaro this is ready for review. The code is working according to this logic:

https://github.com/dmwm/WMCore/issues/12168#issuecomment-2675665611

If there is anything that you don't agree with the logic above please let me know.
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-25T17:40:45Z
body: @khurtado , what do you get when you run it for `CMSSW_12_4_0_pre2,CMSSW_12_4_0_pre3` only ( i.e. releases which do not have any default micro arch set)?
author_association: NONE
user: login: khurtado
created_at: 2025-02-25T17:49:05Z
body: @smuzaffar I get `2`.

```
>>> from WMCore.Services.TagCollector.TagCollector import TagCollector
>>> tc = TagCollector()
>>> tc.getGreaterMicroarchVersionNumber("CMSSW_12_4_0_pre2,CMSSW_12_4_0_pre2,CMSSW_15_0_0_pre3")
3
>>> tc.getGreaterMicroarchVersionNumber("CMSSW_12_4_0_pre2,CMSSW_12_4_0_pre3")
2
```
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-25T17:52:46Z
body: are there any resources which do not support even v2 and do we want to use those for old release cycles e.g. CMSSW_10_6 etc? 

author_association: NONE
user: login: khurtado
created_at: 2025-02-25T17:56:59Z
body: > are there any resources which do not support even v2 and do we want to use those for old release cycles e.g. CMSSW_10_6 etc?

According to the comment below, only T3_US_UMiss. I can change the default to 0 (no constraint) rather than 2 if that's desired (like it is done with non-x86 architectures).

https://github.com/dmwm/WMCore/issues/12168#issuecomment-2532282942

But I think the ideal solution would be to add default michroarchitecture values for all releases. That way, the version is in control of whoever controls the [XML](https://cmssdt.cern.ch/SDT/cgi-bin/ReleasesXML?anytype=1).  
author_association: CONTRIBUTOR
user: login: smuzaffar
created_at: 2025-02-26T10:54:53Z
body: @khurtado , as non-x86-64 releases do not have default_micro_arch and you use 0, so I would suggest that do the same for all releases. If no default mixcro arch is set then use 0 otherwise use the highest
author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-26T18:19:26Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 77 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/397/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-26T18:23:07Z
body: Jenkins results:
 * Python3 Unit tests: failed
   * 1 new failures
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 77 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/398/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-02-26T18:57:22Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 2 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 78 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/400/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: khurtado
created_at: 2025-02-27T20:35:40Z
body: @smuzaffar Ok, I have changed the default to 0
@amaltaro I updates the unit tests to include these 2 new methods.
author_association: CONTRIBUTOR
user: login: dmwm-bot
created_at: 2025-02-27T20:46:51Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 1 tests no longer failing
   * 3 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 78 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/412/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: khurtado
created_at: 2025-03-06T16:16:31Z
body: @amaltaro Good point. If I track the time spent by `tc.getGreaterMicroarchVersionNumber`, which is used by SimpleCondorPlugin, the time spent for a 1k loop is about 10s on one CERN production agent.

```
(WMAgent-2.3.9.2) [cmst1@vocms0258:test]$ python3 test_performance.py
1000 jobs: time to query tc.getGreaterMicroarchVersionNumber("CMSSW_12_4_0_pre2,CMSSW_12_4_0_pre2,CMSSW_15_0_0_pre3") = 9.809740543365479
```

So, roughly 1 second every 100 jobs.
From what I remember, the jobs from jobList can even come from different workflows, and different tasks in the workflows may have different architectures listed, so I'm not sure how to avoid doing this job by job.Do you see workflows often submitting 50K jobs at a single time? Would you consider blocks of 500-1K jobs are more frequent in production? 
If we expect many of the jobs to perform the same query, perhaps we can cache the result itself to use it within the loop. E.g.: If we expect a group of jobs in jobList to carry ~1 to 3 workflow jobs, and each may only carry up to 4 tasks categories (each task category with a few dozen or hundred jobs) with different architectures listed, we would end up calling this method only up to 12 times or so.
author_association: CONTRIBUTOR
user: login: amaltaro
created_at: 2025-03-06T16:33:26Z
body: Uhmm, this result is worse than I expected.

Kenyi, can you change the code slightly (maybe in your local setup only) and test `getGreaterMicroarchVersionNumber()` such that it does not make this call:
```
        rel_microarchs=self.defaultMicroArchVersionNumberByRelease()
```
and instead it just uses a data structure which has already been created for release vs microarch?

If it has an acceptable performance, we could then do the following in SimpleCondorPlugin:
```
# in the beginning of the polling cycle
rel_microarchs=self.defaultMicroArchVersionNumberByRelease()
# then for each job
minMicroArch = self.tc.getGreaterMicroarchVersionNumber(cmsswVersions)
```

Otherwise, as you said, we will have to adopt a caching layer and this `evaluateRSEExpression()` method shows a potential good use of such feature:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/Rucio/Rucio.py#L739

Given that many of those jobs in SimpleCondorPlugin will have belong to the same Workflow + Task, there should be plenty of cache hits.
author_association: CONTRIBUTOR
user: login: dmwm-bot
created_at: 2025-03-07T18:13:31Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 80 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/453/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: dmwm-bot
created_at: 2025-03-07T18:17:59Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 1 changes in unstable tests
 * Python3 Pylint check: succeeded
   * 4 warnings
   * 80 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/454/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: khurtado
created_at: 2025-03-07T18:19:10Z
body: @amaltaro Thanks! I was overcomplicating things. With the first suggestion, we went down to less than 0.05 seconds each 50k jobs.

```
$ python3 test_performance.py
50000 jobs: time to query tc.getGreaterMicroarchVersionNumber("CMSSW_12_4_0_pre2,CMSSW_12_4_0_pre2,CMSSW_15_0_0_pre3") = 0.04307103157043457
```

I squashed both commits into 1 since the unit test is only a single file.
author_association: CONTRIBUTOR
user: login: dmwm-bot
created_at: 2025-03-10T15:16:13Z
body: Jenkins results:
 * Python3 Unit tests: succeeded
   * 2 changes in unstable tests
 * Python3 Pylint check: failed
   * 2 warnings and errors that must be fixed
   * 4 warnings
   * 80 comments to review
 * Pycodestyle check: succeeded
   * 25 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/456/artifact/artifacts/PullRequestReport.html

author_association: NONE
user: login: khurtado
created_at: 2025-03-10T16:08:08Z
body: @amaltaro I made the change above and squashed the commits
author_association: CONTRIBUTOR