Repository: NOAA-EMC/global-workflow
Pull request #3440: Migration to role account for Jenkins on Orion
Author: login: TerrenceMcGuinness-NOAA
Labels: name: CI-Orion-Passed
description: **Bot use only** CI testing on Orion for this PR has completed successfully
name: CI/CD
description: Issue related to CI/CD
Created: 2025-03-12T17:55:47Z | Merged: 2025-03-14T15:44:29Z

# Description

This PR has the necessary updates to the CI scripts in Jenkins for running on **Orion** with the glopara role account.

The Orion role account can not write to PTMP so updates to the `$HOMEgfs/workflow/hosts/orion.yaml` [file](https://github.com/TerrenceMcGuinness-NOAA/global-workflow/blob/ci_jenkins_role_orion2/workflow/hosts/orion.yaml) where made:

```bash
STMP: '/work2/noaa/global/${USER}/ORION/STMP'
PTMP: '/work2/noaa/global/${USER}/ORION/PTMP'
```

**HOMEDIR** was also [updated](https://github.com/TerrenceMcGuinness-NOAA/global-workflow/blob/a65859736f694188131e81e0b77612622dc896d6/workflow/hosts/orion.yaml#L7) to

```
HOMEDIR: '/work2/noaa/global/${USER}/ORION'
```
because the role account on Orion can not write to `/work/noaa/global` (ticket pending)

[PyGithub](https://github.com/PyGithub/PyGithub) is not in **Spack Stack** yet and is needed in support of the GitHub logging and gist creation python codes.  This python module will be installed in `$HOME/.local` in the role accounts until the module is added to **Spack Stack**.

Same with **GitHub CLI** (which will be supported in Spack Stack 1.9) it will be installed in  `${ROLE_HOME}/utils/bin` which should be in role account's `$PATH `where `ROLE_HOME` is at the same level as `fix` and `data`.

This is setup to run in the role account with the following added directories and utils:

`/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI`

Location on disk where the CI test cases actually build and run specified in [Jenkinsfile](https://github.com/TerrenceMcGuinness-NOAA/global-workflow/blob/2043b458b5ee157e3784feed2fcd6b9e5a9d955c/ci/Jenkinsfile#L10)

```
└── GFS_CI_CD
    ├── global-workflow
    │   └── workflow
    ├── GFS_BASH_CI
    ├── GitLab
    └── Jenkins
        ├── agent
        └── workspace

```
Top level CI directories (this will file structure be duplicated for each role account)
```
GFS_CI_CD
```
CI system directory for Jenkins launching scripts and in support of BASH CI
```
GFS_CI_CD/global-workflow
```
Directory where the JAVA agent is launched from
```
Jenkins/agent
```
Jenkins work space (system level workspace)
```
Jenkins/workspace
```

**NOTE:** Update to Jenkinsfile for EPIC with more robust map from machine to Jenkins Node Names without using split on labels.

  Resolves #3427
  Resolves #3373

# Type of change
- [ ] Bug fix (fixes something broken)
- [ ] New feature (adds functionality)
- [x] Maintenance (code refactor, clean-up, new CI test, etc.)

# Change characteristics
<!-- Choose YES or NO from each of the following and delete the other -->
- Is this a breaking change (a change in existing functionality)? NO
- Does this change require a documentation update? NO
- Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
  - [ ] EMC verif-global <!-- NOAA-EMC/EMC_verif-global#1234 -->
  - [ ] GDAS <!-- NOAA-EMC/GDASApp#1234 -->
  - [ ] GFS-utils <!-- NOAA-EMC/gfs-utils#1234 -->
  - [ ] GSI <!-- NOAA-EMC/GSI#1234 -->
  - [ ] GSI-monitor <!-- NOAA-EMC/GSI-Monitor#1234 -->
  - [ ] GSI-utils <!-- NOAA-EMC/GSI-Utils#1234 -->
  - [ ] UFS-utils <!-- ufs-community/UFS_UTILS#1234 -->
  - [ ] UFS-weather-model <!-- ufs-community/ufs-weather-model#1234 -->
  - [ ] wxflow <!-- NOAA-EMC/wxflow#1234 -->

# How has this been tested?

This PR can be tested in place

Linked issue #3427: Migrate Jenkins CI automation to Role Accounts on Orion
### What new functionality do you need?

This is a subtask of https://github.com/NOAA-EMC/global-workflow/issues/3373 for Orion

### What are the requirements for the new functionality?

This is a subtask of https://github.com/NOAA-EMC/global-workflow/issues/3373 for Orion

### Acceptance Criteria

This is a subtask of https://github.com/NOAA-EMC/global-workflow/issues/3373 for Orion

### Suggest a solution (optional)

_No response_

Diff:
diff --git a/ci/Jenkinsfile b/ci/Jenkinsfile
index 50c7503f3eb..c65f2abd598 100644
--- a/ci/Jenkinsfile
+++ b/ci/Jenkinsfile
@@ -4,10 +4,12 @@ def CUSTOM_WORKSPACE = 'none'
 def HOMEgfs = 'none'
 def CI_CASES = ''
 def GH = 'none'
+// Map of the machine names (MACHINE_ID) to the Jenkins Node names
+def NodeName = [hera: 'Hera-EMC', orion: 'Orion-EMC', hercules: 'Hercules-EMC', gaeac5: 'GaeaC5', gaeac6: 'Gaeac6-EMC']
 // Location of the custom workspaces for each machine in the CI system.  They are persistent for each iteration of the PR.
-def NodeName = [hera: 'Hera-EMC', orion: 'Orion-EMC', hercules: 'Hercules-EMC', gaea: 'Gaea', gaeac6: 'Gaeac6-EMC']
-def custom_workspace = [hera: '/scratch1/NCEPDEV/global/glopara/CI', orion: '/work2/noaa/stmp/CI/ORION', hercules: '/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI', gaea: '/gpfs/f5/epic/proj-shared/global/CI', gaeac6: '/gpfs/f6/drsa-precip3/proj-shared/global/CI']
+def custom_workspace = [hera: '/scratch1/NCEPDEV/global/glopara/CI', orion: '/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI', hercules: '/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI', gaeac5: '/gpfs/f5/epic/proj-shared/global/CI', gaeac6: '/gpfs/f6/drsa-precip3/proj-shared/global/CI']
 def repo_url = 'git@github.com:NOAA-EMC/global-workflow.git'
+
 def STATUS = 'Passed'
 
 pipeline {
@@ -22,7 +24,6 @@ pipeline {
     stages { // This initial stage is used to get the Machine name from the GitHub labels on the PR
              // which is used to designate the Nodes in the Jenkins Controller by the agent label
              // Each Jenkins Node is connected to said machine via an JAVA agent via an ssh tunnel
-             // no op 2
 
         stage('1. Get Machine') {
             agent { label 'built-in' }
@@ -35,6 +36,7 @@ pipeline {
                     }
 
                     def run_nodes = []
+                    def machine_names = []
                     if (isSpawnedFromAnotherJob) {
                         echo "machine being set to value passed to this spawned job"
                         echo "passed machine: ${params.machine}"
@@ -42,29 +44,33 @@ pipeline {
                     } else {
                         echo "This is parent job so getting list of nodes matching labels:"
                         for (label in pullRequest.labels) {
+                            echo "label in pullReqest: ${label}"
                             if (label.matches("CI-(.*?)-Ready")) {
+                                echo "Found Ready Label: ${label}"
                                 def machine_name = label.split('-')[1].toString().toLowerCase()
                                 jenkins.model.Jenkins.get().computers.each { c ->
                                     if (c.node.selfLabel.name == NodeName[machine_name]) {
                                         run_nodes.add(c.node.selfLabel.name)
+                                        machine_names.add(machine_name)  // record machine name alongside node
                                     }
                                 }
                             }
                         }
-                        // Spawning all the jobs on the nodes matching the labels
+                        // Spawning jobs using both run_nodes and machine_names arrays
                         if (run_nodes.size() > 1) {
-                            run_nodes.init().each { node ->
-                                def machine_name = node.split('-')[0].toLowerCase()
+                            for (int i = 0; i < run_nodes.size() - 1; i++) {
+                                def node = run_nodes[i]
+                                def machine_name = machine_names[i]  // use the corresponding machine name
                                 echo "Spawning job on node: ${node} with machine name: ${machine_name}"
                                 build job: "/global-workflow/EMC-Global-Pipeline/PR-${env.CHANGE_ID}", parameters: [
                                     string(name: 'machine', value: machine_name),
-                                    string(name: 'Node', value: node) ],
-                                    wait: false
+                                    string(name: 'Node', value: node)
+                                ], wait: false
                             }
-                            machine = run_nodes.last().split('-')[0].toLowerCase()
+                            machine = machine_names[run_nodes.size() - 1]
                             echo "Running parent job: ${machine}"
                         } else {
-                            machine = run_nodes[0].split('-')[0].toLowerCase()
+                            machine = machine_names[0]
                             echo "Running only the parent job: ${machine}"
                         }
                     }
@@ -76,10 +82,11 @@ pipeline {
             agent { label NodeName[machine].toLowerCase() }
             steps {
                 script {
+                    // Capitalize the first letter of the machine name and use if for labels
                     Machine = machine[0].toUpperCase() + machine.substring(1)
                     echo "Getting Common Workspace for ${Machine}"
                     ws("${custom_workspace[machine]}/${env.CHANGE_ID}") {
-                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'Gaea', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
+                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'GaeaC5', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
                         GH = sh(script: "which gh || echo '~/bin/gh'", returnStdout: true).trim()
                         CUSTOM_WORKSPACE = "${WORKSPACE}"
                         HOMEgfs = "${CUSTOM_WORKSPACE}/global-workflow"
@@ -184,18 +191,20 @@ pipeline {
             agent { label NodeName[machine].toLowerCase() }
             steps {
                 script {
+                  ws(HOMEgfs) {
                     def parallelStages = CI_CASES.collectEntries { caseName ->
                         ["${caseName}": {
                             stage("Create ${caseName}") {
                                 script {
                                         env.RUNTESTS = "${CUSTOM_WORKSPACE}/RUNTESTS"
+                                        def error_output = ""
                                         try {
                                               error_output = sh(script: """
                                                 source ${HOMEgfs}/workflow/gw_setup.sh
                                                 ${HOMEgfs}/ci/scripts/utils/ci_utils_wrapper.sh create_experiment ${HOMEgfs}/ci/cases/pr/${caseName}.yaml
                                               """, returnStdout: true).trim()
                                         } catch (Exception error_create) {
-                                            sh(script: """${GH} pr comment ${env.CHANGE_ID} --repo ${repo_url} --body "${Case} **FAILED** to create experiment on ${Machine} in BUILD# ${env.BUILD_NUMBER}\n with the error:\n\\`\\`\\`\n${error_output}\\`\\`\\`" """)
+                                            sh(script: """${GH} pr comment ${env.CHANGE_ID} --repo ${repo_url} --body '${caseName} **FAILED** to create experiment on ${Machine} in BUILD# ${env.BUILD_NUMBER}\n with the error:\n```\n${error_output}```' """)
                                             error("Case ${caseName} failed to create experiment directory")
                                         }
                                      }
@@ -271,8 +280,9 @@ pipeline {
                         }]
                     }
                 parallel parallelStages + [failFast: true]
-                }
-            }
+              }
+            } 
+          }
         }
 
         stage( '5. Finalize' ) {
diff --git a/ci/platforms/config.orion b/ci/platforms/config.orion
index 880546908e6..5188300d159 100644
--- a/ci/platforms/config.orion
+++ b/ci/platforms/config.orion
@@ -1,18 +1,25 @@
 #!/usr/bin/bash
 
-export GFS_CI_ROOT=/work2/noaa/stmp/GFS_CI_ROOT/ORION
+# Main CI root directory
+export GFS_CI_ROOT=/work2/noaa/global/${USER}/GFS_CI_CD/ORION
+# ICSDIR root directory used on the create_experment.py command line
 export ICSDIR_ROOT=/work/noaa/global/glopara/data/ICSDIR
 
-# Jenkins directories
-export JENKINS_AGENT_LAUNCH_DIR=/home/role-nems/GFS_CI_ROOT_JENKINS/AGENT_mterry
-export JENKINS_WORK_DIR=/home/role-nems/GFS_CI_ROOT_JENKINS
+# JENKINS launch directory for agent
+export JENKINS_AGENT_LAUNCH_DIR=${GFS_CI_ROOT}/Jenkins/agent
+# JENKINS internal working directories for CI jobs (not for users use)
+export JENKINS_WORK_DIR=${GFS_CI_ROOT}/Jenkins/workspace
+# NOTE: JENKINS custom_workspace directory where CI jobs are run
+# /work2/noaa/global/role-global/GFS_CI_CD/ORION/CI
+# is defined in the Jenkinsfile
+
+# CTest functional test directories for pre stagged input data
+export STAGED_TESTS_DIR=${GFS_CI_ROOT}/STAGED_TESTS_DIR
 
 # CI BASH test directories
 export GFS_BASH_CI_ROOT=${GFS_CI_ROOT}/GFS_BASH_CI
 
-# CTest functional test directories for pre stagged input data
-export STAGED_TESTS_DIR=/work/noaa/stmp/GFS_CI_ROOT/ORION/STAGED_TESTS_DIR
-
-export HPC_ACCOUNT=nems
+# HPC account which overides the default account
+export HPC_ACCOUNT=fv3-cpu
 export max_concurrent_cases=5
 export max_concurrent_pr=4
diff --git a/ci/scripts/utils/launch_java_agent.sh b/ci/scripts/utils/launch_java_agent.sh
index a4da13b227f..9d9bd130d1f 100755
--- a/ci/scripts/utils/launch_java_agent.sh
+++ b/ci/scripts/utils/launch_java_agent.sh
@@ -164,6 +164,13 @@ check_node_online() {
 
 lauch_agent () {
     echo "Launching Jenkins Agent on ${host} using internal workspace ${JENKINS_WORK_DIR}"
+    
+    # Clear the remoting cache
+    if [[ -d "${JENKINS_WORK_DIR}/remoting" ]]; then
+        echo "Clearing remoting cache in ${JENKINS_WORK_DIR}/remoting"
+        rm -rf "${JENKINS_WORK_DIR}/remoting"
+    fi
+    
     command="nohup ${JAVA} -jar agent.jar -jnlpUrl ${controller_url}/computer/${MACHINE_ID^}-EMC/jenkins-agent.jnlp  -secret @jenkins-secret-file -workDir ${JENKINS_WORK_DIR}"
     echo -e "Launching Jenkins Agent on ${host} with the command:\n${command}" >& "${LOG}"
     ${command} >> "${LOG}" 2>&1 &
diff --git a/workflow/hosts/orion.yaml b/workflow/hosts/orion.yaml
index f1ef2c8b944..adc9547e0a6 100644
--- a/workflow/hosts/orion.yaml
+++ b/workflow/hosts/orion.yaml
@@ -4,9 +4,9 @@ BASE_DATA: '/work/noaa/global/glopara/data'
 BASE_IC: '/work/noaa/global/glopara/data/ICSDIR'
 PACKAGEROOT: '/work/noaa/global/glopara/nwpara'
 COMINsyn: '/work/noaa/global/glopara/com/gfs/prod/syndat'
-HOMEDIR: '/work/noaa/global/${USER}'
-STMP: '/work/noaa/stmp/${USER}/ORION'
-PTMP: '/work/noaa/stmp/${USER}/ORION'
+HOMEDIR: '/work2/noaa/global/${USER}/ORION'
+STMP: '/work2/noaa/global/${USER}/ORION/STMP'
+PTMP: '/work2/noaa/global/${USER}/ORION/PTMP'
 NOSCRUB: $HOMEDIR
 SCHEDULER: slurm
 ACCOUNT: fv3-cpu


Reviews:
user: login: aerorahul
body: Looks fine to me. Thanks Terry!
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T18:03:02Z
user: login: DavidHuber-NOAA
body: Changes look good to me.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T13:30:14Z
user: login: KateFriedman-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T14:02:34Z
user: login: TerrenceMcGuinness-NOAA
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-14T15:38:31Z
user: login: KateFriedman-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T15:41:47Z
user: login: KateFriedman-NOAA
body: Looks good, thanks @TerrenceMcGuinness-NOAA !
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T15:42:14Z

Review comments:
diff_hunk: @@ -76,10 +82,11 @@ pipeline {
             agent { label NodeName[machine].toLowerCase() }
             steps {
                 script {
+                    // Capitalize the first letter of the machine name and use if for labels
                     Machine = machine[0].toUpperCase() + machine.substring(1)
                     echo "Getting Common Workspace for ${Machine}"
                     ws("${custom_workspace[machine]}/${env.CHANGE_ID}") {
-                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'Gaea', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
+                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'GaeaC5', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
path: ci/Jenkinsfile
user: login: KateFriedman-NOAA
body: Should this be `GaeaC5-EMC` with "_EMC" like the others?
created_at: 2025-03-14T14:02:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,10 +82,11 @@ pipeline {
             agent { label NodeName[machine].toLowerCase() }
             steps {
                 script {
+                    // Capitalize the first letter of the machine name and use if for labels
                     Machine = machine[0].toUpperCase() + machine.substring(1)
                     echo "Getting Common Workspace for ${Machine}"
                     ws("${custom_workspace[machine]}/${env.CHANGE_ID}") {
-                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'Gaea', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
+                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'GaeaC5', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
path: ci/Jenkinsfile
user: login: TerrenceMcGuinness-NOAA
body: No, this is the node name for EPIC
created_at: 2025-03-14T15:38:31Z
author_association: COLLABORATOR
diff_hunk: @@ -76,10 +82,11 @@ pipeline {
             agent { label NodeName[machine].toLowerCase() }
             steps {
                 script {
+                    // Capitalize the first letter of the machine name and use if for labels
                     Machine = machine[0].toUpperCase() + machine.substring(1)
                     echo "Getting Common Workspace for ${Machine}"
                     ws("${custom_workspace[machine]}/${env.CHANGE_ID}") {
-                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'Gaea', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
+                        properties([parameters([[$class: 'NodeParameterDefinition', allowedSlaves: ['built-in', 'Hercules-EMC', 'Hera-EMC', 'Orion-EMC', 'GaeaC5', 'GaeaC6-EMC'], defaultSlaves: ['built-in'], name: '', nodeEligibility: [$class: 'AllNodeEligibility'], triggerIfResult: 'allCases']])])
path: ci/Jenkinsfile
user: login: KateFriedman-NOAA
body: Gotcha, good to know, thanks @TerrenceMcGuinness-NOAA !
created_at: 2025-03-14T15:41:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: emcbot
created_at: 2025-03-13T00:47:35Z
body: Experiment C48_ATM **FAILED** on Orion in Build# 5 with error logs:
```
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C48_ATM_b671a626/logs/2021032312/gfs_arch_vrfy.log
```

Follow link here to view the contents of the above file(s): [(link)](https://gist.github.com/emcbot/74e9fc7986af1bea05db47522d83da60)
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T00:47:44Z
body: Experiment C48_ATM **FAILED** on Orion in Build# 5 in
`/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/EXPDIR/C48_ATM_b671a626`
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T01:18:22Z
body: Experiment C96C48_hybatmDA **FAILED** on Orion in Build# 5 with error logs:
```
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C96C48_hybatmDA_b671a626/logs/2021122100/gdas_fit2obs.log
```

Follow link here to view the contents of the above file(s): [(link)](https://gist.github.com/emcbot/388880578b6537f6caa5cec98ecd224f)
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T01:18:23Z
body: Experiment C96_atm3DVar **FAILED** on Orion in Build# 5 with error logs:
```
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C96_atm3DVar_b671a626/logs/2021122100/gdas_fit2obs.log
```

Follow link here to view the contents of the above file(s): [(link)](https://gist.github.com/emcbot/cad954f1cdb170f0eeeef700b032acad)
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T01:18:31Z
body: Experiment C96C48_hybatmDA **FAILED** on Orion in Build# 5 in
`/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/EXPDIR/C96C48_hybatmDA_b671a626`
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T01:18:32Z
body: Experiment C96_atm3DVar **FAILED** on Orion in Build# 5 in
`/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/EXPDIR/C96_atm3DVar_b671a626`
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T02:45:12Z
body: Experiment C48_S2SW **FAILED** on Orion in Build# 5 with error logs:
```
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C48_S2SW_b671a626/logs/2021032312/gfs_arch_vrfy.log
```

Follow link here to view the contents of the above file(s): [(link)](https://gist.github.com/emcbot/ba064e226a28f6cd993bae1d97ef7e1a)
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T02:45:22Z
body: Experiment C48_S2SW **FAILED** on Orion in Build# 5 in
`/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/EXPDIR/C48_S2SW_b671a626`
author_association: NONE
user: login: emcbot
created_at: 2025-03-13T03:03:48Z
body: **CI Failed** on Orion in Build# 5<br>Built and ran in directory `/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440`
```

Experiment C96mx100_S2S_b671a626 Completed 1 Cycles: *SUCCESS* at Wed Mar 12 07:28:36 PM CDT 2025
Experiment C48_ATM_b671a626 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Mar 12 07:47:09 PM CDT 2025
Experiment C48_ATM_b671a626 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C48_ATM_b671a626/logs/2021032312/gfs_arch_vrfy.log
Experiment C96_atm3DVar_b671a626 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Mar 12 08:18:04 PM CDT 2025
Experiment C96_atm3DVar_b671a626 Terminated: *FAIL*
Experiment C96C48_hybatmDA_b671a626 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Mar 12 08:18:04 PM CDT 2025
Experiment C96C48_hybatmDA_b671a626 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C96C48_hybatmDA_b671a626/logs/2021122100/gdas_fit2obs.log
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C96_atm3DVar_b671a626/logs/2021122100/gdas_fit2obs.log
Experiment C48_S2SW_b671a626 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Wed Mar 12 09:44:52 PM CDT 2025
Experiment C48_S2SW_b671a626 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440/RUNTESTS/COMROOT/C48_S2SW_b671a626/logs/2021032312/gfs_arch_vrfy.log
Experiment C48_S2SWA_gefs_b671a626 Completed 1 Cycles: *SUCCESS* at Wed Mar 12 10:03:34 PM CDT 2025

author_association: NONE
user: login: emcbot
created_at: 2025-03-13T21:02:54Z
body: **CI Passed** on Orion in Build# 6<br>Built and ran in directory `/work2/noaa/global/role-global/GFS_CI_CD/ORION/CI/3440`
```

Experiment C96mx100_S2S_8711e590 Completed 1 Cycles: *SUCCESS* at Thu Mar 13 01:29:19 PM CDT 2025
Experiment C48_ATM_8711e590 Completed 1 Cycles: *SUCCESS* at Thu Mar 13 01:41:20 PM CDT 2025
Experiment C96_atm3DVar_8711e590 Completed 3 Cycles: *SUCCESS* at Thu Mar 13 03:31:04 PM CDT 2025
Experiment C96C48_hybatmDA_8711e590 Completed 3 Cycles: *SUCCESS* at Thu Mar 13 03:31:16 PM CDT 2025
Experiment C48_S2SW_8711e590 Completed 1 Cycles: *SUCCESS* at Thu Mar 13 03:37:21 PM CDT 2025
Experiment C48_S2SWA_gefs_8711e590 Completed 1 Cycles: *SUCCESS* at Thu Mar 13 04:02:40 PM CDT 2025

author_association: NONE