Repository: rancher/tests
Pull request #118: Updated the recurring runs Jenkinsfile to use the new qa-infra repo
Author: login: igomez06
Created: 2025-05-20T03:41:20Z | Merged: 2025-05-23T18:54:31Z

Removed corral. Wrote some helper scripts, as well removed corral code that is no longer needed. Fixed two build error in the validation test side.


Diff:
diff --git a/validation/Dockerfile.e2e b/validation/Dockerfile.e2e
index e697c0225b..eba7f7ffad 100644
--- a/validation/Dockerfile.e2e
+++ b/validation/Dockerfile.e2e
@@ -6,20 +6,58 @@ ENV GOPATH=/root/go
 ENV PATH=${PATH}:/root/go/bin
 
 ENV WORKSPACE=${GOPATH}/src/github.com/rancher/tests
-ENV WORKSPACE2=${GOPATH}/src/github.com/rancherlabs/corral-packages
+ENV WORKSPACE2=${GOPATH}/src/github.com/rancher/qa-infra-automation
 
 WORKDIR $WORKSPACE
 
 COPY ["./tests", "$WORKSPACE"]
-COPY ["./corral-packages", "$WORKSPACE2"]
+COPY ["./qa-infra-automation", "$WORKSPACE2"]
 
-ARG CORRAL_VERSION=v1.1.1
+ARG TERRAFORM_VERSION=1.11.3
 
 RUN zypper -n install gcc binutils glibc-devel-static ca-certificates git-core wget curl unzip tar vim less file xz gzip sed gawk iproute2 iptables jq
 RUN zypper install -y -f docker && rpm -e --nodeps --noscripts containerd
+
+# Python related dependencies for the Integration/Validation tests.
+RUN zypper -n install python311-pip python311-base python311 python311-devel python311-tox libffi-devel libopenssl-devel
+
+# Upgrade pip and install Python packages
+RUN python3.11 -m pip install --upgrade pip setuptools && \
+    python3.11 -m pip install ansible kubernetes
+
+# Install ansible-inventory-terraform plugin from GitHub
+RUN ansible-galaxy collection install cloud.terraform
+    
 RUN go mod download && \
-    go install gotest.tools/gotestsum@latest && \
-    go install github.com/rancherlabs/corral@${CORRAL_VERSION}
+    go install gotest.tools/gotestsum@latest 
+
+RUN zypper install -y openssh && \
+    mkdir -p /root/.ssh && chmod 600 .ssh/jenkins-* && \
+    cp -r .ssh/jenkins-* /root/.ssh/
+
+RUN for pem_file in /root/.ssh/jenkins-*; do \
+        ssh-keygen -f "$pem_file" -y > "/root/.ssh/$(basename "$pem_file").pub"; \
+    done
+
+RUN ls -al /root/.ssh
+
+# Install Terraform (manual download and install)
+RUN wget "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" && \
+    unzip "terraform_${TERRAFORM_VERSION}_linux_amd64.zip" && \
+    mv terraform /usr/local/bin/ && \
+    chmod +x /usr/local/bin/terraform && \
+    rm "terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
+
+# Install kubectl
+RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" && \
+    chmod +x kubectl && \
+    mv ./kubectl /usr/local/bin/kubectl
+
+# Install Helm
+RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 && \
+chmod +x get_helm.sh && \
+./get_helm.sh && \
+rm get_helm.sh
 
 # necessary to run if statements using [[ ]]
 SHELL ["/bin/bash", "-c"] 
diff --git a/validation/fleet/upgrade/upgrade.go b/validation/fleet/upgrade/upgrade.go
index a8bed33454..83a4f4296c 100644
--- a/validation/fleet/upgrade/upgrade.go
+++ b/validation/fleet/upgrade/upgrade.go
@@ -47,7 +47,7 @@ func gitPushCommit(client *rancher.Client, sshNode *nodes.Node, repoName string)
 
 func createLocalFleetGitRepo(client *rancher.Client, sshNode *nodes.Node, repoName string, namespaceName string, clusterName string, clusterID string, secretName string) (*v1.SteveAPIObject, error) {
 
-	gitSSHRepo := fmt.Sprintf(fmt.Sprintf("%s@%s:/home/%s/%s", sshNode.SSHUser, sshNode.PublicIPAddress, sshNode.SSHUser, repoName))
+	gitSSHRepo := fmt.Sprintf("%s@%s:/home/%s/%s", sshNode.SSHUser, sshNode.PublicIPAddress, sshNode.SSHUser, repoName)
 
 	fleetGitRepo := &v1alpha1.GitRepo{
 		ObjectMeta: metav1.ObjectMeta{
diff --git a/validation/pipeline/Jenkinsfile.multibranch.recurring b/validation/pipeline/Jenkinsfile.multibranch.recurring
index b028f5af8d..ddb2f85cba 100644
--- a/validation/pipeline/Jenkinsfile.multibranch.recurring
+++ b/validation/pipeline/Jenkinsfile.multibranch.recurring
@@ -5,12 +5,18 @@ node {
     def rancherImageTags = "${env.RANCHER_IMAGE_TAGS}".split(',')
     def jobName = "${JOB_NAME}"
     def config = env.CONFIG
+    def ansibleConfig = env.ANSIBLE_CONFIG
+    def terraformConfig = env.TERRAFORM_CONFIG
     def filename = "config.yaml"
+    def ansibleVarsFilename = "vars.yaml"
+    def terraformVarsfilename = "cluster.tfvars"
     def hostnameSuffix = ["v2.8-head":"v28", "v2.9-head":"v29", "v2.10-head":"v210", "head":"main"] // TODO: update with new versions supported by rancher
     def tagChartMap = ["v2.8-head":"2.8.11", "v2.9-head":"2.9.5", "v2.10-head":"2.10.1", "head":"2.11.0"] // TODO: update with new versions supported by rancher for image tags
     def primeChartList = ["v2.8-head", "v2.9-head"]
     def rancherChartRepo = "prime"
     def configPath = ""
+    def ansibleConfigPath = ""
+    def terraformConfigPath = ""
 
     if (jobName.contains('/')) { 
       jobNames = jobName.split('/')
@@ -77,8 +83,12 @@ node {
             stage('Setup') {
                 dir("./tests/validation") {
                   writeFile file: filename, text: config
-                  env.CATTLE_TEST_CONFIG = "${workPath}${filename}"
+                  writeFile file: ansibleVarsFilename, text: ansibleConfig
+                  writeFile file: terraformVarsfilename, text: terraformConfig
+                  // env.CATTLE_TEST_CONFIG = "${workPath}${filename}"
                   configPath = "./tests/validation/${filename}"
+                  ansibleConfigPath = "./tests/validation/${ansibleVarsFilename}"
+                  terraformConfigPath = "./tests/validation/${terraformVarsfilename}"
                 }
                 sh returnStdout: true, script: 'wget -qO ./yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64'
                 sh returnStdout:true, script: 'chmod a+x ./yq'
@@ -87,7 +97,7 @@ node {
               try {
                   jobs = [:]
                   def hostnamePrefix = sh (
-                    script: "./yq '.corralConfigs.corralConfigVars.aws_hostname_prefix '  ${configPath}",
+                    script: "./yq '.aws_hostname_prefix '  ${configPath}",
                     returnStdout: true
                   ).trim()
                   for( String testTag : rancherImageTags ) {
@@ -98,22 +108,27 @@ node {
                     def hostname = "${hostnamePrefix}${hostnameSuffix[testTag]}"
                     def rancherHostName = "${hostname}.qa.rancher.space"
 
-                    sh "./yq e '.corralConfigs.corralConfigVars.rancher_image_tag = \"${testTag}\"' -i ${configPath}"
-                    sh "./yq e '.corralConfigs.corralConfigVars.rancher_version = \"${tagChartMap[testTag]}\"' -i ${configPath}"
-                    sh "./yq e '.corralConfigs.corralConfigVars.aws_hostname_prefix = \"${hostname}\"' -i ${configPath}"
+                    sh "./yq e '.rancher_version = \"${tagChartMap[testTag]}\"' -i ${ansibleConfigPath}"
+                    sh "./yq e '.rancher_image_tag = \"${tagChartMap[testTag]}\"' -i ${ansibleConfigPath}"
+                    sh "./yq e '.aws_hostname_prefix = \"${hostname}\"' -i ${terraformConfigPath}"
+
                     sh "./yq e '.rancher.host = \"${rancherHostName}\"' -i ${configPath}"
                     
                     if ( primeChartList.contains(testTag) ) {
                       def cattleImageEnvVar = "[\\\"CATTLE_AGENT_IMAGE|${env.PRIME_STAGING_REGISTRY}/rancher/rancher-agent:${testTag}\\\"]"
                       def rancherImage = "${env.PRIME_STAGING_REGISTRY}/rancher/rancher"
-                      sh "./yq e '.corralConfigs.corralConfigVars.rancher_chart_url = \"${env.PRIME_REPO}\"' -i ${configPath}"
-                      sh "./yq e '.corralConfigs.corralConfigVars.rancher_chart_repo = \"${rancherChartRepo}\"' -i ${configPath}"
-                      sh "./yq e '.corralConfigs.corralConfigVars.rancher_image = \"${rancherImage}\"' -i ${configPath}"
-                      sh "./yq e '.corralConfigs.corralConfigVars.env_var_map = \"${cattleImageEnvVar}\"' -i ${configPath}"
+                      sh "./yq e '.rancher_chart_url = \"${env.PRIME_REPO}\"' -i ${ansibleConfigPath}"
+                      sh "./yq e '.rancher_chart_repo = \"${rancherChartRepo}\"' -i ${ansibleConfigPath}"
+                      sh "./yq e '.rancher_image = \"${rancherImage}\"' -i ${ansibleConfigPath}"
+                      sh "./yq e '.env_var_map = \"${cattleImageEnvVar}\"' -i ${ansibleConfigPath}" // TODO: this will update when support multiple rancher version heads are added
                     }
                     def configYaml = readFile(file: "${configPath}")
+                    def ansibleConfigYaml = readFile(file: "${ansibleConfigPath}")
+                    def terraformConfigYaml = readFile(file: "${terraformConfigPath}")
                     def goValidationParams = [
                         text(name: 'CONFIG', value: configYaml),
+                        text(name: 'ANSIBLE_CONFIG', value: ansibleConfigYaml),
+                        text(name: 'TERRAFORM_CONFIG', value: terraformConfigYaml)
                         string(name: 'BRANCH', value: "${branch}"),
                         string(name: 'TEST_RUN_NAME', value: "${testTag}-${env.TEST_RUN_NAME}"),
                       ]
diff --git a/validation/pipeline/Jenkinsfile.recurring b/validation/pipeline/Jenkinsfile.recurring
index 2ba02068ec..fd925fc3ed 100644
--- a/validation/pipeline/Jenkinsfile.recurring
+++ b/validation/pipeline/Jenkinsfile.recurring
@@ -2,6 +2,7 @@
 node {
     def rootPath = "/root/go/src/github.com/rancher/tests/"
     def workPath = "/root/go/src/github.com/rancher/tests/validation/"
+    def qaInfraWorkPath = "/root/go/src/github.com/rancher/qa-infra-automation"
     def jobName = "${JOB_NAME}"
     if (jobName.contains('/')) { 
       jobNames = jobName.split('/')
@@ -22,27 +23,33 @@ node {
     def envFile = ".env"
     def rancherConfig = "rancher_env.config"
     def branch = "main"
-    def corralBranch = "main"
+    def qaInfraBranch = "main"
+    def qaInfraRepo = "https://github.com/rancher/qa-infra-automation"
     def config = env.CONFIG
+    def ansibleConfig = env.ANSIBLE_CONFIG
+    def terraformConfig = env.TERRAFORM_CONFIG 
     def rancherToken = ""
     def filename = "config.yaml"
-    def individualJob = (jobName =~ /(go-recurring-[a-z]+)/)[0][1]
+    def ansibleVarsFilename = "vars.yaml"
+    def terraformVarsfilename = "cluster.tfvars"
+    // def individualJob = (jobName =~ /(go-recurring-[a-z]+)/)[0][1]
+    def individualJob = "go-recurring-daily"
     individualJob = "${individualJob}-individual-job"
 
     if ("${env.BRANCH}" != "null" && "${env.BRANCH}" != "") {
       branch = "${env.BRANCH}"
     }
 
-    if ("${env.RANCHER_CORRAL_PACKAGES_REPO_BRANCH}" != "null" && "${env.RANCHER_CORRAL_PACKAGES_REPO_BRANCH}" != "") {
-      corralBranch = "${env.RANCHER_CORRAL_PACKAGES_REPO_BRANCH}"
-    }
-
     if ("${env.REPO}" != "null" && "${env.REPO}" != "") {
       rancherRepo = "${env.REPO}"
     }
 
-    if ("${env.RANCHER_CORRAL_PACKAGES_REPO_URL}" != "null" && "${env.RANCHER_CORRAL_PACKAGES_REPO_URL}" != "") {
-      corralRepo = "${env.RANCHER_CORRAL_PACKAGES_REPO_URL}"
+    if ("${env.QA_INFRA_REPO_BRANCH}" != "null" && "${env.QA_INFRA_REPO_BRANCH}" != "") {
+      qaInfraBranch = "${env.QA_INFRA_REPO_BRANCH}"
+    }
+
+    if ("${env.QA_INFRA_REPO_URL}" != "null" && "${env.QA_INFRA_REPO_URL}" != "") {
+      qaInfraRepo = "${env.QA_INFRA_REPO_URL}"
     }
   
     def timeout = "60m"
@@ -88,13 +95,13 @@ node {
                       userRemoteConfigs: [[url: rancherRepo]]
                     ])
             }
-            dir('./corral-packages') {
-              echo "cloning corral-packages repo"
+            dir('./qa-infra-automation') {
+              echo "cloning qa-infra-automation repo"
               checkout([
                     $class: 'GitSCM',
-                    branches: [[name: "*/${corralBranch}"]],
+                    branches: [[name: "*/${qaInfraBranch}"]],
                     extensions: scm.extensions + [[$class: 'CleanCheckout']],
-                    userRemoteConfigs: [[url: corralRepo]]
+                    userRemoteConfigs: [[url: qaInfraRepo]]
                   ])
             }
           }
@@ -107,9 +114,16 @@ node {
               config = config.replace('${AWS_VPC}', env.AWS_VPC)
               config = config.replace('${AWS_SECURITY_GROUPS}', env.AWS_SECURITY_GROUPS)
               config = config.replace('${ADMIN_PASSWORD}', env.ADMIN_PASSWORD)
+              terraformConfig = terraformConfig.replace('${AWS_SECRET_ACCESS_KEY}', env.AWS_SECRET_ACCESS_KEY)
+              terraformConfig = terraformConfig.replace('${AWS_ACCESS_KEY_ID}', env.AWS_ACCESS_KEY_ID)
+              terraformConfig = terraformConfig.replace('${AWS_REGION}', env.AWS_REGION)
+              terraformConfig = terraformConfig.replace('${AWS_VPC}', env.AWS_VPC)
+              terraformConfig = terraformConfig.replace('${AWS_SECURITY_GROUPS}', env.AWS_SECURITY_GROUPS)
+              ansibleConfig = ansibleConfig.replace('${ADMIN_PASSWORD}', env.ADMIN_PASSWORD)
+
 
               if (env.AWS_SSH_PEM_KEY && env.AWS_SSH_KEY_NAME) {
-                dir("./tests/validation/.ssh") {
+                dir("./tests/.ssh") {
                   def decoded = new String(AWS_SSH_PEM_KEY.decodeBase64())
                   writeFile file: AWS_SSH_KEY_NAME, text: decoded
                 }
@@ -118,6 +132,14 @@ node {
                 writeFile file: filename, text: config
                 env.CATTLE_TEST_CONFIG = "${workPath}${filename}"
               }
+              dir("./qa-infra-automation") {
+                dir ("./ansible") {
+                  writeFile file: ansibleVarsFilename, text: ansibleConfig
+                }
+                dir ("./terraform/aws/cluster_nodes") {
+                  writeFile file: terraformVarsfilename, text: terraformConfig
+                }
+              }
               
               dir ("./") {
                 sh "./tests/validation/configure.sh"
@@ -132,7 +154,7 @@ node {
             stage("Build Environment") {
               try {
                 sh "docker run -v ${validationVolume}:/root --name ${buildTestContainer} -t --env-file ${envFile} " +
-                "${imageName} sh -c \"${workPath}pipeline/scripts/setup_environment.sh;${workPath}pipeline/scripts/build_qase_auto_testrun.sh\""
+                "${imageName} sh -c \"${workPath}pipeline/scripts/setup_environment.sh && ${workPath}pipeline/scripts/build_qase_auto_testrun.sh\""
               } catch(err) {
                   sh "docker stop ${buildTestContainer}"
                   sh "docker rm -v ${buildTestContainer}"
@@ -141,67 +163,6 @@ node {
               }
             }
             try {
-              stage ('Generate configuration files') {
-                  sh "docker run --volumes-from ${buildTestContainer} --name ${configGeneratorContainer} -t -e CATTLE_TEST_CONFIG=\$CATTLE_TEST_CONFIG " +
-                    "${imageName} sh -c \"${workPath}pipeline/releaseupgrade/releaseupgrade.sh\""
-
-                  sh "docker cp ${configGeneratorContainer}:${rootPath}${configsDir}/ ."
-
-                  def configs = sh(script: "ls -1 ./${configsDir}", returnStdout: true).split()
-
-                  for (int i = 0; i < configs.size(); i++) {
-                      def configName = configs[i]
-                      echo "pushing ${configName} to the global variable"
-                      configFiles << "${configName}"
-                  }
-
-                  println "Building the image again with the config files"
-                  sh "./tests/validation/configure.sh"
-                  sh "docker build . -f ./tests/validation/Dockerfile.e2e -t ${imageName}"
-              }
-              stage('Deploys clusters') {
-                try {
-                  jobs = [:]
-
-                  configFiles.each {
-                    def configPath = "./${configsDir}/$it"
-                    def absoluteConfigPath = "${rootPath}${configsDir}/$it"
-
-                    def testCase = sh (
-                      script: "./yq '.testCases.provisioningTestCase'  ${configPath}",
-                      returnStdout: true
-                    ).trim()
-
-                    def testPackage = sh (
-                      script: "./yq '.testCases.provisioningTestPackage'  ${configPath}",
-                      returnStdout: true
-                    ).trim()
-
-                    jobs["$it-provisioning"] = { 
-                      stage("Parallel Provisioning: $it"){
-                        def containerName = "${jobName}${env.BUILD_NUMBER}_$it-provisioning"
-                        try {
-                          sh "docker run --volumes-from ${buildTestContainer} --name ${containerName} -t -e CATTLE_TEST_CONFIG=\"${absoluteConfigPath}\" " +
-                        "${imageName} sh -c \"gotestsum --format standard-verbose --packages=\"${testsDir}${testPackage}\" --junitfile ${testResultsOut} -- -tags=validation ${testCase} -timeout=\"60m\" -v\""
-
-                          dir("./${patchedConfigsDir}") {
-                            sh "docker cp ${containerName}:${rootPath}${configsDir}/$it ."
-                          }
-                        } catch (err){
-                          sh "docker stop ${containerName}"
-                          sh "docker rm -v ${containerName}"
-                        }
-
-                        sh "docker stop ${containerName}"
-                        sh "docker rm -v ${containerName}"
-                      }
-                    }
-                  }
-                  parallel jobs
-                } catch(err){
-                    echo "Provisioning tests had failures: " + err
-                }
-              }
               stage('Run Test Validations') {
                   try {
                     jobs = [:]
@@ -224,65 +185,36 @@ node {
 
                     sh "./yq e '.flags.desiredflags = \"${desiredflags}\"' -i ${mainRancherConfigPath}"
 
-                    configFiles.each {
-                      try {
-                        def configPath = "./${patchedConfigsDir}/$it"
-
-                        def tags = sh (
-                          script: "./yq '.testCases.tags'  ${configPath}",
-                          returnStdout: true
-                        ).trim()
-
-                        def runFlag = sh (  
-                          script: "./yq '.testCases.runFlag'  ${configPath}",
-                          returnStdout: true
-                        ).trim()
+                    def clusterName = "local"
 
-                        def clusterName = sh (  
-                          script: "./yq '.rancher.clusterName' ${configPath}",
-                          returnStdout: true
-                        ).trim()
+                    sh "./yq e '.rancher.clusterName = \"${clusterName}\"' -i ${mainRancherConfigPath}"
 
-                        sh "./yq e '.rancher.clusterName = \"${clusterName}\"' -i ${mainRancherConfigPath}"
+                    def configYaml = readFile(file: "${mainRancherConfigPath}")
 
-                        def configYaml = readFile(file: "${mainRancherConfigPath}")
+                    def goProvisioningParams = [
+                      string(name: 'TIMEOUT', value: timeout),
+                      text(name: 'CONFIG', value: configYaml),
+                      string(name: 'REPO', value: "${rancherRepo}"),
+                      string(name: 'BRANCH', value: "${branch}"),
+                      string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
+                      string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
+                      string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE_PROVISIONING}"),
+                      string(name: 'TAGS', value: "${env.PROVISIONING_TAGS}"),
+                    ]
 
-                        def goValidationParams = [
-                          string(name: 'TIMEOUT', value: timeout),
-                          text(name: 'CONFIG', value: configYaml),
-                          string(name: 'REPO', value: "${rancherRepo}"),
-                          string(name: 'BRANCH', value: "${branch}"),
-                          string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
-                          string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
-                          string(name: 'GOTEST_TESTCASE', value: "${runFlag} -p 1"),
-                          string(name: 'TAGS', value: tags),
-                        ]
+                    def goValidationParams = [
+                      string(name: 'TIMEOUT', value: timeout),
+                      text(name: 'CONFIG', value: configYaml),
+                      string(name: 'REPO', value: "${rancherRepo}"),
+                      string(name: 'BRANCH', value: "${branch}"),
+                      string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
+                      string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
+                      string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE_VALIDATION} -p 1"),
+                      string(name: 'TAGS', value: "${env.VALIDATION_TEST_TAG}"),
+                    ]
 
-                        jobs["$it-test-validation"] = { build job: individualJob, parameters: goValidationParams }
-                      } catch(err) {
-                        echo "Test Validation $it had failures: " + err
-                      }
-                    }
-
-                    if ("${env.TAGS}" != "") {
-                      sh "./yq e 'del(.provisioningInput.machinePools)' -i ${mainRancherConfigPath}"
-                      sh "./yq e 'del(.provisioningInput.nodePools)' -i ${mainRancherConfigPath}"
-                      def configYaml = readFile(file: "${mainRancherConfigPath}")
-
-                      def goValidationParams = [
-                        string(name: 'TIMEOUT', value: timeout),
-                        text(name: 'CONFIG', value: configYaml),
-                        string(name: 'REPO', value: "${rancherRepo}"),
-                        string(name: 'BRANCH', value: "${branch}"),
-                        string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
-                        string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
-                        string(name: 'TAGS', value: "${env.TAGS}"),
-                        string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE} -p 1"),
-                      ]
-
-                      jobs["provisioning-test-validation"] = { build job: individualJob, parameters: goValidationParams }
-                    
-                    }
+                    jobs["test-provisioning"] = { build job: individualJob, parameters: goProvisioningParams }
+                    jobs["test-validation"] = { build job: individualJob, parameters: goValidationParams }
                     parallel jobs
                   } catch(err) {
                     echo "Test validations had failures: " + err
@@ -301,8 +233,6 @@ node {
                 } catch(err) {
                   sh "docker stop ${buildTestContainer}"
                   sh "docker rm -v ${buildTestContainer}"
-                  sh "docker stop ${configGeneratorContainer}"
-                  sh "docker rm -v ${configGeneratorContainer}"
                   sh "docker stop ${cleanupTestContainer}"
                   sh "docker rm -v ${cleanupTestContainer}"
                   sh "docker rmi -f ${imageName}"
@@ -315,8 +245,6 @@ node {
               echo 'Cleaning test images and volume.'
               sh "docker stop ${buildTestContainer}"
               sh "docker rm -v ${buildTestContainer}"
-              sh "docker stop ${configGeneratorContainer}"
-              sh "docker rm -v ${configGeneratorContainer}"
               if ("${env.CLEANUP_RANCHER}" == "True" || "${env.CLEANUP_RANCHER}" == "true") {
                 sh "docker stop ${cleanupTestContainer}"
                 sh "docker rm -v ${cleanupTestContainer}"
diff --git a/validation/pipeline/rancherha/main.go b/validation/pipeline/rancherha/main.go
index 7222817601..068f8278eb 100644
--- a/validation/pipeline/rancherha/main.go
+++ b/validation/pipeline/rancherha/main.go
@@ -1,88 +1,33 @@
 package main
 
 import (
-	"strings"
-
-	"github.com/rancher/shepherd/clients/corral"
 	"github.com/rancher/shepherd/clients/rancher"
-	"github.com/rancher/shepherd/clients/rkecli"
 	"github.com/rancher/shepherd/pkg/config"
-	"github.com/rancher/shepherd/pkg/environmentflag"
 	"github.com/rancher/shepherd/pkg/session"
 	"github.com/rancher/tests/actions/pipeline"
-	"github.com/rancher/tests/validation/pipeline/rancherha/corralha"
 	"github.com/sirupsen/logrus"
 )
 
-const rke1 = "rke1"
-
 func main() {
-	corralRancherHA := new(corralha.CorralRancherHA)
-	config.LoadConfig(corralha.CorralRancherHAConfigConfigurationFileKey, corralRancherHA)
+	rancherConfig := new(rancher.Config)
+	config.LoadConfig(rancher.ConfigurationFileKey, rancherConfig)
+	bootstrapPassword := rancherConfig.AdminPassword
 
-	corralSession := session.NewSession()
-
-	corralConfig := corral.Configurations()
-	err := corral.SetupCorralConfig(corralConfig.CorralConfigVars, corralConfig.CorralConfigUser, corralConfig.CorralSSHPath)
+	token, err := pipeline.CreateAdminToken(bootstrapPassword, rancherConfig)
 	if err != nil {
-		logrus.Fatalf("error setting up corral: %v", err)
+		logrus.Errorf("error creating the admin token: %v", err)
 	}
 
-	configPackage := corral.PackagesConfig()
-
-	environmentFlags := environmentflag.NewEnvironmentFlags()
-	environmentflag.LoadEnvironmentFlags(environmentflag.ConfigurationFileKey, environmentFlags)
-	installRancher := environmentFlags.GetValue(environmentflag.InstallRancher)
-
-	logrus.Infof("installRancher value is %t", installRancher)
-
-	if installRancher {
-		path := configPackage.CorralPackageImages[corralRancherHA.Name]
-		corralName := corralRancherHA.Name
-
-		_, err = corral.CreateCorral(corralSession, corralName, path, true, configPackage.HasCleanup)
-		if err != nil {
-			logrus.Errorf("error creating corral: %v", err)
-		}
-
-		bootstrapPassword, err := corral.GetCorralEnvVar(corralName, "bootstrap_password")
-		if err != nil {
-			logrus.Errorf("error getting the bootstrap password: %v", err)
-		}
-
-		rancherConfig := new(rancher.Config)
-		config.LoadConfig(rancher.ConfigurationFileKey, rancherConfig)
-
-		token, err := pipeline.CreateAdminToken(bootstrapPassword, rancherConfig)
-		if err != nil {
-			logrus.Errorf("error creating the admin token: %v", err)
-		}
-
-		rancherConfig.AdminToken = token
-		config.UpdateConfig(rancher.ConfigurationFileKey, rancherConfig)
-		rancherSession := session.NewSession()
-		client, err := rancher.NewClient(rancherConfig.AdminToken, rancherSession)
-		if err != nil {
-			logrus.Errorf("error creating the rancher client: %v", err)
-		}
-
-		err = pipeline.PostRancherInstall(client, rancherConfig.AdminPassword)
-		if err != nil {
-			logrus.Errorf("error during post rancher install: %v", err)
-		}
-
-		if strings.Contains(configPackage.CorralPackageImages[corralRancherHA.Name], rke1) {
-			sshkey, err := corral.GetCorralEnvVar(corralName, "corral_private_key")
-			if err != nil {
-				logrus.Errorf("error getting the private SSH key: %v", err)
-			}
+	rancherConfig.AdminToken = token
+	config.UpdateConfig(rancher.ConfigurationFileKey, rancherConfig)
+	rancherSession := session.NewSession()
+	client, err := rancher.NewClient(rancherConfig.AdminToken, rancherSession)
+	if err != nil {
+		logrus.Errorf("error creating the rancher client: %v", err)
+	}
 
-			rkecliConfig := new(rkecli.Config)
-			config.LoadAndUpdateConfig(rkecli.ConfigurationFileKey, rkecliConfig, func() {
-				rkecliConfig.SSHKey = sshkey
-			})
-		}
-	} else {
-		logrus.Infof("Skipped Rancher Install because installRancher is %t", installRancher)
+	err = pipeline.PostRancherInstall(client, rancherConfig.AdminPassword)
+	if err != nil {
+		logrus.Errorf("error during post rancher install: %v", err)
 	}
 }
diff --git a/validation/pipeline/scripts/build_qa_infra.sh b/validation/pipeline/scripts/build_qa_infra.sh
new file mode 100755
index 0000000000..e72e319967
--- /dev/null
+++ b/validation/pipeline/scripts/build_qa_infra.sh
@@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+RKE2_PLAYBOOK_PATH="ansible/rke2/rke2-playbook.yml"
+TERRAFORM_INVENTORY="ansible/rke2/terraform-inventory.yml"
+ANSIBLE_CONFIG="ansible/rke2/ansible.cfg"
+RANCHER_PLAYBOOK_PATH="ansible/rancher/rancher-playbook.yml"
+TFVARS_FILE="cluster.tfvars"
+KUBECONFIG_FILE="$REPO_ROOT/ansible/rke2/kubeconfig.yaml"
+VARS_FILE="./ansible/vars.yaml"
+PRIVATE_KEY_FILE="/root/.ssh/jenkins-elliptic-validation.pem"
+
+# --- Terraform Steps ---
+terraform -chdir="$TERRAFORM_DIR" init -input=false
+
+# Create and select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace new "$WORKSPACE_NAME" || terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+
+# Export the TF_WORKSPACE environment variable
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_CONFIG="$ANSIBLE_CONFIG"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_PRIVATE_KEY_FILE="$PRIVATE_KEY_FILE"
+
+# Apply the Terraform configuration
+terraform -chdir="$TERRAFORM_DIR" apply -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform apply failed."
+    exit 1
+fi
+
+# --- RKE2 Playbook ---
+
+# Run the RKE2 playbook with retries
+max_attempts=3
+attempt=0
+while [ $attempt -lt $max_attempts ]; do
+    attempt=$((attempt + 1))
+    echo "Attempt $attempt: Running RKE2 playbook..."
+    ansible-playbook -i "$TERRAFORM_INVENTORY" "$RKE2_PLAYBOOK_PATH" -vvvv -e "@$VARS_FILE"
+    rke2_exit_code=$?
+    if [ $rke2_exit_code -ne 0 ]; then
+        echo "RKE2 playbook failed on attempt $attempt."
+        if [ $attempt -lt $max_attempts ]; then
+            echo "Waiting 60 seconds before retrying..."
+            sleep 60
+        fi
+    else
+        echo "RKE2 playbook succeeded on attempt $attempt."
+        break
+    fi
+done
+
+if [ $attempt -eq $max_attempts ] && [ $rke2_exit_code -ne 0 ]; then
+    echo "Error: RKE2 playbook failed after $max_attempts attempts."
+    echo "destroy the terraform"
+    # Destroy the Terraform infrastructure
+    terraform -chdir="$TERRAFORM_DIR" destroy -auto-approve -var-file="$TFVARS_FILE"
+    if [ $? -ne 0 ]; then
+        echo "Error: Terraform destroy failed."
+        exit 1
+    fi
+    echo "Terraform infrastructure destroyed successfully!"
+    exit 1
+fi
+
+# Export KUBECONFIG
+export KUBECONFIG="$KUBECONFIG_FILE"
+echo $KUBECONFIG
+
+# --- Rancher Playbook ---
+
+# Run the Rancher playbook
+ansible-playbook "$RANCHER_PLAYBOOK_PATH" -vvvv -e "@$VARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Rancher playbook failed."
+    terraform -chdir="$TERRAFORM_DIR" destroy -auto-approve -var-file="$TFVARS_FILE"
+    if [ $? -ne 0 ]; then
+        echo "Error: Terraform destroy failed."
+        exit 1
+    fi
+    echo "Terraform infrastructure destroyed successfully!"
+    exit 1
+fi
+
+echo "Script completed successfully!"
diff --git a/validation/pipeline/scripts/destroy_qa_infra.sh b/validation/pipeline/scripts/destroy_qa_infra.sh
new file mode 100644
index 0000000000..0cb688638e
--- /dev/null
+++ b/validation/pipeline/scripts/destroy_qa_infra.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -e  # Enable exit on non-zero exit code
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables (consistent with build_qa_infra.sh)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+TFVARS_FILE="cluster.tfvars" # Ensure this matches the build script
+
+# --- Terraform Steps ---
+echo "--- Terraform Destroy ---"
+
+# Select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform workspace select failed."
+    exit 1
+fi
+
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Destroy the Terraform infrastructure
+terraform -chdir="$TERRAFORM_DIR" destroy -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform destroy failed."
+    exit 1
+fi
+
+echo "Terraform infrastructure destroyed successfully!"
diff --git a/validation/pipeline/scripts/rancher_cleanup.sh b/validation/pipeline/scripts/rancher_cleanup.sh
index 6915eb15b8..0c617b8300 100755
--- a/validation/pipeline/scripts/rancher_cleanup.sh
+++ b/validation/pipeline/scripts/rancher_cleanup.sh
@@ -2,11 +2,13 @@
 set -ex
 cd $(dirname $0)/../../../../../rancher/tests
 
-echo | corral config
+# echo | corral config
 
-corral list
+# corral list
 
 ls -al
 pwd
 echo "cleanup rancher"
-validation/registries/bin/ranchercleanup
+sh ./validation/pipeline/scripts/destroy_qa_infra.sh
+# validation/registries/bin/ranchercleanup
+
diff --git a/validation/pipeline/scripts/setup_environment.sh b/validation/pipeline/scripts/setup_environment.sh
index 5c056ef3cf..ec7348bdec 100755
--- a/validation/pipeline/scripts/setup_environment.sh
+++ b/validation/pipeline/scripts/setup_environment.sh
@@ -5,15 +5,27 @@ cd $(dirname $0)/../../../../../rancher/tests
 echo "building rancherversion bin"
 env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o tests/v2/validation/rancherversion ./validation/pipeline/rancherversion
 
-echo "build corral packages"
-sh ./validation/pipeline/scripts/build_corral_packages.sh
+# echo "build corral packages"
+# sh ./validation/pipeline/scripts/build_corral_packages.sh
 
-echo | corral config
+# echo | corral config
 
-echo "build rancherHA images"
-sh ./validation/pipeline/scripts/build_rancherha_images.sh
+echo "build rancher infra"
+sh ./validation/pipeline/scripts/build_qa_infra.sh
 
-corral list
+if [ $? -eq 0 ]; then
+    echo "build rancherHA images"
+    sh ./validation/pipeline/scripts/build_rancherha_images.sh
 
-echo "running rancher corral"
-validation/registries/bin/rancherha
+    if [ $? -eq 0 ]; then
+        # corral list
+
+        echo "running rancher"
+        validation/registries/bin/rancherha
+        if [ $? -ne 0 ]; then
+            echo "rancherha failed. Running cleanup script."
+            sh ./validation/pipeline/scripts/destroy_qa_infra.sh
+            exit 1
+        fi
+    fi
+fi
diff --git a/validation/upgrade/workload.go b/validation/upgrade/workload.go
index 33a5de30f9..63989aa143 100644
--- a/validation/upgrade/workload.go
+++ b/validation/upgrade/workload.go
@@ -2,7 +2,6 @@ package upgrade
 
 import (
 	"context"
-	"fmt"
 	"strings"
 	"testing"
 	"time"
@@ -378,7 +377,7 @@ func createPostUpgradeWorkloads(t *testing.T, client *rancher.Client, clusterNam
 }
 
 func getSteveID(namespaceName, resourceName string) string {
-	return fmt.Sprintf("%s/%s", namespaceName, resourceName)
+	return namespaceName + "/" + resourceName
 }
 
 func getProject(client *rancher.Client, clusterName, projectName string) (project *management.Project, err error) {


Reviews:
user: login: rancher-max
body: Left some various comments. There are also some pieces of commented out code that likely aren't necessary anymore at all and can just be removed. BUT nothing is blocking, so approving! Thank you for this!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-22T02:17:28Z
user: login: igomez06
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T14:22:10Z
user: login: igomez06
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T14:22:43Z
user: login: igomez06
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T14:23:00Z
user: login: caliskanugur
body: A few comments and questions, LGTM
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-22T15:32:37Z
user: login: caliskanugur
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-22T15:49:10Z

Review comments:
diff_hunk: @@ -224,65 +185,36 @@ node {
 
                     sh "./yq e '.flags.desiredflags = \"${desiredflags}\"' -i ${mainRancherConfigPath}"
 
-                    configFiles.each {
-                      try {
-                        def configPath = "./${patchedConfigsDir}/$it"
-
-                        def tags = sh (
-                          script: "./yq '.testCases.tags'  ${configPath}",
-                          returnStdout: true
-                        ).trim()
-
-                        def runFlag = sh (  
-                          script: "./yq '.testCases.runFlag'  ${configPath}",
-                          returnStdout: true
-                        ).trim()
+                    def clusterName = "local"
 
-                        def clusterName = sh (  
-                          script: "./yq '.rancher.clusterName' ${configPath}",
-                          returnStdout: true
-                        ).trim()
+                    sh "./yq e '.rancher.clusterName = \"${clusterName}\"' -i ${mainRancherConfigPath}"
 
-                        sh "./yq e '.rancher.clusterName = \"${clusterName}\"' -i ${mainRancherConfigPath}"
+                    def configYaml = readFile(file: "${mainRancherConfigPath}")
 
-                        def configYaml = readFile(file: "${mainRancherConfigPath}")
+                    def goProvisioningParams = [
+                      string(name: 'TIMEOUT', value: timeout),
+                      text(name: 'CONFIG', value: configYaml),
+                      string(name: 'REPO', value: "${rancherRepo}"),
+                      string(name: 'BRANCH', value: "${branch}"),
+                      string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
+                      string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
+                      string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE_PROVISIONING}"),
+                      string(name: 'TAGS', value: "${env.PROVISIONING_TAGS}"),
+                    ]
 
-                        def goValidationParams = [
-                          string(name: 'TIMEOUT', value: timeout),
-                          text(name: 'CONFIG', value: configYaml),
-                          string(name: 'REPO', value: "${rancherRepo}"),
-                          string(name: 'BRANCH', value: "${branch}"),
-                          string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
-                          string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
-                          string(name: 'GOTEST_TESTCASE', value: "${runFlag} -p 1"),
-                          string(name: 'TAGS', value: tags),
-                        ]
+                    def goValidationParams = [
+                      string(name: 'TIMEOUT', value: timeout),
+                      text(name: 'CONFIG', value: configYaml),
+                      string(name: 'REPO', value: "${rancherRepo}"),
+                      string(name: 'BRANCH', value: "${branch}"),
+                      string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
+                      string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
+                      string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE_VALIDATION} -p 1"),
+                      string(name: 'TAGS', value: "${env.VALIDATION_TEST_TAG}"),
+                    ]
 
-                        jobs["$it-test-validation"] = { build job: individualJob, parameters: goValidationParams }
-                      } catch(err) {
-                        echo "Test Validation $it had failures: " + err
-                      }
-                    }
-
-                    if ("${env.TAGS}" != "") {
-                      sh "./yq e 'del(.provisioningInput.machinePools)' -i ${mainRancherConfigPath}"
-                      sh "./yq e 'del(.provisioningInput.nodePools)' -i ${mainRancherConfigPath}"
-                      def configYaml = readFile(file: "${mainRancherConfigPath}")
-
-                      def goValidationParams = [
-                        string(name: 'TIMEOUT', value: timeout),
-                        text(name: 'CONFIG', value: configYaml),
-                        string(name: 'REPO', value: "${rancherRepo}"),
-                        string(name: 'BRANCH', value: "${branch}"),
-                        string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
-                        string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
-                        string(name: 'TAGS', value: "${env.TAGS}"),
-                        string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE} -p 1"),
-                      ]
-
-                      jobs["provisioning-test-validation"] = { build job: individualJob, parameters: goValidationParams }
-                    
-                    }
+                    jobs["test-provisioning"] = { build job: individualJob, parameters: goProvisioningParams }
+                    jobs["test-validation"] = { build job: individualJob, parameters: goValidationParams }
path: validation/pipeline/Jenkinsfile.recurring
user: login: rancher-max
body: Probably should parameterize these job names in case they ever change for any reason.
created_at: 2025-05-22T02:03:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+RKE2_PLAYBOOK_PATH="ansible/rke2/rke2-playbook.yml"
+TERRAFORM_INVENTORY="ansible/rke2/terraform-inventory.yml"
+ANSIBLE_CONFIG="ansible/rke2/ansible.cfg"
+RANCHER_PLAYBOOK_PATH="ansible/rancher/rancher-playbook.yml"
+TFVARS_FILE="cluster.tfvars"
+KUBECONFIG_FILE="$REPO_ROOT/ansible/rke2/kubeconfig.yaml"
+VARS_FILE="./ansible/vars.yaml"
+PRIVATE_KEY_FILE="/root/.ssh/jenkins-elliptic-validation.pem"
+
+# --- Terraform Steps ---
+terraform -chdir="$TERRAFORM_DIR" init -input=false
+
+# Create and select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace new "$WORKSPACE_NAME" || terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+
+# Export the TF_WORKSPACE environment variable
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_CONFIG="$ANSIBLE_CONFIG"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_PRIVATE_KEY_FILE="$PRIVATE_KEY_FILE"
+
+# Apply the Terraform configuration
+terraform -chdir="$TERRAFORM_DIR" apply -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform apply failed."
+    exit 1
+fi
+
+# --- RKE2 Playbook ---
path: validation/pipeline/scripts/build_qa_infra.sh
user: login: rancher-max
body: I'm almost wondering if the playbooks should be its own separate script (and stage in the Jenkinsfile) so that it's easier to create other jobs in the future in case we want anything that just builds infra or just deploys on provided infra. Right now I don't think it's a big deal since it's currently built kind of coupled, but as we decouple to add more providers to the qa-infra-automation, it will make more sense to have these decouple as well.
created_at: 2025-05-22T02:12:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+RKE2_PLAYBOOK_PATH="ansible/rke2/rke2-playbook.yml"
+TERRAFORM_INVENTORY="ansible/rke2/terraform-inventory.yml"
+ANSIBLE_CONFIG="ansible/rke2/ansible.cfg"
+RANCHER_PLAYBOOK_PATH="ansible/rancher/rancher-playbook.yml"
+TFVARS_FILE="cluster.tfvars"
+KUBECONFIG_FILE="$REPO_ROOT/ansible/rke2/kubeconfig.yaml"
+VARS_FILE="./ansible/vars.yaml"
+PRIVATE_KEY_FILE="/root/.ssh/jenkins-elliptic-validation.pem"
+
+# --- Terraform Steps ---
+terraform -chdir="$TERRAFORM_DIR" init -input=false
+
+# Create and select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace new "$WORKSPACE_NAME" || terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+
+# Export the TF_WORKSPACE environment variable
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_CONFIG="$ANSIBLE_CONFIG"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_PRIVATE_KEY_FILE="$PRIVATE_KEY_FILE"
+
+# Apply the Terraform configuration
+terraform -chdir="$TERRAFORM_DIR" apply -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform apply failed."
+    exit 1
+fi
+
+# --- RKE2 Playbook ---
+
+# Run the RKE2 playbook with retries
+max_attempts=3
+attempt=0
+while [ $attempt -lt $max_attempts ]; do
+    attempt=$((attempt + 1))
+    echo "Attempt $attempt: Running RKE2 playbook..."
+    ansible-playbook -i "$TERRAFORM_INVENTORY" "$RKE2_PLAYBOOK_PATH" -vvvv -e "@$VARS_FILE"
+    rke2_exit_code=$?
+    if [ $rke2_exit_code -ne 0 ]; then
+        echo "RKE2 playbook failed on attempt $attempt."
+        if [ $attempt -lt $max_attempts ]; then
+            echo "Waiting 60 seconds before retrying..."
+            sleep 60
+        fi
+    else
+        echo "RKE2 playbook succeeded on attempt $attempt."
+        break
+    fi
+done
+
+if [ $attempt -eq $max_attempts ] && [ $rke2_exit_code -ne 0 ]; then
+    echo "Error: RKE2 playbook failed after $max_attempts attempts."
+    echo "destroy the terraform"
+    # Destroy the Terraform infrastructure
+    terraform -chdir="$TERRAFORM_DIR" destroy -auto-approve -var-file="$TFVARS_FILE"
+    if [ $? -ne 0 ]; then
+        echo "Error: Terraform destroy failed."
+        exit 1
+    fi
+    echo "Terraform infrastructure destroyed successfully!"
+    exit 1
+fi
+
+# Export KUBECONFIG
+export KUBECONFIG="$KUBECONFIG_FILE"
+echo $KUBECONFIG
+
+# --- Rancher Playbook ---
path: validation/pipeline/scripts/build_qa_infra.sh
user: login: rancher-max
body: ~Does this one need various attempts as well, or does it fail with multiple attempts anyway since it will likely deploy some resources into the cluster and need to destroy those on subsequent tries?~ It looks like there are retries built into this module actually, so ignore!
created_at: 2025-05-22T02:15:30Z
author_association: MEMBER
diff_hunk: @@ -224,65 +185,36 @@ node {
 
                     sh "./yq e '.flags.desiredflags = \"${desiredflags}\"' -i ${mainRancherConfigPath}"
 
-                    configFiles.each {
-                      try {
-                        def configPath = "./${patchedConfigsDir}/$it"
-
-                        def tags = sh (
-                          script: "./yq '.testCases.tags'  ${configPath}",
-                          returnStdout: true
-                        ).trim()
-
-                        def runFlag = sh (  
-                          script: "./yq '.testCases.runFlag'  ${configPath}",
-                          returnStdout: true
-                        ).trim()
+                    def clusterName = "local"
 
-                        def clusterName = sh (  
-                          script: "./yq '.rancher.clusterName' ${configPath}",
-                          returnStdout: true
-                        ).trim()
+                    sh "./yq e '.rancher.clusterName = \"${clusterName}\"' -i ${mainRancherConfigPath}"
 
-                        sh "./yq e '.rancher.clusterName = \"${clusterName}\"' -i ${mainRancherConfigPath}"
+                    def configYaml = readFile(file: "${mainRancherConfigPath}")
 
-                        def configYaml = readFile(file: "${mainRancherConfigPath}")
+                    def goProvisioningParams = [
+                      string(name: 'TIMEOUT', value: timeout),
+                      text(name: 'CONFIG', value: configYaml),
+                      string(name: 'REPO', value: "${rancherRepo}"),
+                      string(name: 'BRANCH', value: "${branch}"),
+                      string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
+                      string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
+                      string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE_PROVISIONING}"),
+                      string(name: 'TAGS', value: "${env.PROVISIONING_TAGS}"),
+                    ]
 
-                        def goValidationParams = [
-                          string(name: 'TIMEOUT', value: timeout),
-                          text(name: 'CONFIG', value: configYaml),
-                          string(name: 'REPO', value: "${rancherRepo}"),
-                          string(name: 'BRANCH', value: "${branch}"),
-                          string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
-                          string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
-                          string(name: 'GOTEST_TESTCASE', value: "${runFlag} -p 1"),
-                          string(name: 'TAGS', value: tags),
-                        ]
+                    def goValidationParams = [
+                      string(name: 'TIMEOUT', value: timeout),
+                      text(name: 'CONFIG', value: configYaml),
+                      string(name: 'REPO', value: "${rancherRepo}"),
+                      string(name: 'BRANCH', value: "${branch}"),
+                      string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
+                      string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
+                      string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE_VALIDATION} -p 1"),
+                      string(name: 'TAGS', value: "${env.VALIDATION_TEST_TAG}"),
+                    ]
 
-                        jobs["$it-test-validation"] = { build job: individualJob, parameters: goValidationParams }
-                      } catch(err) {
-                        echo "Test Validation $it had failures: " + err
-                      }
-                    }
-
-                    if ("${env.TAGS}" != "") {
-                      sh "./yq e 'del(.provisioningInput.machinePools)' -i ${mainRancherConfigPath}"
-                      sh "./yq e 'del(.provisioningInput.nodePools)' -i ${mainRancherConfigPath}"
-                      def configYaml = readFile(file: "${mainRancherConfigPath}")
-
-                      def goValidationParams = [
-                        string(name: 'TIMEOUT', value: timeout),
-                        text(name: 'CONFIG', value: configYaml),
-                        string(name: 'REPO', value: "${rancherRepo}"),
-                        string(name: 'BRANCH', value: "${branch}"),
-                        string(name: 'QASE_TEST_RUN_ID', value: "${testRunID}"),
-                        string(name: 'TEST_PACKAGE', value: "${env.TEST_PACKAGE}"),
-                        string(name: 'TAGS', value: "${env.TAGS}"),
-                        string(name: 'GOTEST_TESTCASE', value: "${env.GOTEST_TESTCASE} -p 1"),
-                      ]
-
-                      jobs["provisioning-test-validation"] = { build job: individualJob, parameters: goValidationParams }
-                    
-                    }
+                    jobs["test-provisioning"] = { build job: individualJob, parameters: goProvisioningParams }
+                    jobs["test-validation"] = { build job: individualJob, parameters: goValidationParams }
path: validation/pipeline/Jenkinsfile.recurring
user: login: igomez06
body: Hey Max I will change this, when I refactor the job with the new shared library stuff that Ugur has been working on. I have task in the backlog for it. For this go around I just wanted to be functional.
created_at: 2025-05-22T14:22:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+RKE2_PLAYBOOK_PATH="ansible/rke2/rke2-playbook.yml"
+TERRAFORM_INVENTORY="ansible/rke2/terraform-inventory.yml"
+ANSIBLE_CONFIG="ansible/rke2/ansible.cfg"
+RANCHER_PLAYBOOK_PATH="ansible/rancher/rancher-playbook.yml"
+TFVARS_FILE="cluster.tfvars"
+KUBECONFIG_FILE="$REPO_ROOT/ansible/rke2/kubeconfig.yaml"
+VARS_FILE="./ansible/vars.yaml"
+PRIVATE_KEY_FILE="/root/.ssh/jenkins-elliptic-validation.pem"
+
+# --- Terraform Steps ---
+terraform -chdir="$TERRAFORM_DIR" init -input=false
+
+# Create and select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace new "$WORKSPACE_NAME" || terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+
+# Export the TF_WORKSPACE environment variable
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_CONFIG="$ANSIBLE_CONFIG"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_PRIVATE_KEY_FILE="$PRIVATE_KEY_FILE"
+
+# Apply the Terraform configuration
+terraform -chdir="$TERRAFORM_DIR" apply -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform apply failed."
+    exit 1
+fi
+
+# --- RKE2 Playbook ---
path: validation/pipeline/scripts/build_qa_infra.sh
user: login: igomez06
body: I will add this to the backlog. 
created_at: 2025-05-22T14:22:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+RKE2_PLAYBOOK_PATH="ansible/rke2/rke2-playbook.yml"
+TERRAFORM_INVENTORY="ansible/rke2/terraform-inventory.yml"
+ANSIBLE_CONFIG="ansible/rke2/ansible.cfg"
+RANCHER_PLAYBOOK_PATH="ansible/rancher/rancher-playbook.yml"
+TFVARS_FILE="cluster.tfvars"
+KUBECONFIG_FILE="$REPO_ROOT/ansible/rke2/kubeconfig.yaml"
+VARS_FILE="./ansible/vars.yaml"
+PRIVATE_KEY_FILE="/root/.ssh/jenkins-elliptic-validation.pem"
+
+# --- Terraform Steps ---
+terraform -chdir="$TERRAFORM_DIR" init -input=false
+
+# Create and select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace new "$WORKSPACE_NAME" || terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+
+# Export the TF_WORKSPACE environment variable
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_CONFIG="$ANSIBLE_CONFIG"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_PRIVATE_KEY_FILE="$PRIVATE_KEY_FILE"
+
+# Apply the Terraform configuration
+terraform -chdir="$TERRAFORM_DIR" apply -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform apply failed."
+    exit 1
+fi
+
+# --- RKE2 Playbook ---
+
+# Run the RKE2 playbook with retries
+max_attempts=3
+attempt=0
+while [ $attempt -lt $max_attempts ]; do
+    attempt=$((attempt + 1))
+    echo "Attempt $attempt: Running RKE2 playbook..."
+    ansible-playbook -i "$TERRAFORM_INVENTORY" "$RKE2_PLAYBOOK_PATH" -vvvv -e "@$VARS_FILE"
+    rke2_exit_code=$?
+    if [ $rke2_exit_code -ne 0 ]; then
+        echo "RKE2 playbook failed on attempt $attempt."
+        if [ $attempt -lt $max_attempts ]; then
+            echo "Waiting 60 seconds before retrying..."
+            sleep 60
+        fi
+    else
+        echo "RKE2 playbook succeeded on attempt $attempt."
+        break
+    fi
+done
+
+if [ $attempt -eq $max_attempts ] && [ $rke2_exit_code -ne 0 ]; then
+    echo "Error: RKE2 playbook failed after $max_attempts attempts."
+    echo "destroy the terraform"
+    # Destroy the Terraform infrastructure
+    terraform -chdir="$TERRAFORM_DIR" destroy -auto-approve -var-file="$TFVARS_FILE"
+    if [ $? -ne 0 ]; then
+        echo "Error: Terraform destroy failed."
+        exit 1
+    fi
+    echo "Terraform infrastructure destroyed successfully!"
+    exit 1
+fi
+
+# Export KUBECONFIG
+export KUBECONFIG="$KUBECONFIG_FILE"
+echo $KUBECONFIG
+
+# --- Rancher Playbook ---
path: validation/pipeline/scripts/build_qa_infra.sh
user: login: igomez06
body: Yes! Retries are built in.
created_at: 2025-05-22T14:23:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
path: validation/pipeline/scripts/build_qa_infra.sh
user: login: caliskanugur
body: Just a nit for this one, we have Jenkins workspaces and terraform workspaces, so it may be easier to distinguish this in the future.

I think most of these variables can dynamically come from a shared library function so I don't think it's worth making the change right now
created_at: 2025-05-22T15:14:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+#!/bin/bash
+
+cd  /root/go/src/github.com/rancher/qa-infra-automation
+
+# Set variables
+REPO_ROOT=$(pwd)
+WORKSPACE_NAME="jenkins_workspace"
+TERRAFORM_DIR="terraform/aws/cluster_nodes"
+RKE2_PLAYBOOK_PATH="ansible/rke2/rke2-playbook.yml"
+TERRAFORM_INVENTORY="ansible/rke2/terraform-inventory.yml"
+ANSIBLE_CONFIG="ansible/rke2/ansible.cfg"
+RANCHER_PLAYBOOK_PATH="ansible/rancher/rancher-playbook.yml"
+TFVARS_FILE="cluster.tfvars"
+KUBECONFIG_FILE="$REPO_ROOT/ansible/rke2/kubeconfig.yaml"
+VARS_FILE="./ansible/vars.yaml"
+PRIVATE_KEY_FILE="/root/.ssh/jenkins-elliptic-validation.pem"
+
+# --- Terraform Steps ---
+terraform -chdir="$TERRAFORM_DIR" init -input=false
+
+# Create and select the Terraform workspace
+terraform -chdir="$TERRAFORM_DIR" workspace new "$WORKSPACE_NAME" || terraform -chdir="$TERRAFORM_DIR" workspace select "$WORKSPACE_NAME"
+
+# Export the TF_WORKSPACE environment variable
+export TF_WORKSPACE="$WORKSPACE_NAME"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_CONFIG="$ANSIBLE_CONFIG"
+
+# Export the ANSIBLE_CONFIG environment variable
+export ANSIBLE_PRIVATE_KEY_FILE="$PRIVATE_KEY_FILE"
+
+# Apply the Terraform configuration
+terraform -chdir="$TERRAFORM_DIR" apply -auto-approve -var-file="$TFVARS_FILE"
+if [ $? -ne 0 ]; then
+    echo "Error: Terraform apply failed."
+    exit 1
+fi
+
+# --- RKE2 Playbook ---
path: validation/pipeline/scripts/build_qa_infra.sh
user: login: caliskanugur
body: I agree, I think there are two different functionalities here. I do think the defaults at the very top can be removed, and be more dynamic with a few shared library functions - except retries.

Retries are only for rke2 installation since rancher playbook is more granular and it has scoped retries for each task on the playbook, right?

And the reason for rke2 having retries here is cause rke2 installation depends more on scripts from corral transition rather than the playbook?
created_at: 2025-05-22T15:21:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,31 @@
+#!/bin/bash
path: validation/pipeline/scripts/destroy_qa_infra.sh
user: login: caliskanugur
body: Same for this functionality as well, I believe this can be a shared library function in the future.
created_at: 2025-05-22T15:31:53Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-20T15:55:54Z
body: 

TestSuites above were modified. TestSuites below use modified code from this PR.

UpgradeTestSuite TestNewCommitFleetRepo
UpgradeTestSuite TestGitRepoForceUpdate
UpgradeTestSuite TestPauseFleetRepo
UpgradeWorkloadTestSuite TestWorkloadPreUpgrade
UpgradeWorkloadTestSuite TestWorkloadPostUpgrade
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-05-20T17:51:56Z
body: 

TestSuites above were modified. TestSuites below use modified code from this PR.

UpgradeTestSuite TestNewCommitFleetRepo
UpgradeTestSuite TestGitRepoForceUpdate
UpgradeTestSuite TestPauseFleetRepo
UpgradeWorkloadTestSuite TestWorkloadPreUpgrade
UpgradeWorkloadTestSuite TestWorkloadPostUpgrade
author_association: NONE