Repository: NVIDIA/spark-rapids-tools
Pull request #1673: Add support for custom target GPU cluster info in Profiling Tool
Author: login: parthosa
Labels: name: feature request
description: New feature or request
name: user_tools
description: Scope the wrapper module running CSP, QualX, and reports (python)
name: core_tools
description: Scope the core module (scala)
name: api_change
description: A change affecting the output (add/remove/rename files, add/remove/rename columns)
Created: 2025-05-14T22:43:32Z | Merged: 2025-05-19T22:58:56Z

Fixes #1664 

This PR introduces a new argument `--target-cluster-info` to the Profiling Tools in JAR, which accepts a YAML file describing the target GPU environment. Currently, it is supported only for CSPs.

This allows the profiling tool to generate cluster recommendations and corresponding AutoTuner configurations tied to the specified target instance type.

### Usage via Python CLI
1. Build and install the Python CLI in non-fat mode
   - This builds the Tools JAR and embeds it into the Python CLI wheel.

```shell
$ cd spark-rapids-tools/user_tools
$ ./build.sh non-fat
$ pip install dist/spark_rapids_user_tools-25.2.4-py3-none-any.whl
```

2. Create a target cluster YAML file
   - Refer to [this comment](https://github.com/NVIDIA/spark-rapids-tools/issues/1664#issuecomment-2881787628) for context and future direction.
```yaml
workerInfo:
  instanceType: g2-standard-48
``` 

4. Run the Profile CLI as:
```shell
spark_rapids profiling \
  --platform $PLATFORM \
  --eventlogs $EVENT_LOG \
  --target-cluster-info $PATH_TO_TARGET_YAML \
```
 
5. Output
   1. Cluster Information will be saved at `prof_2025xxx/rapids_4_spark_profile/application_174xxx/cluster_information.json`

      <details>
      <summary>
      cluster_information.json
      </summary>
      <pre>
      [ {
        "appName" : "mock-app-name",
        "appId" : "application_174xxx",
        "eventLogPath" : "/path/to/eventlog",
        "clusterInfo" : {
          "vendor" : "dataproc",
          "coresPerExecutor" : 7,
          "numExecsPerNode" : -1,
          "numExecutors" : 300,
          "numWorkerNodes" : 300,
          "executorHeapMemory" : 40960,
          "dynamicAllocationEnabled" : true,
          "dynamicAllocationMaxExecutors" : "10000",
          "dynamicAllocationMinExecutors" : "1",
          "dynamicAllocationInitialExecutors" : "2",
          "driverHost" : "mock.driver.host"
        },
        "recommendedClusterInfo" : {
          "vendor" : "dataproc",
          "coresPerExecutor" : 12,
          "numWorkerNodes" : 44,
          "numGpusPerNode" : 4,
          "numExecutors" : 175,
          "gpuDevice" : "l4",
          "dynamicAllocationEnabled" : true,
          "dynamicAllocationMaxExecutors" : "10000",
          "dynamicAllocationMinExecutors" : "1",
          "dynamicAllocationInitialExecutors" : "2",
          "workerNodeType" : "g2-standard-48"
        }
      } ]
      </pre>
      </details>

   1. Corresponding AutoTuner Recommendations will be saved at `prof_2025xxx/profiling_summary.log`

## Changes

- Added `--target-cluster-info` option in ProfileArgs to support target system specification.
- Introduced `TargetClusterProps` class to encapsulate cluster properties.
- Added `PropertiesLoader` utility to parse and load target cluster YAML files.
- **[Key Changes]** Updated logic in the Platform class to support:
    - Create recommended node instance:
        a. Using provided target properties (if available), else falling back to platform defaults (for CSPs).
        b. Or during AutoTuner, based on event log-derived information.
- Generating final recommended cluster configuration.

### Tests
- Added unit tests to validate various scenarios where users provide cluster information using the `--target-cluster-info ` YAML file

## Challenges and Future Scope
- There are multiple ways to currently specify cluster info to the Tools JAR:
    - Embedding GPU type in platform name (e.g., `dataproc-l4`, `emr-a100`)
    - Using the `--worker-info` file (currently overloaded — used sometimes for source info, sometimes for target)
- Removing legacy usage of `--worker-info` led to significant refactoring and test changes.
- To maintain backward compatibility, those options remain unchanged for now.
- In future, as we proceed with the new `--target-cluster-info` mechanism (see [comment](https://github.com/NVIDIA/spark-rapids-tools/issues/1664#issuecomment-2881787628)), we will deprecate/remove these legacy paths incrementally.
- Created follow up issue to benchmark tuning configurations for L4 https://github.com/NVIDIA/spark-rapids-tools/issues/1682


Linked issue #1664: [FEA] Allow users to specify custom GPU instance series in Profiling Tool
**Describe the bug**
Profiling Tool currently recommends spark configurations based on a fixed instance series (e.g., `n1-standard` for Dataproc, `g6` for EMR). However, a user may be running the GPU job on a different instance series. For example, users may be using `g2-standard` series on Dataproc.

**Expected behavior**
- Profiling Tool should allow users to specify a GPU instance series for CSPs.
   - Not all series will be supported initially

_Stretch Goal:_
- For OnPrem setups, users should be able to provide executor size in the following format:

```yaml
system:
  numCores:  /* per executor CPU core number = */
  memory:  /* per executor memory size = */
gpu:
  name: L20
```
Linked issue comments: user: login: amahussein
created_at: 2025-05-12T14:51:50Z
body: What is the plan to support this configuration? Piggy-back on the worker-info input provided to the AutoTuner?
author_association: COLLABORATOR
user: login: parthosa
created_at: 2025-05-12T17:35:09Z
body: > What is the plan to support this configuration? 

From internal discussion, the requirements are:

_Given GPU event logs, recommend an initial GPU cluster shape and corresponding Spark configurations for a specified instance type (e.g., g2-standard-48)_

Based on the requirement, the plan is to allow CSP platforms to provide a supported instance type. Users will provide a `--target_worker_info` input to the python CLI, which can be either:
- A string: the name of the instance type (for CSPs)
- A file path: containing worker info (only for onprem)

 The AutoTuner will then recommend:
1. Spark configurations (e.g., executor cores, memory, etc.)
1. Cluster shape for the given instance type

Can you share your thoughts on the above?

author_association: COLLABORATOR
user: login: amahussein
created_at: 2025-05-12T18:56:18Z
body: Thanks @parthosa 
  
> * A string: the name of the instance type (for CSPs)
> * A file path: containing worker info (only for onprem)

I am not sure a single string for CSP instance type is sufficient. This means that we will have a new argument for each new configuration. For example, if users want to configure the instance-type, then we can expect that other configurations to follow. For example, the users might also want to specify number of workers...etc. 
A configuration file for such configuration is more practical for teh following reasons:

- It works for all CSPs including onPrem
- The autotuner already supports reading an input file. So, the changes will be easier to apply.


**Regarding the workflow:**

The challenge is that we have CSP aware code in the core-tools. We previously removed any SDK that will pull machine type information. Then, we hardcoded them in the core-tools autotuner.

If we want to handle this feature the right way then we might consider the following:

_Option-1_
1. make all the CSP related configurations in user-tools. This means some few refactor between scala/python. Mainly, modify the step in the user-tools that creates the worker-info. And move hardcoded-values to conf/yaml file so that it can be easily overriden by users; especially if we are targetting users with custom QualX model and custom Spark.
2. the user-tools will pass the CSP related information to the core-tools AutoTuner, including the worker-info file.
3. the core-tools AutoTuner does the tuning based on the configs of the CSP it receives.

_Option-2_

Convert the AutoTuner code to the user-tools (poython wrapper). The pain point in that we have to figure an efficient way to pipe the AppInfo scala metadata to the Python wrapper. The CSV files would be difficult to get right and everytime we need a new information, then the output has to be modified to be consumed by the Python AutoTuner.
Eventually, Python's AutoTuner seems to be the right way on the long run.
author_association: COLLABORATOR
user: login: parthosa
created_at: 2025-05-14T23:04:56Z
body: Discussed offline with @amahussein and @viadea regarding how users can specify the target cluster/worker configuration. Currently, we do not support providing details about the target GPU cluster, worker, or executor very clearly.


To enable this, users can provide a `--target-cluster-info.yaml` file with the following structure:

```yaml
clusterInfo:
  numWorkers: 1

# Users can specify worker node size
workerInfo:
   # For CSP environments
  instanceType: g2-standard-48

   # For OnPrem
  system:                                      
    numCores: 16
    memory: 42g

  # GPU details (optional)
  gpu: 
    count: 1
    name: L20

# In some case(e.g. for k8s setups), executor sizing may be more useful than the worker node size
executorInfo:
  numCores: 8
  memory: 16g

# Known Spark properties to be respected by Bootstrap/AutoTuner
sparkProperties:
  spark.driver.maxResultSize: 2g
```

This format is designed to support both OnPrem and CSP environments. 

_However, as an initial step, we plan to support a minimal version of this only in the Profiler:_

```yaml
workerInfo:
  instanceType: g2-standard-48
```

 
author_association: COLLABORATOR

Diff:
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/ClusterConfigurationStrategy.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/ClusterConfigurationStrategy.scala
index b2950bf09..c7a44c88d 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/ClusterConfigurationStrategy.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/ClusterConfigurationStrategy.scala
@@ -17,7 +17,7 @@
 package com.nvidia.spark.rapids.tool
 
 import org.apache.spark.network.util.ByteUnit
-import org.apache.spark.sql.rapids.tool.ExistingClusterInfo
+import org.apache.spark.sql.rapids.tool.SourceClusterInfo
 import org.apache.spark.sql.rapids.tool.util.StringUtils
 
 /**
@@ -26,8 +26,9 @@ import org.apache.spark.sql.rapids.tool.util.StringUtils
 case class RecommendedClusterConfig(
     numExecutors: Int,
     coresPerExec: Int,
-    numGpusPerNode: Int,
-    memoryPerNodeMb: Long // For onprem or cases where a matching CSP instance type is unavailable
+    memoryPerNodeMb: Long, // For onprem or cases where a matching CSP instance type is unavailable
+    gpuDevice: GpuDevice,
+    numGpusPerNode: Int
 ) {
   def execsPerNode: Int = {
     numGpusPerNode
@@ -81,6 +82,10 @@ abstract class ClusterConfigurationStrategy(
 
   protected def getMemoryPerNodeMb: Long
 
+  protected def getNumGpus: Int
+
+  protected def getGpuDevice: GpuDevice
+
   /**
    * Generates the recommended cluster configuration based on the strategy.
    *
@@ -111,8 +116,9 @@ abstract class ClusterConfigurationStrategy(
       Some(RecommendedClusterConfig(
         numExecutors = recommendedNumExecutors,
         coresPerExec = recommendedCoresPerExec,
-        numGpusPerNode = platform.recommendedGpusPerNode,
-        memoryPerNodeMb = getMemoryPerNodeMb))
+        memoryPerNodeMb = getMemoryPerNodeMb,
+        gpuDevice = getGpuDevice,
+        numGpusPerNode = getNumGpus))
     }
   }
 }
@@ -157,6 +163,15 @@ class ClusterPropertyBasedStrategy(
   override protected def getMemoryPerNodeMb: Long = {
     StringUtils.convertToMB(clusterProperties.system.getMemory, Some(ByteUnit.BYTE))
   }
+
+  override def getGpuDevice: GpuDevice = {
+    GpuDevice.createInstance(clusterProperties.getGpu.name)
+      .getOrElse(platform.defaultGpuDevice)
+  }
+
+  override def getNumGpus: Int = {
+    numGpusFromProps
+  }
 }
 
 /**
@@ -167,7 +182,7 @@ class EventLogBasedStrategy(
     sparkProperties: Map[String, String]
   ) extends ClusterConfigurationStrategy(platform, sparkProperties) {
 
-  private val clusterInfoFromEventLog: ExistingClusterInfo = {
+  private val clusterInfoFromEventLog: SourceClusterInfo = {
     platform.clusterInfoFromEventLog.getOrElse(
       throw new IllegalArgumentException("Cluster information from event log must be defined"))
   }
@@ -187,6 +202,16 @@ class EventLogBasedStrategy(
   override def calculateInitialCoresPerExec: Int = {
     clusterInfoFromEventLog.coresPerExecutor
   }
+
+  // TODO: Add information about the existing GPU on the node
+  override def getGpuDevice: GpuDevice = {
+    platform.getGpuOrDefault
+  }
+
+  // TODO: Add information about the existing GPU count on the node
+  override def getNumGpus: Int = {
+    platform.defaultNumGpus
+  }
 }
 
 /**
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
index 33c7b2d8e..f1ddb2b92 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
@@ -18,11 +18,11 @@ package com.nvidia.spark.rapids.tool
 import scala.annotation.tailrec
 
 import com.nvidia.spark.rapids.tool.planparser.DatabricksParseHelper
-import com.nvidia.spark.rapids.tool.tuning.{ClusterProperties, ProfilingAutoTunerConfigsProvider, SparkMaster}
+import com.nvidia.spark.rapids.tool.tuning.{ClusterProperties, SparkMaster, TargetClusterProps}
 
 import org.apache.spark.internal.Logging
 import org.apache.spark.network.util.ByteUnit
-import org.apache.spark.sql.rapids.tool.{ExistingClusterInfo, RecommendedClusterInfo}
+import org.apache.spark.sql.rapids.tool.{MatchingInstanceTypeNotFoundException, RecommendedClusterInfo, SourceClusterInfo}
 import org.apache.spark.sql.rapids.tool.util.{SparkRuntime, StringUtils}
 
 /**
@@ -54,9 +54,24 @@ object PlatformNames {
 
 case class DynamicAllocationInfo(enabled: Boolean, max: String, min: String, initial: String)
 
+/**
+ * Case class representing the key for the node instance map.
+ * @param instanceType Name of the instance type
+ * @param gpuCount Count of GPUs on the instance. This is required for certain
+ *                 cases where a given instance type can have multiple GPU counts
+ *                 (e.g. n1-standard-16 with 1, 2, or 4 GPUs).
+ */
+case class NodeInstanceMapKey(instanceType: String, gpuCount: Option[Int] = None) {
+  override def toString: String = gpuCount match {
+    case Some(count) => s"NodeInstanceMapKey(instanceType='$instanceType', gpuCount=$count)"
+    case None => s"NodeInstanceMapKey(instanceType='$instanceType')"
+  }
+}
+
 // resource information and name of the CSP instance types, or for onprem its
 // the executor information since we can't recommend node types
-case class InstanceInfo(cores: Int, memoryMB: Long, name: String, numGpus: Int) {
+case class InstanceInfo(cores: Int, memoryMB: Long, name: String,
+      numGpus: Int, gpuDevice: GpuDevice) {
   /**
    * Get the memory per executor based on the instance type.
    * Note: For GPU instances, num of executors is same as the number of GPUs.
@@ -67,73 +82,120 @@ case class InstanceInfo(cores: Int, memoryMB: Long, name: String, numGpus: Int)
 }
 
 object InstanceInfo {
-    def createDefaultInstance(cores: Int, memoryMB: Long, numGpus: Int): InstanceInfo = {
-      InstanceInfo(cores, memoryMB, "N/A", numGpus)
-    }
+  def createDefaultInstance(cores: Int, memoryMB: Long,
+      numGpus: Int, gpuDevice: GpuDevice): InstanceInfo = {
+    InstanceInfo(cores, memoryMB, "N/A", numGpus, gpuDevice)
+  }
 }
 
-// This is meant to be temporary mapping to figure out instance type based
-// on the number of GPUs and cores.  Eventually this should be read from files
-// generated based on CSP instance information.
-// format (numGpus, numCores) -> InstanceInfo about that CSP node instance type
+// This is a map of the instance types to the InstanceInfo.
+// format (instance type name, gpu count) -> InstanceInfo about that CSP node instance type
 object PlatformInstanceTypes {
 
   // Using G6 instances for EMR
-  val EMR_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1),
-    (1, 8) -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1),
-    (1, 16) -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1),
-    (1, 32) -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1),
-    (4, 48) -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 1),
-    (1, 64) -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1)
+  val EMR_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("g6.xlarge") -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.2xlarge") -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.4xlarge") -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.8xlarge") -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.12xlarge") -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 4, L4Gpu),
+    NodeInstanceMapKey("g6.16xlarge") -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1, L4Gpu)
   )
 
   // Using G5 instances. To be updated once G6 availability on Databricks
   // is consistent
-  val DATABRICKS_AWS_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1),
-    (1, 8) -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1),
-    (1, 16) -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1),
-    (1, 32) -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1),
-    (4, 48) -> InstanceInfo(48, 192 * 1024, "g5.12xlarge", 1),
-    (1, 64) -> InstanceInfo(64, 256 * 1024, "g5.16xlarge", 1)
+  val DATABRICKS_AWS_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("g5.xlarge") -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.2xlarge") -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.4xlarge") -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.8xlarge") -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.12xlarge") -> InstanceInfo(48, 192 * 1024, "g5.12xlarge", 4, A10GGpu),
+    NodeInstanceMapKey("g5.16xlarge") -> InstanceInfo(64, 256 * 1024, "g5.16xlarge", 1, A10GGpu)
   )
 
   // Standard_NC4as_T4_v3 - only recommending nodes with T4's for now, add more later
-  val AZURE_NCAS_T4_V3_BY_GPUS_CORES = Map(
-    (1, 4) -> InstanceInfo(4, 28 * 1024, "Standard_NC4as_T4_v3", 1), // 1 GPU
-    (1, 8) -> InstanceInfo(8, 56 * 1024, "Standard_NC8as_T4_v3", 1), // 1 GPU
-    (1, 16) -> InstanceInfo(16, 110 * 1024, "Standard_NC16as_T4_v3", 1), // 1 GPU
-    (4, 64) -> InstanceInfo(64, 440 * 1024, "Standard_NC64as_T4_v3", 4)  // 4 GPUs
+  val AZURE_NCAS_T4_V3_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("Standard_NC4as_T4_v3") ->
+      InstanceInfo(4, 28 * 1024, "Standard_NC4as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC8as_T4_v3") ->
+      InstanceInfo(8, 56 * 1024, "Standard_NC8as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC16as_T4_v3") ->
+      InstanceInfo(16, 110 * 1024, "Standard_NC16as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC64as_T4_v3") ->
+      InstanceInfo(64, 440 * 1024, "Standard_NC64as_T4_v3", 4, T4Gpu)  // 4 GPUs
   )
 
   // dataproc and dataproc-gke
   // Google supports 1, 2, or 4 Gpus of most types
   // added to n1-standard boxes. You may be able to add 8 v100's but we
   // are going to ignore that.
-  val DATAPROC_BY_GPUS_CORES = Map(
-    (1, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 1),
-    (1, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 1),
-    (1, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 1),
-    (1, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 1),
-    (1, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 1),
-    (1, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 1),
-    (1, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 1),
-    (1, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 1),
-    (2, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 2),
-    (2, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 2),
-    (2, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 2),
-    (2, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 2),
-    (2, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 2),
-    (2, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 2),
-    (2, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 2),
-    (2, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 2),
-    (4, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 4),
-    (4, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 4),
-    (4, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 4),
-    (4, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 4),
-    (4, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 4),
-    (4, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 4),
-    (4, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 4),
-    (4, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 4)
+  val DATAPROC_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    // g2-standard instances with Intel GPUs
+    NodeInstanceMapKey("g2-standard-4") ->
+      InstanceInfo(4, 16 * 1024, "g2-standard-4", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-8") ->
+      InstanceInfo(8, 32 * 1024, "g2-standard-8", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-12") ->
+      InstanceInfo(12, 48 * 1024, "g2-standard-12", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-16") ->
+      InstanceInfo(16, 64 * 1024, "g2-standard-16", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-24") ->
+      InstanceInfo(24, 96 * 1024, "g2-standard-24", 2, L4Gpu),
+    NodeInstanceMapKey("g2-standard-32") ->
+      InstanceInfo(32, 128 * 1024, "g2-standard-32", 2, L4Gpu),
+    NodeInstanceMapKey("g2-standard-48") ->
+      InstanceInfo(48, 192 * 1024, "g2-standard-48", 4, L4Gpu),
+    NodeInstanceMapKey("g2-standard-96") ->
+      InstanceInfo(96, 384 * 1024, "g2-standard-96", 8, L4Gpu),
+    // Existing n1-standard instances
+    NodeInstanceMapKey(instanceType = "n1-standard-1", gpuCount = Some(1)) ->
+      InstanceInfo(1, 1 * 3840, "n1-standard-1", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-2", gpuCount = Some(1)) ->
+      InstanceInfo(2, 2 * 3840, "n1-standard-2", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-4", gpuCount = Some(1)) ->
+      InstanceInfo(4, 4 * 3840, "n1-standard-4", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-8", gpuCount = Some(1)) ->
+      InstanceInfo(8, 8 * 3840, "n1-standard-8", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-16", gpuCount = Some(1)) ->
+      InstanceInfo(16, 16 * 3840, "n1-standard-16", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-32", gpuCount = Some(1)) ->
+      InstanceInfo(32, 32 * 3840, "n1-standard-32", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-64", gpuCount = Some(1)) ->
+      InstanceInfo(64, 64 * 3840, "n1-standard-64", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-96", gpuCount = Some(1)) ->
+      InstanceInfo(96, 96 * 3840, "n1-standard-96", 1, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-1", gpuCount = Some(2)) ->
+      InstanceInfo(1, 1 * 3840, "n1-standard-1", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-2", gpuCount = Some(2)) ->
+      InstanceInfo(2, 2 * 3840, "n1-standard-2", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-4", gpuCount = Some(2)) ->
+      InstanceInfo(4, 4 * 3840, "n1-standard-4", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-8", gpuCount = Some(2)) ->
+      InstanceInfo(8, 8 * 3840, "n1-standard-8", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-16", gpuCount = Some(2)) ->
+      InstanceInfo(16, 16 * 3840, "n1-standard-16", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-32", gpuCount = Some(2)) ->
+      InstanceInfo(32, 32 * 3840, "n1-standard-32", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-64", gpuCount = Some(2)) ->
+      InstanceInfo(64, 64 * 3840, "n1-standard-64", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-96", gpuCount = Some(2)) ->
+      InstanceInfo(96, 96 * 3840, "n1-standard-96", 2, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-1", gpuCount = Some(4)) ->
+      InstanceInfo(1, 1 * 3840, "n1-standard-1", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-2", gpuCount = Some(4)) ->
+      InstanceInfo(2, 2 * 3840, "n1-standard-2", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-4", gpuCount = Some(4)) ->
+      InstanceInfo(4, 4 * 3840, "n1-standard-4", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-8", gpuCount = Some(4)) ->
+      InstanceInfo(8, 8 * 3840, "n1-standard-8", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-16", gpuCount = Some(4)) ->
+      InstanceInfo(16, 16 * 3840, "n1-standard-16", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-32", gpuCount = Some(4)) ->
+      InstanceInfo(32, 32 * 3840, "n1-standard-32", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-64", gpuCount = Some(4)) ->
+      InstanceInfo(64, 64 * 3840, "n1-standard-64", 4, T4Gpu),
+    NodeInstanceMapKey(instanceType = "n1-standard-96", gpuCount = Some(4)) ->
+      InstanceInfo(96, 96 * 3840, "n1-standard-96", 4, T4Gpu)
   )
 }
 
@@ -142,27 +204,58 @@ object PlatformInstanceTypes {
  *
  * @param gpuDevice Gpu Device present in the platform
  * @param clusterProperties Cluster Properties passed into the tool as worker info
+ * @param targetCluster Target cluster information (e.g. instance type, GPU type)
  */
 abstract class Platform(var gpuDevice: Option[GpuDevice],
-    val clusterProperties: Option[ClusterProperties]) extends Logging {
+    val clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps]) extends Logging {
   val platformName: String
-  val defaultGpuDevice: GpuDevice
+  def defaultRecommendedNodeInstanceMapKey: Option[NodeInstanceMapKey] = None
+  def defaultRecommendedCoresPerExec: Int = 16
+  def defaultGpuDevice: GpuDevice = T4Gpu
+  def defaultNumGpus: Int = 1
+
   val sparkVersionLabel: String = "Spark version"
 
   // It's not deal to use vars here but to minimize changes and
   // keep backwards compatibility we put them here for now and hopefully
   // in future we can refactor.
-  var clusterInfoFromEventLog: Option[ExistingClusterInfo] = None
+  var clusterInfoFromEventLog: Option[SourceClusterInfo] = None
   // instance information for the gpu node type we will use to run with
-  var recommendedNodeInstanceInfo: Option[InstanceInfo] = None
+  var recommendedNodeInstanceInfo: Option[InstanceInfo] = {
+    targetCluster.map(_.getWorkerInfo.getNodeInstanceMapKey) match {
+      case Some(nodeInstanceMapKey) =>
+        getInstanceMapByName.get(nodeInstanceMapKey).orElse {
+          // scalastyle:off line.size.limit
+          throw new MatchingInstanceTypeNotFoundException(
+            s"""
+               |Could not find matching instance type in resources map.
+               |Requested: $nodeInstanceMapKey
+               |Supported: ${getInstanceMapByName.keys.mkString(", ")}
+               |
+               |Next Steps:
+               |Update the target cluster YAML with a valid instance type and GPU count, or skip it to use default: ${defaultRecommendedNodeInstanceMapKey.getOrElse("None")}
+               |""".stripMargin.trim
+          )
+          // scalastyle:on line.size.limit
+        }
+      case None =>
+        defaultRecommendedNodeInstanceMapKey.flatMap(getInstanceMapByName.get)
+    }
+  }
+
   // overall final recommended cluster configuration
   var recommendedClusterInfo: Option[RecommendedClusterInfo] = None
 
   // Default recommendation based on NDS benchmarks (note: this could be platform specific)
-  def recommendedCoresPerExec: Int = ProfilingAutoTunerConfigsProvider.DEF_CORES_PER_EXECUTOR
-  // Default number of GPUs to use, currently we do not support multiple GPUs per node
-  def recommendedGpusPerNode = 1
-  def defaultNumGpus: Int = 1
+  final def recommendedCoresPerExec: Int = {
+    recommendedNodeInstanceInfo.map(instInfo => instInfo.cores / instInfo.numGpus)
+      .getOrElse(defaultRecommendedCoresPerExec)
+  }
+
+  final def recommendedGpuDevice: GpuDevice = {
+    recommendedNodeInstanceInfo.map(_.gpuDevice).getOrElse(defaultGpuDevice)
+  }
 
   // Default runtime for the platform
   val defaultRuntime: SparkRuntime.SparkRuntime = SparkRuntime.SPARK
@@ -174,6 +267,7 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
   // scalastyle:off line.size.limit
   // Supported Spark version to RapidsShuffleManager version mapping.
   // Reference: https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/rapids-shuffle.html#rapids-shuffle-manager
+  // TODO: Issue to automate this https://github.com/NVIDIA/spark-rapids-tools/issues/1676
   // scalastyle:on line.size.limit
   val supportedShuffleManagerVersionMap: Array[(String, String)] = Array(
     "3.2.0" -> "320",
@@ -191,7 +285,12 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
     "3.4.2" -> "342",
     "3.4.3" -> "343",
     "3.5.0" -> "350",
-    "3.5.1" -> "351"
+    "3.5.1" -> "351",
+    "3.5.2" -> "352",
+    "3.5.3" -> "353",
+    "3.5.4" -> "354",
+    "3.5.5" -> "355",
+    "4.0.0" -> "400"
   )
 
   /**
@@ -349,11 +448,11 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
       numExecs: Int,
       numWorkerNodes: Int,
       sparkProperties: Map[String, String],
-      systemProperties: Map[String, String]): ExistingClusterInfo = {
+      systemProperties: Map[String, String]): SourceClusterInfo = {
     val driverHost = sparkProperties.get("spark.driver.host")
     val executorHeapMem = getExecutorHeapMemoryMB(sparkProperties)
     val dynamicAllocSettings = Platform.getDynamicAllocationSettings(sparkProperties)
-    ExistingClusterInfo(platformName, coresPerExecutor, numExecsPerNode, numExecs, numWorkerNodes,
+    SourceClusterInfo(platformName, coresPerExecutor, numExecsPerNode, numExecs, numWorkerNodes,
       executorHeapMem, dynamicAllocSettings.enabled, dynamicAllocSettings.max,
       dynamicAllocSettings.min, dynamicAllocSettings.initial, driverHost = driverHost)
   }
@@ -370,11 +469,6 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
       numExecs, numExecutorNodes, sparkProperties, systemProperties))
   }
 
-  override def toString: String = {
-    val gpuStr = gpuDevice.fold("")(gpu => s"-$gpu")
-    s"$platformName$gpuStr"
-  }
-
   /**
    * Indicate if the platform is a cloud service provider.
    */
@@ -391,51 +485,9 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
   def maxGpusSupported: Int = 1
 
   /**
-   * Get the mapping of number of GPUs and cores to the instance type.
-   * Format (numGpus, numCores) -> InstanceInfo about the matching CSP node instance type.
+   * Get the mapping of instance names to their corresponding instance information.
    */
-  def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = Map.empty
-
-  /**
-   * Get the instance type based on the core and gpu requirements.
-   * Case A: Find the exact instance type based on the required resources.
-   * Case B: If the exact instance type is not found, find the instance type with the smallest
-   *         combined GPU and core count that meets the requirements.
-   * Case C: For onprem or cases where a matching CSP instance type is unavailable.
-   */
-  private def getInstanceByResources(
-      recommendedClusterConfig: RecommendedClusterConfig): InstanceInfo = {
-    val requiredGpus = recommendedClusterConfig.numGpusPerNode
-    val requiredCores = recommendedClusterConfig.coresPerNode
-
-    // A. Find the exact instance type based on the required resources
-    getInstanceByResourcesMap.get((requiredGpus, requiredCores)).orElse {
-      // B. If the exact instance type is not found, find the instance type with the smallest
-      //    combined GPU and core count that meets the requirements
-      val suitableInstances = getInstanceByResourcesMap.filterKeys { case (gpus, cores) =>
-        gpus >= requiredGpus && cores >= requiredCores
-      }
-      if (suitableInstances.isEmpty) {
-        None
-      } else {
-        val optimalNumGpusAndCoresPair = suitableInstances.keys.minBy { case (gpus, cores) =>
-          gpus + cores
-        }
-        suitableInstances.get(optimalNumGpusAndCoresPair)
-      }
-    }.getOrElse {
-      // C. For onprem or cases where a matching CSP instance type is unavailable
-      logDebug(s"Could not find a matching instance with requiredGpus=$requiredGpus," +
-        s" requiredCores=$requiredCores. Falling back to create a default instance info: \n" +
-        s"coresPerExec=${recommendedClusterConfig.coresPerExec}, " +
-        s"memoryPerNodeMb=${recommendedClusterConfig.memoryPerNodeMb}, " +
-        s"numGpusPerNode=${recommendedClusterConfig.numGpusPerNode}")
-      InstanceInfo.createDefaultInstance(
-        cores = recommendedClusterConfig.coresPerExec,
-        memoryMB = recommendedClusterConfig.memoryPerNodeMb,
-        numGpus = recommendedClusterConfig.numGpusPerNode)
-    }
-  }
+  def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = Map.empty
 
   /**
    * Creates the recommended GPU cluster info based on Spark properties and
@@ -460,7 +512,15 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
         // cores per executor, num executors per node).
         strategy.getRecommendedConfig match {
           case Some(clusterConfig) =>
-            val recommendedNodeInstance = getInstanceByResources(clusterConfig)
+            // If recommended node instance information is not already set, create it based on the
+            // existing cluster configuration and platform defaults.
+            val recommendedNodeInstance = recommendedNodeInstanceInfo.getOrElse {
+              InstanceInfo.createDefaultInstance(
+                cores = clusterConfig.coresPerNode,
+                memoryMB = clusterConfig.memoryPerNodeMb,
+                numGpus = clusterConfig.numGpusPerNode,
+                gpuDevice = clusterConfig.gpuDevice)
+            }
             val vendor = clusterInfoFromEventLog.map(_.vendor).getOrElse("")
             val numWorkerNodes = if (!isPlatformCSP) {
               // For on-prem, we do not have the concept of worker nodes.
@@ -479,7 +539,7 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
               numWorkerNodes = numWorkerNodes,
               numGpusPerNode = recommendedNodeInstance.numGpus,
               numExecutors = clusterConfig.numExecutors,
-              gpuDevice = getGpuOrDefault.toString,
+              gpuDevice = recommendedNodeInstance.gpuDevice.toString,
               dynamicAllocationEnabled = dynamicAllocSettings.enabled,
               dynamicAllocationMaxExecutors = dynamicAllocSettings.max,
               dynamicAllocationMinExecutors = dynamicAllocSettings.min,
@@ -503,8 +563,9 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
 }
 
 abstract class DatabricksPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties]) extends Platform(gpuDevice, clusterProperties) {
-  override val defaultGpuDevice: GpuDevice = T4Gpu
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends Platform(gpuDevice, clusterProperties, targetCluster) {
   override val sparkVersionLabel: String = "Databricks runtime"
   override def isPlatformCSP: Boolean = true
 
@@ -523,8 +584,8 @@ abstract class DatabricksPlatform(gpuDevice: Option[GpuDevice],
   )
 
   // Supported Databricks version to RapidsShuffleManager version mapping.
+  // TODO: Issue to automate this https://github.com/NVIDIA/spark-rapids-tools/issues/1676
   override val supportedShuffleManagerVersionMap: Array[(String, String)] = Array(
-    "11.3" -> "330db",
     "12.2" -> "332db",
     "13.3" -> "341db"
   )
@@ -534,7 +595,7 @@ abstract class DatabricksPlatform(gpuDevice: Option[GpuDevice],
       numExecs: Int,
       numWorkerNodes: Int,
       sparkProperties: Map[String, String],
-      systemProperties: Map[String, String]): ExistingClusterInfo = {
+      systemProperties: Map[String, String]): SourceClusterInfo = {
     val workerNodeType = sparkProperties.get(DatabricksParseHelper.PROP_WORKER_TYPE_ID_KEY)
     val driverNodeType = sparkProperties.get(DatabricksParseHelper.PROP_DRIVER_TYPE_ID_KEY)
     val clusterId = sparkProperties.get(DatabricksParseHelper.PROP_TAG_CLUSTER_ID_KEY)
@@ -542,7 +603,7 @@ abstract class DatabricksPlatform(gpuDevice: Option[GpuDevice],
     val clusterName = sparkProperties.get(DatabricksParseHelper.PROP_TAG_CLUSTER_NAME_KEY)
     val executorHeapMem = getExecutorHeapMemoryMB(sparkProperties)
     val dynamicAllocSettings = Platform.getDynamicAllocationSettings(sparkProperties)
-    ExistingClusterInfo(platformName, coresPerExecutor, numExecsPerNode, numExecs, numWorkerNodes,
+    SourceClusterInfo(platformName, coresPerExecutor, numExecsPerNode, numExecs, numWorkerNodes,
       executorHeapMem, dynamicAllocSettings.enabled, dynamicAllocSettings.max,
       dynamicAllocSettings.min, dynamicAllocSettings.initial, driverNodeType,
       workerNodeType, driverHost, clusterId, clusterName)
@@ -550,33 +611,45 @@ abstract class DatabricksPlatform(gpuDevice: Option[GpuDevice],
 }
 
 class DatabricksAwsPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties])
-  extends DatabricksPlatform(gpuDevice, clusterProperties)
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends DatabricksPlatform(gpuDevice, clusterProperties, targetCluster)
   with Logging {
   override val platformName: String = PlatformNames.DATABRICKS_AWS
-  override val defaultGpuDevice: GpuDevice = A10GGpu
+  override def defaultRecommendedNodeInstanceMapKey: Option[NodeInstanceMapKey] = {
+    Some(NodeInstanceMapKey(instanceType = "g5.8xlarge"))
+  }
 
-  override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
-    PlatformInstanceTypes.DATABRICKS_AWS_BY_GPUS_CORES
+  override def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = {
+    PlatformInstanceTypes.DATABRICKS_AWS_BY_INSTANCE_NAME
   }
 }
 
 class DatabricksAzurePlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties])
-  extends DatabricksPlatform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends DatabricksPlatform(gpuDevice, clusterProperties, targetCluster) {
   override val platformName: String = PlatformNames.DATABRICKS_AZURE
+  override def defaultRecommendedNodeInstanceMapKey: Option[NodeInstanceMapKey] = {
+    Some(NodeInstanceMapKey(instanceType = "Standard_NC8as_T4_v3"))
+  }
 
   override def maxGpusSupported: Int = 4
 
-  override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
-    PlatformInstanceTypes.AZURE_NCAS_T4_V3_BY_GPUS_CORES
+  override def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = {
+    PlatformInstanceTypes.AZURE_NCAS_T4_V3_BY_INSTANCE_NAME
   }
 }
 
 class DataprocPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties]) extends Platform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends Platform(gpuDevice, clusterProperties, targetCluster) {
   override val platformName: String = PlatformNames.DATAPROC
-  override val defaultGpuDevice: GpuDevice = T4Gpu
+  override def defaultRecommendedNodeInstanceMapKey: Option[NodeInstanceMapKey] = {
+    Some(NodeInstanceMapKey(instanceType = "g2-standard-16"))
+  }
+
   override val recommendationsToInclude: Seq[(String, String)] = Seq(
     // Keep disabled. This property does not work well with GPU clusters.
     "spark.dataproc.enhanced.optimizer.enabled" -> "false",
@@ -587,30 +660,38 @@ class DataprocPlatform(gpuDevice: Option[GpuDevice],
   override def isPlatformCSP: Boolean = true
   override def maxGpusSupported: Int = 4
 
-  override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
-    PlatformInstanceTypes.DATAPROC_BY_GPUS_CORES
+  override def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = {
+    PlatformInstanceTypes.DATAPROC_BY_INSTANCE_NAME
   }
 }
 
 class DataprocServerlessPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties])
-  extends DataprocPlatform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends DataprocPlatform(gpuDevice, clusterProperties, targetCluster) {
+
   override val platformName: String = PlatformNames.DATAPROC_SL
-  override val defaultGpuDevice: GpuDevice = L4Gpu
+  override def defaultGpuDevice: GpuDevice = L4Gpu
   override def isPlatformCSP: Boolean = true
 }
 
 class DataprocGkePlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties])
-  extends DataprocPlatform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends DataprocPlatform(gpuDevice, clusterProperties, targetCluster) {
+
   override val platformName: String = PlatformNames.DATAPROC_GKE
   override def isPlatformCSP: Boolean = true
 }
 
 class EmrPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties]) extends Platform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends Platform(gpuDevice, clusterProperties, targetCluster) {
   override val platformName: String = PlatformNames.EMR
-  override val defaultGpuDevice: GpuDevice = A10GGpu
+  override def defaultRecommendedNodeInstanceMapKey: Option[NodeInstanceMapKey] = {
+    Some(NodeInstanceMapKey(instanceType = "g6.4xlarge"))
+  }
 
   override def isPlatformCSP: Boolean = true
   override def requirePathRecommendations: Boolean = false
@@ -622,28 +703,33 @@ class EmrPlatform(gpuDevice: Option[GpuDevice],
       numExecs: Int,
       numWorkerNodes: Int,
       sparkProperties: Map[String, String],
-      systemProperties: Map[String, String]): ExistingClusterInfo = {
+      systemProperties: Map[String, String]): SourceClusterInfo = {
     val clusterId = systemProperties.get("EMR_CLUSTER_ID")
     val driverHost = sparkProperties.get("spark.driver.host")
     val executorHeapMem = getExecutorHeapMemoryMB(sparkProperties)
     val dynamicAllocSettings = Platform.getDynamicAllocationSettings(sparkProperties)
-    ExistingClusterInfo(platformName, coresPerExecutor, numExecsPerNode, numExecs,
+    SourceClusterInfo(platformName, coresPerExecutor, numExecsPerNode, numExecs,
       numWorkerNodes, executorHeapMem, dynamicAllocSettings.enabled, dynamicAllocSettings.max,
       dynamicAllocSettings.min, dynamicAllocSettings.initial, clusterId = clusterId,
       driverHost = driverHost)
   }
 
-  override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
-    PlatformInstanceTypes.EMR_BY_GPUS_CORES
+  override def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = {
+    PlatformInstanceTypes.EMR_BY_INSTANCE_NAME
   }
 }
 
 class OnPremPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties]) extends Platform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends Platform(gpuDevice, clusterProperties, targetCluster) {
+  // TODO: Add support for providing target cluster properties for OnPrem
+  require(targetCluster.isEmpty,
+    "OnPrem platform does not support target cluster properties yet. " +
+      "Use `--worker-info` option instead.")
+
   override val platformName: String = PlatformNames.ONPREM
-  // Note we don't have an speedup factor file for onprem l4's but we want auto tuner
-  // to use L4.
-  override val defaultGpuDevice: GpuDevice = L4Gpu
+  override def defaultGpuDevice: GpuDevice = L4Gpu
   override val defaultGpuForSpeedupFactor: GpuDevice = A100Gpu
   // on prem is hard since we don't know what node configurations they have
   // assume 1 for now. We should have them pass this information in the future.
@@ -705,17 +791,25 @@ object PlatformFactory extends Logging {
   @tailrec
   private def createPlatformInstance(platformName: String,
       gpuDevice: Option[GpuDevice],
-      clusterProperties: Option[ClusterProperties]): Platform = platformName match {
-    case PlatformNames.DATABRICKS_AWS => new DatabricksAwsPlatform(gpuDevice, clusterProperties)
-    case PlatformNames.DATABRICKS_AZURE => new DatabricksAzurePlatform(gpuDevice, clusterProperties)
-    case PlatformNames.DATAPROC => new DataprocPlatform(gpuDevice, clusterProperties)
-    case PlatformNames.DATAPROC_GKE => new DataprocGkePlatform(gpuDevice, clusterProperties)
-    case PlatformNames.DATAPROC_SL => new DataprocServerlessPlatform(gpuDevice, clusterProperties)
-    case PlatformNames.EMR => new EmrPlatform(gpuDevice, clusterProperties)
-    case PlatformNames.ONPREM => new OnPremPlatform(gpuDevice, clusterProperties)
+      clusterProperties: Option[ClusterProperties],
+      targetCluster: Option[TargetClusterProps]): Platform = platformName match {
+    case PlatformNames.DATABRICKS_AWS =>
+      new DatabricksAwsPlatform(gpuDevice, clusterProperties, targetCluster)
+    case PlatformNames.DATABRICKS_AZURE =>
+      new DatabricksAzurePlatform(gpuDevice, clusterProperties, targetCluster)
+    case PlatformNames.DATAPROC =>
+      new DataprocPlatform(gpuDevice, clusterProperties, targetCluster)
+    case PlatformNames.DATAPROC_GKE =>
+      new DataprocGkePlatform(gpuDevice, clusterProperties, targetCluster)
+    case PlatformNames.DATAPROC_SL =>
+      new DataprocServerlessPlatform(gpuDevice, clusterProperties, targetCluster)
+    case PlatformNames.EMR =>
+      new EmrPlatform(gpuDevice, clusterProperties, targetCluster)
+    case PlatformNames.ONPREM =>
+      new OnPremPlatform(gpuDevice, clusterProperties, targetCluster)
     case p if p.isEmpty =>
       logInfo(s"Platform is not specified. Using ${PlatformNames.DEFAULT} as default.")
-      createPlatformInstance(PlatformNames.DEFAULT, gpuDevice, clusterProperties)
+      createPlatformInstance(PlatformNames.DEFAULT, gpuDevice, clusterProperties, targetCluster)
     case _ =>
       throw new IllegalArgumentException(s"Unsupported platform: $platformName. " +
         s"Options include ${PlatformNames.getAllNames.mkString(", ")}.")
@@ -726,17 +820,25 @@ object PlatformFactory extends Logging {
    *
    * @param platformKey The key identifying the platform. Defaults to `PlatformNames.DEFAULT`.
    * @param clusterProperties Optional cluster properties if the user specified them.
+   * @param targetClusterProps Optional target cluster properties if the user specified them.
    */
   def createInstance(platformKey: String = PlatformNames.DEFAULT,
-      clusterProperties: Option[ClusterProperties] = None): Platform = {
+      clusterProperties: Option[ClusterProperties] = None,
+      targetClusterProps: Option[TargetClusterProps] = None): Platform = {
+    // TODO: Remove platformKey containing GPU name
     val (platformName, gpuName) = extractPlatformGpuName(platformKey)
+    if (gpuName.nonEmpty) {
+      logWarning("Using GPU name in platform key will be deprecated in future. " +
+        "Please use the '--target-worker-info' option to specify the GPU type.")
+    }
     val gpuDevice = gpuName.flatMap(GpuDevice.createInstance)
     // case when gpu name is detected but not in device map
     if (gpuName.isDefined && gpuDevice.isEmpty) {
       throw new IllegalArgumentException(s"Unsupported GPU device: ${gpuName.get}. " +
           s"Supported GPU devices are: ${GpuDevice.deviceMap.keys.mkString(", ")}.")
     }
-    val platform = createPlatformInstance(platformName, gpuDevice, clusterProperties)
+    val platform = createPlatformInstance(platformName, gpuDevice,
+      clusterProperties, targetClusterProps)
     logInfo(s"Using platform: $platform")
     platform
   }
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileArgs.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileArgs.scala
index d3651f709..5a3cfe0db 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileArgs.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileArgs.scala
@@ -16,7 +16,6 @@
 package com.nvidia.spark.rapids.tool.profiling
 
 import com.nvidia.spark.rapids.tool.PlatformNames
-import com.nvidia.spark.rapids.tool.tuning.ProfilingAutoTunerConfigsProvider
 import org.rogach.scallop.{ScallopConf, ScallopOption}
 import org.rogach.scallop.exceptions.ScallopException
 
@@ -109,9 +108,11 @@ Usage: java -cp rapids-4-spark-tools_2.12-<version>.jar:$SPARK_HOME/jars/*
   val workerInfo: ScallopOption[String] =
     opt[String](required = false,
       descr = "File path containing the system information of a worker node. It is assumed " +
-        "that all workers are homogenous. It requires the AutoTuner to be enabled. Default is " +
-        "./worker_info.yaml",
-      default = Some(ProfilingAutoTunerConfigsProvider.DEFAULT_WORKER_INFO_PATH))
+        "that all workers are homogenous. It requires the AutoTuner to be enabled")
+  val targetClusterInfo: ScallopOption[String] =
+    opt[String](required = false,
+      descr = "File path containing the system information of the target cluster. " +
+        "This is supported only for CSP platform yet.")
 
   validate(filterCriteria) {
     case crit if crit.endsWith("-newest-filesystem") ||
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileClassWarehouse.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileClassWarehouse.scala
index 062aa54f6..f09e95696 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileClassWarehouse.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileClassWarehouse.scala
@@ -216,7 +216,7 @@ case class AppStatusResult(
   }
 
   override def convertToSeq(): Array[String] = {
-    Array(path, status, appId, message)
+    Array(path, status, appId, StringUtils.reformatCSVString(message.replaceAll("\\R", "\\\\n")))
   }
 
   override def convertToCSVSeq(): Array[String] = convertToSeq()
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileOutputWriter.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileOutputWriter.scala
index 5e7c49f2d..687b9a32c 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileOutputWriter.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileOutputWriter.scala
@@ -15,7 +15,6 @@
  */
 package com.nvidia.spark.rapids.tool.profiling
 
-import com.nvidia.spark.rapids.SparkRapidsBuildInfoEvent
 import com.nvidia.spark.rapids.tool.ToolTextFileWriter
 import org.apache.commons.lang3.StringUtils
 import org.json4s.DefaultFormats
@@ -55,12 +54,11 @@ class ProfileOutputWriter(outputDir: String, filePrefix: String, numOutputRows:
     }
   }
 
-  def writeSparkRapidsBuildInfo(headerText: String,
-      sparkRapidsBuildInfo: Seq[SparkRapidsBuildInfoEvent]): Unit = {
+  def writeJsonFile(headerText: String, objToWrite: AnyRef): Unit = {
     val fileName = headerText.replace(" ", "_").toLowerCase
     val jsonWriter = new ToolTextFileWriter(outputDir, s"$fileName.json", s"$headerText JSON:")
     try {
-      jsonWriter.write(Serialization.writePretty(sparkRapidsBuildInfo) + "\n")
+      jsonWriter.write(Serialization.writePretty(objToWrite) + "\n")
     } finally {
       jsonWriter.close()
     }
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala
index cbd6edc3c..5ad334d80 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/Profiler.scala
@@ -22,11 +22,11 @@ import scala.collection.JavaConverters._
 import scala.util.control.NonFatal
 
 import com.nvidia.spark.rapids.tool.{AppSummaryInfoBaseProvider, EventLogInfo, EventLogPathProcessor, FailedEventLog, PlatformFactory, ToolBase}
-import com.nvidia.spark.rapids.tool.tuning.{AutoTuner, ProfilingAutoTunerConfigsProvider, TuningEntryTrait}
+import com.nvidia.spark.rapids.tool.tuning.{AutoTuner, ClusterProperties, ProfilingAutoTunerConfigsProvider, TargetClusterProps, TuningEntryTrait}
 import com.nvidia.spark.rapids.tool.views._
 import org.apache.hadoop.conf.Configuration
 
-import org.apache.spark.sql.rapids.tool.{AppBase, FailureApp, IncorrectAppStatusException}
+import org.apache.spark.sql.rapids.tool.{AppBase, ClusterSummary, FailureApp, IncorrectAppStatusException}
 import org.apache.spark.sql.rapids.tool.profiling.ApplicationInfo
 import org.apache.spark.sql.rapids.tool.ui.ConsoleProgressBar
 import org.apache.spark.sql.rapids.tool.util._
@@ -200,10 +200,12 @@ class Profiler(hadoopConf: Configuration, appArgs: ProfileArgs, enablePB: Boolea
       // we need a platform per application because it is storing cluster information, which could
       // vary between applications, especially when using dynamic allocation.
       val platform = {
-        val workerInfoPath = appArgs.workerInfo
-          .getOrElse(ProfilingAutoTunerConfigsProvider.DEFAULT_WORKER_INFO_PATH)
-        val clusterPropsOpt = ProfilingAutoTunerConfigsProvider.loadClusterProps(workerInfoPath)
-        PlatformFactory.createInstance(appArgs.platform(), clusterPropsOpt)
+        val clusterPropsOpt = appArgs.workerInfo.toOption.flatMap(
+          PropertiesLoader[ClusterProperties].loadFromFile)
+        val targetClusterPropsOpt = appArgs.targetClusterInfo.toOption.flatMap(
+          PropertiesLoader[TargetClusterProps].loadFromFile)
+        PlatformFactory.createInstance(appArgs.platform(),
+          clusterPropsOpt, targetClusterPropsOpt)
       }
       val app = new ApplicationInfo(hadoopConf, path, platform = platform)
       EventLogPathProcessor.logApplicationInfo(app)
@@ -400,13 +402,21 @@ class Profiler(hadoopConf: Configuration, appArgs: ProfileArgs, enablePB: Boolea
       profileOutputWriter.writeText(Profiler.getAutoTunerResultsAsString(properties, comments))
     }
 
-    profileOutputWriter.writeSparkRapidsBuildInfo("Spark Rapids Build Info",
+    profileOutputWriter.writeJsonFile("Spark Rapids Build Info",
       app.sparkRapidsBuildInfo)
     profileOutputWriter.writeCSVTable(STAGE_DIAGNOSTICS_LABEL,
       profilerResult.diagnostics.stageDiagnostics)
     profileOutputWriter.writeCSVTable(ProfIODiagnosticMetricsView.getLabel,
       profilerResult.diagnostics.IODiagnostics)
-
+    // Construct the cluster summary information
+    // Note: This is available only after AutoTuner is run
+    val clusterSummary = app.appLogPath.map { logInfo =>
+      ClusterSummary(logInfo.appName, logInfo.appId.get, Some(logInfo.eventLogPath),
+        profilerResult.app.platform.flatMap(_.clusterInfoFromEventLog),
+        profilerResult.app.platform.flatMap(_.recommendedClusterInfo))
+    }
+    // Kept the file name as "Cluster Information" to be consistent with Qualification Tool
+    profileOutputWriter.writeJsonFile(CLUSTER_INFORMATION_LABEL, clusterSummary)
   }
 
   /**
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualOutputWriter.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualOutputWriter.scala
index 77b62077a..4b1f35147 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualOutputWriter.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualOutputWriter.scala
@@ -977,7 +977,7 @@ object QualOutputWriter {
       prettyPrint: Boolean,
       reformatCSV: Boolean = true): Seq[String] = {
     val reformatCSVFunc = getReformatCSVFunc(reformatCSV)
-    val clusterInfo = sumInfo.clusterSummary.clusterInfo
+    val sourceClusterInfo = sumInfo.clusterSummary.sourceClusterInfo
     val recClusterInfo = sumInfo.clusterSummary.recommendedClusterInfo
 
     // Wrapper function around reformatCSVFunc() to handle optional fields and
@@ -989,24 +989,27 @@ object QualOutputWriter {
       refactorCSVFuncWithOption(Some(sumInfo.clusterSummary.appId), APP_ID_STR),
       refactorCSVFuncWithOption(Some(sumInfo.clusterSummary.appName), APP_NAME_STR),
       refactorCSVFuncWithOption(sumInfo.clusterSummary.eventLogPath, STATUS_REPORT_PATH_STR),
-      refactorCSVFuncWithOption(clusterInfo.map(_.vendor), VENDOR),
-      refactorCSVFuncWithOption(clusterInfo.flatMap(_.driverHost), DRIVER_HOST),
-      refactorCSVFuncWithOption(clusterInfo.flatMap(_.clusterId), CLUSTER_ID_STR),
-      refactorCSVFuncWithOption(clusterInfo.flatMap(_.clusterName), CLUSTER_NAME),
-      refactorCSVFuncWithOption(clusterInfo.flatMap(_.workerNodeType), WORKER_NODE_TYPE),
-      refactorCSVFuncWithOption(clusterInfo.flatMap(_.driverNodeType), DRIVER_NODE_TYPE),
-      refactorCSVFuncWithOption(clusterInfo.map(_.numWorkerNodes.toString), NUM_WORKER_NODES),
-      refactorCSVFuncWithOption(clusterInfo.map(_.numExecsPerNode.toString), NUM_EXECS_PER_NODE),
-      refactorCSVFuncWithOption(clusterInfo.map(_.numExecutors.toString), NUM_EXECS),
-      refactorCSVFuncWithOption(clusterInfo.map(_.executorHeapMemory.toString),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.vendor), VENDOR),
+      refactorCSVFuncWithOption(sourceClusterInfo.flatMap(_.driverHost), DRIVER_HOST),
+      refactorCSVFuncWithOption(sourceClusterInfo.flatMap(_.clusterId), CLUSTER_ID_STR),
+      refactorCSVFuncWithOption(sourceClusterInfo.flatMap(_.clusterName), CLUSTER_NAME),
+      refactorCSVFuncWithOption(sourceClusterInfo.flatMap(_.workerNodeType), WORKER_NODE_TYPE),
+      refactorCSVFuncWithOption(sourceClusterInfo.flatMap(_.driverNodeType), DRIVER_NODE_TYPE),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.numWorkerNodes.toString), NUM_WORKER_NODES),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.numExecsPerNode.toString),
+        NUM_EXECS_PER_NODE),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.numExecutors.toString), NUM_EXECS),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.executorHeapMemory.toString),
         EXECUTOR_HEAP_MEMORY),
-      refactorCSVFuncWithOption(clusterInfo.map(_.dynamicAllocationEnabled.toString),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.dynamicAllocationEnabled.toString),
         DYN_ALLOC_ENABLED),
-      refactorCSVFuncWithOption(clusterInfo.map(_.dynamicAllocationMaxExecutors), DYN_ALLOC_MAX),
-      refactorCSVFuncWithOption(clusterInfo.map(_.dynamicAllocationMinExecutors), DYN_ALLOC_MIN),
-      refactorCSVFuncWithOption(clusterInfo.map(_.dynamicAllocationInitialExecutors),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.dynamicAllocationMaxExecutors),
+        DYN_ALLOC_MAX),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.dynamicAllocationMinExecutors),
+        DYN_ALLOC_MIN),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.dynamicAllocationInitialExecutors),
         DYN_ALLOC_INIT),
-      refactorCSVFuncWithOption(clusterInfo.map(_.coresPerExecutor.toString), CORES_PER_EXEC),
+      refactorCSVFuncWithOption(sourceClusterInfo.map(_.coresPerExecutor.toString), CORES_PER_EXEC),
       refactorCSVFuncWithOption(recClusterInfo.flatMap(_.workerNodeType),
         RECOMMENDED_WORKER_NODE_TYPE),
       refactorCSVFuncWithOption(recClusterInfo.map(_.numExecutors.toString),
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala
index fb9295d1c..a505ab6a6 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/Qualification.scala
@@ -21,7 +21,7 @@ import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
 import scala.collection.JavaConverters._
 
 import com.nvidia.spark.rapids.tool.{EventLogInfo, FailedEventLog, PlatformFactory, ToolBase}
-import com.nvidia.spark.rapids.tool.tuning.{QualificationAutoTunerConfigsProvider, TunerContext}
+import com.nvidia.spark.rapids.tool.tuning.{ClusterProperties, TunerContext}
 import com.nvidia.spark.rapids.tool.views.QualRawReportGenerator
 import org.apache.hadoop.conf.Configuration
 
@@ -36,7 +36,8 @@ class Qualification(outputPath: String, numRows: Int, hadoopConf: Configuration,
     printStdout: Boolean, enablePB: Boolean,
     reportSqlLevel: Boolean, maxSQLDescLength: Int, mlOpsEnabled: Boolean,
     penalizeTransitions: Boolean, tunerContext: Option[TunerContext],
-    clusterReport: Boolean, platformArg: String, workerInfoPath: String) extends ToolBase(timeout) {
+    clusterReport: Boolean, platformArg: String, workerInfoPath: Option[String])
+  extends ToolBase(timeout) {
 
   override val simpleName: String = "qualTool"
   override val outputDir = s"$outputPath/rapids_4_spark_qualification_output"
@@ -115,7 +116,8 @@ class Qualification(outputPath: String, numRows: Int, hadoopConf: Configuration,
       // we need a platform per application because it's storing cluster information which could
       // vary between applications, especially when using dynamic allocation
       val platform = {
-        val clusterPropsOpt = QualificationAutoTunerConfigsProvider.loadClusterProps(workerInfoPath)
+        val clusterPropsOpt = workerInfoPath.flatMap(
+          PropertiesLoader[ClusterProperties].loadFromFile)
         PlatformFactory.createInstance(platformArg, clusterPropsOpt)
       }
       val appResult = QualificationAppInfo.createApp(path, hadoopConf, pluginTypeChecker,
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala
index 4d5de33a4..3c495d35f 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationArgs.scala
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2021-2024, NVIDIA CORPORATION.
+ * Copyright (c) 2021-2025, NVIDIA CORPORATION.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -16,7 +16,6 @@
 package com.nvidia.spark.rapids.tool.qualification
 
 import com.nvidia.spark.rapids.tool.PlatformNames
-import com.nvidia.spark.rapids.tool.tuning.QualificationAutoTunerConfigsProvider
 import org.rogach.scallop.{ScallopConf, ScallopOption}
 import org.rogach.scallop.exceptions.ScallopException
 
@@ -193,9 +192,7 @@ Usage: java -cp rapids-4-spark-tools_2.12-<version>.jar:$SPARK_HOME/jars/*
   val workerInfo: ScallopOption[String] =
     opt[String](required = false,
       descr = "File path containing the system information of a worker node. It is assumed " +
-        "that all workers are homogenous. It requires the AutoTuner to be enabled. Default is " +
-        "./worker_info.yaml",
-      default = Some(QualificationAutoTunerConfigsProvider.DEFAULT_WORKER_INFO_PATH))
+        "that all workers are homogenous. It requires the AutoTuner to be enabled.")
   val clusterReport: ScallopOption[Boolean] =
     toggle("cluster-report",
       default = Some(true),
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala
index adfb07b4a..46f35b420 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/qualification/QualificationMain.scala
@@ -19,12 +19,12 @@ package com.nvidia.spark.rapids.tool.qualification
 import scala.util.control.NonFatal
 
 import com.nvidia.spark.rapids.tool.{EventLogPathProcessor, PlatformFactory}
-import com.nvidia.spark.rapids.tool.tuning.{QualificationAutoTunerConfigsProvider, TunerContext}
+import com.nvidia.spark.rapids.tool.tuning.{ClusterProperties, TunerContext}
 
 import org.apache.spark.internal.Logging
 import org.apache.spark.sql.rapids.tool.AppFilterImpl
 import org.apache.spark.sql.rapids.tool.qualification.QualificationSummaryInfo
-import org.apache.spark.sql.rapids.tool.util.RapidsToolsConfUtil
+import org.apache.spark.sql.rapids.tool.util.{PropertiesLoader, RapidsToolsConfUtil}
 
 /**
  * A tool to analyze Spark event logs and determine if
@@ -74,8 +74,8 @@ object QualificationMain extends Logging {
     // This platform instance should not be used for anything other then referencing the
     // files for this particular Platform.
     val referencePlatform = try {
-      val clusterPropsOpt =
-        QualificationAutoTunerConfigsProvider.loadClusterProps(appArgs.workerInfo())
+      val clusterPropsOpt = appArgs.workerInfo.toOption.flatMap(
+        PropertiesLoader[ClusterProperties].loadFromFile)
       PlatformFactory.createInstance(appArgs.platform(), clusterPropsOpt)
     } catch {
       case NonFatal(e) =>
@@ -122,7 +122,7 @@ object QualificationMain extends Logging {
     val qual = new Qualification(outputDirectory, numOutputRows, hadoopConf, timeout,
       nThreads, order, pluginTypeChecker, reportReadSchema, printStdout,
       enablePB, reportSqlLevel, maxSQLDescLength, mlOpsEnabled, penalizeTransitions,
-      tunerContext, appArgs.clusterReport(), appArgs.platform(), appArgs.workerInfo())
+      tunerContext, appArgs.clusterReport(), appArgs.platform(), appArgs.workerInfo.toOption)
     val res = qual.qualifyApps(filteredLogs)
     (0, res)
   }
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/AutoTuner.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/AutoTuner.scala
index 8290ccaf0..01f84d501 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/AutoTuner.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/AutoTuner.scala
@@ -16,7 +16,6 @@
 
 package com.nvidia.spark.rapids.tool.tuning
 
-import java.io.{BufferedReader, InputStreamReader, IOException}
 import java.util
 
 import scala.beans.BeanProperty
@@ -27,16 +26,12 @@ import scala.util.matching.Regex
 
 import com.nvidia.spark.rapids.tool.{AppSummaryInfoBaseProvider, GpuDevice, Platform, PlatformFactory}
 import com.nvidia.spark.rapids.tool.profiling._
-import org.apache.hadoop.conf.Configuration
-import org.apache.hadoop.fs.{FileSystem, FSDataInputStream, Path}
-import org.yaml.snakeyaml.{DumperOptions, LoaderOptions, Yaml}
-import org.yaml.snakeyaml.constructor.{Constructor, ConstructorException}
-import org.yaml.snakeyaml.representer.Representer
+import org.yaml.snakeyaml.constructor.ConstructorException
 
 import org.apache.spark.internal.Logging
 import org.apache.spark.network.util.ByteUnit
 import org.apache.spark.sql.rapids.tool.ToolUtils
-import org.apache.spark.sql.rapids.tool.util.{StringUtils, WebCrawlerUtil}
+import org.apache.spark.sql.rapids.tool.util.{PropertiesLoader, StringUtils, WebCrawlerUtil}
 
 /**
  * A wrapper class that stores all the GPU properties.
@@ -440,6 +435,7 @@ class AutoTuner(
   private def configureGPURecommendedInstanceType(): Unit = {
     platform.createRecommendedGpuClusterInfo(getAllProperties.toMap)
     platform.recommendedClusterInfo.foreach { gpuClusterRec =>
+      // TODO: Should we skip recommendation if cores per executor is lower than a min value?
       appendRecommendation("spark.executor.cores", gpuClusterRec.coresPerExecutor)
       if (gpuClusterRec.numExecutors > 0) {
         appendRecommendation("spark.executor.instances", gpuClusterRec.numExecutors)
@@ -456,8 +452,9 @@ class AutoTuner(
    * Recommendation for 'spark.rapids.sql.concurrentGpuTasks' based on gpu memory.
    * Assumption - cluster properties were updated to have a default values if missing.
    */
-  def calcGpuConcTasks(): Long = {
-    Math.min(autoTunerConfigsProvider.MAX_CONC_GPU_TASKS, platform.getGpuOrDefault.getGpuConcTasks)
+  private def calcGpuConcTasks(): Long = {
+    Math.min(autoTunerConfigsProvider.MAX_CONC_GPU_TASKS,
+      platform.recommendedGpuDevice.getGpuConcTasks)
   }
 
   /**
@@ -802,14 +799,14 @@ class AutoTuner(
       appInfoProvider.getMeanShuffleRead >
         autoTunerConfigsProvider.AQE_SHUFFLE_READ_BYTES_THRESHOLD) {
       // AQE Recommendations for large input and large shuffle reads
-      platform.getGpuOrDefault.getAdvisoryPartitionSizeInBytes.foreach { size =>
+      platform.recommendedGpuDevice.getAdvisoryPartitionSizeInBytes.foreach { size =>
         appendRecommendation("spark.sql.adaptive.advisoryPartitionSizeInBytes", size)
       }
       val initialPartitionNumProperty =
         getPropertyValue("spark.sql.adaptive.coalescePartitions.initialPartitionNum").map(_.toInt)
       if (initialPartitionNumProperty.getOrElse(0) <=
             autoTunerConfigsProvider.AQE_MIN_INITIAL_PARTITION_NUM) {
-        recInitialPartitionNum = platform.getGpuOrDefault.getInitialPartitionNum.getOrElse(0)
+        recInitialPartitionNum = platform.recommendedGpuDevice.getInitialPartitionNum.getOrElse(0)
       }
       // We need to set this to false, else Spark ignores the target size specified by
       // spark.sql.adaptive.advisoryPartitionSizeInBytes.
@@ -1328,7 +1325,6 @@ trait AutoTunerConfigsProvider extends Logging {
   val DEF_DISTINCT_READ_THRESHOLD = 50.0
   // Default file cache size minimum is 100 GB
   val DEF_READ_SIZE_THRESHOLD = 100 * 1024L * 1024L * 1024L
-  val DEFAULT_WORKER_INFO_PATH = "./worker_info.yaml"
   val SUPPORTED_SIZE_UNITS: Seq[String] = Seq("b", "k", "m", "g", "t", "p")
   private val DOC_URL: String = "https://nvidia.github.io/spark-rapids/docs/" +
     "additional-functionality/advanced_configs.html#advanced-configuration"
@@ -1430,41 +1426,6 @@ trait AutoTunerConfigsProvider extends Logging {
     tuning
   }
 
-  def loadClusterPropertiesFromContent(clusterProps: String): Option[ClusterProperties] = {
-    val representer = new Representer(new DumperOptions())
-    representer.getPropertyUtils.setSkipMissingProperties(true)
-    val constructor = new Constructor(classOf[ClusterProperties], new LoaderOptions())
-    val yamlObjNested = new Yaml(constructor, representer)
-    val loadedClusterProps = yamlObjNested.load(clusterProps).asInstanceOf[ClusterProperties]
-    if (loadedClusterProps != null && loadedClusterProps.softwareProperties == null) {
-      logInfo("softwareProperties is empty from input worker_info file")
-      loadedClusterProps.softwareProperties = new util.LinkedHashMap[String, String]()
-    }
-    Option(loadedClusterProps)
-  }
-
-  def loadClusterProps(filePath: String): Option[ClusterProperties] = {
-    val path = new Path(filePath)
-    var fsIs: FSDataInputStream = null
-    try {
-      val fs = FileSystem.get(path.toUri, new Configuration())
-      fsIs = fs.open(path)
-      val reader = new BufferedReader(new InputStreamReader(fsIs))
-      val fileContent = Stream.continually(reader.readLine()).takeWhile(_ != null).mkString("\n")
-      loadClusterPropertiesFromContent(fileContent)
-    } catch {
-      // In case of missing file/malformed for cluster properties, default properties are used.
-      // Hence, catching and logging as a warning
-      case _: IOException =>
-        logWarning(s"No file found for input workerInfo path: $filePath")
-        None
-    } finally {
-      if (fsIs != null) {
-        fsIs.close()
-      }
-    }
-  }
-
   /**
    * Similar to [[buildAutoTuner]] but it allows constructing the AutoTuner without an
    * existing file. This can be used in testing.
@@ -1483,7 +1444,7 @@ trait AutoTunerConfigsProvider extends Logging {
       driverInfoProvider: DriverLogInfoProvider = BaseDriverLogInfoProvider.noneDriverLog
   ): AutoTuner = {
     try {
-      val clusterPropsOpt = loadClusterPropertiesFromContent(clusterProps)
+      val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(clusterProps)
       createAutoTunerInstance(clusterPropsOpt.getOrElse(new ClusterProperties()),
         singleAppProvider, platform, driverInfoProvider)
     } catch {
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/TargetClusterProps.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/TargetClusterProps.scala
new file mode 100644
index 000000000..1098ccfa7
--- /dev/null
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/TargetClusterProps.scala
@@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.nvidia.spark.rapids.tool.tuning
+
+import com.nvidia.spark.rapids.tool.NodeInstanceMapKey
+import scala.beans.BeanProperty
+
+class WorkerInfo(
+  @BeanProperty var instanceType: String,
+  @BeanProperty var gpu: GpuWorkerProps) {
+
+  def this() = this("", new GpuWorkerProps())
+
+  /**
+   * Helper method to get the NodeInstanceMapKey for this worker. This
+   * will be used as the key in the following map:
+   * [[com.nvidia.spark.rapids.tool.Platform.getInstanceMapByName]]
+   * @return
+   */
+  def getNodeInstanceMapKey: NodeInstanceMapKey = {
+    val gpuCount = this.getGpu.getCount
+    if (gpuCount > 0) {
+      NodeInstanceMapKey(instanceType = this.getInstanceType, gpuCount = Some(gpuCount))
+    } else {
+      NodeInstanceMapKey(instanceType = this.getInstanceType)
+    }
+  }
+}
+
+class TargetClusterProps(
+  @BeanProperty var workerInfo: WorkerInfo) {
+  def this() = this(new WorkerInfo())
+
+  override def toString: String = {
+    s"${getClass.getSimpleName}(workerInfo=$workerInfo)"
+  }
+}
diff --git a/core/src/main/scala/com/nvidia/spark/rapids/tool/views/package.scala b/core/src/main/scala/com/nvidia/spark/rapids/tool/views/package.scala
index a509ae3f7..bab4b8a2b 100644
--- a/core/src/main/scala/com/nvidia/spark/rapids/tool/views/package.scala
+++ b/core/src/main/scala/com/nvidia/spark/rapids/tool/views/package.scala
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2024, NVIDIA CORPORATION.
+ * Copyright (c) 2024-2025, NVIDIA CORPORATION.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -29,6 +29,7 @@ package object views {
   val SQL_DUR_LABEL = "SQL Duration and Executor CPU Time Percent"
   val SQL_MAX_INPUT_SIZE = "SQL Max Task Input Size"
   val STAGE_DIAGNOSTICS_LABEL = "Stage Level Diagnostic Metrics"
+  val CLUSTER_INFORMATION_LABEL = "Cluster Information"
 
   val AGG_DESCRIPTION = Map(
     STAGE_AGG_LABEL -> "Stage metrics",
diff --git a/core/src/main/scala/org/apache/spark/sql/rapids/tool/ClassWarehouse.scala b/core/src/main/scala/org/apache/spark/sql/rapids/tool/ClassWarehouse.scala
index bb70d62fd..e3975725f 100644
--- a/core/src/main/scala/org/apache/spark/sql/rapids/tool/ClassWarehouse.scala
+++ b/core/src/main/scala/org/apache/spark/sql/rapids/tool/ClassWarehouse.scala
@@ -51,7 +51,8 @@ sealed trait ClusterInfo {
 // Information about the cluster used with to run the application we
 // are qualifying or profiling. This is compared to what we might recommend
 // for a cluster.
-case class ExistingClusterInfo(
+// TODO: Add information about the GPUs on the source cluster
+case class SourceClusterInfo(
     vendor: String,
     coresPerExecutor: Int,
     numExecsPerNode: Int,
@@ -89,5 +90,5 @@ case class ClusterSummary(
     appName: String,
     appId: String,
     eventLogPath: Option[String],
-    clusterInfo: Option[ExistingClusterInfo],
+    sourceClusterInfo: Option[SourceClusterInfo],
     recommendedClusterInfo: Option[RecommendedClusterInfo])
diff --git a/core/src/main/scala/org/apache/spark/sql/rapids/tool/ToolUtils.scala b/core/src/main/scala/org/apache/spark/sql/rapids/tool/ToolUtils.scala
index 75b76646f..3d5aebc4b 100644
--- a/core/src/main/scala/org/apache/spark/sql/rapids/tool/ToolUtils.scala
+++ b/core/src/main/scala/org/apache/spark/sql/rapids/tool/ToolUtils.scala
@@ -467,6 +467,9 @@ case class UnsupportedSparkRuntimeException(
 class InvalidMemoryUnitFormatException(message: String)
   extends IllegalArgumentException(message)
 
+class MatchingInstanceTypeNotFoundException(message: String)
+  extends Exception(message)
+
 // Class used a container to hold the information of the Tuple<sqlID, PlanInfo, SparkGraph>
 // to simplify arguments of methods and caching.
 case class SqlPlanInfoGraphEntry(
diff --git a/core/src/main/scala/org/apache/spark/sql/rapids/tool/util/PropertiesLoader.scala b/core/src/main/scala/org/apache/spark/sql/rapids/tool/util/PropertiesLoader.scala
new file mode 100644
index 000000000..5bb6b71d6
--- /dev/null
+++ b/core/src/main/scala/org/apache/spark/sql/rapids/tool/util/PropertiesLoader.scala
@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.rapids.tool.util
+
+import java.io.{BufferedReader, InputStreamReader, IOException}
+
+import scala.reflect.ClassTag
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, FSDataInputStream, Path}
+import org.yaml.snakeyaml.{DumperOptions, LoaderOptions, Yaml}
+import org.yaml.snakeyaml.constructor.Constructor
+import org.yaml.snakeyaml.representer.Representer
+
+import org.apache.spark.internal.Logging
+
+
+/**
+ * Helper class to load YAML properties files into a specific type.
+ *
+ * @tparam T The type of configuration to load
+ */
+class PropertiesLoader[T](implicit classTag: ClassTag[T]) extends Logging {
+  def loadFromContent(content: String): Option[T] = {
+    val representer = new Representer(new DumperOptions())
+    representer.getPropertyUtils.setSkipMissingProperties(true)
+    val constructor = new Constructor(classTag.runtimeClass.asInstanceOf[Class[T]],
+      new LoaderOptions())
+    val yamlObjNested = new Yaml(constructor, representer)
+    Option(yamlObjNested.load(content).asInstanceOf[T])
+  }
+
+  def loadFromFile(filePath: String): Option[T] = {
+    val path = new Path(filePath)
+    var fsIs: FSDataInputStream = null
+    try {
+      val fs = FileSystem.get(path.toUri, new Configuration())
+      fsIs = fs.open(path)
+      val reader = new BufferedReader(new InputStreamReader(fsIs))
+      val fileContent = Stream.continually(reader.readLine()).takeWhile(_ != null).mkString("\n")
+      loadFromContent(fileContent)
+    } catch {
+      case _: IOException =>
+        logWarning(s"No file found for input path: $filePath")
+        None
+    } finally {
+      if (fsIs != null) {
+        fsIs.close()
+      }
+    }
+  }
+}
+
+object PropertiesLoader {
+  def apply[T](implicit classTag: ClassTag[T]): PropertiesLoader[T] = new PropertiesLoader[T]
+}
diff --git a/core/src/test/scala/com/nvidia/spark/rapids/tool/ToolTestUtils.scala b/core/src/test/scala/com/nvidia/spark/rapids/tool/ToolTestUtils.scala
index 80bf28596..13d17b004 100644
--- a/core/src/test/scala/com/nvidia/spark/rapids/tool/ToolTestUtils.scala
+++ b/core/src/test/scala/com/nvidia/spark/rapids/tool/ToolTestUtils.scala
@@ -18,14 +18,19 @@ package com.nvidia.spark.rapids.tool
 
 import java.io.{File, FilenameFilter, FileNotFoundException}
 
-import scala.collection.mutable.ArrayBuffer
-
+import com.fasterxml.jackson.databind.ObjectMapper
+import com.fasterxml.jackson.module.scala.DefaultScalaModule
 import com.nvidia.spark.rapids.tool.profiling.ProfileArgs
 import com.nvidia.spark.rapids.tool.qualification.QualOutputWriter
+import com.nvidia.spark.rapids.tool.tuning.{GpuWorkerProps, TargetClusterProps, WorkerInfo}
+import org.apache.hadoop.fs.Path
+import org.yaml.snakeyaml.{DumperOptions, Yaml}
+import scala.collection.mutable.ArrayBuffer
 
 import org.apache.spark.internal.Logging
 import org.apache.spark.sql.{DataFrame, SparkSession, TrampolineUtil}
 import org.apache.spark.sql.functions.{col, lit}
+import org.apache.spark.sql.rapids.tool.ClusterSummary
 import org.apache.spark.sql.rapids.tool.profiling.ApplicationInfo
 import org.apache.spark.sql.rapids.tool.util.RapidsToolsConfUtil
 import org.apache.spark.sql.types._
@@ -187,6 +192,62 @@ object ToolTestUtils extends Logging {
           s"but got: $actualStatusReportCount")
     }
   }
+
+  /**
+   * Load a JSON file containing an array of ClusterSummary objects.
+   * @param path The path to the JSON file.
+   * @return An array of ClusterSummary objects.
+   */
+  def loadClusterSummaryFromJson(path: String): Array[ClusterSummary] = {
+    val mapper = new ObjectMapper().registerModule(DefaultScalaModule)
+    mapper.readValue(new File(path), classOf[Array[ClusterSummary]])
+  }
+
+  def buildTargetClusterInfo(
+      instanceType: String,
+      gpuCount: Option[Int] = None,
+      gpuMemory: Option[String] = None,
+      gpuDevice: Option[String] = None): TargetClusterProps = {
+    val gpuWorkerProps = new GpuWorkerProps(
+      gpuMemory.getOrElse(""), gpuCount.getOrElse(0), gpuDevice.getOrElse(""))
+    val workerProps = new WorkerInfo(instanceType, gpuWorkerProps)
+    new TargetClusterProps(workerProps)
+  }
+
+  def buildTargetClusterInfoAsString(
+      instanceType: String,
+      gpuCount: Option[Int] = None,
+      gpuMemory: Option[String] = None,
+      gpuDevice: Option[String] = None): String = {
+    val targetCluster = buildTargetClusterInfo(instanceType, gpuCount, gpuMemory, gpuDevice)
+    // set the options to convert the object into formatted yaml content
+    val options = new DumperOptions()
+    options.setIndent(2)
+    options.setPrettyFlow(true)
+    options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK)
+    val yaml = new Yaml(options)
+    val rawString = yaml.dump(targetCluster)
+    // Skip the first line as it contains "the name of the class"
+    rawString.split("\n").drop(1).mkString("\n")
+  }
+
+  def createTargetClusterInfoFile(
+      outputDirectory: String,
+      instanceType: String,
+      gpuCount: Option[Int] = None,
+      gpuMemory: Option[String] = None,
+      gpuDevice: Option[String] = None): Path = {
+    val fileWriter = new ToolTextFileWriter(outputDirectory, "targetClusterInfo.yaml",
+      "Target Cluster Info")
+    try {
+      val targetClusterInfoString =
+        buildTargetClusterInfoAsString(instanceType, gpuCount, gpuMemory, gpuDevice)
+      fileWriter.write(targetClusterInfoString)
+      fileWriter.getFileOutputPath
+    } finally {
+      fileWriter.close()
+    }
+  }
 }
 
 /**
diff --git a/core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/ClusterRecommendationSuite.scala b/core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/ClusterRecommendationSuite.scala
new file mode 100644
index 000000000..d39025e84
--- /dev/null
+++ b/core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/ClusterRecommendationSuite.scala
@@ -0,0 +1,269 @@
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.nvidia.spark.rapids.tool.profiling
+
+import java.io.File
+
+import com.nvidia.spark.rapids.tool.{GpuTypes, PlatformNames, StatusReportCounts, ToolTestUtils}
+import com.nvidia.spark.rapids.tool.views.CLUSTER_INFORMATION_LABEL
+import org.scalatest.FunSuite
+import org.scalatest.exceptions.TestFailedException
+import org.scalatest.prop.{TableDrivenPropertyChecks, TableFor3, TableFor4}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.{SparkSession, TrampolineUtil}
+import org.apache.spark.sql.rapids.tool.RecommendedClusterInfo
+
+
+class ClusterRecommendationSuite extends FunSuite with Logging with TableDrivenPropertyChecks {
+  lazy val sparkSession: SparkSession = {
+    SparkSession
+      .builder()
+      .master("local[*]")
+      .appName("Rapids Spark Profiling Tool Unit Tests")
+      .getOrCreate()
+  }
+
+  private val logDir = ToolTestUtils.getTestResourcePath("spark-events-profiling")
+
+  val validClusterRecommendationScenarios: TableFor4[
+      String, String, Option[Int], RecommendedClusterInfo] = Table(
+    // Define the column headers
+    ("platform", "instanceType", "gpuCount", "expectedClusterInfo"),
+    // Test scenarios
+    (
+      PlatformNames.DATAPROC,
+      "n1-standard-16",
+      Some(1),
+      RecommendedClusterInfo(
+        vendor = PlatformNames.DATAPROC,
+        coresPerExecutor = 16,
+        numWorkerNodes = 8,
+        numGpusPerNode = 1,
+        numExecutors = 8,
+        gpuDevice = GpuTypes.T4,
+        dynamicAllocationEnabled = false,
+        dynamicAllocationMaxExecutors = "N/A",
+        dynamicAllocationMinExecutors = "N/A",
+        dynamicAllocationInitialExecutors = "N/A",
+        workerNodeType = Some("n1-standard-16"))
+    ),
+    (
+      PlatformNames.DATAPROC,
+      "n1-standard-32",
+      Some(2),
+      RecommendedClusterInfo(
+        vendor = PlatformNames.DATAPROC,
+        coresPerExecutor = 16,
+        numWorkerNodes = 4,
+        numGpusPerNode = 2,
+        numExecutors = 8,
+        gpuDevice = GpuTypes.T4,
+        dynamicAllocationEnabled = false,
+        dynamicAllocationMaxExecutors = "N/A",
+        dynamicAllocationMinExecutors = "N/A",
+        dynamicAllocationInitialExecutors = "N/A",
+        workerNodeType = Some("n1-standard-32"))
+    ),
+    (
+      PlatformNames.DATAPROC,
+      "g2-standard-16",
+      None,
+      RecommendedClusterInfo(
+        vendor = PlatformNames.DATAPROC,
+        coresPerExecutor = 16,
+        numWorkerNodes = 8,
+        numGpusPerNode = 1,
+        numExecutors = 8,
+        gpuDevice = GpuTypes.L4,
+        dynamicAllocationEnabled = false,
+        dynamicAllocationMaxExecutors = "N/A",
+        dynamicAllocationMinExecutors = "N/A",
+        dynamicAllocationInitialExecutors = "N/A",
+        workerNodeType = Some("g2-standard-16"))
+    ),
+    (
+      PlatformNames.EMR,
+      "g6.8xlarge",
+      None,
+      RecommendedClusterInfo(
+        vendor = PlatformNames.EMR,
+        coresPerExecutor = 32,
+        numWorkerNodes = 4,
+        numGpusPerNode = 1,
+        numExecutors = 4,
+        gpuDevice = GpuTypes.L4,
+        dynamicAllocationEnabled = false,
+        dynamicAllocationMaxExecutors = "N/A",
+        dynamicAllocationMinExecutors = "N/A",
+        dynamicAllocationInitialExecutors = "N/A",
+        workerNodeType = Some("g6.8xlarge"))
+    ),
+    (
+      PlatformNames.DATABRICKS_AWS,
+      "g5.8xlarge",
+      None,
+      RecommendedClusterInfo(
+        vendor = PlatformNames.DATABRICKS_AWS,
+        coresPerExecutor = 32,
+        numWorkerNodes = 4,
+        numGpusPerNode = 1,
+        numExecutors = 4,
+        gpuDevice = GpuTypes.A10G,
+        dynamicAllocationEnabled = false,
+        dynamicAllocationMaxExecutors = "N/A",
+        dynamicAllocationMinExecutors = "N/A",
+        dynamicAllocationInitialExecutors = "N/A",
+        workerNodeType = Some("g5.8xlarge"))
+    )
+  )
+  // scalastyle:on line.size.limit
+  forAll(validClusterRecommendationScenarios) {
+    (platform, instanceType, gpuCount, expectedClusterInfo) =>
+      val scenarioName = gpuCount match {
+        case Some(count) => s"$instanceType with $count GPUs"
+        case None => instanceType
+      }
+      test(s"test valid cluster shape recommendation on $platform - $scenarioName") {
+        TrampolineUtil.withTempDir { tempDir =>
+          val targetClusterInfoFile = ToolTestUtils.createTargetClusterInfoFile(
+            tempDir.getAbsolutePath,
+            instanceType = instanceType,
+            gpuCount = gpuCount)
+
+          val appArgs = new ProfileArgs(Array(
+            "--platform",
+            platform,
+            "--target-cluster-info",
+            targetClusterInfoFile.toString,
+            "--output-directory",
+            tempDir.getAbsolutePath,
+            "--csv",
+            "--auto-tuner",
+            s"$logDir/nds_q66_gpu.zstd"))
+
+          val (exit, _) = ProfileMain.mainInternal(appArgs)
+          assert(exit == 0)
+          val tempSubDir = new File(tempDir, s"${Profiler.SUBDIR}/application_1701368813061_0008")
+          val fileName = CLUSTER_INFORMATION_LABEL.replace(" ", "_").toLowerCase
+          val actualFilePath = s"${tempSubDir.getAbsolutePath}/$fileName.json"
+
+          val recommendedClusterInfo = ToolTestUtils.loadClusterSummaryFromJson(actualFilePath)
+            .headOption.flatMap(_.recommendedClusterInfo).getOrElse {
+              throw new TestFailedException(
+                s"Failed to load recommended cluster info from $actualFilePath", 0)
+            }
+
+          val clusterInfoMatches = recommendedClusterInfo == expectedClusterInfo
+          assert(clusterInfoMatches,
+            s"""
+               |Actual cluster info does not match the expected cluster info.
+               |Actual: $recommendedClusterInfo
+               |Expected: $expectedClusterInfo
+               |""".stripMargin)
+        }
+      }
+  }
+
+  test(s"test valid cluster shape recommendation on onprem") {
+    val expectedClusterInfo = RecommendedClusterInfo(
+      vendor = PlatformNames.ONPREM,
+      coresPerExecutor = 16,
+      numWorkerNodes = -1,
+      numGpusPerNode = 1,
+      numExecutors = 8,
+      gpuDevice = GpuTypes.L4,
+      dynamicAllocationEnabled = false,
+      dynamicAllocationMaxExecutors = "N/A",
+      dynamicAllocationMinExecutors = "N/A",
+      dynamicAllocationInitialExecutors = "N/A",
+      workerNodeType = Some("N/A"))
+    TrampolineUtil.withTempDir { tempDir =>
+      val appArgs = new ProfileArgs(Array(
+        "--platform",
+        PlatformNames.ONPREM,
+        "--output-directory",
+        tempDir.getAbsolutePath,
+        "--csv",
+        "--auto-tuner",
+        s"$logDir/nds_q66_gpu.zstd"))
+
+      val (exit, _) = ProfileMain.mainInternal(appArgs)
+      assert(exit == 0)
+      val tempSubDir = new File(tempDir, s"${Profiler.SUBDIR}/application_1701368813061_0008")
+      val fileName = CLUSTER_INFORMATION_LABEL.replace(" ", "_").toLowerCase
+      val actualFilePath = s"${tempSubDir.getAbsolutePath}/$fileName.json"
+
+      val recommendedClusterInfo = ToolTestUtils.loadClusterSummaryFromJson(actualFilePath)
+        .headOption.flatMap(_.recommendedClusterInfo).getOrElse {
+          throw new TestFailedException(
+            s"Failed to load recommended cluster info from $actualFilePath", 0)
+        }
+
+      val clusterInfoMatches = recommendedClusterInfo == expectedClusterInfo
+      assert(clusterInfoMatches,
+        s"""
+           |Actual cluster info does not match the expected cluster info.
+           |Actual: $recommendedClusterInfo
+           |Expected: $expectedClusterInfo
+           |""".stripMargin)
+    }
+  }
+
+  val invalidClusterInfoScenarios: TableFor3[String, String, Option[Int]] = Table(
+    // Define the column headers
+    ("platform", "instanceType", "gpuCount"),
+    // Test scenarios
+    (PlatformNames.DATAPROC, "n1-standard-4", None),
+    (PlatformNames.EMR, "abc.8xlarge", None),
+    (PlatformNames.DATABRICKS_AWS, "g5.2xlarge", Some(4))
+  )
+
+  forAll(invalidClusterInfoScenarios) {
+    (platform, instanceType, gpuCount) =>
+      val scenarioName = gpuCount match {
+        case Some(count) => s"$instanceType with $count GPUs"
+        case None => instanceType
+      }
+      test(s"test invalid cluster shape recommendation on $platform - $scenarioName") {
+        TrampolineUtil.withTempDir { tempDir =>
+          val targetClusterInfoFile = ToolTestUtils.createTargetClusterInfoFile(
+            tempDir.getAbsolutePath,
+            instanceType = instanceType,
+            gpuCount = gpuCount)
+
+          val appArgs = new ProfileArgs(Array(
+            "--platform",
+            platform,
+            "--target-cluster-info",
+            targetClusterInfoFile.toString,
+            "--output-directory",
+            tempDir.getAbsolutePath,
+            "--csv",
+            "--auto-tuner",
+            s"$logDir/nds_q66_gpu.zstd"))
+
+          val (exit, _) = ProfileMain.mainInternal(appArgs)
+          assert(exit == 0)
+          // Status counts: 0 SUCCESS, 0 FAILURE, 0 SKIPPED, 1 UNKNOWN
+          val expectedStatusCount = StatusReportCounts(0, 0, 0, 1)
+          ToolTestUtils.compareStatusReport(sparkSession, expectedStatusCount,
+            s"${tempDir.getAbsolutePath}/${Profiler.SUBDIR}/profiling_status.csv")
+        }
+      }
+  }
+}
diff --git a/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/PluginTypeCheckerSuite.scala b/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/PluginTypeCheckerSuite.scala
index c75f3087a..84601e47e 100644
--- a/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/PluginTypeCheckerSuite.scala
+++ b/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/PluginTypeCheckerSuite.scala
@@ -151,18 +151,18 @@ class PluginTypeCheckerSuite extends FunSuite with Logging {
   }
 
   val platformSpeedupEntries: Seq[(Platform, Map[String, Double])] = Seq(
-    (new OnPremPlatform(Some(A100Gpu), None), Map("UnionExec" -> 3.0, "Ceil" -> 4.0)),
-    (new DataprocPlatform(Some(T4Gpu), None), Map("UnionExec" -> 4.88, "Ceil" -> 4.88)),
-    (new EmrPlatform(Some(T4Gpu), None), Map("UnionExec" -> 2.07, "Ceil" -> 2.07)),
-    (new DatabricksAwsPlatform(Some(T4Gpu), None), Map("UnionExec" -> 2.45, "Ceil" -> 2.45)),
-    (new DatabricksAzurePlatform(Some(T4Gpu), None),
+    (new OnPremPlatform(Some(A100Gpu), None, None), Map("UnionExec" -> 3.0, "Ceil" -> 4.0)),
+    (new DataprocPlatform(Some(T4Gpu), None, None), Map("UnionExec" -> 4.88, "Ceil" -> 4.88)),
+    (new EmrPlatform(Some(T4Gpu), None, None), Map("UnionExec" -> 2.07, "Ceil" -> 2.07)),
+    (new DatabricksAwsPlatform(Some(T4Gpu), None, None), Map("UnionExec" -> 2.45, "Ceil" -> 2.45)),
+    (new DatabricksAzurePlatform(Some(T4Gpu), None, None),
       Map("UnionExec" -> 2.73, "Ceil" -> 2.73)),
-    (new DataprocServerlessPlatform(Some(L4Gpu), None),
+    (new DataprocServerlessPlatform(Some(L4Gpu), None, None),
       Map("WindowExec" -> 4.25, "Ceil" -> 4.25)),
-    (new DataprocPlatform(Some(L4Gpu), None), Map("UnionExec" -> 4.16, "Ceil" -> 4.16)),
-    (new DataprocGkePlatform(Some(T4Gpu), None), Map("WindowExec" -> 3.65, "Ceil" -> 3.65)),
-    (new DataprocGkePlatform(Some(L4Gpu), None), Map("WindowExec" -> 3.74, "Ceil" -> 3.74)),
-    (new EmrPlatform(Some(A10Gpu), None), Map("UnionExec" -> 2.59, "Ceil" -> 2.59))
+    (new DataprocPlatform(Some(L4Gpu), None, None), Map("UnionExec" -> 4.16, "Ceil" -> 4.16)),
+    (new DataprocGkePlatform(Some(T4Gpu), None, None), Map("WindowExec" -> 3.65, "Ceil" -> 3.65)),
+    (new DataprocGkePlatform(Some(L4Gpu), None, None), Map("WindowExec" -> 3.74, "Ceil" -> 3.74)),
+    (new EmrPlatform(Some(A10Gpu), None, None), Map("UnionExec" -> 2.59, "Ceil" -> 2.59))
   )
 
   platformSpeedupEntries.foreach { case (platform, speedupMap) =>
diff --git a/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/QualificationSuite.scala b/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/QualificationSuite.scala
index bd2dfc82e..e78d5ead3 100644
--- a/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/QualificationSuite.scala
+++ b/core/src/test/scala/com/nvidia/spark/rapids/tool/qualification/QualificationSuite.scala
@@ -21,8 +21,6 @@ import java.util.concurrent.TimeUnit.NANOSECONDS
 
 import scala.collection.mutable.{ArrayBuffer, ListBuffer}
 
-import com.fasterxml.jackson.databind.ObjectMapper
-import com.fasterxml.jackson.module.scala.DefaultScalaModule
 import com.nvidia.spark.rapids.BaseTestSuite
 import com.nvidia.spark.rapids.tool.{EventLogPathProcessor, PlatformNames, StatusReportCounts, ToolTestUtils}
 import com.nvidia.spark.rapids.tool.planparser.DatabricksParseHelper
@@ -33,7 +31,7 @@ import org.apache.spark.ml.linalg.Vectors
 import org.apache.spark.scheduler.{SparkListener, SparkListenerStageCompleted, SparkListenerTaskEnd}
 import org.apache.spark.sql.{DataFrame, Dataset, SparkSession, TrampolineUtil}
 import org.apache.spark.sql.functions.{desc, hex, to_json, udf}
-import org.apache.spark.sql.rapids.tool.{AppBase, AppFilterImpl, ClusterSummary, ExistingClusterInfo, ToolUtils}
+import org.apache.spark.sql.rapids.tool.{AppBase, AppFilterImpl, SourceClusterInfo, ToolUtils}
 import org.apache.spark.sql.rapids.tool.qualification.{QualificationSummaryInfo, RunningQualificationEventProcessor}
 import org.apache.spark.sql.rapids.tool.util.{FSUtils, RapidsToolsConfUtil, UTF8Source}
 import org.apache.spark.sql.types._
@@ -1275,9 +1273,9 @@ class QualificationSuite extends BaseTestSuite {
         // or Json.
         val sqlIdToLookup = allSQLIds.max
         val (csvOut, txtOut) = qualApp.getPerSqlTextAndCSVSummary(sqlIdToLookup)
-        assert(csvOut.contains("collect at ToolTestUtils.scala:68") && csvOut.contains(","),
+        assert(csvOut.contains("collect at ToolTestUtils.scala:73") && csvOut.contains(","),
           s"CSV output was: $csvOut")
-        assert(txtOut.contains("collect at ToolTestUtils.scala:68") && txtOut.contains("|"),
+        assert(txtOut.contains("collect at ToolTestUtils.scala:73") && txtOut.contains("|"),
           s"TXT output was: $txtOut")
         val sqlOut = qualApp.getPerSQLSummary(sqlIdToLookup, ":", true, 5)
         assert(sqlOut.contains("colle:"), s"SQL output was: $sqlOut")
@@ -1591,28 +1589,28 @@ class QualificationSuite extends BaseTestSuite {
 
   // Expected results as a map of event log -> cluster info.
   // scalastyle:off line.size.limit
-  val expectedClusterInfoMap: Seq[(String, Option[ExistingClusterInfo])] = Seq(
+  val expectedClusterInfoMap: Seq[(String, Option[SourceClusterInfo])] = Seq(
     "eventlog_2nodes_8cores" -> // 2 executor nodes with 8 cores.
-      Some(ExistingClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 8,
+      Some(SourceClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 8,
         numExecsPerNode = 1, numExecutors = 2, numWorkerNodes = 2, executorHeapMemory = 0L,
         dynamicAllocationEnabled = false, "N/A", "N/A", "N/A", driverHost = Some("10.10.10.100"))),
     "eventlog_3nodes_12cores_multiple_executors" -> // 3 nodes, each with 2 executors having 12 cores.
-      Some(ExistingClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 12,
+      Some(SourceClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 12,
         numExecsPerNode = -1, numExecutors = 4, numWorkerNodes = 3, executorHeapMemory = 0L,
         dynamicAllocationEnabled = false, "N/A", "N/A", "N/A", driverHost = Some("10.59.184.210"))),
     "eventlog_4nodes_8cores_dynamic_alloc.zstd" -> // using dynamic allocation, total of 5 nodes, each with max 7
       // executor running having 4 cores. At the end it had 1 active executor.
-      Some(ExistingClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 4,
+      Some(SourceClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 4,
         numExecsPerNode = -1, numExecutors = 7, numWorkerNodes = 5, executorHeapMemory = 20480L,
         dynamicAllocationEnabled = true, dynamicAllocationMaxExecutors = "2147483647",
         dynamicAllocationMinExecutors = "0", dynamicAllocationInitialExecutors = "2",
         driverHost = Some("10.10.6.9"))),
     "eventlog_3nodes_12cores_variable_cores" -> // 3 nodes with varying cores: 8, 12, and 8, each with 1 executor.
-      Some(ExistingClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 12,
+      Some(SourceClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 12,
         numExecsPerNode = 1, numExecutors = 3, numWorkerNodes = 3, executorHeapMemory = 0L,
         dynamicAllocationEnabled = false, "N/A", "N/A", "N/A", driverHost = Some("10.10.10.100"))),
     "eventlog_3nodes_12cores_exec_removed" -> // 2 nodes, each with 1 executor having 12 cores, 1 executor removed.
-      Some(ExistingClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 12,
+      Some(SourceClusterInfo(vendor = PlatformNames.DEFAULT, coresPerExecutor = 12,
         numExecsPerNode = 1, numExecutors = 2, numWorkerNodes = 2, executorHeapMemory = 0L,
         dynamicAllocationEnabled = false, "N/A", "N/A", "N/A", driverHost = Some("10.10.10.100"))),
     "eventlog_driver_only" -> None // Event log with driver only
@@ -1627,9 +1625,9 @@ class QualificationSuite extends BaseTestSuite {
   }
 
   // Expected results as a map of platform -> cluster info.
-  val expectedPlatformClusterInfoMap: Seq[(String, ExistingClusterInfo)] = Seq(
+  val expectedPlatformClusterInfoMap: Seq[(String, SourceClusterInfo)] = Seq(
     PlatformNames.DATABRICKS_AWS ->
-        ExistingClusterInfo(vendor = PlatformNames.DATABRICKS_AWS,
+        SourceClusterInfo(vendor = PlatformNames.DATABRICKS_AWS,
           coresPerExecutor = 8,
           numExecsPerNode = 1,
           numExecutors = 2,
@@ -1643,7 +1641,7 @@ class QualificationSuite extends BaseTestSuite {
           clusterId = Some("1212-214324-test"),
           clusterName = Some("test-db-aws-cluster")),
     PlatformNames.DATABRICKS_AZURE ->
-      ExistingClusterInfo(vendor = PlatformNames.DATABRICKS_AZURE,
+      SourceClusterInfo(vendor = PlatformNames.DATABRICKS_AZURE,
         coresPerExecutor = 8,
         numExecsPerNode = 1,
         numExecutors = 2,
@@ -1657,7 +1655,7 @@ class QualificationSuite extends BaseTestSuite {
         clusterId = Some("1212-214324-test"),
         clusterName = Some("test-db-azure-cluster")),
     PlatformNames.DATAPROC ->
-      ExistingClusterInfo(vendor = PlatformNames.DATAPROC,
+      SourceClusterInfo(vendor = PlatformNames.DATAPROC,
         coresPerExecutor = 8,
         numExecsPerNode = 1,
         numExecutors = 2,
@@ -1667,7 +1665,7 @@ class QualificationSuite extends BaseTestSuite {
         "N/A", "N/A", "N/A",
         driverHost = Some("dataproc-test-m.c.internal")),
     PlatformNames.EMR ->
-      ExistingClusterInfo(vendor = PlatformNames.EMR,
+      SourceClusterInfo(vendor = PlatformNames.EMR,
         coresPerExecutor = 8,
         numExecsPerNode = 1,
         numExecutors = 2,
@@ -1678,7 +1676,7 @@ class QualificationSuite extends BaseTestSuite {
         driverHost = Some("10.10.10.100"),
         clusterId = Some("j-123AB678XY321")),
     PlatformNames.ONPREM ->
-      ExistingClusterInfo(vendor = PlatformNames.ONPREM,
+      SourceClusterInfo(vendor = PlatformNames.ONPREM,
         coresPerExecutor = 8,
         numExecsPerNode = 1,
         numExecutors = 2,
@@ -1700,7 +1698,7 @@ class QualificationSuite extends BaseTestSuite {
    * Runs the qualification tool and verifies cluster information against expected values.
    */
   private def runQualificationAndTestClusterInfo(eventlogPath: String, platform: String,
-      expectedClusterInfo: Option[ExistingClusterInfo]): Unit = {
+      expectedClusterInfo: Option[SourceClusterInfo]): Unit = {
     TrampolineUtil.withTempDir { outPath =>
       val baseArgs = Array("--output-directory", outPath.getAbsolutePath, "--platform", platform)
       val appArgs = new QualificationArgs(baseArgs :+ eventlogPath)
@@ -1708,16 +1706,11 @@ class QualificationSuite extends BaseTestSuite {
       assert(exitCode == 0 && result.size == 1,
         "Qualification tool returned unexpected results.")
 
-      // Read JSON as [{'appId': 'app-id-1', ..}]
-      def readJson(path: String): Array[ClusterSummary] = {
-        val mapper = new ObjectMapper().registerModule(DefaultScalaModule)
-        mapper.readValue(new File(path), classOf[Array[ClusterSummary]])
-      }
-
       // Read output JSON and create a set of (event log, cluster info)
       val outputResultFile = s"$outPath/${QualOutputWriter.LOGFILE_NAME}/" +
         s"${QualOutputWriter.LOGFILE_NAME}_cluster_information.json"
-      val actualClusterInfo = readJson(outputResultFile).headOption.flatMap(_.clusterInfo)
+      val actualClusterInfo = ToolTestUtils.loadClusterSummaryFromJson(outputResultFile)
+        .headOption.flatMap(_.sourceClusterInfo)
       assert(actualClusterInfo == expectedClusterInfo,
         "Actual cluster info does not match the expected cluster info.")
     }
diff --git a/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala b/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
index 2ebeaf0cd..b5f3dfcd1 100644
--- a/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
+++ b/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
@@ -20,14 +20,14 @@ import java.io.{File, FileNotFoundException}
 
 import scala.collection.mutable
 
-import com.nvidia.spark.rapids.tool.{A100Gpu, AppSummaryInfoBaseProvider, GpuDevice, PlatformFactory, PlatformNames, T4Gpu, ToolTestUtils}
+import com.nvidia.spark.rapids.tool.{A100Gpu, AppSummaryInfoBaseProvider, GpuDevice, Platform, PlatformFactory, PlatformNames, T4Gpu, ToolTestUtils}
 import com.nvidia.spark.rapids.tool.planparser.DatabricksParseHelper
 import com.nvidia.spark.rapids.tool.profiling.{DriverLogUnsupportedOperators, ProfileArgs, ProfileMain, Profiler}
 import org.scalatest.prop.TableDrivenPropertyChecks._
 import org.scalatest.prop.TableFor4
 
 import org.apache.spark.sql.TrampolineUtil
-import org.apache.spark.sql.rapids.tool.util.{FSUtils, WebCrawlerUtil}
+import org.apache.spark.sql.rapids.tool.util.{FSUtils, PropertiesLoader, WebCrawlerUtil}
 
 /**
  * Suite to test the Profiling Tool's AutoTuner
@@ -106,8 +106,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
       logEventsProps, Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
   }
@@ -130,8 +129,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       mutable.Map("spark.rapids.sql.enabled" -> "true",
         "spark.plugins" -> "com.nvidia.spark.AnotherPlugin, com.nvidia.spark.SQLPlugin"),
       Some("3.2.0"), Seq())
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -152,7 +150,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -231,8 +229,8 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
   }
 
   test("Load cluster properties with CPU cores 0") {
-    val dataprocWorkerInfo = buildGpuWorkerInfoAsString(None, Some(0))
-    val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, getGpuAppMockInfoProvider)
+    val workerInfo = buildGpuWorkerInfoAsString(None, Some(0))
+    val autoTuner = buildAutoTunerForTests(workerInfo, getGpuAppMockInfoProvider)
     val (properties, comments) = autoTuner.getRecommendedProperties()
     val autoTunerOutput = Profiler.getAutoTunerResultsAsString(properties, comments)
     // scalastyle:off line.size.limit
@@ -266,8 +264,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
 
   test("Load cluster properties with memory to cores ratio to small") {
     val workerInfo = buildGpuWorkerInfoAsString(None, Some(8), Some("14000MiB"))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(workerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.ONPREM, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(workerInfo,
         getGpuAppMockInfoProvider, platform)
@@ -321,8 +318,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
 
   test("Load cluster properties with CPU memory 0") {
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(None, Some(32), Some("0m"))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -344,7 +340,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -388,8 +384,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
 
   test("Load cluster properties with number of workers 0") {
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(None, Some(32), Some("122880MiB"), Some(0))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -411,7 +406,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -461,14 +456,13 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.executor.memory" -> "32768m",
       "spark.executor.memoryOverhead" -> "7372m",
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.sql.files.maxPartitionBytes" -> "512m",
       "spark.task.resource.gpu.amount" -> "0.001")
     val sparkProps = defaultDataprocProps.++(customProps)
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(sparkProps), Some(32),
       Some("122880MiB"), Some(4), Some(0))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -530,14 +524,13 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.sql.files.maxPartitionBytes" -> "512m",
       "spark.task.resource.gpu.amount" -> "0.001")
     val sparkProps = defaultDataprocProps.++(customProps)
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(sparkProps), Some(32),
       Some("122880MiB"), Some(4), Some(2), None)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -591,7 +584,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "16",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "16",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -602,8 +595,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val sparkProps = defaultDataprocProps.++(customProps)
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(sparkProps), Some(32),
       Some("122880MiB"), Some(4), Some(2), Some("0M"))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -653,7 +645,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "16",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "16",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -662,8 +654,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val sparkProps = defaultDataprocProps.++(customProps)
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(sparkProps), Some(32),
       Some("122880MiB"), Some(4), Some(2), Some("0MiB"), None)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -718,7 +709,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "16",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "16",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -727,8 +718,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val sparkProps = defaultDataprocProps.++(customProps)
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(sparkProps), Some(32),
       Some("122880MiB"), Some(4), Some(2), Some("0MiB"), Some("GPU-X"))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -776,11 +766,11 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
 
   // This test should ignore executor memory specified in the properties as
   // CSP nodes have fixed memory configurations.
+  // TODO: Revisit this test
   test("Test executor memory on CSP where executor memory/cpu ratio is small") {
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(None, Some(8), Some("15360MiB"),
       Some(4), Some(1))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -802,7 +792,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -852,7 +842,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "16",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "16",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -895,8 +885,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |- ${ProfilingAutoTunerConfigsProvider.classPathComments("rapids.shuffle.jars")}
           |""".stripMargin
     // scalastyle:on line.size.limit
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -916,7 +905,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "16",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "16",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -960,8 +949,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |- ${ProfilingAutoTunerConfigsProvider.classPathComments("rapids.shuffle.jars")}
           |""".stripMargin
     // scalastyle:on line.size.limit
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -988,7 +976,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1027,8 +1015,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |- ${ProfilingAutoTunerConfigsProvider.classPathComments("rapids.shuffle.jars")}
           |""".stripMargin
     // scalastyle:on line.size.limit
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
       getGpuAppMockInfoProvider, platform)
@@ -1046,7 +1033,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.executor.memory" -> "47222m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "8",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "8",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -1072,8 +1059,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(8126464.0, Seq(0), Seq(0.004), logEventsProps,
       Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1093,7 +1079,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1131,7 +1117,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.executor.memory" -> "47222m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "8",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "8",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -1155,8 +1141,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(8126464.0, Seq(0), Seq(0.004), logEventsProps,
       Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1176,7 +1161,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1209,7 +1194,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -1227,8 +1212,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
         "spark.rapids.sql.concurrentGpuTasks" -> "4")
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val infoProvider = getMockInfoProvider(8126464.0, Seq(0), Seq(0.004), logEventsProps,
       Some(testSparkVersion))
@@ -1250,7 +1234,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -1293,7 +1277,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -1314,8 +1298,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(8126464.0, Seq(0), Seq(0.004), logEventsProps,
       Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1335,7 +1318,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -1376,7 +1359,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -1401,8 +1384,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
         "spark.rapids.sql.concurrentGpuTasks" -> "1")
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
         getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.01, 0.0), logEventsProps,
@@ -1424,7 +1406,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1461,7 +1443,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -1488,8 +1470,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1509,7 +1490,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1568,8 +1549,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1589,7 +1569,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1627,7 +1607,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.rapids.memory.pinnedPool.size" -> "4096m",
       "spark.rapids.shuffle.multiThreaded.reader.threads" -> "16",
       "spark.rapids.shuffle.multiThreaded.writer.threads" -> "16",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.rapids.sql.multiThreadedRead.numThreads" -> "20",
       "spark.shuffle.manager" ->
         s"com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager",
@@ -1635,8 +1615,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       "spark.task.resource.gpu.amount" -> "0.001")
     val sparkProps = defaultDataprocProps.++(customProps)
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(sparkProps))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo,
         getGpuAppMockInfoWithJars(rapidsJars), platform)
@@ -1808,8 +1787,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testSparkVersion), Seq(), 40.0, 200000000000L)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1830,7 +1808,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1890,8 +1868,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testSparkVersion), Seq(), 40.0, 2000000L)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -1911,7 +1888,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -1943,8 +1920,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
 
   test("test recommendations for databricks-aws platform argument") {
     val databricksWorkerInfo = buildGpuWorkerInfoAsString()
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(databricksWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(databricksWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(databricksWorkerInfo,
         getGpuAppMockInfoProvider, platform)
@@ -1991,8 +1967,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       val dataprocWorkerInfo = buildGpuWorkerInfoAsString()
       val infoProvider = getMockInfoProvider(8126464.0, Seq(0),
         Seq(0.004), logEventsProps, Some(testSparkVersion))
-      val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-        .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+      val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
       val platform = PlatformFactory.createInstance(PlatformNames.ONPREM, clusterPropsOpt)
       val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform,
         sparkMaster)
@@ -2013,7 +1988,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     val unsupportedDriverOperators = Seq(
       DriverLogUnsupportedOperators(
@@ -2025,8 +2000,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     )
     val driverInfoProvider = DriverInfoProviderMockTest(unsupportedDriverOperators)
     val workerInfo = buildGpuWorkerInfoAsString(Some(customProps))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(workerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DEFAULT, clusterPropsOpt)
     val autoTuner: AutoTuner = ProfilingAutoTunerConfigsProvider
       .buildAutoTunerFromProps(workerInfo,
@@ -2064,8 +2038,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     )
     val driverInfoProvider = DriverInfoProviderMockTest(unsupportedDriverOperators)
     val workerInfo = buildGpuWorkerInfoAsString(Some(customProps))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(workerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DEFAULT, clusterPropsOpt)
     val autoTuner: AutoTuner = ProfilingAutoTunerConfigsProvider
       .buildAutoTunerFromProps(workerInfo,
@@ -2117,11 +2090,10 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     val workerInfo = buildGpuWorkerInfoAsString(Some(customProps))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(workerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DEFAULT, clusterPropsOpt)
     val autoTuner: AutoTuner = ProfilingAutoTunerConfigsProvider
       .buildAutoTunerFromProps(workerInfo,
@@ -2142,7 +2114,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     val unsupportedDriverOperators = Seq(
       DriverLogUnsupportedOperators(
@@ -2151,8 +2123,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     )
     val driverInfoProvider = DriverInfoProviderMockTest(unsupportedDriverOperators)
     val workerInfo = buildGpuWorkerInfoAsString(Some(customProps))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(workerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DEFAULT, clusterPropsOpt)
     val autoTuner: AutoTuner = ProfilingAutoTunerConfigsProvider
       .buildAutoTunerFromProps(workerInfo,
@@ -2172,7 +2143,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -2196,8 +2167,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(8126464.0, Seq(0), Seq(0.004), logEventsProps,
       Some(testSparkVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -2217,7 +2187,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -2256,7 +2226,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -2277,9 +2247,8 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(gpuDevice.getMemory), Some(gpuDevice.toString))
     val infoProvider = getMockInfoProvider(8126464.0, Seq(0), Seq(0.004), logEventsProps,
       Some(testSparkVersion), meanInput = inputSize, meanShuffleRead = shuffleRead)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
-    val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
+    val platform = PlatformFactory.createInstance(PlatformNames.ONPREM, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
     val autoTunerOutput = Profiler.getAutoTunerResultsAsString(properties, comments)
@@ -2339,14 +2308,15 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
         "spark.rapids.sql.enabled" -> "true",
         "spark.plugins" -> "com.nvidia.spark.SQLPlugin",
         "spark.rapids.sql.concurrentGpuTasks" -> "1")
-    val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
+    val databricksWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testDatabricksVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
-    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS, clusterPropsOpt)
-    val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(
+      databricksWorkerInfo)
+    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS,
+      clusterProperties = clusterPropsOpt)
+    val autoTuner = buildAutoTunerForTests(databricksWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
     val autoTunerOutput = Profiler.getAutoTunerResultsAsString(properties, comments)
     // scalastyle:off line.size.limit
@@ -2354,16 +2324,16 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       s"""|
           |Spark Properties:
           |--conf spark.databricks.adaptive.autoOptimizeShuffle.enabled=false
-          |--conf spark.executor.memory=32g
+          |--conf spark.executor.memory=64g
           |--conf spark.locality.wait=0
           |--conf spark.rapids.memory.pinnedPool.size=4g
           |--conf spark.rapids.shuffle.multiThreaded.maxBytesInFlight=4g
-          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
-          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
+          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=48
+          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=48
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
-          |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
+          |--conf spark.rapids.sql.multiThreadedRead.numThreads=64
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
           |--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark$testSmVersionDatabricks.RapidsShuffleManager
           |--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128m
@@ -2403,32 +2373,37 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     }
   }
 
-  test("test shuffle manager version for supported databricks version") {
-    val databricksVersion = "11.3.x-gpu-ml-scala2.12"
-    val databricksWorkerInfo = buildGpuWorkerInfoAsString(None)
-    val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
-      mutable.Map("spark.rapids.sql.enabled" -> "true",
-        "spark.plugins" -> "com.nvidia.spark.AnotherPlugin, com.nvidia.spark.SQLPlugin",
-        DatabricksParseHelper.PROP_TAG_CLUSTER_SPARK_VERSION_KEY -> databricksVersion),
-      Some(databricksVersion), Seq())
-    // Do not set the platform as DB to see if it can work correctly irrespective
-    val autoTuner = buildAutoTunerForTests(databricksWorkerInfo,
+  val dbPlatform: Platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS)
+  dbPlatform.supportedShuffleManagerVersionMap.foreach { case (dbVersion, smVersion) =>
+    test(s"test shuffle manager version for supported databricks version - $dbVersion") {
+      val databricksVersion = s"$dbVersion.x-gpu-ml-scala2.12"
+      val databricksWorkerInfo = buildGpuWorkerInfoAsString(None)
+      val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
+        mutable.Map("spark.rapids.sql.enabled" -> "true",
+          "spark.plugins" -> "com.nvidia.spark.AnotherPlugin, com.nvidia.spark.SQLPlugin",
+          DatabricksParseHelper.PROP_TAG_CLUSTER_SPARK_VERSION_KEY -> databricksVersion),
+        Some(databricksVersion), Seq())
+      // Do not set the platform as DB to see if it can work correctly irrespective
+      val autoTuner = buildAutoTunerForTests(databricksWorkerInfo,
         infoProvider, PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS))
-    // Assert shuffle manager string for DB 11.3 tag
-    verifyRecommendedShuffleManagerVersion(autoTuner, expectedSmVersion = "330db")
+      // Assert shuffle manager string for given Databricks version
+      verifyRecommendedShuffleManagerVersion(autoTuner, expectedSmVersion = smVersion)
+    }
   }
 
-  test("test shuffle manager version for supported spark version") {
-    val sparkVersion = "3.3.0"
-    val workerInfo = buildGpuWorkerInfoAsString(None)
-    val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
-      mutable.Map("spark.rapids.sql.enabled" -> "true",
-        "spark.plugins" -> "com.nvidia.spark.AnotherPlugin, com.nvidia.spark.SQLPlugin"),
-      Some(sparkVersion), Seq())
-    val autoTuner = buildAutoTunerForTests(workerInfo,
+  val sparkPlatform: Platform = PlatformFactory.createInstance(PlatformNames.DEFAULT)
+  sparkPlatform.supportedShuffleManagerVersionMap.foreach { case (sparkVersion, smVersion) =>
+    test(s"test shuffle manager version for supported spark version - $sparkVersion") {
+      val workerInfo = buildGpuWorkerInfoAsString(None)
+      val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
+        mutable.Map("spark.rapids.sql.enabled" -> "true",
+          "spark.plugins" -> "com.nvidia.spark.AnotherPlugin, com.nvidia.spark.SQLPlugin"),
+        Some(sparkVersion), Seq())
+      val autoTuner = buildAutoTunerForTests(workerInfo,
         infoProvider, PlatformFactory.createInstance())
-    // Assert shuffle manager string for supported Spark v3.3.0
-    verifyRecommendedShuffleManagerVersion(autoTuner, expectedSmVersion = "330")
+      // Assert shuffle manager string for given Spark version
+      verifyRecommendedShuffleManagerVersion(autoTuner, expectedSmVersion = smVersion)
+    }
   }
 
   test("test shuffle manager version for supported custom spark version") {
@@ -2519,7 +2494,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -2543,8 +2518,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(1000L, 1000L), Seq(0.4, 0.4),
       logEventsProps, Some(testSparkVersion), shuffleStagesWithPosSpilling = Set(1))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -2564,7 +2538,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -2602,7 +2576,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val customProps = mutable.LinkedHashMap(
       "spark.executor.cores" -> "8",
       "spark.executor.memory" -> "47222m",
-      "spark.rapids.sql.concurrentGpuTasks" -> "2",
+      "spark.rapids.sql.concurrentGpuTasks" -> "3",
       "spark.task.resource.gpu.amount" -> "0.001")
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -2627,8 +2601,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(1000L, 1000L), Seq(0.4, 0.4),
       logEventsProps, Some(testSparkVersion), shuffleStagesWithPosSpilling = Set(1, 5),
       shuffleSkewStages = Set(1))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -2648,7 +2621,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -2714,7 +2687,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -2790,7 +2763,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -2866,7 +2839,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -2924,8 +2897,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
       logEventsProps, Some("3.4.1-amzn-1"))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(emrWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(emrWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.EMR, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(emrWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -2944,7 +2916,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -3009,7 +2981,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -3079,7 +3051,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.enabled=true
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
@@ -3137,8 +3109,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
       val dataprocWorkerInfo = buildGpuWorkerInfoAsString()
       val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
         logEventsProps, Some(testSparkVersion), scanStagesWithGpuOom = hasGpuOOm)
-      val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-        .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+      val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
       val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
       val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
       val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -3158,7 +3129,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
             |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
             |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
             |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-            |--conf spark.rapids.sql.concurrentGpuTasks=2
+            |--conf spark.rapids.sql.concurrentGpuTasks=3
             |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
             |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
             |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
@@ -3253,7 +3224,9 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     }
   }
 
-  test(s"test AutoTuner recommends increasing shuffle partition when shuffle stages " +
+  // TODO: Revisit this test once optimal tuning for AQE for dataproc on g2 instances is determined
+  //       See https://github.com/NVIDIA/spark-rapids-tools/issues/1682
+  ignore(s"test AutoTuner recommends increasing shuffle partition when shuffle stages " +
     s"have OOM failures") {
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -3272,8 +3245,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val dataprocWorkerInfo = buildGpuWorkerInfoAsString()
     val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
       logEventsProps, Some(testSparkVersion), shuffleStagesWithOom = true)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
@@ -3330,7 +3302,9 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     compareOutput(expectedResults, autoTunerOutput)
   }
 
-  test(s"test AutoTuner recommends increasing shuffle partition and AQE initial partition num " +
+  // TODO: Revisit this test once optimal tuning for AQE for dataproc on g2 instances is determined
+  //       See https://github.com/NVIDIA/spark-rapids-tools/issues/1682
+  ignore(s"test AutoTuner recommends increasing shuffle partition and AQE initial partition num " +
     s"when shuffle stages have OOM failures") {
     // mock the properties loaded from eventLog
     val logEventsProps: mutable.Map[String, String] =
@@ -3350,8 +3324,7 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
     val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
       logEventsProps, Some(testSparkVersion), shuffleStagesWithOom = true,
       meanInput = 50000, meanShuffleRead = 80000)
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(dataprocWorkerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.DATAPROC, clusterPropsOpt)
     val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
diff --git a/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/QualificationAutoTunerSuite.scala b/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/QualificationAutoTunerSuite.scala
index 6ceea9471..69605d902 100644
--- a/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/QualificationAutoTunerSuite.scala
+++ b/core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/QualificationAutoTunerSuite.scala
@@ -23,6 +23,8 @@ import com.nvidia.spark.rapids.tool.profiling.Profiler
 import org.scalatest.prop.TableDrivenPropertyChecks._
 import org.scalatest.prop.TableFor3
 
+import org.apache.spark.sql.rapids.tool.util.PropertiesLoader
+
 /**
  * Suite to test the Qualification Tool's AutoTuner
  */
@@ -62,8 +64,7 @@ class QualificationAutoTunerSuite extends BaseAutoTunerSuite {
       Some("212992MiB"), Some(5))
     val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
       logEventsProps, Some(testSparkVersion))
-    val clusterPropsOpt = QualificationAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(workerInfo)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
     val platform = PlatformFactory.createInstance(PlatformNames.EMR, clusterPropsOpt)
     buildAutoTunerForTests(workerInfo, infoProvider, platform)
   }
@@ -149,8 +150,7 @@ class QualificationAutoTunerSuite extends BaseAutoTunerSuite {
           "spark.executor.memory" -> "8g",
           "spark.executor.memoryOverhead" -> "2g"
         )
-      val clusterPropsOpt = QualificationAutoTunerConfigsProvider
-        .loadClusterPropertiesFromContent(workerInfo)
+      val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(workerInfo)
       val infoProvider = getMockInfoProvider(0, Seq(0), Seq(0.0),
         logEventsProps, Some(testSparkVersion))
       val platform = PlatformFactory.createInstance(PlatformNames.ONPREM, clusterPropsOpt)
diff --git a/user_tools/src/spark_rapids_pytools/resources/profiling-conf.yaml b/user_tools/src/spark_rapids_pytools/resources/profiling-conf.yaml
index b183a332f..43aae5a9d 100644
--- a/user_tools/src/spark_rapids_pytools/resources/profiling-conf.yaml
+++ b/user_tools/src/spark_rapids_pytools/resources/profiling-conf.yaml
@@ -1,4 +1,4 @@
-# Copyright (c) 2024, NVIDIA CORPORATION.
+# Copyright (c) 2024-2025, NVIDIA CORPORATION.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -52,6 +52,7 @@ sparkRapids:
       - s
       - start-app-time
       - t
+      - target-cluster-info
       - timeout
 local:
   output:
diff --git a/user_tools/tests/spark_rapids_tools_e2e/features/installation_checks.feature b/user_tools/tests/spark_rapids_tools_e2e/features/installation_checks.feature
index 719e5b7ae..549723eef 100644
--- a/user_tools/tests/spark_rapids_tools_e2e/features/installation_checks.feature
+++ b/user_tools/tests/spark_rapids_tools_e2e/features/installation_checks.feature
@@ -28,10 +28,10 @@ Feature: Tool Installation Checks
 
     Examples:
       | platform         | cli    | expected_stdout                 |
-      | dataproc         | gcloud | 2 x n1-standard-16 (1 T4 each)  |
+      | dataproc         | gcloud | 2 x g2-standard-16 (1 L4 each)  |
       | emr              | aws    | 2 x g6.4xlarge                  |
-      | databricks-aws   | aws    | 2 x g5.4xlarge                  |
-      | databricks-azure | az     | 2 x Standard_NC16as_T4_v3       |
+      | databricks-aws   | aws    | 1 x g5.8xlarge                  |
+      | databricks-azure | az     | 4 x Standard_NC8as_T4_v3        |
 
   @test_id_IC_0002
   Scenario: Environment has missing java


Reviews:
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T23:54:08Z
user: login: amahussein
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T21:06:16Z
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T07:29:22Z
user: login: parthosa
body: @amahussein Based on our offline discussion, added the following changes - 

- [x] Remove warning for missing `WorkerInfo`
- [x] Ensure default instances match those used by QualX models. Updated unit tests for this
- [x] Create a internal issue to track documentation updates for this PR
- [x] Rename `ExistingClusterInfo` to `SourceClusterInfo` 
- [x] Create an issue to automate generation of supported shims for RAPIDS Shuffle Manager [#1676](https://github.com/NVIDIA/spark-rapids-tools/issues/1676)
- [x] Sync with @viadea on usability experience. Started discussion on internal thread.


#### Default instances recommended (aligned with instances used by QualX):

- Dataproc - `g2-standard-16`  (L4)
- EMR - `g6.4xlarge`  (L4)
- DB AWS - `g5.8xlarge`  (A10g)
- DB Azure - `Standard_NC8as_T4_v3`  (T4)
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-16T08:13:06Z
user: login: cindyyuanjiang
body: Thanks @parthosa! Some minor nits and questions.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-17T00:32:24Z
user: login: amahussein
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T15:21:39Z
user: login: amahussein
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T16:16:39Z
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T17:03:29Z
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T18:17:37Z
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T18:43:04Z
user: login: amahussein
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T21:47:53Z
user: login: amahussein
body: Thanks @parthosa !
I am skeptical about the PR in general.

One thing later we want to watch for is that adding new machine type should not alter existing unit tests because the existing unit tests are supposed to match the behavior of other node-types. so, it is either have expectations loaded depending on the node-instance or that we do not verify non-default node-types.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-19T21:52:34Z
user: login: cindyyuanjiang
body: Thanks @parthosa! One quick question, otherwise LGTM.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-19T22:09:20Z
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T22:43:12Z
user: login: cindyyuanjiang
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T22:53:50Z
user: login: parthosa
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T22:58:01Z

Review comments:
diff_hunk: @@ -191,7 +283,12 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
     "3.4.2" -> "342",
     "3.4.3" -> "343",
     "3.5.0" -> "350",
-    "3.5.1" -> "351"
+    "3.5.1" -> "351",
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: parthosa
body: Updated list of supported shuffle managers ([link](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/rapids-shuffle.html#supported-spark-versions))
created_at: 2025-05-14T23:11:22Z
author_association: COLLABORATOR
diff_hunk: @@ -191,7 +283,12 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
     "3.4.2" -> "342",
     "3.4.3" -> "343",
     "3.5.0" -> "350",
-    "3.5.1" -> "351"
+    "3.5.1" -> "351",
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: amahussein
body: We should create an issue to generate the supported shims as part of the generated files. Then it can be part of the syncPlugin jenkins job.
Easiest way is to piggyback on the sync-plugin python scripts to dump all the directoreis (which represent the shims except for DB). A cleaner way would be to modify the plugin repo to generate a separate resource file.
created_at: 2025-05-15T19:28:44Z
author_association: COLLABORATOR
diff_hunk: @@ -51,6 +51,7 @@ sealed trait ClusterInfo {
 // Information about the cluster used with to run the application we
 // are qualifying or profiling. This is compared to what we might recommend
 // for a cluster.
+// TODO: Add information about the existing GPUs on the cluster
path: core/src/main/scala/org/apache/spark/sql/rapids/tool/ClassWarehouse.scala
user: login: amahussein
body: should we also rename the name of the class `ExistingClusterInfo`? 
created_at: 2025-05-15T19:36:54Z
author_association: COLLABORATOR
diff_hunk: @@ -633,17 +712,22 @@ class EmrPlatform(gpuDevice: Option[GpuDevice],
       driverHost = driverHost)
   }
 
-  override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
-    PlatformInstanceTypes.EMR_BY_GPUS_CORES
+  override def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = {
+    PlatformInstanceTypes.EMR_BY_INSTANCE_NAME
   }
 }
 
 class OnPremPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties]) extends Platform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends Platform(gpuDevice, clusterProperties, targetCluster) {
+  // TODO: Add support for providing target cluster properties for OnPrem
+  require(targetCluster.isEmpty,
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: amahussein
body: require works like an assert. do we want to halt the execution if the conditions are not satisfied?
created_at: 2025-05-15T19:44:59Z
author_association: COLLABORATOR
diff_hunk: @@ -191,7 +283,12 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
     "3.4.2" -> "342",
     "3.4.3" -> "343",
     "3.5.0" -> "350",
-    "3.5.1" -> "351"
+    "3.5.1" -> "351",
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: parthosa
body: Created issue - https://github.com/NVIDIA/spark-rapids-tools/issues/1676 for automating generation of supported shims
created_at: 2025-05-16T07:29:22Z
author_association: COLLABORATOR
diff_hunk: @@ -51,6 +51,7 @@ sealed trait ClusterInfo {
 // Information about the cluster used with to run the application we
 // are qualifying or profiling. This is compared to what we might recommend
 // for a cluster.
+// TODO: Add information about the existing GPUs on the cluster
path: core/src/main/scala/org/apache/spark/sql/rapids/tool/ClassWarehouse.scala
user: login: parthosa
body: Renamed to `SourceClusterInfo`
created_at: 2025-05-16T07:29:50Z
author_association: COLLABORATOR
diff_hunk: @@ -633,17 +712,22 @@ class EmrPlatform(gpuDevice: Option[GpuDevice],
       driverHost = driverHost)
   }
 
-  override def getInstanceByResourcesMap: Map[(Int, Int), InstanceInfo] = {
-    PlatformInstanceTypes.EMR_BY_GPUS_CORES
+  override def getInstanceMapByName: Map[NodeInstanceMapKey, InstanceInfo] = {
+    PlatformInstanceTypes.EMR_BY_INSTANCE_NAME
   }
 }
 
 class OnPremPlatform(gpuDevice: Option[GpuDevice],
-    clusterProperties: Option[ClusterProperties]) extends Platform(gpuDevice, clusterProperties) {
+    clusterProperties: Option[ClusterProperties],
+    targetCluster: Option[TargetClusterProps])
+  extends Platform(gpuDevice, clusterProperties, targetCluster) {
+  // TODO: Add support for providing target cluster properties for OnPrem
+  require(targetCluster.isEmpty,
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: parthosa
body: Discussed offline — this approach is acceptable as the target cluster info is shared across all event logs. Hence, if the user provides this information for onprem, it is safe to halt execution.
created_at: 2025-05-16T07:32:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.nvidia.spark.rapids.tool.tuning
+
+import com.nvidia.spark.rapids.tool.NodeInstanceMapKey
+import scala.beans.BeanProperty
+
+class WorkerInfo(
+  @BeanProperty var instanceType: String,
+  @BeanProperty var gpu: GpuWorkerProps) {
path: core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/TargetClusterProps.scala
user: login: cindyyuanjiang
body: Question: is there a reason for using `BeanProperty` here?
created_at: 2025-05-16T23:15:05Z
author_association: COLLABORATOR
diff_hunk: @@ -54,9 +54,24 @@ object PlatformNames {
 
 case class DynamicAllocationInfo(enabled: Boolean, max: String, min: String, initial: String)
 
+/**
+ * Case class representing the key for the node instance map.
+ * @param instanceType Name of the instance type
+ * @param gpuCount Count of GPUs on the instance. This is required for certain
+ *                 cases where a given instance type can have multiple GPU counts
+ *                 (e.g. n1-standard-16 with 1, 2, or 4 GPUs).
+ */
+case class    NodeInstanceMapKey(instanceType: String, gpuCount: Option[Int] = None) {
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: cindyyuanjiang
body: nit: extra spaces here?
created_at: 2025-05-17T00:04:48Z
author_association: COLLABORATOR
diff_hunk: @@ -67,73 +82,120 @@ case class InstanceInfo(cores: Int, memoryMB: Long, name: String, numGpus: Int)
 }
 
 object InstanceInfo {
-    def createDefaultInstance(cores: Int, memoryMB: Long, numGpus: Int): InstanceInfo = {
-      InstanceInfo(cores, memoryMB, "N/A", numGpus)
-    }
+  def createDefaultInstance(cores: Int, memoryMB: Long,
+      numGpus: Int, gpuDevice: GpuDevice): InstanceInfo = {
+    InstanceInfo(cores, memoryMB, "N/A", numGpus, gpuDevice)
+  }
 }
 
-// This is meant to be temporary mapping to figure out instance type based
-// on the number of GPUs and cores.  Eventually this should be read from files
-// generated based on CSP instance information.
-// format (numGpus, numCores) -> InstanceInfo about that CSP node instance type
+// This is a map of the instance types to the InstanceInfo.
+// format (instance type name, gpu count) -> InstanceInfo about that CSP node instance type
 object PlatformInstanceTypes {
 
   // Using G6 instances for EMR
-  val EMR_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1),
-    (1, 8) -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1),
-    (1, 16) -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1),
-    (1, 32) -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1),
-    (4, 48) -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 1),
-    (1, 64) -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1)
+  val EMR_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("g6.xlarge") -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.2xlarge") -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.4xlarge") -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.8xlarge") -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.12xlarge") -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 4, L4Gpu),
+    NodeInstanceMapKey("g6.16xlarge") -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1, L4Gpu)
   )
 
   // Using G5 instances. To be updated once G6 availability on Databricks
   // is consistent
-  val DATABRICKS_AWS_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1),
-    (1, 8) -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1),
-    (1, 16) -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1),
-    (1, 32) -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1),
-    (4, 48) -> InstanceInfo(48, 192 * 1024, "g5.12xlarge", 1),
-    (1, 64) -> InstanceInfo(64, 256 * 1024, "g5.16xlarge", 1)
+  val DATABRICKS_AWS_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("g5.xlarge") -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.2xlarge") -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.4xlarge") -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.8xlarge") -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.12xlarge") -> InstanceInfo(48, 192 * 1024, "g5.12xlarge", 4, A10GGpu),
+    NodeInstanceMapKey("g5.16xlarge") -> InstanceInfo(64, 256 * 1024, "g5.16xlarge", 1, A10GGpu)
   )
 
   // Standard_NC4as_T4_v3 - only recommending nodes with T4's for now, add more later
-  val AZURE_NCAS_T4_V3_BY_GPUS_CORES = Map(
-    (1, 4) -> InstanceInfo(4, 28 * 1024, "Standard_NC4as_T4_v3", 1), // 1 GPU
-    (1, 8) -> InstanceInfo(8, 56 * 1024, "Standard_NC8as_T4_v3", 1), // 1 GPU
-    (1, 16) -> InstanceInfo(16, 110 * 1024, "Standard_NC16as_T4_v3", 1), // 1 GPU
-    (4, 64) -> InstanceInfo(64, 440 * 1024, "Standard_NC64as_T4_v3", 4)  // 4 GPUs
+  val AZURE_NCAS_T4_V3_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("Standard_NC4as_T4_v3") ->
+      InstanceInfo(4, 28 * 1024, "Standard_NC4as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC8as_T4_v3") ->
+      InstanceInfo(8, 56 * 1024, "Standard_NC8as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC16as_T4_v3") ->
+      InstanceInfo(16, 110 * 1024, "Standard_NC16as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC64as_T4_v3") ->
+      InstanceInfo(64, 440 * 1024, "Standard_NC64as_T4_v3", 4, T4Gpu)  // 4 GPUs
   )
 
   // dataproc and dataproc-gke
   // Google supports 1, 2, or 4 Gpus of most types
   // added to n1-standard boxes. You may be able to add 8 v100's but we
   // are going to ignore that.
-  val DATAPROC_BY_GPUS_CORES = Map(
-    (1, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 1),
-    (1, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 1),
-    (1, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 1),
-    (1, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 1),
-    (1, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 1),
-    (1, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 1),
-    (1, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 1),
-    (1, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 1),
-    (2, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 2),
-    (2, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 2),
-    (2, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 2),
-    (2, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 2),
-    (2, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 2),
-    (2, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 2),
-    (2, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 2),
-    (2, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 2),
-    (4, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 4),
-    (4, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 4),
-    (4, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 4),
-    (4, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 4),
-    (4, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 4),
-    (4, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 4),
-    (4, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 4),
-    (4, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 4)
+  val DATAPROC_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    // g2-standard instances with Intel GPUs
+    NodeInstanceMapKey("g2-standard-4") ->
+      InstanceInfo(4, 16 * 1024, "g2-standard-4", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-8") ->
+      InstanceInfo(8, 32 * 1024, "g2-standard-8", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-12") ->
+      InstanceInfo(12, 48 * 1024, "g2-standard-12", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-16") ->
+      InstanceInfo(16, 64 * 1024, "g2-standard-16", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-24") ->
+      InstanceInfo(24, 96 * 1024, "g2-standard-24", 2, L4Gpu),
+    NodeInstanceMapKey("g2-standard-32") ->
+      InstanceInfo(32, 128 * 1024, "g2-standard-32", 2, L4Gpu),
+    NodeInstanceMapKey("g2-standard-48") ->
+      InstanceInfo(48, 192 * 1024, "g2-standard-48", 4, L4Gpu),
+    NodeInstanceMapKey("g2-standard-96") ->
+      InstanceInfo(96, 384 * 1024, "g2-standard-96", 8, L4Gpu),
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: cindyyuanjiang
body: QQ: we only added the `g2-standard` series now. Is this map intended to include all GPU instance types?
created_at: 2025-05-17T00:09:32Z
author_association: COLLABORATOR
diff_hunk: @@ -3371,18 +3335,16 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
           |--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager
-          |--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=32m
           |--conf spark.sql.adaptive.autoBroadcastJoinThreshold=[FILL_IN_VALUE]
-          |--conf spark.sql.adaptive.coalescePartitions.initialPartitionNum=800
           |--conf spark.sql.adaptive.coalescePartitions.minPartitionSize=4m
           |--conf spark.sql.adaptive.coalescePartitions.parallelismFirst=false
           |--conf spark.sql.files.maxPartitionBytes=512m
-          |--conf spark.sql.shuffle.partitions=800
+          |--conf spark.sql.shuffle.partitions=100
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: cindyyuanjiang
body: QQ: why would the autotuner result change for this test case (and for the previous test cases too)?
created_at: 2025-05-17T00:30:32Z
author_association: COLLABORATOR
diff_hunk: @@ -26,15 +26,12 @@ import org.apache.spark.sql.rapids.tool.util.StringUtils
 case class RecommendedClusterConfig(
     numExecutors: Int,
     coresPerExec: Int,
-    numGpusPerNode: Int,
-    memoryPerNodeMb: Long // For onprem or cases where a matching CSP instance type is unavailable
+    memoryPerNodeMb: Long, // For onprem or cases where a matching CSP instance type is unavailable
+    gpuDevice: GpuDevice,
+    numGpusPerNode: Int
 ) {
-  def execsPerNode: Int = {
-    numGpusPerNode
-  }
-
   def coresPerNode: Int = {
-    coresPerExec * execsPerNode
+    coresPerExec * numGpusPerNode
path: core/src/main/scala/com/nvidia/spark/rapids/tool/ClusterConfigurationStrategy.scala
user: login: amahussein
body: This calculation sounds weird. does it assume that the number of gpus represent the number of execs?
created_at: 2025-05-19T15:21:28Z
author_association: COLLABORATOR
diff_hunk: @@ -191,7 +284,12 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
     "3.4.2" -> "342",
     "3.4.3" -> "343",
     "3.5.0" -> "350",
-    "3.5.1" -> "351"
+    "3.5.1" -> "351",
+    "3.5.2" -> "352",
+    "3.5.3" -> "353",
+    "3.5.4" -> "354",
+    "3.5.5" -> "355",
+    "4.0.0" -> "400"
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: amahussein
body: Regarding #1678, perhaps we should also fix the shim for DB in this PR as well. This will help avoid a bug sooner while #1676 is still being implemented.
created_at: 2025-05-19T16:16:36Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2025, NVIDIA CORPORATION.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.nvidia.spark.rapids.tool.tuning
+
+import com.nvidia.spark.rapids.tool.NodeInstanceMapKey
+import scala.beans.BeanProperty
+
+class WorkerInfo(
+  @BeanProperty var instanceType: String,
+  @BeanProperty var gpu: GpuWorkerProps) {
path: core/src/main/scala/com/nvidia/spark/rapids/tool/tuning/TargetClusterProps.scala
user: login: parthosa
body: Using `@BeanProperty` automatically generates getters and setters (e.g., `getInstanceType()`, `setInstanceType()`). These are required for YAML serialization/deserialization, as YAML processing library use these getter/setter methods to convert between YAML files and Scala objects.
created_at: 2025-05-19T17:03:28Z
author_association: COLLABORATOR
diff_hunk: @@ -26,15 +26,12 @@ import org.apache.spark.sql.rapids.tool.util.StringUtils
 case class RecommendedClusterConfig(
     numExecutors: Int,
     coresPerExec: Int,
-    numGpusPerNode: Int,
-    memoryPerNodeMb: Long // For onprem or cases where a matching CSP instance type is unavailable
+    memoryPerNodeMb: Long, // For onprem or cases where a matching CSP instance type is unavailable
+    gpuDevice: GpuDevice,
+    numGpusPerNode: Int
 ) {
-  def execsPerNode: Int = {
-    numGpusPerNode
-  }
-
   def coresPerNode: Int = {
-    coresPerExec * execsPerNode
+    coresPerExec * numGpusPerNode
path: core/src/main/scala/com/nvidia/spark/rapids/tool/ClusterConfigurationStrategy.scala
user: login: parthosa
body: Yes, currently plugin does not support multiple GPUs to a single executor — each executor is limited to one GPU. 

Hence, `numExecsPerNode` is same as `numGpusPerNode` (same as before). I will revert this change to make this explicit.
created_at: 2025-05-19T17:07:58Z
author_association: COLLABORATOR
diff_hunk: @@ -191,7 +284,12 @@ abstract class Platform(var gpuDevice: Option[GpuDevice],
     "3.4.2" -> "342",
     "3.4.3" -> "343",
     "3.5.0" -> "350",
-    "3.5.1" -> "351"
+    "3.5.1" -> "351",
+    "3.5.2" -> "352",
+    "3.5.3" -> "353",
+    "3.5.4" -> "354",
+    "3.5.5" -> "355",
+    "4.0.0" -> "400"
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: parthosa
body: Removed 11.3. Support for 14.3 is not enabled by default yet and requires another config to be set. Refer [comment](https://github.com/NVIDIA/spark-rapids-tools/issues/1678#issuecomment-2891901735) 
created_at: 2025-05-19T17:18:15Z
author_association: COLLABORATOR
diff_hunk: @@ -67,73 +82,120 @@ case class InstanceInfo(cores: Int, memoryMB: Long, name: String, numGpus: Int)
 }
 
 object InstanceInfo {
-    def createDefaultInstance(cores: Int, memoryMB: Long, numGpus: Int): InstanceInfo = {
-      InstanceInfo(cores, memoryMB, "N/A", numGpus)
-    }
+  def createDefaultInstance(cores: Int, memoryMB: Long,
+      numGpus: Int, gpuDevice: GpuDevice): InstanceInfo = {
+    InstanceInfo(cores, memoryMB, "N/A", numGpus, gpuDevice)
+  }
 }
 
-// This is meant to be temporary mapping to figure out instance type based
-// on the number of GPUs and cores.  Eventually this should be read from files
-// generated based on CSP instance information.
-// format (numGpus, numCores) -> InstanceInfo about that CSP node instance type
+// This is a map of the instance types to the InstanceInfo.
+// format (instance type name, gpu count) -> InstanceInfo about that CSP node instance type
 object PlatformInstanceTypes {
 
   // Using G6 instances for EMR
-  val EMR_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1),
-    (1, 8) -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1),
-    (1, 16) -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1),
-    (1, 32) -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1),
-    (4, 48) -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 1),
-    (1, 64) -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1)
+  val EMR_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("g6.xlarge") -> InstanceInfo(4, 16 * 1024, "g6.xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.2xlarge") -> InstanceInfo(8, 32 * 1024, "g6.2xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.4xlarge") -> InstanceInfo(16, 64 * 1024, "g6.4xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.8xlarge") -> InstanceInfo(32, 128 * 1024, "g6.8xlarge", 1, L4Gpu),
+    NodeInstanceMapKey("g6.12xlarge") -> InstanceInfo(48, 192 * 1024, "g6.12xlarge", 4, L4Gpu),
+    NodeInstanceMapKey("g6.16xlarge") -> InstanceInfo(64, 256 * 1024, "g6.16xlarge", 1, L4Gpu)
   )
 
   // Using G5 instances. To be updated once G6 availability on Databricks
   // is consistent
-  val DATABRICKS_AWS_BY_GPUS_CORES = Map((1, 4) -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1),
-    (1, 8) -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1),
-    (1, 16) -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1),
-    (1, 32) -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1),
-    (4, 48) -> InstanceInfo(48, 192 * 1024, "g5.12xlarge", 1),
-    (1, 64) -> InstanceInfo(64, 256 * 1024, "g5.16xlarge", 1)
+  val DATABRICKS_AWS_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("g5.xlarge") -> InstanceInfo(4, 16 * 1024, "g5.xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.2xlarge") -> InstanceInfo(8, 32 * 1024, "g5.2xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.4xlarge") -> InstanceInfo(16, 64 * 1024, "g5.4xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.8xlarge") -> InstanceInfo(32, 128 * 1024, "g5.8xlarge", 1, A10GGpu),
+    NodeInstanceMapKey("g5.12xlarge") -> InstanceInfo(48, 192 * 1024, "g5.12xlarge", 4, A10GGpu),
+    NodeInstanceMapKey("g5.16xlarge") -> InstanceInfo(64, 256 * 1024, "g5.16xlarge", 1, A10GGpu)
   )
 
   // Standard_NC4as_T4_v3 - only recommending nodes with T4's for now, add more later
-  val AZURE_NCAS_T4_V3_BY_GPUS_CORES = Map(
-    (1, 4) -> InstanceInfo(4, 28 * 1024, "Standard_NC4as_T4_v3", 1), // 1 GPU
-    (1, 8) -> InstanceInfo(8, 56 * 1024, "Standard_NC8as_T4_v3", 1), // 1 GPU
-    (1, 16) -> InstanceInfo(16, 110 * 1024, "Standard_NC16as_T4_v3", 1), // 1 GPU
-    (4, 64) -> InstanceInfo(64, 440 * 1024, "Standard_NC64as_T4_v3", 4)  // 4 GPUs
+  val AZURE_NCAS_T4_V3_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    NodeInstanceMapKey("Standard_NC4as_T4_v3") ->
+      InstanceInfo(4, 28 * 1024, "Standard_NC4as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC8as_T4_v3") ->
+      InstanceInfo(8, 56 * 1024, "Standard_NC8as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC16as_T4_v3") ->
+      InstanceInfo(16, 110 * 1024, "Standard_NC16as_T4_v3", 1, T4Gpu), // 1 GPU
+    NodeInstanceMapKey("Standard_NC64as_T4_v3") ->
+      InstanceInfo(64, 440 * 1024, "Standard_NC64as_T4_v3", 4, T4Gpu)  // 4 GPUs
   )
 
   // dataproc and dataproc-gke
   // Google supports 1, 2, or 4 Gpus of most types
   // added to n1-standard boxes. You may be able to add 8 v100's but we
   // are going to ignore that.
-  val DATAPROC_BY_GPUS_CORES = Map(
-    (1, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 1),
-    (1, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 1),
-    (1, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 1),
-    (1, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 1),
-    (1, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 1),
-    (1, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 1),
-    (1, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 1),
-    (1, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 1),
-    (2, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 2),
-    (2, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 2),
-    (2, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 2),
-    (2, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 2),
-    (2, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 2),
-    (2, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 2),
-    (2, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 2),
-    (2, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 2),
-    (4, 1) -> InstanceInfo(1, 1 * 3840, "n1-standard-1", 4),
-    (4, 2) -> InstanceInfo(2, 2 * 3840, "n1-standard-2", 4),
-    (4, 4) -> InstanceInfo(4, 4 * 3840, "n1-standard-4", 4),
-    (4, 8) -> InstanceInfo(8, 8 * 3840, "n1-standard-8", 4),
-    (4, 16) -> InstanceInfo(16, 16 * 3840, "n1-standard-16", 4),
-    (4, 32) -> InstanceInfo(32, 32 * 3840, "n1-standard-32", 4),
-    (4, 64) -> InstanceInfo(64, 64 * 3840, "n1-standard-64", 4),
-    (4, 96) -> InstanceInfo(96, 96 * 3840, "n1-standard-96", 4)
+  val DATAPROC_BY_INSTANCE_NAME: Map[NodeInstanceMapKey, InstanceInfo] = Map(
+    // g2-standard instances with Intel GPUs
+    NodeInstanceMapKey("g2-standard-4") ->
+      InstanceInfo(4, 16 * 1024, "g2-standard-4", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-8") ->
+      InstanceInfo(8, 32 * 1024, "g2-standard-8", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-12") ->
+      InstanceInfo(12, 48 * 1024, "g2-standard-12", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-16") ->
+      InstanceInfo(16, 64 * 1024, "g2-standard-16", 1, L4Gpu),
+    NodeInstanceMapKey("g2-standard-24") ->
+      InstanceInfo(24, 96 * 1024, "g2-standard-24", 2, L4Gpu),
+    NodeInstanceMapKey("g2-standard-32") ->
+      InstanceInfo(32, 128 * 1024, "g2-standard-32", 2, L4Gpu),
+    NodeInstanceMapKey("g2-standard-48") ->
+      InstanceInfo(48, 192 * 1024, "g2-standard-48", 4, L4Gpu),
+    NodeInstanceMapKey("g2-standard-96") ->
+      InstanceInfo(96, 384 * 1024, "g2-standard-96", 8, L4Gpu),
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: parthosa
body: Yes, in future we could include all GPU instance types. For now, we limit it to a few for which we have known/relevant tuning configurations.
created_at: 2025-05-19T17:23:12Z
author_association: COLLABORATOR
diff_hunk: @@ -54,9 +54,24 @@ object PlatformNames {
 
 case class DynamicAllocationInfo(enabled: Boolean, max: String, min: String, initial: String)
 
+/**
+ * Case class representing the key for the node instance map.
+ * @param instanceType Name of the instance type
+ * @param gpuCount Count of GPUs on the instance. This is required for certain
+ *                 cases where a given instance type can have multiple GPU counts
+ *                 (e.g. n1-standard-16 with 1, 2, or 4 GPUs).
+ */
+case class    NodeInstanceMapKey(instanceType: String, gpuCount: Option[Int] = None) {
path: core/src/main/scala/com/nvidia/spark/rapids/tool/Platform.scala
user: login: parthosa
body: Fixed
created_at: 2025-05-19T17:23:21Z
author_association: COLLABORATOR
diff_hunk: @@ -3371,18 +3335,16 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
           |--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager
-          |--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=32m
           |--conf spark.sql.adaptive.autoBroadcastJoinThreshold=[FILL_IN_VALUE]
-          |--conf spark.sql.adaptive.coalescePartitions.initialPartitionNum=800
           |--conf spark.sql.adaptive.coalescePartitions.minPartitionSize=4m
           |--conf spark.sql.adaptive.coalescePartitions.parallelismFirst=false
           |--conf spark.sql.files.maxPartitionBytes=512m
-          |--conf spark.sql.shuffle.partitions=800
+          |--conf spark.sql.shuffle.partitions=100
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: parthosa
body: That is a good question. We do not have benchmark for AQE and related shuffle partition L4. Created a parent issue to benchmark tuning configs for L4 - https://github.com/NVIDIA/spark-rapids-tools/issues/1682.

Updated these tests to use `ignore()` and TODO.

cc: @amahussein  
created_at: 2025-05-19T18:43:04Z
author_association: COLLABORATOR
diff_hunk: @@ -3371,18 +3335,16 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
           |--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager
-          |--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=32m
           |--conf spark.sql.adaptive.autoBroadcastJoinThreshold=[FILL_IN_VALUE]
-          |--conf spark.sql.adaptive.coalescePartitions.initialPartitionNum=800
           |--conf spark.sql.adaptive.coalescePartitions.minPartitionSize=4m
           |--conf spark.sql.adaptive.coalescePartitions.parallelismFirst=false
           |--conf spark.sql.files.maxPartitionBytes=512m
-          |--conf spark.sql.shuffle.partitions=800
+          |--conf spark.sql.shuffle.partitions=100
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: amahussein
body: scary!
created_at: 2025-05-19T21:47:53Z
author_association: COLLABORATOR
diff_hunk: @@ -2339,31 +2308,32 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
         "spark.rapids.sql.enabled" -> "true",
         "spark.plugins" -> "com.nvidia.spark.SQLPlugin",
         "spark.rapids.sql.concurrentGpuTasks" -> "1")
-    val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
+    val databricksWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testDatabricksVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
-    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS, clusterPropsOpt)
-    val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(
+      databricksWorkerInfo)
+    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS,
+      clusterProperties = clusterPropsOpt)
+    val autoTuner = buildAutoTunerForTests(databricksWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
     val autoTunerOutput = Profiler.getAutoTunerResultsAsString(properties, comments)
     // scalastyle:off line.size.limit
     val expectedResults =
       s"""|
           |Spark Properties:
           |--conf spark.databricks.adaptive.autoOptimizeShuffle.enabled=false
-          |--conf spark.executor.memory=32g
+          |--conf spark.executor.memory=64g
           |--conf spark.locality.wait=0
           |--conf spark.rapids.memory.pinnedPool.size=4g
           |--conf spark.rapids.shuffle.multiThreaded.maxBytesInFlight=4g
-          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
-          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
+          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=48
+          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=48
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
-          |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
+          |--conf spark.rapids.sql.multiThreadedRead.numThreads=64
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: cindyyuanjiang
body: This test case also have shuffle related changes, should we ignore this too?
created_at: 2025-05-19T22:08:02Z
author_association: COLLABORATOR
diff_hunk: @@ -2339,31 +2308,32 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
         "spark.rapids.sql.enabled" -> "true",
         "spark.plugins" -> "com.nvidia.spark.SQLPlugin",
         "spark.rapids.sql.concurrentGpuTasks" -> "1")
-    val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
+    val databricksWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testDatabricksVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
-    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS, clusterPropsOpt)
-    val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(
+      databricksWorkerInfo)
+    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS,
+      clusterProperties = clusterPropsOpt)
+    val autoTuner = buildAutoTunerForTests(databricksWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
     val autoTunerOutput = Profiler.getAutoTunerResultsAsString(properties, comments)
     // scalastyle:off line.size.limit
     val expectedResults =
       s"""|
           |Spark Properties:
           |--conf spark.databricks.adaptive.autoOptimizeShuffle.enabled=false
-          |--conf spark.executor.memory=32g
+          |--conf spark.executor.memory=64g
           |--conf spark.locality.wait=0
           |--conf spark.rapids.memory.pinnedPool.size=4g
           |--conf spark.rapids.shuffle.multiThreaded.maxBytesInFlight=4g
-          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
-          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
+          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=48
+          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=48
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
-          |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
+          |--conf spark.rapids.sql.multiThreadedRead.numThreads=64
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: parthosa
body: This is expected and is due to the change in the default Databricks AWS instance type from `g5.4xlarge` (16 vCPUs) to `g5.8xlarge` (32 vCPUs). This is to align with the instance used by the Qualification model.

Now for this unit test, with more available cores, AutoTuner now recommends higher thread counts for multithreaded readers and writers.



created_at: 2025-05-19T22:35:59Z
author_association: COLLABORATOR
diff_hunk: @@ -2339,31 +2308,32 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
         "spark.rapids.sql.enabled" -> "true",
         "spark.plugins" -> "com.nvidia.spark.SQLPlugin",
         "spark.rapids.sql.concurrentGpuTasks" -> "1")
-    val dataprocWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
+    val databricksWorkerInfo = buildGpuWorkerInfoAsString(Some(customProps), Some(32),
       Some("212992MiB"), Some(5), Some(4), Some(T4Gpu.getMemory), Some(T4Gpu.toString))
     val infoProvider = getMockInfoProvider(3.7449728E7, Seq(0, 0), Seq(0.4, 0.4), logEventsProps,
       Some(testDatabricksVersion))
-    val clusterPropsOpt = ProfilingAutoTunerConfigsProvider
-      .loadClusterPropertiesFromContent(dataprocWorkerInfo)
-    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS, clusterPropsOpt)
-    val autoTuner = buildAutoTunerForTests(dataprocWorkerInfo, infoProvider, platform)
+    val clusterPropsOpt = PropertiesLoader[ClusterProperties].loadFromContent(
+      databricksWorkerInfo)
+    val platform = PlatformFactory.createInstance(PlatformNames.DATABRICKS_AWS,
+      clusterProperties = clusterPropsOpt)
+    val autoTuner = buildAutoTunerForTests(databricksWorkerInfo, infoProvider, platform)
     val (properties, comments) = autoTuner.getRecommendedProperties()
     val autoTunerOutput = Profiler.getAutoTunerResultsAsString(properties, comments)
     // scalastyle:off line.size.limit
     val expectedResults =
       s"""|
           |Spark Properties:
           |--conf spark.databricks.adaptive.autoOptimizeShuffle.enabled=false
-          |--conf spark.executor.memory=32g
+          |--conf spark.executor.memory=64g
           |--conf spark.locality.wait=0
           |--conf spark.rapids.memory.pinnedPool.size=4g
           |--conf spark.rapids.shuffle.multiThreaded.maxBytesInFlight=4g
-          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
-          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
+          |--conf spark.rapids.shuffle.multiThreaded.reader.threads=48
+          |--conf spark.rapids.shuffle.multiThreaded.writer.threads=48
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
-          |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
+          |--conf spark.rapids.sql.multiThreadedRead.numThreads=64
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: cindyyuanjiang
body: understood, thanks!
created_at: 2025-05-19T22:53:50Z
author_association: COLLABORATOR
diff_hunk: @@ -3371,18 +3335,16 @@ class ProfilingAutoTunerSuite extends BaseAutoTunerSuite {
           |--conf spark.rapids.shuffle.multiThreaded.reader.threads=28
           |--conf spark.rapids.shuffle.multiThreaded.writer.threads=28
           |--conf spark.rapids.sql.batchSizeBytes=2147483647b
-          |--conf spark.rapids.sql.concurrentGpuTasks=2
+          |--conf spark.rapids.sql.concurrentGpuTasks=3
           |--conf spark.rapids.sql.format.parquet.multithreaded.combine.waitTime=1000
           |--conf spark.rapids.sql.multiThreadedRead.numThreads=80
           |--conf spark.rapids.sql.reader.multithreaded.combine.sizeBytes=10m
           |--conf spark.shuffle.manager=com.nvidia.spark.rapids.spark$testSmVersion.RapidsShuffleManager
-          |--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=32m
           |--conf spark.sql.adaptive.autoBroadcastJoinThreshold=[FILL_IN_VALUE]
-          |--conf spark.sql.adaptive.coalescePartitions.initialPartitionNum=800
           |--conf spark.sql.adaptive.coalescePartitions.minPartitionSize=4m
           |--conf spark.sql.adaptive.coalescePartitions.parallelismFirst=false
           |--conf spark.sql.files.maxPartitionBytes=512m
-          |--conf spark.sql.shuffle.partitions=800
+          |--conf spark.sql.shuffle.partitions=100
path: core/src/test/scala/com/nvidia/spark/rapids/tool/tuning/ProfilingAutoTunerSuite.scala
user: login: parthosa
body: The UT changes in this PR are a due to updating the default instance types used by CSPs to match those in the Qualification model. 
Ideally, I could have made these changes in a separate PR which might have reduced the impact on the UTs.
created_at: 2025-05-19T22:58:01Z
author_association: COLLABORATOR

PR comments:
user: login: parthosa
created_at: 2025-05-19T22:43:02Z
body: > One thing later we want to watch for is that adding new machine type should not alter existing unit tests because the existing unit tests are supposed to match the behavior of other node-types.

That is a very good point. Adding support for other node-types should not change the existing UTs. The UT changes in this PR are a due to updating the default instance types used by CSPs to match those in the Qualification model. Ideally, I could have made these changes in a separate PR which might have reduced the impact on the UTs.
author_association: COLLABORATOR
user: login: parthosa
created_at: 2025-05-19T22:58:52Z
body: Thank you @amahussein and @cindyyuanjiang for reviewing the PR. 
author_association: COLLABORATOR