Repository: deeporiginbio/deeporigin-client
Pull request #223: feat: support for client pattern in dd toolkit
Author: login: sg-s
Created: 2025-05-21T17:51:32Z | Merged: 2025-05-23T12:59:06Z

## problem description

balto (and other machine users) need to be able to use the client in a way that allows for multi-org, multi-user. basically, it should be possible to make a list of calls to various functions in the drug discovery toolkit, where each call is for a different user/different org

this is a problem because the current version is designed for a single-user, single org paradigm. tokens, org ids and other config information is read from a file that exists on the home dir

this cannot work for this use case

## solution

assuming you have some org_id and token, you can create an instance of `PlatformClients` that contains all clients to interact with the platform:


```python
clients = PlatformClients(token=..., base_url=...)
```

and this can be used in"

```python
sim = Complex.from_dir(
    EXAMPLE_DATA_DIR,
   _platform_clients=clients,
)

```

this client pattern also works for all functions, e.g.:

```
get_dataframe(_platform_clients=clients)
```



Diff:
diff --git a/src/drug_discovery/abfe.py b/src/drug_discovery/abfe.py
index d5f07099..4fc0ad31 100644
--- a/src/drug_discovery/abfe.py
+++ b/src/drug_discovery/abfe.py
@@ -41,9 +41,12 @@ def get_results(self) -> pd.DataFrame | None:
 
         This method returns a dataframe showing the results of ABFE runs associated with this simulation session. The ligand file name and ΔG are shown, together with user-supplied properties"""
 
+        files_client = getattr(self.parent._platform_clients, "FilesApi", None)
+
         files = utils.find_files_on_ufa(
             tool="ABFE",
             protein=self.parent.protein.file_path.name,
+            client=files_client,
         )
 
         results_files = [file for file in files if file.endswith("/results.csv")]
@@ -52,7 +55,10 @@ def get_results(self) -> pd.DataFrame | None:
             print("No ABFE results found for this protein.")
             return None
 
-        files_api.download_files(results_files)
+        files_api.download_files(
+            results_files,
+            client=files_client,
+        )
 
         # read all the CSV files using pandas and
         # set Ligand1 column to ligand name (parent dir of results.csv)
@@ -125,7 +131,7 @@ def run_end_to_end(
         *,
         ligands: Optional[list[Ligand]] = None,
         re_run: bool = False,
-    ):
+    ) -> list[Job] | None:
         """Method to run an end-to-end ABFE run.
 
         Args:
@@ -141,6 +147,7 @@ def run_end_to_end(
             only_with_status=["Succeeded", "Running", "Queued", "Created"],
             include_metadata=True,
             resolve_user_names=False,
+            _platform_clients=self.parent._platform_clients,
         )
 
         # filter to find relevant jobs
@@ -155,9 +162,14 @@ def run_end_to_end(
             df["metadata"].apply(lambda d: isinstance(d, dict) and d.get("ligand_file"))
         )
 
-        ligands_to_run = [
-            ligand for ligand in ligand_names if ligand not in ligands_already_run
-        ]
+        if re_run:
+            # need to re-run, so don't remove already run ligands
+            ligands_to_run = ligand_names
+        else:
+            # no re-run, remove already run ligands
+            ligands_to_run = [
+                ligand for ligand in ligand_names if ligand not in ligands_already_run
+            ]
 
         self.parent._sync_protein_and_ligands()
 
@@ -174,6 +186,8 @@ def run_end_to_end(
         if self.jobs is None:
             self.jobs = []
 
+        jobs_for_this_run = []
+
         for ligand_name in ligands_to_run:
             ligand_path = "entities/ligands/" + os.path.basename(ligand_name)
 
@@ -189,9 +203,10 @@ def run_end_to_end(
                 params=self._params.end_to_end,
                 tool="ABFE",
                 tool_version=self.tool_version,
+                _platform_clients=self.parent._platform_clients,
             )
 
-            job = Job.from_ids([job_id])
+            job = Job.from_id(job_id, _platform_clients=self.parent._platform_clients)
 
             job._viz_func = self._render_progress
             job._name_func = self._name_job
@@ -199,6 +214,9 @@ def run_end_to_end(
             job.sync()
 
             self.jobs.append(job)
+            jobs_for_this_run.append(job)
+
+        return jobs_for_this_run
 
     @beartype
     def show_trajectory(
@@ -225,6 +243,8 @@ def show_trajectory(
         )
         files_to_download = [remote_pdb_file]
 
+        files_client = getattr(self.parent._platform_clients, "FilesApi", None)
+
         if step == "binding":
             # Check for valid windows
 
@@ -233,6 +253,7 @@ def show_trajectory(
                 tool="ABFE",
                 protein=self.parent.protein.file_path.name,
                 ligand=Path(ligand.file_path).name,
+                client=files_client,
             )
             xtc_files = [
                 file
@@ -247,8 +268,6 @@ def show_trajectory(
                 int(re.search(r"window_(\d+)", path).group(1)) for path in xtc_files
             ]
 
-            print(valid_windows)
-
             if window not in valid_windows:
                 raise DeepOriginException(
                     f"Invalid window number: {window}. Valid windows are: {sorted(valid_windows)}"
@@ -267,7 +286,7 @@ def show_trajectory(
 
         files_to_download.append(remote_xtc_file)
 
-        files_api.download_files(files_to_download)
+        files_api.download_files(files_to_download, client=files_client)
 
         from deeporigin_molstar.src.viewers import ProteinViewer
 
@@ -313,6 +332,11 @@ def parse_progress(cls, job: Job) -> dict:
             if "cmd" in data and data["cmd"] == "FEP Results":
                 return {step: "Succeeded" for step in steps}
 
+            if "status" in data and data["status"] == "Initiating":
+                progress = {step: "NotStarted" for step in steps}
+                progress["init"] = "Running"
+                return progress
+
             status_value = job._status[0]
 
             # If the overall status is Succeeded, return a dictionary with every key set to "Succeeded".
diff --git a/src/drug_discovery/complex.py b/src/drug_discovery/complex.py
index 2a098879..1f116b53 100644
--- a/src/drug_discovery/complex.py
+++ b/src/drug_discovery/complex.py
@@ -11,6 +11,8 @@
 from deeporigin.drug_discovery.docking import Docking
 from deeporigin.drug_discovery.rbfe import RBFE
 from deeporigin.drug_discovery.structures import Ligand, Protein
+from deeporigin.files import FilesClient
+from deeporigin.platform.utils import PlatformClients
 
 
 @dataclass
@@ -21,23 +23,22 @@ class Complex:
     protein: Protein
     _ligands: list[Ligand] = field(default_factory=list, repr=False)
 
-    # these params are not user facing
-    _db: Optional[dict] = None
-
-    """stores a hash of all ligands and the protein. This will be computed post initialization"""
-    _hash: Optional[str] = None
+    _platform_clients: Optional[PlatformClients] = None
 
+    @beartype
     def __init__(
         self,
         protein: Protein,
         *,
         ligands: list[Ligand] | None = None,
-        **kwargs,
+        _platform_clients: Optional[PlatformClients] = None,
     ):
         """Initialize a Complex with either ligands or _ligands parameter"""
         self._ligands = ligands if ligands is not None else []
         self.protein = protein
 
+        self._platform_clients = _platform_clients
+
         self.__post_init__()
 
     def __post_init__(self):
@@ -60,7 +61,12 @@ def ligands(self, new_ligands: list[Ligand]):
         self._ligands = new_ligands
 
     @classmethod
-    def from_dir(cls, directory: str) -> "Complex":
+    def from_dir(
+        cls,
+        directory: str,
+        *,
+        _platform_clients: Optional[PlatformClients] = None,
+    ) -> "Complex":
         """Initialize a Complex from a directory containing protein and ligand files.
 
         Args:
@@ -112,6 +118,7 @@ def from_dir(cls, directory: str) -> "Complex":
         instance = cls(
             protein=protein,
             ligands=ligands,
+            _platform_clients=_platform_clients,
         )
 
         return instance
@@ -156,9 +163,12 @@ def _sync_protein_and_ligands(self) -> None:
         Internal method. Do not use."""
 
         # get a list of all files in the entities directory
-        from deeporigin.files import FilesClient
 
-        files_client = FilesClient()
+        if self._platform_clients is None:
+            files_client = FilesClient()
+        else:
+            files_client = self._platform_clients.FilesApi
+
         remote_files = files_client.list_folder("entities", recursive=True)
         remote_files = list(remote_files.keys())
 
diff --git a/src/drug_discovery/utils.py b/src/drug_discovery/utils.py
index fd54e531..b566869b 100644
--- a/src/drug_discovery/utils.py
+++ b/src/drug_discovery/utils.py
@@ -8,7 +8,9 @@
 from beartype import beartype
 
 from deeporigin.drug_discovery.constants import tool_mapper, valid_tools
+from deeporigin.files import FilesClient
 from deeporigin.platform import tools_api
+from deeporigin.platform.utils import PlatformClients
 from deeporigin.utils.core import PrettyDict
 
 DATA_DIRS = dict()
@@ -36,6 +38,8 @@ def _start_tool_run(
     ligand2_path: Optional[str] = None,
     tool: valid_tools,
     tool_version: str,
+    provider: tools_api.PROVIDER = "ufa",
+    _platform_clients: Optional[PlatformClients] = None,
 ) -> str:
     """starts a single run of ABFE end to end and logs it in the ABFE database. Internal function. Do not use.
 
@@ -61,24 +65,24 @@ def _start_tool_run(
 
     # a protein is needed for ABFE, RBFE, and docking
     params["protein"] = {
-        "$provider": "ufa",
+        "$provider": provider,
         "key": protein_path,
     }
 
     # input ligand files
     if tool == "RBFE":
         params["ligand1"] = {
-            "$provider": "ufa",
+            "$provider": provider,
             "key": ligand1_path,
         }
 
         params["ligand2"] = {
-            "$provider": "ufa",
+            "$provider": provider,
             "key": ligand2_path,
         }
     elif tool == "ABFE":
         params["ligand"] = {
-            "$provider": "ufa",
+            "$provider": provider,
             "key": ligand1_path,
         }
 
@@ -86,22 +90,22 @@ def _start_tool_run(
     if tool == "RBFE":
         outputs = {
             "output_file": {
-                "$provider": "ufa",
+                "$provider": provider,
                 "key": output_dir_path + "output/",
             },
             "rbfe_results_summary": {
-                "$provider": "ufa",
+                "$provider": provider,
                 "key": output_dir_path + "results.csv",
             },
         }
     elif tool == "ABFE":
         outputs = {
             "output_file": {
-                "$provider": "ufa",
+                "$provider": provider,
                 "key": output_dir_path + "output/",
             },
             "abfe_results_summary": {
-                "$provider": "ufa",
+                "$provider": provider,
                 "key": output_dir_path + "results.csv",
             },
         }
@@ -109,11 +113,11 @@ def _start_tool_run(
         raise NotImplementedError("Docking is not implemented yet")
         outputs = {
             "data_file": {
-                "$provider": "ufa",
+                "$provider": provider,
                 "key": output_dir_path,
             },
             "results_sdf": {
-                "$provider": "ufa",
+                "$provider": provider,
                 "key": "mason/inputs/message_file.txt",
             },
         }
@@ -123,13 +127,28 @@ def _start_tool_run(
             "⚠️ Warning: test_run=1 in these parameters. Results will not be accurate."
         )
 
+    tools_client = getattr(_platform_clients, "ToolsApi", None)
+    clusters_client = getattr(_platform_clients, "ClustersApi", None)
+
+    if _platform_clients is None:
+        from deeporigin.config import get_value
+
+        org_friendly_id = get_value()["organization_id"]
+    else:
+        org_friendly_id = _platform_clients.org_friendly_id
+
     job_id = tools_api._process_job(
         inputs=params,
         outputs=outputs,
         tool_key=tool_mapper[tool],
         metadata=metadata,
+        client=tools_client,
+        org_friendly_id=org_friendly_id,
+        clusters_client=clusters_client,
     )
 
+    print(f"Job started with Job ID: {job_id}")
+
     return job_id
 
 
@@ -171,6 +190,7 @@ def find_files_on_ufa(
     tool: str,
     protein: str,
     ligand: Optional[str] = None,
+    client: Optional[FilesClient] = None,
 ) -> list:
     """
     Find files on the UFA (Unified File API) storage for a given tool run.
@@ -188,9 +208,12 @@ def find_files_on_ufa(
     Returns:
         List[str]: A list of file paths found in the specified UFA directory.
     """
-    from deeporigin.files import FilesClient
 
-    client = FilesClient()
+    if client is None:
+        from deeporigin.files import FilesClient
+
+        client = FilesClient()
+
     if ligand is not None:
         search_str = f"tool-runs/{tool}/{protein}/{ligand}/"
     else:
diff --git a/src/drug_discovery/workflow_step.py b/src/drug_discovery/workflow_step.py
index c634a0ad..7b5c84b6 100644
--- a/src/drug_discovery/workflow_step.py
+++ b/src/drug_discovery/workflow_step.py
@@ -27,15 +27,30 @@ def __init__(self, parent):
 
     def get_jobs(self):
         """Get the jobs for this workflow step."""
-        df = get_dataframe()
+        df = get_dataframe(
+            _platform_clients=self.parent._platform_clients,
+        )
         df = df[df["tool_key"].str.contains(self._tool_key)]
 
         df = df[df["protein_id"].str.contains(self.parent.protein.file_path.name)]
 
+        job_ids = df["id"].tolist()
+
         if self._fuse_jobs:
-            self.jobs = [Job.from_ids(list(map(str, df["id"].tolist())))]
+            self.jobs = [
+                Job.from_ids(
+                    job_ids,
+                    _platform_clients=self.parent._platform_clients,
+                )
+            ]
         else:
-            self.jobs = [Job.from_ids([str(row["id"])]) for _, row in df.iterrows()]
+            self.jobs = [
+                Job.from_ids(
+                    [job_id],
+                    _platform_clients=self.parent._platform_clients,
+                )
+                for job_id in job_ids
+            ]
 
         for job in self.jobs:
             job._viz_func = self._render_progress
diff --git a/src/platform/clusters_api.py b/src/platform/clusters_api.py
index 5911f536..69238a73 100644
--- a/src/platform/clusters_api.py
+++ b/src/platform/clusters_api.py
@@ -1,6 +1,10 @@
 """bridge module to interact with the platform tools api"""
 
+import functools
 import sys
+from typing import Optional
+
+from beartype import beartype
 
 from deeporigin.platform.utils import _add_functions_to_module
 
@@ -8,3 +12,28 @@
     module=sys.modules[__name__],
     api_name="ClustersApi",
 )
+
+
+@functools.cache
+@beartype
+def _get_cluster_id(
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> str:
+    """gets a valid cluster ID to run tools on
+
+    this defaults to pulling us-west-2"""
+
+    available_clusters = list_clusters(  # noqa: F821
+        client=client,
+        org_friendly_id=org_friendly_id,
+    )
+
+    cluster = [
+        cluster
+        for cluster in available_clusters
+        if "us-west-2" in cluster.attributes.name
+    ]
+    cluster = cluster[0]
+    return cluster.id
diff --git a/src/platform/files_api.py b/src/platform/files_api.py
index e93ecfe3..de8829ae 100644
--- a/src/platform/files_api.py
+++ b/src/platform/files_api.py
@@ -5,10 +5,11 @@
 from deeporigin.files import FilesClient
 
 
-def download_files(files: list[str]):
+def download_files(files: list[str], *, client=None):
     """download files from UFA, if needed, and save to ~/.deeporigin/"""
 
-    client = FilesClient()
+    if client is None:
+        client = FilesClient()
 
     # Construct src_to_dest mapping for download_files
     home_dir = os.path.expanduser("~")
diff --git a/src/platform/tools_api.py b/src/platform/tools_api.py
index c5b68611..4dcdfa0d 100644
--- a/src/platform/tools_api.py
+++ b/src/platform/tools_api.py
@@ -1,17 +1,15 @@
 """bridge module to interact with the platform tools api"""
 
 import concurrent.futures
-import functools
 from functools import wraps
 import inspect
 import sys
-from typing import Any, Callable, Optional
+from typing import Any, Callable, Literal, Optional
 
 from beartype import beartype
 import pandas as pd
 
-from deeporigin.config import get_value
-from deeporigin.platform import clusters_api, tools_api
+from deeporigin.platform import clusters_api
 from deeporigin.platform.utils import _add_functions_to_module
 
 __all__ = _add_functions_to_module(
@@ -20,12 +18,20 @@
 )
 
 
-TERMINAL_STATES = {"Succeeded", "Failed"}
+TERMINAL_STATES = {"Succeeded", "Failed", "Cancelled"}
 NON_TERMINAL_STATES = {"Created", "Queued", "Running"}
 
+# possible providers for files that work with the tools API
+PROVIDER = Literal["ufa", "s3"]
+
 
 @beartype
-def get_status_and_progress(execution_id: str) -> dict:
+def get_status_and_progress(
+    execution_id: str,
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> dict:
     """Determine the status of a run, identified by job ID
 
     Args:
@@ -34,8 +40,10 @@ def get_status_and_progress(execution_id: str) -> dict:
 
     """
 
-    data = tools_api.get_tool_execution(
+    data = get_tool_execution(  # noqa: F821
         execution_id=execution_id,
+        org_friendly_id=org_friendly_id,
+        client=client,
     )
 
     return dict(
@@ -49,7 +57,12 @@ def get_status_and_progress(execution_id: str) -> dict:
 
 
 @beartype
-def get_statuses_and_progress(job_ids: list[str]) -> list:
+def get_statuses_and_progress(
+    job_ids: list[str],
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> list:
     """get statuses and progress reports for multiple jobs in parallel
 
     Args:
@@ -57,20 +70,33 @@ def get_statuses_and_progress(job_ids: list[str]) -> list:
 
     """
     results = []
+    errors = []
 
     with concurrent.futures.ThreadPoolExecutor() as executor:
         # Submit all jobs and create a mapping from future to job_id
         future_to_job_id = {
-            executor.submit(get_status_and_progress, job_id): job_id
+            executor.submit(
+                get_status_and_progress,
+                job_id,
+                client=client,
+                org_friendly_id=org_friendly_id,
+            ): job_id
             for job_id in job_ids
         }
 
         # As each future completes, store the result in the status dictionary
         for future in concurrent.futures.as_completed(future_to_job_id):
+            job_id = future_to_job_id[future]
             try:
                 results.append(future.result())
-            except Exception:
-                pass
+            except Exception as e:
+                errors.append((job_id, e))
+
+    if errors:
+        error_msgs = "\n".join([f"Job {jid}: {str(err)}" for jid, err in errors])
+        raise RuntimeError(
+            f"Some jobs failed in get_status_and_progress:\n{error_msgs}"
+        )
 
     return results
 
@@ -78,6 +104,9 @@ def get_statuses_and_progress(job_ids: list[str]) -> list:
 @beartype
 def cancel_runs(
     job_ids: list[str] | pd.core.series.Series | pd.core.frame.DataFrame,
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
 ) -> None:
     """Cancel multiple jobs in parallel.
 
@@ -91,7 +120,15 @@ def cancel_runs(
         job_ids = job_ids["id"].tolist()
 
     with concurrent.futures.ThreadPoolExecutor() as executor:
-        futures = [executor.submit(cancel_run, job_id) for job_id in job_ids]
+        futures = [
+            executor.submit(
+                cancel_run,
+                job_id,
+                client=client,
+                org_friendly_id=org_friendly_id,
+            )
+            for job_id in job_ids
+        ]
         concurrent.futures.wait(futures)
         # Raise the first exception encountered
         for future in futures:
@@ -101,25 +138,41 @@ def cancel_runs(
 
 
 @beartype
-def cancel_run(execution_id: str) -> None:
+def cancel_run(
+    execution_id: str,
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> None:
     """cancel a run
 
     Args:
         execution_id (str): execution ID
     """
 
-    data = get_status_and_progress(execution_id)
+    data = get_status_and_progress(
+        execution_id,
+        client=client,
+        org_friendly_id=org_friendly_id,
+    )
     if data["status"] in ["Cancelled", "Failed", "Succeeded"]:
         return
 
-    tools_api.action_tool_execution(
+    action_tool_execution(  # noqa: F821
         execution_id=execution_id,
         action="cancel",
+        client=client,
+        org_friendly_id=org_friendly_id,
     )
 
 
 @beartype
-def query_run_statuses(job_ids: list[str]) -> dict:
+def query_run_statuses(
+    job_ids: list[str],
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> dict:
     """get statuses for multiple jobs in parallel
 
     Args:
@@ -131,7 +184,13 @@ def query_run_statuses(job_ids: list[str]) -> dict:
     with concurrent.futures.ThreadPoolExecutor() as executor:
         # Submit all jobs and create a mapping from future to job_id
         future_to_job_id = {
-            executor.submit(query_run_status, job_id): job_id for job_id in job_ids
+            executor.submit(
+                query_run_status,
+                job_id,
+                client=client,
+                org_friendly_id=org_friendly_id,
+            ): job_id
+            for job_id in job_ids
         }
 
         # As each future completes, store the result in the status dictionary
@@ -146,7 +205,12 @@ def query_run_statuses(job_ids: list[str]) -> dict:
 
 
 @beartype
-def query_run_status(execution_id: str) -> str:
+def query_run_status(
+    execution_id: str,
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> str:
     """Determine the status of a run, identified by execution_id ID
 
     Args:
@@ -157,7 +221,11 @@ def query_run_status(execution_id: str) -> str:
 
     """
 
-    data = tools_api.get_tool_execution(execution_id=execution_id)
+    data = get_tool_execution(  # noqa: F821
+        execution_id=execution_id,
+        client=client,
+        org_friendly_id=org_friendly_id,
+    )
 
     return data.attributes.status
 
@@ -189,32 +257,14 @@ def _resolve_column_name(column_name: str, cols: list) -> str:
     return column_id
 
 
-@functools.cache
-@beartype
-def _get_cluster_id() -> str:
-    """gets a valid cluster ID to run tools on
-
-    this defaults to pulling us-west-2"""
-
-    available_clusters = clusters_api.list_clusters(
-        org_friendly_id=get_value()["organization_id"]
-    )
-
-    cluster = [
-        cluster
-        for cluster in available_clusters
-        if "us-west-2" in cluster.attributes.name
-    ]
-    cluster = cluster[0]
-    cluster_id = cluster.id
-    return cluster_id
-
-
 @beartype
 def run_tool(
     *,
     data: dict,
     tool_key: str,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+    clusters_client=None,
 ):
     """run any tool using provided data transfer object (DTO)
 
@@ -222,20 +272,57 @@ def run_tool(
         data (dict): data transfer object. This is typically generated by the `make_payload` function."""
 
     if "clusterId" not in data.keys():
-        data["clusterId"] = _get_cluster_id()
+        data["clusterId"] = clusters_api._get_cluster_id(
+            client=clusters_client,
+            org_friendly_id=org_friendly_id,
+        )
 
-    return tools_api.execute_tool(
+    return execute_tool(  # noqa: F821
         tool_key=tool_key,
         execute_tool_dto=data,
+        client=client,
+        org_friendly_id=org_friendly_id,
+    )
+
+
+@beartype
+def _process_job(
+    *,
+    inputs: dict,
+    outputs: dict,
+    tool_key: str,
+    cols: Optional[list] = None,
+    metadata: Optional[dict] = None,
+    client=None,  # this is the tools client
+    org_friendly_id: Optional[str] = None,
+    clusters_client=None,
+) -> str:
+    """helper function that uses inputs and outputs to construct a payload and run a tool"""
+
+    payload = make_payload(
+        outputs=outputs,
+        inputs=inputs,
+        cols=cols,
+        metadata=metadata,
     )
 
+    response = run_tool(
+        data=payload,
+        tool_key=tool_key,
+        client=client,
+        org_friendly_id=org_friendly_id,
+        clusters_client=clusters_client,
+    )
+    job_id = response.id
+
+    return job_id
+
 
 @beartype
 def make_payload(
     *,
     inputs: dict,
     outputs: dict,
-    cluster_id: Optional[str] = None,
     cols: Optional[list] = None,
     metadata: Optional[dict] = None,
 ) -> dict:
@@ -244,7 +331,6 @@ def make_payload(
     Args:
         inputs (dict): inputs
         outputs (dict): outputs
-        cluster_id (Optional[str], optional): cluster ID. Defaults to None. If not provided, the default cluster (us-west-2) is used.
         cols: (Optional[list], optional): list of columns. Defaults to None. If provided, column names (in inputs or outputs) are converted to column IDs.
         metadata: (Optional[dict], optional): metadata to be added to the payload. Defaults to None.
 
@@ -252,13 +338,11 @@ def make_payload(
         dict: correctly formatted payload, ready to be passed to run_tool
     """
 
-    if cluster_id is None:
-        cluster_id = _get_cluster_id()
-
+    # note that clusterId is not included here
+    # it need to be included for this payload to be valid
     payload = dict(
         inputs=inputs,
         outputs=outputs,
-        clusterId=cluster_id,
         metadata=metadata,
     )
 
@@ -329,34 +413,12 @@ def _column_name_to_column_id(
     return data
 
 
-@beartype
-def _process_job(
+def generate_tool_function(
     *,
-    inputs: dict,
-    outputs: dict,
     tool_key: str,
-    cols: Optional[list] = None,
-    metadata: Optional[dict] = None,
-) -> str:
-    """helper function that uses inputs and outputs to construct a payload and run a tool"""
-
-    payload = make_payload(
-        outputs=outputs,
-        inputs=inputs,
-        cols=cols,
-        metadata=metadata,
-    )
-
-    response = run_tool(
-        data=payload,
-        tool_key=tool_key,
-    )
-    job_id = response.id
-
-    return job_id
-
-
-def generate_tool_function(*, tool_key: str, tool_version: str) -> Callable:
+    tool_version: str,
+    org_friendly_id: Optional[str] = None,
+) -> Callable:
     """utility function that generates a function that can be used to run a too, from schema
 
     Example usage:
@@ -370,7 +432,7 @@ def generate_tool_function(*, tool_key: str, tool_version: str) -> Callable:
 
     """
 
-    data = tools_api.get_tool(tool_key=tool_key)
+    data = get_tool(tool_key=tool_key)  # noqa: F821
     data = [item for item in data if item.version == tool_version]
 
     if len(data) == 0:
@@ -431,6 +493,14 @@ def generate_tool_function(*, tool_key: str, tool_version: str) -> Callable:
             annotation=dict[str, Any] | None,
         )
     )
+    parameters.append(
+        inspect.Parameter(
+            "client",
+            inspect.Parameter.KEYWORD_ONLY,
+            default=None,
+            annotation=Any,
+        )
+    )
 
     sig = inspect.Signature(parameters)
 
@@ -454,12 +524,15 @@ def generated_function(*args, **kwargs):
         }
 
         metadata = arguments.get("metadata")
+        client = arguments.get("client")
 
-        return tools_api._process_job(
+        return _process_job(
             inputs=inputs,
             outputs=outputs,
             tool_key=tool_key,
             metadata=metadata,
+            client=client,
+            org_friendly_id=org_friendly_id,
         )
 
     # Sanitize function name to be a valid Python identifier
diff --git a/src/platform/utils.py b/src/platform/utils.py
index 6da47ce0..76e52d17 100644
--- a/src/platform/utils.py
+++ b/src/platform/utils.py
@@ -1,7 +1,7 @@
 """module to automatically wrap methods in autogenerated low-level code and re-expose them as high-level functions"""
 
 import inspect
-from typing import Literal
+from typing import Optional
 from urllib.parse import urljoin
 
 from beartype import beartype
@@ -15,11 +15,59 @@
 from deeporigin.utils.core import _get_method
 
 
+class PlatformClients:
+    """
+    A container for all DeepOrigin platform API clients, instantiated with a token, base_url, and org_friendly_id.
+
+    Each API client is available as an attribute of this class, e.g., self.FilesApi, self.ToolsApi, etc.
+
+    Args:
+        token (str): The authentication token to use for all API clients.
+        base_url (str): The base URL of the DeepOrigin platform (e.g., 'https://os.deeporigin.io').
+        org_friendly_id (str): The organization-friendly ID to use for API requests.
+    """
+
+    def __init__(
+        self,
+        *,
+        token: str,
+        base_url: str,
+        org_friendly_id: str,
+    ) -> None:
+        from deeporigin.files import FilesClient
+
+        self.org_friendly_id = org_friendly_id
+
+        # FilesApi is a special case
+        self.FilesApi = FilesClient(
+            token=token,
+            base_url=base_url,
+            organization_id=org_friendly_id,
+        )
+
+        api_endpoint = base_url + "/api/"
+        apis = [attr for attr in do_sdk_platform.__dir__() if attr.endswith("Api")]
+
+        for api in apis:
+            setattr(
+                self,
+                api,
+                _get_api_client(
+                    api_name=api,
+                    token=token,
+                    api_endpoint=api_endpoint,
+                ),
+            )
+
+    def list_clients(self) -> list:
+        """Return a list of all API client attribute names available on this instance."""
+        return [attr for attr in self.__dict__.keys() if attr.endswith("Api")]
+
+
 @beartype
 def _add_functions_to_module(
     module,
     api_name: str,
-    sdk_name: str = "platform",
 ) -> list:
     """utility function to dynamically add functions to a module
 
@@ -35,7 +83,6 @@ def _add_functions_to_module(
     methods = _get_client_methods(
         _get_api_client(
             api_name=api_name,
-            sdk_name=sdk_name,
             configure=False,
         )
     )
@@ -60,7 +107,6 @@ def _add_functions_to_module(
             _create_function(
                 method_path=method,
                 api_name=api_name,
-                sdk_name=sdk_name,
             ),
         )
 
@@ -70,14 +116,14 @@ def _add_functions_to_module(
 @beartype
 def _get_api_client(
     *,
-    sdk_name: Literal["platform", "files"],
     api_name: str,
     configure: bool = True,
+    token: Optional[str] = None,
+    api_endpoint: Optional[str] = None,
 ):
     """return a configured client for the API we want to access
 
     Args:
-        sdk_name (str): name of the SDK to use ('platform' or 'files')
         api_name (str): name of the API
         configure (bool): whether to configure the client with authentication
 
@@ -86,27 +132,25 @@ def _get_api_client(
 
     """
 
-    if sdk_name == "platform":
-        sdk = do_sdk_platform
-    else:
-        # sdk = do_sdk_files
-        raise NotImplementedError("Files API is not implemented yet.")
     if configure:
-        if sdk_name == "platform":
-            host = urljoin(get_value()["api_endpoint"], "/api")
-        elif sdk_name == "files":
-            host = urljoin(get_value()["api_endpoint"], "/file-api")
+        if api_endpoint is None:
+            api_endpoint = get_value()["api_endpoint"]
+
+        host = urljoin(api_endpoint, "/api")
 
-        configuration = sdk.Configuration(
+        if token is None:
+            token = get_tokens()["access"]
+
+        configuration = do_sdk_platform.Configuration(
             host=host,
-            access_token=get_tokens()["access"],
+            access_token=token,
         )
 
-        client = sdk.ApiClient(configuration=configuration)
+        client = do_sdk_platform.ApiClient(configuration=configuration)
     else:
-        client = sdk.ApiClient()
+        client = do_sdk_platform.ApiClient()
 
-    api_class = getattr(sdk, api_name)
+    api_class = getattr(do_sdk_platform, api_name)
     client = api_class(api_client=client)
     return client
 
@@ -131,7 +175,6 @@ def _create_function(
     *,
     method_path: str,
     api_name: str,
-    sdk_name: str,
 ):
     """utility function the dynamically creates functions
     that wrap low-level functions in the DeepOrigin data API"""
@@ -140,11 +183,9 @@ def _create_function(
     # of inspecting its methods and extracting
     # function signatures. So we don't need any
     # authentication
-
     client = _get_api_client(
         configure=False,
         api_name=api_name,
-        sdk_name=sdk_name,
     )
 
     method = _get_method(client, method_path)
@@ -161,16 +202,14 @@ def dynamic_function(
         if client is None:
             client = _get_api_client(
                 api_name=api_name,
-                sdk_name=sdk_name,
             )
         method = _get_method(client, method_path)
 
         # Insert org_friendly_id if not present in kwargs, and
         # if it's required by the method
         method_sig = inspect.signature(method)
-        if (
-            "org_friendly_id" not in kwargs
-            and "org_friendly_id" in method_sig.parameters
+        if "org_friendly_id" in method_sig.parameters and (
+            kwargs.get("org_friendly_id") is None
         ):
             kwargs["org_friendly_id"] = get_value()["organization_id"]
 
diff --git a/src/tools/job.py b/src/tools/job.py
index eceeed87..73360041 100644
--- a/src/tools/job.py
+++ b/src/tools/job.py
@@ -17,6 +17,7 @@
 import pandas as pd
 
 from deeporigin.platform import tools_api
+from deeporigin.platform.utils import PlatformClients
 from deeporigin.utils.core import elapsed_minutes
 
 # Enable nested event loops for Jupyter
@@ -58,8 +59,16 @@ class Job:
     _execution_ids: list = field(default_factory=list)
     _metadata: list = field(default_factory=list)
 
+    # clients
+    _platform_clients: Optional[PlatformClients] = None
+
     @classmethod
-    def from_ids(cls, ids: list[str]) -> "Job":
+    def from_ids(
+        cls,
+        ids: list[str],
+        *,
+        _platform_clients: Optional[PlatformClients] = None,
+    ) -> "Job":
         """Create a Job instance from a list of IDs.
 
         Args:
@@ -68,10 +77,19 @@ def from_ids(cls, ids: list[str]) -> "Job":
         Returns:
             A new Job instance with the given IDs.
         """
-        return cls(name="job", _ids=ids)
+        return cls(
+            name="job",
+            _ids=ids,
+            _platform_clients=_platform_clients,
+        )
 
     @classmethod
-    def from_id(cls, id: str) -> "Job":
+    def from_id(
+        cls,
+        id: str,
+        *,
+        _platform_clients: Optional[PlatformClients] = None,
+    ) -> "Job":
         """Create a Job instance from a single ID.
 
         Args:
@@ -80,7 +98,11 @@ def from_id(cls, id: str) -> "Job":
         Returns:
             A new Job instance with the given ID.
         """
-        return cls(name="job", _ids=[id])
+        return cls(
+            name="job",
+            _ids=[id],
+            _platform_clients=_platform_clients,
+        )
 
     def sync(self):
         """Synchronize the job status and progress reports.
@@ -90,8 +112,16 @@ def sync(self):
         reached a terminal state (Succeeded or Failed).
         """
 
+        # get the org_friendly_id from the platform clients if it's not None
+        org_friendly_id = getattr(self._platform_clients, "org_friendly_id", None)
+        tools_client = getattr(self._platform_clients, "ToolsApi", None)
+
         # use
-        results = tools_api.get_statuses_and_progress(self._ids)
+        results = tools_api.get_statuses_and_progress(
+            self._ids,
+            client=tools_client,
+            org_friendly_id=org_friendly_id,
+        )
 
         self._status = [result["status"] for result in results]
         self._progress_reports = [result["progress"] for result in results]
@@ -279,7 +309,7 @@ def stop_watching(self):
         or can be called manually to stop monitoring.
         """
         if self._task is not None:
-            self._task.cancel()
+            self._task.cancel()  # note that this is not job.cancel() -- we're cancelling the asyncio task
             self._task = None
 
     def _repr_html_(self) -> str:
@@ -304,7 +334,15 @@ def cancel(self):
         Returns:
             The result of the cancellation operation from utils.cancel_runs.
         """
-        tools_api.cancel_runs(self._ids)
+
+        org_friendly_id = getattr(self._platform_clients, "org_friendly_id", None)
+        tools_client = getattr(self._platform_clients, "ToolsApi", None)
+
+        tools_api.cancel_runs(
+            self._ids,
+            client=tools_client,
+            org_friendly_id=org_friendly_id,
+        )
 
 
 def get_dataframe(
@@ -313,6 +351,7 @@ def get_dataframe(
     only_with_status: Optional[list[str]] = None,
     include_metadata: bool = False,
     resolve_user_names: bool = False,
+    _platform_clients: Optional[PlatformClients] = None,
 ) -> pd.DataFrame:
     """Get a dataframe of the job statuses and progress reports.
 
@@ -338,20 +377,28 @@ def get_dataframe(
             },
         }
 
-    from deeporigin.config import get_value
-    from deeporigin.platform import organizations_api
+    if _platform_clients is None:
+        from deeporigin.config import get_value
 
-    org_id = get_value()["organization_id"]
+        org_friendly_id = get_value()["organization_id"]
+    else:
+        org_friendly_id = _platform_clients.org_friendly_id
 
     response = tools_api.get_tool_executions(
-        org_friendly_id=org_id,
+        org_friendly_id=org_friendly_id,
         filter=_filter,
         page_size=10000,
+        client=_platform_clients.ToolsApi,
     )
     jobs = response["data"]
 
     if resolve_user_names:
-        users = organizations_api.get_organization_users(org_friendly_id=org_id)
+        from deeporigin.platform import organizations_api
+
+        users = organizations_api.get_organization_users(
+            org_friendly_id=org_friendly_id,
+            client=_platform_clients.OrganizationsApi,
+        )
 
         # Create a mapping of user IDs to user names
         user_id_to_name = {user["id"]: user["attributes"]["name"] for user in users}


Reviews:
user: login: laionazeredo
body: This is a big step ahead for our sdk. Thanks for this implementation @sg-s 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T12:25:21Z
user: login: sg-s
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T13:35:20Z
user: login: sg-s
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T17:53:16Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR introduces a client-pattern to support multi-org, multi-user workflows by injecting a `PlatformClients` container throughout the toolkit and updating API calls to accept explicit client and organization ID parameters.
- Add `PlatformClients` class to centralize SDK client creation
- Update existing wrappers (`tools_api`, `files_api`, `clusters_api`, `job`, `get_dataframe`, etc.) to accept `client` and `org_friendly_id`
- Propagate `_platform_clients` through all user-facing constructors and workflow utilities

### Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

<details>
<summary>Show a summary per file</summary>

| File                         | Description                                                      |
| ---------------------------- | ---------------------------------------------------------------- |
| src/tools/job.py             | Added `_platform_clients` support to `Job` methods (`from_ids`, `from_id`, `sync`, `cancel`) |
| src/platform/utils.py        | Introduced `PlatformClients`; extended `_get_api_client` with token/api_endpoint parameters |
| src/platform/tools_api.py    | Extended status and cancel utilities to accept `client`/`org_friendly_id` |
| src/platform/files_api.py    | Added optional `client` param to `download_files`               |
| src/platform/clusters_api.py | Added cached `_get_cluster_id` supporting `client`/`org_friendly_id` |
| src/drug_discovery/workflow_step.py | Pass `_platform_clients` into `get_dataframe` and `Job.from_ids` |
| src/drug_discovery/utils.py  | Propagate `provider` and `_platform_clients` into `_start_tool_run` |
| src/drug_discovery/complex.py| Carry `_platform_clients` through `Complex` constructors and file sync |
| src/drug_discovery/abfe.py   | Use `FilesApi` from `_platform_clients` in ABFE result fetching and downloads |
</details>



<details>
<summary>Comments suppressed due to low confidence (5)</summary>

**src/tools/job.py:87**
* Using the parameter name `id` shadows the built-in `id()` function. Consider renaming it to `job_id` for clarity and to avoid unexpected shadowing.
```
def from_id(
```
**src/tools/job.py:72**
* The docstring does not document the new `_platform_clients` parameter. Please update the docstring to include `_platform_clients` and its purpose.
```
"""Create a Job instance from a list of IDs.
```
**src/platform/utils.py:49**
* The name `do_sdk_platform` is not imported or defined in this module, causing a NameError. Add `import do_sdk_platform` (or the correct SDK module) at the top.
```
apis = [attr for attr in do_sdk_platform.__dir__() if attr.endswith("Api")]
```
**src/platform/utils.py:142**
* The function `get_tokens` is used but not imported in this module, which will raise a NameError. Import it from its defining module (e.g., `from deeporigin.config import get_tokens`).
```
token = get_tokens()["access"]
```
**src/platform/clusters_api.py:9**
* This import shadows the current module (`clusters_api`), leading to recursive imports. You likely need to import the SDK client module (e.g., `import do_sdk_platform.clusters_api as sdk_clusters_api`).
```
from deeporigin.platform import clusters_api
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T17:55:14Z
user: login: smbatyankh
body: LGTM:

Two changes that would be nice to add as well
1) Ability to provide UFA paths as inputs for simulation
2) Abilit to provide output directory 

state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T10:44:35Z

Review comments:
diff_hunk: @@ -15,11 +15,105 @@
 from deeporigin.utils.core import _get_method
 
 
+class PlatformClients:
+    """
+    A container for all DeepOrigin platform API clients, instantiated with a token, base_url, and org_friendly_id.
+
+    Each API client is available as an attribute of this class, e.g., self.FilesApi, self.ToolsApi, etc.
+
+    Args:
+        token (str): The authentication token to use for all API clients.
+        base_url (str): The base URL of the DeepOrigin platform (e.g., 'https://os.deeporigin.io').
+        org_friendly_id (str): The organization-friendly ID to use for API requests.
+    """
+
+    def __init__(
+        self,
+        *,
+        token: str,
+        base_url: str,
+        org_friendly_id: str,
+    ) -> None:
path: src/platform/utils.py
user: login: sg-s
body: this class is a handy wrapper around all clients that we need to do anything on the platform. all clients are configured, instantiated, and ready to go
created_at: 2025-05-22T13:32:21Z
author_association: COLLABORATOR
diff_hunk: @@ -58,8 +59,16 @@ class Job:
     _execution_ids: list = field(default_factory=list)
     _metadata: list = field(default_factory=list)
 
+    # clients
+    _platform_clients: Optional[PlatformClients] = None
+
path: src/tools/job.py
user: login: sg-s
body: job now supports _platform_clients
created_at: 2025-05-22T13:33:39Z
author_association: COLLABORATOR
diff_hunk: @@ -304,7 +334,15 @@ def cancel(self):
         Returns:
             The result of the cancellation operation from utils.cancel_runs.
         """
-        tools_api.cancel_runs(self._ids)
+
+        org_friendly_id = getattr(self._platform_clients, "org_friendly_id", None)
+        tools_client = getattr(self._platform_clients, "ToolsApi", None)
+
+        tools_api.cancel_runs(
+            self._ids,
+            client=tools_client,
+            org_friendly_id=org_friendly_id,
+        )
path: src/tools/job.py
user: login: sg-s
body: note: in calls like this, client and org_friendly_id can be None (if client is not configured). then, it simply passes through, and it's the resp of the lower level function to figure this out (which it can, for end users)
created_at: 2025-05-22T13:35:12Z
author_association: COLLABORATOR
diff_hunk: @@ -41,9 +41,12 @@ def get_results(self) -> pd.DataFrame | None:
 
         This method returns a dataframe showing the results of ABFE runs associated with this simulation session. The ligand file name and ΔG are shown, together with user-supplied properties"""
 
+        files_client = getattr(self.parent._platform_clients, "FilesApi", None)
+
path: src/drug_discovery/abfe.py
user: login: sg-s
body: this will be None or a valid files_client
created_at: 2025-05-22T17:49:15Z
author_association: COLLABORATOR
diff_hunk: @@ -155,9 +162,14 @@ def run_end_to_end(
             df["metadata"].apply(lambda d: isinstance(d, dict) and d.get("ligand_file"))
         )
 
-        ligands_to_run = [
-            ligand for ligand in ligand_names if ligand not in ligands_already_run
-        ]
+        if re_run:
+            # need to re-run, so don't remove already run ligands
+            ligands_to_run = ligand_names
+        else:
+            # no re-run, remove already run ligands
+            ligands_to_run = [
+                ligand for ligand in ligand_names if ligand not in ligands_already_run
+            ]
path: src/drug_discovery/abfe.py
user: login: sg-s
body: corrected a bug in the logic for re-running
created_at: 2025-05-22T17:49:38Z
author_association: COLLABORATOR
diff_hunk: @@ -21,23 +23,22 @@ class Complex:
     protein: Protein
     _ligands: list[Ligand] = field(default_factory=list, repr=False)
 
-    # these params are not user facing
-    _db: Optional[dict] = None
-
-    """stores a hash of all ligands and the protein. This will be computed post initialization"""
-    _hash: Optional[str] = None
path: src/drug_discovery/complex.py
user: login: sg-s
body: removed hash, and _db. these are not used anymore
created_at: 2025-05-22T17:50:02Z
author_association: COLLABORATOR
diff_hunk: @@ -61,59 +65,59 @@ def _start_tool_run(
 
     # a protein is needed for ABFE, RBFE, and docking
     params["protein"] = {
-        "$provider": "ufa",
+        "$provider": provider,
         "key": protein_path,
     }
 
     # input ligand files
     if tool == "RBFE":
         params["ligand1"] = {
-            "$provider": "ufa",
+            "$provider": provider,
             "key": ligand1_path,
         }
 
         params["ligand2"] = {
-            "$provider": "ufa",
+            "$provider": provider,
             "key": ligand2_path,
         }
     elif tool == "ABFE":
         params["ligand"] = {
-            "$provider": "ufa",
+            "$provider": provider,
             "key": ligand1_path,
         }
path: src/drug_discovery/utils.py
user: login: sg-s
body: this opens the door to potentially supporting s3
created_at: 2025-05-22T17:50:19Z
author_association: COLLABORATOR
diff_hunk: @@ -1,10 +1,40 @@
 """bridge module to interact with the platform tools api"""
 
+import functools
 import sys
+from typing import Optional
 
+from beartype import beartype
+
+from deeporigin.platform import clusters_api
 from deeporigin.platform.utils import _add_functions_to_module
 
 __all__ = _add_functions_to_module(
     module=sys.modules[__name__],
     api_name="ClustersApi",
 )
+
+
+@functools.cache
+@beartype
+def _get_cluster_id(
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> str:
+    """gets a valid cluster ID to run tools on
+
path: src/platform/clusters_api.py
user: login: sg-s
body: this was moved here from tools_api. it makes more sense here
created_at: 2025-05-22T17:50:58Z
author_association: COLLABORATOR
diff_hunk: @@ -20,12 +18,20 @@
 )
 
 
-TERMINAL_STATES = {"Succeeded", "Failed"}
+TERMINAL_STATES = {"Succeeded", "Failed", "Cancelled"}
 NON_TERMINAL_STATES = {"Created", "Queued", "Running"}
path: src/platform/tools_api.py
user: login: sg-s
body: Cancelled is a valid terminal state
created_at: 2025-05-22T17:51:13Z
author_association: COLLABORATOR
diff_hunk: @@ -49,35 +57,56 @@ def get_status_and_progress(execution_id: str) -> dict:
 
 
 @beartype
-def get_statuses_and_progress(job_ids: list[str]) -> list:
+def get_statuses_and_progress(
+    job_ids: list[str],
+    *,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+) -> list:
     """get statuses and progress reports for multiple jobs in parallel
 
     Args:
         job_ids (list[str]): list of job IDs
 
     """
     results = []
+    errors = []
 
     with concurrent.futures.ThreadPoolExecutor() as executor:
         # Submit all jobs and create a mapping from future to job_id
         future_to_job_id = {
-            executor.submit(get_status_and_progress, job_id): job_id
+            executor.submit(
+                get_status_and_progress,
+                job_id,
+                client=client,
+                org_friendly_id=org_friendly_id,
+            ): job_id
             for job_id in job_ids
         }
 
         # As each future completes, store the result in the status dictionary
         for future in concurrent.futures.as_completed(future_to_job_id):
+            job_id = future_to_job_id[future]
             try:
                 results.append(future.result())
-            except Exception:
-                pass
+            except Exception as e:
+                errors.append((job_id, e))
+
+    if errors:
+        error_msgs = "\n".join([f"Job {jid}: {str(err)}" for jid, err in errors])
+        raise RuntimeError(
+            f"Some jobs failed in get_status_and_progress:\n{error_msgs}"
+        )
path: src/platform/tools_api.py
user: login: sg-s
body: this now raises an error if something goes wrong
created_at: 2025-05-22T17:51:50Z
author_association: COLLABORATOR
diff_hunk: @@ -189,53 +257,72 @@ def _resolve_column_name(column_name: str, cols: list) -> str:
     return column_id
 
 
-@functools.cache
-@beartype
-def _get_cluster_id() -> str:
-    """gets a valid cluster ID to run tools on
-
-    this defaults to pulling us-west-2"""
-
-    available_clusters = clusters_api.list_clusters(
-        org_friendly_id=get_value()["organization_id"]
-    )
-
-    cluster = [
-        cluster
-        for cluster in available_clusters
-        if "us-west-2" in cluster.attributes.name
-    ]
-    cluster = cluster[0]
-    cluster_id = cluster.id
-    return cluster_id
-
-
 @beartype
 def run_tool(
     *,
     data: dict,
     tool_key: str,
+    client=None,
+    org_friendly_id: Optional[str] = None,
+    clusters_client=None,
 ):
     """run any tool using provided data transfer object (DTO)
 
     Args:
         data (dict): data transfer object. This is typically generated by the `make_payload` function."""
 
     if "clusterId" not in data.keys():
-        data["clusterId"] = _get_cluster_id()
+        data["clusterId"] = clusters_api._get_cluster_id(
+            client=clusters_client,
+            org_friendly_id=org_friendly_id,
+        )
 
     return tools_api.execute_tool(
         tool_key=tool_key,
         execute_tool_dto=data,
+        client=client,
+        org_friendly_id=org_friendly_id,
     )
 
 
+@beartype
+def _process_job(
+    *,
+    inputs: dict,
+    outputs: dict,
+    tool_key: str,
+    cols: Optional[list] = None,
+    metadata: Optional[dict] = None,
+    client=None,  # this is the tools client
+    org_friendly_id: Optional[str] = None,
+    clusters_client=None,
+) -> str:
+    """helper function that uses inputs and outputs to construct a payload and run a tool"""
+
+    payload = make_payload(
+        outputs=outputs,
path: src/platform/tools_api.py
user: login: sg-s
body: i moved this within the file
created_at: 2025-05-22T17:52:59Z
author_association: COLLABORATOR

PR comments:
user: login: ghost
created_at: 2025-05-21T17:54:06Z
body: ![Logo](https://cdn.ast.checkmarx.net/integrations/logo/Checkmarx.png)
**Checkmarx One – Scan Summary & Details** – [6730a20a\-cfa2\-46b5\-a492\-b788f5ac3ccd](https://us.ast.checkmarx.net/projects/cc182d6a-a56c-4683-afdf-6bd4377eebfb/scans?id=6730a20a-cfa2-46b5-a492-b788f5ac3ccd&branch=client-support)
### Great job, no security vulnerabilities found in this Pull Request

author_association: NONE
user: login: sg-s
created_at: 2025-05-23T12:59:02Z
body: hi @smbatyankh both those changes will happen in a different pr
author_association: COLLABORATOR
user: login: mike9ant
created_at: 2025-05-23T17:20:59Z
body: I think this PlatformClients is not a good name for this (t describes what it does but not from users perspective). From user's perspective, it's more of a ConnectionConfig.

Also, passing this around for every call is not the best/most convenient interface; it will distract users of the APi. I imagine that there would be a central object on the whole interface (accessible through shared reference) that you could set_connection_config() on that would swap the connection for the entire API. Then Balto could just set that, and call all of the apis as-is.
author_association: CONTRIBUTOR
user: login: sg-s
created_at: 2025-05-23T17:46:13Z
body: @mike9ant end users will never use this. this is only meant for use within balto. this PR doesn't change anything about how end users use this
author_association: COLLABORATOR