Repository: pytorch/test-infra
Pull request #6371: [Mobile Benchmark Test] [Feature Request] Add os, job_arn, and job_conclusion to artifact
Author: login: yangw-dev
Labels: name: CLA Signed
description: This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Created: 2025-03-06T19:18:35Z | Merged: 2025-03-12T03:02:08Z

https://github.com/pytorch/test-infra/issues/6294

Details:
- print `os` in test spect printout section
- store os, job_arn (mobile job) and job_conclusion to each artifact metadata. Notice this is not github job conclusion, this is mobile job conclusion.
- wrap post-test logics into ReportProcessor, 
    - pass aws client as parameter for test-driven purpose
    - add unit test for ReportProcessor


Diff:
diff --git a/tools/device-farm-runner/run_on_aws_devicefarm.py b/tools/device-farm-runner/run_on_aws_devicefarm.py
index 37c947c632..c8a55d1a94 100755
--- a/tools/device-farm-runner/run_on_aws_devicefarm.py
+++ b/tools/device-farm-runner/run_on_aws_devicefarm.py
@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 
+import copy
 import datetime
 import json
 import logging
@@ -9,8 +10,10 @@
 import sys
 import time
 from argparse import Action, ArgumentParser, Namespace
+from dataclasses import asdict, dataclass
 from enum import Enum
 from logging import info
+from math import inf
 from typing import Any, Dict, List, Optional
 from warnings import warn
 
@@ -18,6 +21,7 @@
 import requests
 
 
+# TODO(elainewy): refactor and add unit tests for benchmark test logic
 POLLING_DELAY_IN_SECOND = 5
 MAX_UPLOAD_WAIT_IN_SECOND = 600
 
@@ -35,6 +39,7 @@ class ReportType(Enum):
     JOB = "job"
     SUITE = "suite"
     TEST = "test"
+    UNKNOWN = "unknown"
 
 
 DEVICE_FARM_BUCKET = "gha-artifacts"
@@ -196,6 +201,18 @@ def parse_args() -> Any:
         help="an optional file to write the list of artifacts from AWS in JSON format",
     )
 
+    parser.add_argument(
+        "--debug",
+        action="store_true",
+        help="debug mode, the artifacts won't be uploaded to s3, it should mainly used in local env",
+    )
+
+    parser.add_argument(
+        "--new-json-output",
+        action="store_true",
+        help="enable new json artifact output format with jobrun, and list of artifacts, this is temporary ",
+    )
+
     return parser.parse_args()
 
 
@@ -262,21 +279,6 @@ def download_artifact(artifact_url: str, local_filename: str) -> str:
     return local_filename
 
 
-def upload_file_to_s3(
-    file_name: str,
-    bucket: str,
-    key: str,
-) -> None:
-    """
-    Upload a local file to S3
-    """
-    boto3.client("s3").upload_file(
-        file_name,
-        bucket,
-        key,
-    )
-
-
 def set_output(val: Any, gh_var_name: str, filename: Optional[str]) -> None:
     if os.getenv("GITHUB_OUTPUT"):
         with open(str(os.getenv("GITHUB_OUTPUT")), "a") as env:
@@ -290,138 +292,6 @@ def set_output(val: Any, gh_var_name: str, filename: Optional[str]) -> None:
             print(val, file=f)
 
 
-def print_testspec(
-    job_name: Optional[str],
-    file_name: str,
-    indent: int = 0,
-) -> None:
-    """
-    The test spec output from AWS Device Farm is the main output of the test job.
-    """
-    print(f"::group::{job_name} test output")
-    with open(file_name) as f:
-        print(f.read())
-    print("::endgroup::")
-
-
-def print_test_artifacts(
-    client: Any,
-    app_type: str,
-    test_arn: str,
-    workflow_id: str,
-    workflow_attempt: int,
-    job_name: Optional[str],
-    indent: int = 0,
-) -> List[Dict[str, str]]:
-    """
-    Return all artifacts from this specific test. There are 3 types of artifacts
-    from Device Farm including FILE, LOG, and SCREENSHOT
-    """
-    gathered_artifacts = []
-
-    for artifact_type in ["FILE", "LOG", "SCREENSHOT"]:
-        r = client.list_artifacts(arn=test_arn, type=artifact_type)
-        for artifact in r.get("artifacts", []):
-            filetype = artifact["type"]
-            filename = artifact["name"].replace(" ", "_")
-            extension = artifact["extension"].replace(" ", "_")
-
-            local_filename = (
-                artifact["arn"].replace(":", "_").replace("/", "_")
-                + f"_{filename}.{extension}"
-            )
-            s3_key = f"device_farm/{workflow_id}/{workflow_attempt}/{local_filename}"
-            # Download the artifact locally
-            upload_file_to_s3(
-                download_artifact(artifact["url"], local_filename),
-                DEVICE_FARM_BUCKET,
-                s3_key,
-            )
-
-            s3_url = f"https://{DEVICE_FARM_BUCKET}.s3.amazonaws.com/{s3_key}"
-            artifact["s3_url"] = s3_url
-
-            info(
-                f"{' ' * indent}Saving {artifact_type} {filename}.{extension} ({filetype}) "
-                + f"at {s3_url}"
-            )
-
-            # Some more metadata to identify where the artifact comes from
-            artifact["app_type"] = app_type
-            artifact["job_name"] = job_name
-            gathered_artifacts.append(artifact)
-
-            # Additional step to print the test output
-            if filetype == "TESTSPEC_OUTPUT":
-                print_testspec(job_name, local_filename, indent + 2)
-
-    return gathered_artifacts
-
-
-def print_report(
-    client: Any,
-    app_type: str,
-    report: Dict[str, Any],
-    report_type: ReportType,
-    job_name: Optional[str],
-    workflow_id: str,
-    workflow_attempt: int,
-    indent: int = 0,
-) -> List[Dict[str, str]]:
-    """
-    Print the test report from Device Farm in a friendly way and return the list
-    of any notable artifacts from the test run, i.e. logs and screenshots
-    """
-    if not report:
-        warn("Missing report, returning...")
-        return []
-
-    name = report["name"]
-    # Keep the top-level job name as the name of the whole report, this
-    # is used to connect all artifacts from one job together
-    if report_type == ReportType.JOB:
-        job_name = name
-    result = report["result"]
-
-    extra_msg = ""
-    if report_type == ReportType.SUITE or is_success(result):
-        counters = report["counters"]
-        extra_msg = f"with stats {counters}"
-
-    info(f"{' ' * indent}{name} {result} {extra_msg}")
-
-    arn = report["arn"]
-    if report_type == ReportType.RUN:
-        more_reports = client.list_jobs(arn=arn)
-        next_report_type = ReportType.JOB
-    elif report_type == ReportType.JOB:
-        more_reports = client.list_suites(arn=arn)
-        next_report_type = ReportType.SUITE
-    elif report_type == ReportType.SUITE:
-        more_reports = client.list_tests(arn=arn)
-        next_report_type = ReportType.TEST
-    elif report_type == ReportType.TEST:
-        return print_test_artifacts(
-            client, app_type, arn, workflow_id, workflow_attempt, job_name, indent + 2
-        )
-
-    artifacts = []
-    for more_report in more_reports.get(f"{next_report_type.value}s", []):
-        artifacts.extend(
-            print_report(
-                client,
-                app_type,
-                more_report,
-                next_report_type,
-                job_name,
-                workflow_id,
-                workflow_attempt,
-                indent + 2,
-            )
-        )
-    return artifacts
-
-
 def generate_ios_xctestrun(
     client: Any, project_arn: str, prefix: str, ios_xctestrun: str, test_spec: str
 ) -> Dict[str, str]:
@@ -531,6 +401,304 @@ def generate_test_configuration(
     return {"extraDataPackageArn": extra_data_arn}
 
 
+@dataclass
+class DeviceFarmReport:
+    name: str
+    arn: str
+    report_type: str
+    status: str
+    result: str
+    counters: Dict[str, str]
+    infos: Dict[str, str]
+    parent_arn: str
+
+
+@dataclass
+class JobReport(DeviceFarmReport):
+    os: str
+
+
+class ReportProcessor:
+    """
+    A helper class to process the modebile test result from AWS Device Farm.
+
+    Usage:
+        processor = ReportProcessor(...) \n
+        processor.start(mobile_run_report)
+    """
+
+    def __init__(
+        self,
+        device_farm_client: Any,
+        s3_client,
+        app_type: str,
+        workflow_id: str,
+        workflow_attempt: int,
+        is_debug: bool = False,
+    ):
+        self.aws_client = device_farm_client
+        self.s3_client = s3_client
+        self.app_type = app_type
+        self.workflow_id = workflow_id
+        self.workflow_attempt = workflow_attempt
+        self.run_report: Optional[DeviceFarmReport] = None
+        self.job_reports: list[JobReport] = []
+        self.test_spec_info_list: list[Dict] = []
+        self.is_debug = is_debug
+
+    # todo(elainewy): add main method to pass run arn
+    def start(self, report: Dict[str, Any]) -> List[Dict[str, str]]:
+        if not report:
+            warn("Missing report, returning...")
+            return []
+
+        run_arn = report.get("arn", "")
+        if not run_arn:
+            warn("Missing arn from input report, returning...")
+            return []
+
+        if self.is_debug:
+            info(
+                "[DEBUG MODE] the artifacts won't be uploaded to s3, it should mainly used in local env"
+            )
+
+        self.run_report = self._to_run_report(report)
+
+        # fetch mobile job report from the run
+        job_reports_resp = self.aws_client.list_jobs(arn=run_arn)
+        res = []
+
+        # fetch artifacts, and sub-reports for each mobile job
+        for job_report in job_reports_resp.get(ReportType.JOB.value + "s", []):
+            # info(f"Job Report: {jreport}")
+            metadata = self._to_job_report(job_report, run_arn)
+            self.job_reports.append(metadata)
+            artifacts = self._fetch_artifacts_and_reports(
+                job_report,
+                ReportType(metadata.report_type),
+                metadata,
+            )
+            res.extend(artifacts)
+        return res
+
+    def _fetch_artifacts_and_reports(
+        self,
+        report: Dict[str, Any],
+        report_type: ReportType,
+        job_metadata: JobReport,
+        indent: int = 0,
+    ) -> List[Dict[str, str]]:
+        """
+        DFS method that tranverse DeviceFarm report from the mobile job level,
+        identifies and uploads significant artifacts (such as logs and screenshots) to AWS S3,
+        and returns a comprehensive list of artifact metadata, including relevant mobile job report information.
+        """
+        if not report:
+            warn("Missing report, returning...")
+            return []
+
+        name = report["name"]
+        result = report["result"]
+
+        extra_msg = ""
+        if report_type == ReportType.SUITE or is_success(result):
+            counters = report["counters"]
+            extra_msg = f"with stats {counters}"
+
+        info(f"{' ' * indent}{name} {result} {extra_msg}")
+
+        arn = report["arn"]
+        more_reports = {}
+        if report_type == ReportType.JOB:
+            more_reports = self.aws_client.list_suites(arn=arn)
+            next_report_type = ReportType.SUITE
+        elif report_type == ReportType.SUITE:
+            more_reports = self.aws_client.list_tests(arn=arn)
+            next_report_type = ReportType.TEST
+        elif report_type == ReportType.TEST:
+            return self._fetch_test_artifacts(arn, job_metadata, indent + 2)
+        else:
+            warn(f"Unknown report type {report_type}")
+            return []
+
+        artifacts = []
+        for more_report in more_reports.get(f"{next_report_type.value}s", []):
+            artifacts.extend(
+                self._fetch_artifacts_and_reports(
+                    more_report,
+                    next_report_type,
+                    job_metadata,
+                    indent + 2,
+                )
+            )
+        return artifacts
+
+    def _to_job_report(
+        self, report: Dict[str, Any], parent_arn: str, infos: Dict[str, str] = dict()
+    ) -> JobReport:
+        arn = report.get("arn", "")
+        status = report.get("status", "")
+        name = report.get("name", "")
+        result = report.get("result", "")
+        counters = report.get("counters", "{}")
+        os = report.get("device", {}).get("os", "")
+        return JobReport(
+            arn=arn,
+            name=name,
+            report_type=ReportType.JOB.value,
+            status=status,
+            result=result,
+            parent_arn=parent_arn,
+            counters=counters,
+            infos=infos,
+            os=os,
+        )
+
+    def _to_run_report(self, report: Dict[str, Any], infos: Dict[str, str] = dict()):
+        arn = report.get("arn", "")
+        status = report.get("status", "")
+        name = report.get("name", "")
+        result = report.get("result", "")
+        counters = report.get("counters", "{}")
+
+        return DeviceFarmReport(
+            name=name,
+            arn=arn,
+            report_type=ReportType.RUN.value,
+            status=status,
+            result=result,
+            counters=counters,
+            infos=infos,
+            parent_arn="",
+        )
+
+    def _fetch_test_artifacts(
+        self, test_arn: str, job_metadata: JobReport, indent: int = 0
+    ) -> List[Dict[str, str]]:
+        """
+        Return all artifacts from this specific test. There are 3 types of artifacts
+        from Device Farm including FILE, LOG, and SCREENSHOT
+        """
+        gathered_artifacts = []
+        info(f"{' ' * indent} start gathering artifacts")
+        for artifact_type in ["FILE", "LOG", "SCREENSHOT"]:
+            r = self.aws_client.list_artifacts(arn=test_arn, type=artifact_type)
+            for artifact in r.get("artifacts", []):
+                filetype = artifact["type"]
+                filename = artifact["name"].replace(" ", "_")
+                extension = artifact["extension"].replace(" ", "_")
+
+                local_filename = (
+                    artifact["arn"].replace(":", "_").replace("/", "_")
+                    + f"_{filename}.{extension}"
+                )
+                s3_key = f"device_farm/{self.workflow_id}/{self.workflow_attempt}/{local_filename}"
+
+                # Download the artifact locally
+                artifact_file = download_artifact(artifact["url"], local_filename)
+
+                if not self.is_debug:
+                    # upload artifacts to s3 bucket
+                    self._upload_file_to_s3(artifact_file, DEVICE_FARM_BUCKET, s3_key)
+                s3_url = f"https://{DEVICE_FARM_BUCKET}.s3.amazonaws.com/{s3_key}"
+                artifact["s3_url"] = s3_url
+
+                info(
+                    f"{' ' * indent} Saving {artifact_type} {filename}.{extension} ({filetype}) "
+                    + f"at {s3_url}"
+                )
+
+                # Some more metadata to identify where the artifact comes from
+                artifact["app_type"] = self.app_type
+                artifact["job_name"] = job_metadata.name
+                artifact["os"] = job_metadata.os
+                artifact["job_arn"] = job_metadata.arn
+                artifact["job_conclusion"] = job_metadata.result
+                gathered_artifacts.append(artifact)
+                # Additional step to print the test output
+                if filetype == "TESTSPEC_OUTPUT":
+                    self.test_spec_info_list.append(
+                        {
+                            "job_name": job_metadata.name,
+                            "os": job_metadata.os,
+                            "job_arn": job_metadata.arn,
+                            "job_conclusion": job_metadata.result,
+                            "local_filename": local_filename,
+                        }
+                    )
+        return gathered_artifacts
+
+    def print_test_spec(self) -> None:
+        info(f"Test Spec Outputs:")
+        for test_spec_info in self.test_spec_info_list:
+            self.print_single_testspec(
+                test_spec_info["job_name"],
+                test_spec_info["os"],
+                test_spec_info["job_conclusion"],
+                test_spec_info["local_filename"],
+            )
+
+    def print_single_testspec(
+        self,
+        job_name: str,
+        os: str,
+        job_conclusion: str,
+        file_name: str,
+    ) -> None:
+        """
+        The test spec output from AWS Device Farm is the main output of the test job.
+        """
+        print(f"::group::{job_name} {os} test output [Job Result: {job_conclusion}]")
+        with open(file_name) as f:
+            print(f.read())
+        print("::endgroup::")
+
+    def get_run_report(self):
+        if not self.run_report:
+            warn(
+                "cannot print run report, run_report is empty, make sure you call start() first"
+            )
+            return DeviceFarmReport(
+                name="",
+                arn="",
+                report_type="",
+                status="",
+                result="",
+                counters={},
+                infos={},
+                parent_arn="",
+            )
+        return copy.deepcopy(self.run_report)
+
+    def get_job_reports(self):
+        return copy.deepcopy(self.job_reports)
+
+    def print_run_report(self) -> None:
+        if not self.run_report:
+            warn(
+                "cannot print run report, run_report is empty, make sure you call start() first"
+            )
+            return
+        d = asdict(self.run_report)
+        info(f"Run Report Output: {d}")
+
+    def print_job_reports(self) -> None:
+        info("Job Report Output:")
+        for r in self.job_reports:
+            d = json.dumps(asdict(r))
+            info(f"{d}")
+
+    def _upload_file_to_s3(self, file_name: str, bucket: str, key: str) -> None:
+        """
+        Upload a local file to S3
+        """
+        self.s3_client.upload_file(
+            file_name,
+            bucket,
+            key,
+        )
+
+
 def main() -> None:
     args = parse_args()
 
@@ -540,7 +708,8 @@ def main() -> None:
     workflow_attempt = args.workflow_attempt
 
     # NB: Device Farm is only available in us-west-2 region atm
-    client = boto3.client("devicefarm", region_name=AWS_REGION)
+    device_farm_client = boto3.client("devicefarm", region_name=AWS_REGION)
+
     unique_prefix = (
         f"{name_prefix}-{workflow_id}-{workflow_attempt}-"
         + f"{datetime.date.today().isoformat()}-{''.join(random.sample(string.ascii_letters, 8))}"
@@ -555,7 +724,7 @@ def main() -> None:
         app_type = "ANDROID_APP" if args.app.endswith(".apk") else "IOS_APP"
         # Upload the test app
         appfile_arn = upload_file(
-            client=client,
+            client=device_farm_client,
             project_arn=project_arn,
             prefix=unique_prefix,
             filename=args.app,
@@ -563,16 +732,22 @@ def main() -> None:
         )
         info(f"Uploaded app: {appfile_arn}")
 
+    test_to_run = {}
+
     if args.ios_xctestrun:
         app_type = "IOS_APP"
         test_to_run = generate_ios_xctestrun(
-            client, project_arn, unique_prefix, args.ios_xctestrun, args.test_spec
+            device_farm_client,
+            project_arn,
+            unique_prefix,
+            args.ios_xctestrun,
+            args.test_spec,
         )
 
     if args.android_instrumentation_test:
         app_type = "ANDROID_APP"
         test_to_run = generate_android_instrumentation_test(
-            client,
+            device_farm_client,
             project_arn,
             unique_prefix,
             args.android_instrumentation_test,
@@ -582,11 +757,11 @@ def main() -> None:
     configuration = {}
     if args.extra_data:
         configuration = generate_test_configuration(
-            client, project_arn, unique_prefix, args.extra_data
+            device_farm_client, project_arn, unique_prefix, args.extra_data
         )
 
     # Schedule the test
-    r = client.schedule_run(
+    r = device_farm_client.schedule_run(
         projectArn=project_arn,
         name=unique_prefix,
         appArn=appfile_arn,
@@ -603,13 +778,11 @@ def main() -> None:
     result = ""
     try:
         while True:
-            r = client.get_run(arn=run_arn)
+            r = device_farm_client.get_run(arn=run_arn)
             state = r["run"]["status"]
-
             if state == "COMPLETED":
                 result = r["run"]["result"]
                 break
-
             waiting_time = datetime.datetime.now() - start_time
             info(f"Run {unique_prefix} in state {state} after {waiting_time}")
             time.sleep(30)
@@ -617,20 +790,39 @@ def main() -> None:
         warn(f"Failed to run {unique_prefix}: {error}")
         sys.exit(1)
     finally:
-        artifacts = print_report(
-            client,
-            app_type,
-            r.get("run"),
-            ReportType.RUN,
-            None,
-            workflow_id,
-            workflow_attempt,
+        info(f"Run {unique_prefix} finished with state {state} and result {result}")
+        s3_client = boto3.client("s3")
+        processor = ReportProcessor(
+            device_farm_client, s3_client, app_type, workflow_id, workflow_attempt
         )
-        set_output(json.dumps(artifacts), "artifacts", args.output)
+        artifacts = processor.start(r.get("run"))
 
+        if args.new_json_output:
+            info("Generating new json output")
+            output = generate_artifacts_output(
+                artifacts, processor.get_run_report(), processor.get_job_reports()
+            )
+            set_output(json.dumps(output), "artifacts", args.output)
+        else:
+            info("Generating legacy json output")
+            set_output(json.dumps(artifacts), "artifacts", args.output)
+        processor.print_test_spec()
     if not is_success(result):
         sys.exit(1)
 
 
+def generate_artifacts_output(
+    artifacts: List[Dict[str, str]],
+    run_report: DeviceFarmReport,
+    job_reports: List[JobReport],
+):
+    output = {
+        "artifacts": artifacts,
+        "run_report": asdict(run_report),
+        "job_reports": [asdict(job_report) for job_report in job_reports],
+    }
+    return output
+
+
 if __name__ == "__main__":
     main()
diff --git a/tools/device-farm-runner/test_run_on_aws_devicefarm.py b/tools/device-farm-runner/test_run_on_aws_devicefarm.py
new file mode 100644
index 0000000000..20c1dbd565
--- /dev/null
+++ b/tools/device-farm-runner/test_run_on_aws_devicefarm.py
@@ -0,0 +1,314 @@
+import copy
+import json
+import unittest
+from re import M
+from typing import Any, Dict
+from unittest import mock
+from unittest.mock import MagicMock
+
+from run_on_aws_devicefarm import download_artifact, ReportProcessor
+
+
+class MockS3Client:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.upload_file.return_value = None
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+
+class MockDeviceFarmClient:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.list_jobs.return_value = self.mockJobs()
+        self.mock_aws_client.list_suites.return_value = self.mockSuites()
+        self.mock_aws_client.list_tests.return_value = self.mockTests()
+        self.mock_aws_client.list_artifacts.side_effect = (
+            lambda arn, type: self.getArtifacts(arn, type)
+        )
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+    def getArtifacts(self, arn: str, type: str):
+        if type == "FILE":
+            return {
+                "artifacts": [
+                    {
+                        "type": "TESTSPEC_OUTPUT",
+                        "name": "test spec output",
+                        "extension": "output",
+                        "arn": "arn-artifact1",
+                        "url": "url-artifact1",
+                    },
+                ]
+            }
+        elif type == "LOG":
+            return {
+                "artifacts": [
+                    {
+                        "type": "LOG",
+                        "name": "test log",
+                        "extension": "output",
+                        "arn": "arn-artifact2",
+                        "url": "url-artifact2",
+                    },
+                ]
+            }
+        else:
+            return {
+                "artifacts": [
+                    {
+                        "type": "VIDEO",
+                        "name": "test video",
+                        "extension": "output",
+                        "arn": "arn-artifac3",
+                        "url": "url-artifact3",
+                    },
+                ]
+            }
+
+    def mockTests(self) -> Any:
+        return {
+            "tests": [
+                {
+                    "arn": "arn-test1",
+                    "name": "Setup Test",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful test lifecycle of Setup Test",
+                }
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockSuites(self):
+        return {
+            "suites": [
+                {
+                    "arn": "arn-suite1",
+                    "name": "Setup Suite",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful",
+                },
+                {
+                    "arn": "arn-suite2",
+                    "name": "Tests Suite",
+                    "status": "COMPLETED",
+                    "result": "FAILED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Tests passed",
+                },
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockJobs(self):
+        return {
+            "jobs": [
+                {
+                    "arn": "arn-job-1",
+                    "name": "Apple iPhone 15",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "18.0",
+                    },
+                },
+                {
+                    "arn": "arn-job-2",
+                    "name": "Apple iPhone 17 Pro",
+                    "status": "FAILED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "11.0",
+                    },
+                },
+            ]
+        }
+
+
+class Test(unittest.TestCase):
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor_when_happyFlow_then_returnArtifacts(
+        self, download_artifact_mock
+    ):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        expectedNumArtifacts = 12
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertGreaterEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertGreaterEqual(
+            m_s3.getMockClient().upload_file.call_count, expectedNumArtifacts
+        )
+
+        # assert artifacts
+        self.assertGreaterEqual(len(artifacts), expectedNumArtifacts)
+
+        job1_artifacts = [
+            artifact for artifact in artifacts if artifact.get("job_arn") == "arn-job-1"
+        ]
+        a1 = job1_artifacts[0]
+        self.assertEqual(a1["app_type"], "IOS")
+        self.assertEqual(a1["job_arn"], "arn-job-1")
+        self.assertEqual(a1["job_conclusion"], "PASSED")
+        self.assertEqual(a1["job_name"], "Apple iPhone 15")
+        self.assertEqual(a1["os"], "18.0")
+        self.assertEqual(a1["name"], "test spec output")
+
+        job2_artifacts = [
+            artifact for artifact in artifacts if artifact["job_arn"] == "arn-job-2"
+        ]
+        a2 = job2_artifacts[0]
+        self.assertEqual(a2["app_type"], "IOS")
+        self.assertEqual(a2["job_arn"], "arn-job-2")
+        self.assertEqual(a2["job_conclusion"], "PASSED")
+        self.assertEqual(a2["job_name"], "Apple iPhone 17 Pro")
+        self.assertEqual(a2["os"], "11.0")
+        self.assertEqual(a2["name"], "test spec output")
+
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor_when_enableDebugMode_then_noArtifactsAreUploaded(
+        self, download_artifact_mock
+    ):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1, True
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        expectedNumArtifacts = 12
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertGreaterEqual(download_artifact_mock.call_count, expectedNumArtifacts)
+        self.assertGreaterEqual(len(artifacts), expectedNumArtifacts)
+
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 0)
+
+    def test_reportProcessor_when_hasEmptyReportInput_then_returnEmptyList(self):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {}
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1, True
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert
+        self.assertEqual(m_df.getMockClient().list_jobs.call_count, 0)
+        self.assertEqual(len(artifacts), 0)
+
+    def test_reportProcessor_when_missingArnInReportInput_then_returnEmptyList(self):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "status": "COMPLETED",
+            "result": "PASSED",
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert
+        self.assertEqual(m_df.getMockClient().list_jobs.call_count, 0)
+        self.assertEqual(len(artifacts), 0)
+
+
+if __name__ == "__main__":
+    unittest.main()


Reviews:
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-06T19:19:48Z
user: login: ZainRizvi
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:42:42Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T01:44:25Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T01:59:53Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T02:07:34Z
user: login: yangw-dev
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T19:42:14Z
user: login: yangw-dev
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T19:42:30Z
user: login: yangw-dev
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T19:55:33Z
user: login: yangw-dev
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T19:56:24Z

Review comments:
diff_hunk: @@ -0,0 +1,231 @@
+import json
path: tools/device-farm-runner/test_run_on_aws_devicefarm.py
user: login: github-advanced-security[bot]
body: ## PYFMT/format

Run `lintrunner -a` to apply this patch.

[Show more details](https://github.com/pytorch/test-infra/security/code-scanning/3256)
created_at: 2025-03-06T19:19:47Z
author_association: NONE
diff_hunk: @@ -0,0 +1,314 @@
+import copy
+import json
+import unittest
+from re import M
+from typing import Any, Dict
+from unittest import mock
+from unittest.mock import MagicMock
+
+from run_on_aws_devicefarm import download_artifact, ReportProcessor
+
+
+class MockS3Client:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.upload_file.return_value = None
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+
+class MockDeviceFarmClient:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.list_jobs.return_value = self.mockJobs()
+        self.mock_aws_client.list_suites.return_value = self.mockSuites()
+        self.mock_aws_client.list_tests.return_value = self.mockTests()
+        self.mock_aws_client.list_artifacts.side_effect = (
+            lambda arn, type: self.getArtifacts(arn, type)
+        )
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+    def getArtifacts(self, arn: str, type: str):
+        if type == "FILE":
+            return {
+                "artifacts": [
+                    {
+                        "type": "TESTSPEC_OUTPUT",
+                        "name": "test spec output",
+                        "extension": "output",
+                        "arn": "arn-artifact1",
+                        "url": "url-artifact1",
+                    },
+                ]
+            }
+        elif type == "LOG":
+            return {
+                "artifacts": [
+                    {
+                        "type": "LOG",
+                        "name": "test log",
+                        "extension": "output",
+                        "arn": "arn-artifact2",
+                        "url": "url-artifact2",
+                    },
+                ]
+            }
+        else:
+            return {
+                "artifacts": [
+                    {
+                        "type": "VIDEO",
+                        "name": "test video",
+                        "extension": "output",
+                        "arn": "arn-artifac3",
+                        "url": "url-artifact3",
+                    },
+                ]
+            }
+
+    def mockTests(self) -> Any:
+        return {
+            "tests": [
+                {
+                    "arn": "arn-test1",
+                    "name": "Setup Test",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful test lifecycle of Setup Test",
+                }
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockSuites(self):
+        return {
+            "suites": [
+                {
+                    "arn": "arn-suite1",
+                    "name": "Setup Suite",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful",
+                },
+                {
+                    "arn": "arn-suite2",
+                    "name": "Tests Suite",
+                    "status": "COMPLETED",
+                    "result": "FAILED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Tests passed",
+                },
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockJobs(self):
+        return {
+            "jobs": [
+                {
+                    "arn": "arn-job-1",
+                    "name": "Apple iPhone 15",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "18.0",
+                    },
+                },
+                {
+                    "arn": "arn-job-2",
+                    "name": "Apple iPhone 17 Pro",
+                    "status": "FAILED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "11.0",
+                    },
+                },
+            ]
+        }
+
+
+class Test(unittest.TestCase):
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(download_artifact_mock.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+
+        # assert artifacts
+        self.assertEqual(len(artifacts), 12)
+
+        job1_artifacts = [
+            artifact for artifact in artifacts if artifact.get("job_arn") == "arn-job-1"
+        ]
+        self.assertEqual(len(job1_artifacts), 6)
+        a1 = job1_artifacts[0]
+        self.assertEqual(a1["app_type"], "IOS")
+        self.assertEqual(a1["job_arn"], "arn-job-1")
+        self.assertEqual(a1["job_conclusion"], "PASSED")
+        self.assertEqual(a1["job_name"], "Apple iPhone 15")
+        self.assertEqual(a1["os"], "18.0")
+        self.assertEqual(a1["name"], "test spec output")
+
+        job2_artifacts = [
+            artifact for artifact in artifacts if artifact["job_arn"] == "arn-job-2"
+        ]
+        self.assertEqual(len(job2_artifacts), 6)
+        a2 = job2_artifacts[0]
+        self.assertEqual(a2["app_type"], "IOS")
+        self.assertEqual(a2["job_arn"], "arn-job-2")
+        self.assertEqual(a2["job_conclusion"], "PASSED")
+        self.assertEqual(a2["job_name"], "Apple iPhone 17 Pro")
+        self.assertEqual(a2["os"], "11.0")
+        self.assertEqual(a2["name"], "test spec output")
+
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor_debug(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1, True
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(download_artifact_mock.call_count, 12)
path: tools/device-farm-runner/test_run_on_aws_devicefarm.py
user: login: ZainRizvi
body: these numbers are very precise and look like they might break the test when the code changes slightly.  Can these constraints be relaxed a bit to just checking the most critical api calls and perhaps using greaterThan/lessThan type of comparisons?

The idea is to make the test more resilient to design changes in the code (so that future authors don't have to go around updating a bunch of tests) while still verifying that the core functionality desired is maintained.
created_at: 2025-03-07T16:28:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,314 @@
+import copy
+import json
+import unittest
+from re import M
+from typing import Any, Dict
+from unittest import mock
+from unittest.mock import MagicMock
+
+from run_on_aws_devicefarm import download_artifact, ReportProcessor
+
+
+class MockS3Client:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.upload_file.return_value = None
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+
+class MockDeviceFarmClient:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.list_jobs.return_value = self.mockJobs()
+        self.mock_aws_client.list_suites.return_value = self.mockSuites()
+        self.mock_aws_client.list_tests.return_value = self.mockTests()
+        self.mock_aws_client.list_artifacts.side_effect = (
+            lambda arn, type: self.getArtifacts(arn, type)
+        )
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+    def getArtifacts(self, arn: str, type: str):
+        if type == "FILE":
+            return {
+                "artifacts": [
+                    {
+                        "type": "TESTSPEC_OUTPUT",
+                        "name": "test spec output",
+                        "extension": "output",
+                        "arn": "arn-artifact1",
+                        "url": "url-artifact1",
+                    },
+                ]
+            }
+        elif type == "LOG":
+            return {
+                "artifacts": [
+                    {
+                        "type": "LOG",
+                        "name": "test log",
+                        "extension": "output",
+                        "arn": "arn-artifact2",
+                        "url": "url-artifact2",
+                    },
+                ]
+            }
+        else:
+            return {
+                "artifacts": [
+                    {
+                        "type": "VIDEO",
+                        "name": "test video",
+                        "extension": "output",
+                        "arn": "arn-artifac3",
+                        "url": "url-artifact3",
+                    },
+                ]
+            }
+
+    def mockTests(self) -> Any:
+        return {
+            "tests": [
+                {
+                    "arn": "arn-test1",
+                    "name": "Setup Test",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful test lifecycle of Setup Test",
+                }
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockSuites(self):
+        return {
+            "suites": [
+                {
+                    "arn": "arn-suite1",
+                    "name": "Setup Suite",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful",
+                },
+                {
+                    "arn": "arn-suite2",
+                    "name": "Tests Suite",
+                    "status": "COMPLETED",
+                    "result": "FAILED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Tests passed",
+                },
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockJobs(self):
+        return {
+            "jobs": [
+                {
+                    "arn": "arn-job-1",
+                    "name": "Apple iPhone 15",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "18.0",
+                    },
+                },
+                {
+                    "arn": "arn-job-2",
+                    "name": "Apple iPhone 17 Pro",
+                    "status": "FAILED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "11.0",
+                    },
+                },
+            ]
+        }
+
+
+class Test(unittest.TestCase):
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(download_artifact_mock.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+
+        # assert artifacts
+        self.assertEqual(len(artifacts), 12)
+
+        job1_artifacts = [
+            artifact for artifact in artifacts if artifact.get("job_arn") == "arn-job-1"
+        ]
+        self.assertEqual(len(job1_artifacts), 6)
+        a1 = job1_artifacts[0]
+        self.assertEqual(a1["app_type"], "IOS")
+        self.assertEqual(a1["job_arn"], "arn-job-1")
+        self.assertEqual(a1["job_conclusion"], "PASSED")
+        self.assertEqual(a1["job_name"], "Apple iPhone 15")
+        self.assertEqual(a1["os"], "18.0")
+        self.assertEqual(a1["name"], "test spec output")
+
+        job2_artifacts = [
+            artifact for artifact in artifacts if artifact["job_arn"] == "arn-job-2"
+        ]
+        self.assertEqual(len(job2_artifacts), 6)
+        a2 = job2_artifacts[0]
+        self.assertEqual(a2["app_type"], "IOS")
+        self.assertEqual(a2["job_arn"], "arn-job-2")
+        self.assertEqual(a2["job_conclusion"], "PASSED")
+        self.assertEqual(a2["job_name"], "Apple iPhone 17 Pro")
+        self.assertEqual(a2["os"], "11.0")
+        self.assertEqual(a2["name"], "test spec output")
+
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor_debug(self, download_artifact_mock):
path: tools/device-farm-runner/test_run_on_aws_devicefarm.py
user: login: ZainRizvi
body: nit: Can you please rename this test to match the "test_when_then" format the tests further down have? 

That would help explain to future maintainers what exactly this test is intended to validate (which TBH isn't clear to me right now)
created_at: 2025-03-07T16:32:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,314 @@
+import copy
+import json
+import unittest
+from re import M
+from typing import Any, Dict
+from unittest import mock
+from unittest.mock import MagicMock
+
+from run_on_aws_devicefarm import download_artifact, ReportProcessor
+
+
+class MockS3Client:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.upload_file.return_value = None
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+
+class MockDeviceFarmClient:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.list_jobs.return_value = self.mockJobs()
+        self.mock_aws_client.list_suites.return_value = self.mockSuites()
+        self.mock_aws_client.list_tests.return_value = self.mockTests()
+        self.mock_aws_client.list_artifacts.side_effect = (
+            lambda arn, type: self.getArtifacts(arn, type)
+        )
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+    def getArtifacts(self, arn: str, type: str):
+        if type == "FILE":
+            return {
+                "artifacts": [
+                    {
+                        "type": "TESTSPEC_OUTPUT",
+                        "name": "test spec output",
+                        "extension": "output",
+                        "arn": "arn-artifact1",
+                        "url": "url-artifact1",
+                    },
+                ]
+            }
+        elif type == "LOG":
+            return {
+                "artifacts": [
+                    {
+                        "type": "LOG",
+                        "name": "test log",
+                        "extension": "output",
+                        "arn": "arn-artifact2",
+                        "url": "url-artifact2",
+                    },
+                ]
+            }
+        else:
+            return {
+                "artifacts": [
+                    {
+                        "type": "VIDEO",
+                        "name": "test video",
+                        "extension": "output",
+                        "arn": "arn-artifac3",
+                        "url": "url-artifact3",
+                    },
+                ]
+            }
+
+    def mockTests(self) -> Any:
+        return {
+            "tests": [
+                {
+                    "arn": "arn-test1",
+                    "name": "Setup Test",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful test lifecycle of Setup Test",
+                }
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockSuites(self):
+        return {
+            "suites": [
+                {
+                    "arn": "arn-suite1",
+                    "name": "Setup Suite",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful",
+                },
+                {
+                    "arn": "arn-suite2",
+                    "name": "Tests Suite",
+                    "status": "COMPLETED",
+                    "result": "FAILED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Tests passed",
+                },
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockJobs(self):
+        return {
+            "jobs": [
+                {
+                    "arn": "arn-job-1",
+                    "name": "Apple iPhone 15",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "18.0",
+                    },
+                },
+                {
+                    "arn": "arn-job-2",
+                    "name": "Apple iPhone 17 Pro",
+                    "status": "FAILED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "11.0",
+                    },
+                },
+            ]
+        }
+
+
+class Test(unittest.TestCase):
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(download_artifact_mock.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
path: tools/device-farm-runner/test_run_on_aws_devicefarm.py
user: login: ZainRizvi
body: same as comment below, about making these more resilient
created_at: 2025-03-07T16:35:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -531,6 +390,295 @@ def generate_test_configuration(
     return {"extraDataPackageArn": extra_data_arn}
 
 
+@dataclass
+class DeviceFarmReport:
+    name: str
+    arn: str
+    report_type: str
+    status: str
+    result: str
+    counters: Dict[str, str]
+    infos: Dict[str, str]
+    parent_arn: str
+
+
+@dataclass
+class JobReport(DeviceFarmReport):
+    os: str
+
+
+class ReportProcessor:
+    """
+    A helper class to process the modebile test result from AWS Device Farm.
+
+    Usage:
+        processor = ReportProcessor(...) \n
+        processor.start(mobile_run_report)
+    """
+
+    def __init__(
+        self,
+        device_farm_client: Any,
+        s3_client,
+        app_type: str,
+        workflow_id: str,
+        workflow_attempt: int,
+        is_debug: bool = False,
+    ):
+        self.aws_client = device_farm_client
+        self.s3_client = s3_client
+        self.app_type = app_type
+        self.workflow_id = workflow_id
+        self.workflow_attempt = workflow_attempt
+        self.run_report: Optional[DeviceFarmReport] = None
+        self.job_reports: list[JobReport] = []
+        self.test_spec_info_list: list[Dict] = []
+        self.is_debug = is_debug
+
+    # todo(elainewy): add main method to pass run arn
+    def start(self, report: Dict[str, Any]) -> List[Dict[str, str]]:
+        if not report:
+            warn("Missing report, returning...")
+            return []
+
+        arn = report.get("arn", "")
+        if not arn:
+            warn("Missing arn from input report, returning...")
+            return []
+
+        if self.is_debug:
+            info(
+                "[DEBUG MODE] the artifacts won't be uploaded to s3, it should mainly used in local env"
+            )
+        run_report = self._to_run_report(report)
+        self.run_report = run_report
path: tools/device-farm-runner/run_on_aws_devicefarm.py
user: login: ZainRizvi
body: very minor nit, but why not:

```suggestion
        self.run_report = self._to_run_report(report)
```

(feel free to ignore if you prefer the current format)
created_at: 2025-03-07T16:37:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -196,6 +200,8 @@ def parse_args() -> Any:
         help="an optional file to write the list of artifacts from AWS in JSON format",
     )
 
+    parser.add_argument("--debug", action="store_true")
path: tools/device-farm-runner/run_on_aws_devicefarm.py
user: login: huydhn
body: Oh, I didn't realize until I read your comment later that debug mode doesn't upload the artifacts to S3.  This is a nice feature, but I think a help description is needed here to make that clearer.  You could also make some parameters like `--workflow-id` or `--workflow-attempt` optional in debug mode
created_at: 2025-03-08T01:44:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -555,38 +704,50 @@ def main() -> None:
         app_type = "ANDROID_APP" if args.app.endswith(".apk") else "IOS_APP"
         # Upload the test app
         appfile_arn = upload_file(
-            client=client,
+            client=device_farm_client,
             project_arn=project_arn,
             prefix=unique_prefix,
             filename=args.app,
             filetype=app_type,
         )
         info(f"Uploaded app: {appfile_arn}")
 
+    test_to_run = {}
+
     if args.ios_xctestrun:
         app_type = "IOS_APP"
         test_to_run = generate_ios_xctestrun(
-            client, project_arn, unique_prefix, args.ios_xctestrun, args.test_spec
+            device_farm_client,
+            project_arn,
+            unique_prefix,
+            args.ios_xctestrun,
+            args.test_spec,
         )
 
     if args.android_instrumentation_test:
         app_type = "ANDROID_APP"
         test_to_run = generate_android_instrumentation_test(
-            client,
+            device_farm_client,
             project_arn,
             unique_prefix,
             args.android_instrumentation_test,
             args.test_spec,
         )
 
+    if test_to_run == {}:
path: tools/device-farm-runner/run_on_aws_devicefarm.py
user: login: huydhn
body: Strictly speaking, the test suite is not a requirement because Device Farm can fuzz the app to make sure it doesn't crash.  So, only the app is required.  All the current use cases on CI need an test suite though, so having this check is ok.  However, you could achieve this just by setting `required=True` in the `test_group` mutual exclusive group in `argparse` instead of doing an explicit check here
created_at: 2025-03-08T01:59:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -196,6 +200,8 @@ def parse_args() -> Any:
         help="an optional file to write the list of artifacts from AWS in JSON format",
     )
 
+    parser.add_argument("--debug", action="store_true")
path: tools/device-farm-runner/run_on_aws_devicefarm.py
user: login: huydhn
body: On the other hand, another option I think would be useful here is `--quiet`.  I know that I print too many messages like `INFO:root:Run mobile-job-android-13711065730-1-2025-03-07-wrRJMDcB in state RUNNING` in the console log, which could be silent
created_at: 2025-03-08T02:07:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,314 @@
+import copy
+import json
+import unittest
+from re import M
+from typing import Any, Dict
+from unittest import mock
+from unittest.mock import MagicMock
+
+from run_on_aws_devicefarm import download_artifact, ReportProcessor
+
+
+class MockS3Client:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.upload_file.return_value = None
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+
+class MockDeviceFarmClient:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.list_jobs.return_value = self.mockJobs()
+        self.mock_aws_client.list_suites.return_value = self.mockSuites()
+        self.mock_aws_client.list_tests.return_value = self.mockTests()
+        self.mock_aws_client.list_artifacts.side_effect = (
+            lambda arn, type: self.getArtifacts(arn, type)
+        )
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+    def getArtifacts(self, arn: str, type: str):
+        if type == "FILE":
+            return {
+                "artifacts": [
+                    {
+                        "type": "TESTSPEC_OUTPUT",
+                        "name": "test spec output",
+                        "extension": "output",
+                        "arn": "arn-artifact1",
+                        "url": "url-artifact1",
+                    },
+                ]
+            }
+        elif type == "LOG":
+            return {
+                "artifacts": [
+                    {
+                        "type": "LOG",
+                        "name": "test log",
+                        "extension": "output",
+                        "arn": "arn-artifact2",
+                        "url": "url-artifact2",
+                    },
+                ]
+            }
+        else:
+            return {
+                "artifacts": [
+                    {
+                        "type": "VIDEO",
+                        "name": "test video",
+                        "extension": "output",
+                        "arn": "arn-artifac3",
+                        "url": "url-artifact3",
+                    },
+                ]
+            }
+
+    def mockTests(self) -> Any:
+        return {
+            "tests": [
+                {
+                    "arn": "arn-test1",
+                    "name": "Setup Test",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful test lifecycle of Setup Test",
+                }
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockSuites(self):
+        return {
+            "suites": [
+                {
+                    "arn": "arn-suite1",
+                    "name": "Setup Suite",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful",
+                },
+                {
+                    "arn": "arn-suite2",
+                    "name": "Tests Suite",
+                    "status": "COMPLETED",
+                    "result": "FAILED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Tests passed",
+                },
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockJobs(self):
+        return {
+            "jobs": [
+                {
+                    "arn": "arn-job-1",
+                    "name": "Apple iPhone 15",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "18.0",
+                    },
+                },
+                {
+                    "arn": "arn-job-2",
+                    "name": "Apple iPhone 17 Pro",
+                    "status": "FAILED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "11.0",
+                    },
+                },
+            ]
+        }
+
+
+class Test(unittest.TestCase):
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(download_artifact_mock.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+
+        # assert artifacts
+        self.assertEqual(len(artifacts), 12)
+
+        job1_artifacts = [
+            artifact for artifact in artifacts if artifact.get("job_arn") == "arn-job-1"
+        ]
+        self.assertEqual(len(job1_artifacts), 6)
+        a1 = job1_artifacts[0]
+        self.assertEqual(a1["app_type"], "IOS")
+        self.assertEqual(a1["job_arn"], "arn-job-1")
+        self.assertEqual(a1["job_conclusion"], "PASSED")
+        self.assertEqual(a1["job_name"], "Apple iPhone 15")
+        self.assertEqual(a1["os"], "18.0")
+        self.assertEqual(a1["name"], "test spec output")
+
+        job2_artifacts = [
+            artifact for artifact in artifacts if artifact["job_arn"] == "arn-job-2"
+        ]
+        self.assertEqual(len(job2_artifacts), 6)
+        a2 = job2_artifacts[0]
+        self.assertEqual(a2["app_type"], "IOS")
+        self.assertEqual(a2["job_arn"], "arn-job-2")
+        self.assertEqual(a2["job_conclusion"], "PASSED")
+        self.assertEqual(a2["job_name"], "Apple iPhone 17 Pro")
+        self.assertEqual(a2["os"], "11.0")
+        self.assertEqual(a2["name"], "test spec output")
+
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor_debug(self, download_artifact_mock):
path: tools/device-farm-runner/test_run_on_aws_devicefarm.py
user: login: yangw-dev
body: sounds good!
created_at: 2025-03-11T19:42:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,314 @@
+import copy
+import json
+import unittest
+from re import M
+from typing import Any, Dict
+from unittest import mock
+from unittest.mock import MagicMock
+
+from run_on_aws_devicefarm import download_artifact, ReportProcessor
+
+
+class MockS3Client:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.upload_file.return_value = None
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+
+class MockDeviceFarmClient:
+    def __init__(self):
+        self.mock_aws_client = MagicMock()
+        self.mock_aws_client.list_jobs.return_value = self.mockJobs()
+        self.mock_aws_client.list_suites.return_value = self.mockSuites()
+        self.mock_aws_client.list_tests.return_value = self.mockTests()
+        self.mock_aws_client.list_artifacts.side_effect = (
+            lambda arn, type: self.getArtifacts(arn, type)
+        )
+
+    def getMockClient(self) -> MagicMock:
+        return self.mock_aws_client
+
+    def getArtifacts(self, arn: str, type: str):
+        if type == "FILE":
+            return {
+                "artifacts": [
+                    {
+                        "type": "TESTSPEC_OUTPUT",
+                        "name": "test spec output",
+                        "extension": "output",
+                        "arn": "arn-artifact1",
+                        "url": "url-artifact1",
+                    },
+                ]
+            }
+        elif type == "LOG":
+            return {
+                "artifacts": [
+                    {
+                        "type": "LOG",
+                        "name": "test log",
+                        "extension": "output",
+                        "arn": "arn-artifact2",
+                        "url": "url-artifact2",
+                    },
+                ]
+            }
+        else:
+            return {
+                "artifacts": [
+                    {
+                        "type": "VIDEO",
+                        "name": "test video",
+                        "extension": "output",
+                        "arn": "arn-artifac3",
+                        "url": "url-artifact3",
+                    },
+                ]
+            }
+
+    def mockTests(self) -> Any:
+        return {
+            "tests": [
+                {
+                    "arn": "arn-test1",
+                    "name": "Setup Test",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful test lifecycle of Setup Test",
+                }
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockSuites(self):
+        return {
+            "suites": [
+                {
+                    "arn": "arn-suite1",
+                    "name": "Setup Suite",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Successful",
+                },
+                {
+                    "arn": "arn-suite2",
+                    "name": "Tests Suite",
+                    "status": "COMPLETED",
+                    "result": "FAILED",
+                    "counters": {
+                        "total": 1,
+                        "passed": 1,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "Tests passed",
+                },
+            ],
+            "ResponseMetadata": {},
+        }
+
+    def mockJobs(self):
+        return {
+            "jobs": [
+                {
+                    "arn": "arn-job-1",
+                    "name": "Apple iPhone 15",
+                    "status": "COMPLETED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "18.0",
+                    },
+                },
+                {
+                    "arn": "arn-job-2",
+                    "name": "Apple iPhone 17 Pro",
+                    "status": "FAILED",
+                    "result": "PASSED",
+                    "counters": {
+                        "total": 3,
+                        "passed": 3,
+                        "failed": 0,
+                        "warned": 0,
+                        "errored": 0,
+                        "stopped": 0,
+                        "skipped": 0,
+                    },
+                    "message": "fake1",
+                    "device": {
+                        "arn": "device:00",
+                        "name": "Apple iPhone 15",
+                        "manufacturer": "Apple",
+                        "model": "Apple iPhone 15",
+                        "modelId": "A2846",
+                        "formFactor": "PHONE",
+                        "platform": "IOS",
+                        "os": "11.0",
+                    },
+                },
+            ]
+        }
+
+
+class Test(unittest.TestCase):
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+        self.assertEqual(download_artifact_mock.call_count, 12)
+        self.assertEqual(m_s3.getMockClient().upload_file.call_count, 12)
+
+        # assert artifacts
+        self.assertEqual(len(artifacts), 12)
+
+        job1_artifacts = [
+            artifact for artifact in artifacts if artifact.get("job_arn") == "arn-job-1"
+        ]
+        self.assertEqual(len(job1_artifacts), 6)
+        a1 = job1_artifacts[0]
+        self.assertEqual(a1["app_type"], "IOS")
+        self.assertEqual(a1["job_arn"], "arn-job-1")
+        self.assertEqual(a1["job_conclusion"], "PASSED")
+        self.assertEqual(a1["job_name"], "Apple iPhone 15")
+        self.assertEqual(a1["os"], "18.0")
+        self.assertEqual(a1["name"], "test spec output")
+
+        job2_artifacts = [
+            artifact for artifact in artifacts if artifact["job_arn"] == "arn-job-2"
+        ]
+        self.assertEqual(len(job2_artifacts), 6)
+        a2 = job2_artifacts[0]
+        self.assertEqual(a2["app_type"], "IOS")
+        self.assertEqual(a2["job_arn"], "arn-job-2")
+        self.assertEqual(a2["job_conclusion"], "PASSED")
+        self.assertEqual(a2["job_name"], "Apple iPhone 17 Pro")
+        self.assertEqual(a2["os"], "11.0")
+        self.assertEqual(a2["name"], "test spec output")
+
+    @mock.patch("run_on_aws_devicefarm.download_artifact")
+    def test_reportProcessor_debug(self, download_artifact_mock):
+        # setup
+        m_df = MockDeviceFarmClient()
+        m_s3 = MockS3Client()
+        fakeReport = {
+            "name": "test",
+            "arn": "arn-run-report",
+            "status": "COMPLETED",
+            "result": "PASSED",
+            "counters": {"total": 3, "passed": 3, "failed": 0, "warned": 0},
+        }
+        processor = ReportProcessor(
+            m_df.getMockClient(), m_s3.getMockClient(), "IOS", "wf1", 1, True
+        )
+
+        # execute
+        artifacts = processor.start(fakeReport)
+
+        # assert aws client calls
+        m_df.getMockClient().list_jobs.assert_called_once()
+        self.assertEqual(m_df.getMockClient().list_suites.call_count, 2)
+        self.assertEqual(m_df.getMockClient().list_tests.call_count, 4)
+        self.assertEqual(download_artifact_mock.call_count, 12)
path: tools/device-farm-runner/test_run_on_aws_devicefarm.py
user: login: yangw-dev
body: sounds good!
created_at: 2025-03-11T19:42:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -196,6 +200,8 @@ def parse_args() -> Any:
         help="an optional file to write the list of artifacts from AWS in JSON format",
     )
 
+    parser.add_argument("--debug", action="store_true")
path: tools/device-farm-runner/run_on_aws_devicefarm.py
user: login: yangw-dev
body: added the help:)
created_at: 2025-03-11T19:55:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -555,38 +704,50 @@ def main() -> None:
         app_type = "ANDROID_APP" if args.app.endswith(".apk") else "IOS_APP"
         # Upload the test app
         appfile_arn = upload_file(
-            client=client,
+            client=device_farm_client,
             project_arn=project_arn,
             prefix=unique_prefix,
             filename=args.app,
             filetype=app_type,
         )
         info(f"Uploaded app: {appfile_arn}")
 
+    test_to_run = {}
+
     if args.ios_xctestrun:
         app_type = "IOS_APP"
         test_to_run = generate_ios_xctestrun(
-            client, project_arn, unique_prefix, args.ios_xctestrun, args.test_spec
+            device_farm_client,
+            project_arn,
+            unique_prefix,
+            args.ios_xctestrun,
+            args.test_spec,
         )
 
     if args.android_instrumentation_test:
         app_type = "ANDROID_APP"
         test_to_run = generate_android_instrumentation_test(
-            client,
+            device_farm_client,
             project_arn,
             unique_prefix,
             args.android_instrumentation_test,
             args.test_spec,
         )
 
+    if test_to_run == {}:
path: tools/device-farm-runner/run_on_aws_devicefarm.py
user: login: yangw-dev
body: sounds good, remove this check
created_at: 2025-03-11T19:56:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: vercel[bot]
created_at: 2025-03-06T19:18:39Z
body: [vc]: #Apzobu7Qfp2J1Sgi/4v2CuI2zmLgWAWJJBjmjCoH3NM=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJ0b3JjaGNpIiwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2Zib3BlbnNvdXJjZS90b3JjaGNpLzdaQkZXeHdpZDZKdVFvS2VxQlBGeDlhbUd6YWkiLCJuZXh0Q29tbWl0U3RhdHVzIjoiSUdOT1JFRCIsInJvb3REaXJlY3RvcnkiOiJ0b3JjaGNpIiwicHJldmlld1VybCI6InRvcmNoY2ktZ2l0LWFkZHNwZWMtZmJvcGVuc291cmNlLnZlcmNlbC5hcHAifV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)


<details><summary>1 Skipped Deployment</summary>

| Name | Status | Preview | Updated (UTC) |
| :--- | :----- | :------ | :------ |
| **torchci** | ⬜️ Ignored ([Inspect](https://vercel.com/fbopensource/torchci/7ZBFWxwid6JuQoKeqBPFx9amGzai)) | [Visit Preview](https://torchci-git-addspec-fbopensource.vercel.app) | Mar 11, 2025 11:47pm |
</details>


author_association: NONE
user: login: huydhn
created_at: 2025-03-08T02:05:13Z
body: Does the console log from the mobile test jobs look correct https://github.com/pytorch/test-infra/actions/runs/13711065730/job/38347515619?pr=6371?

You could also test this on ExecuTorch side (where it really matters) by quickly submitting a mock PR to ExecuTorch to use your branch `addSpec`:

* https://github.com/pytorch/executorch/blob/main/.github/workflows/android-perf.yml#L374
* and https://github.com/pytorch/executorch/blob/main/.github/workflows/apple-perf.yml#L438 to your branch
author_association: CONTRIBUTOR
user: login: huydhn
created_at: 2025-03-08T02:08:32Z
body: Look like a small bug on this part https://github.com/pytorch/test-infra/actions/runs/13711065730/job/38347515619?pr=6371#step:15:81, f-string something I guess

```
INFO:root:{' ' * indent} start gathering artifacts
```
author_association: CONTRIBUTOR
user: login: huydhn
created_at: 2025-03-10T18:29:44Z
body: cc @Camyll You probably want to take a look at this script because it's related to https://github.com/pytorch/executorch/issues/8245.  This is the script we use to run Device Farm job on CI.  In the current form, it's not easy to use locally but I think we can improve / refactor it to implement https://github.com/pytorch/executorch/issues/8245
author_association: CONTRIBUTOR
user: login: yangw-dev
created_at: 2025-03-11T23:48:53Z
body: > Does the console log from the mobile test jobs look correct https://github.com/pytorch/test-infra/actions/runs/13711065730/job/38347515619?pr=6371?
> 
> You could also test this on ExecuTorch side (where it really matters) by quickly submitting a mock PR to ExecuTorch to use your branch `addSpec`:
> 
> * https://github.com/pytorch/executorch/blob/main/.github/workflows/android-perf.yml#L374
> * and https://github.com/pytorch/executorch/blob/main/.github/workflows/apple-perf.yml#L438 to your branch

It seems like the excuTorch side of run has issues in both android and apple
author_association: CONTRIBUTOR
user: login: yangw-dev
created_at: 2025-03-12T02:34:09Z
body: > Look like a small bug on this part https://github.com/pytorch/test-infra/actions/runs/13711065730/job/38347515619?pr=6371#step:15:81, f-string something I guess
> 
> ```
> INFO:root:{' ' * indent} start gathering artifacts
> ```

solved
author_association: CONTRIBUTOR