Repository: codemagic-ci-cd/cli-tools (language: python)

Issue:
Error when running `xcode-project run-tests` tool (models/xctests/converter.py): could not convert string to float
Hey, we encountered `ValueError: could not convert string to float: '1m 4'` error when using `xcode-project run-tests` CLI tool
Seems to only happen when tests succeed.

Xcode version: 16.0
Failed Workflow ID if it helps: `671fb636267ec43c813905f2`

```
[16:18:13 28-10-2024] WARNING cli_app.py:135 > [31mExecuting "xcode-project run-tests" failed unexpectedly. Detailed logs are available at "/var/folders/w2/rrf5p87d1bbfyphxc7jdnyvh0000gn/T/codemagic-28-10-24.log". [0m
[16:18:13 28-10-2024] ERROR cli_app.py:137 > Exception traceback:
Traceback (most recent call last):
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/cli/cli_app.py", line 243, in invoke_cli
    CliApp._running_app._invoke_action(args)
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/cli/cli_app.py", line 184, in _invoke_action
    return cli_action(**action_args)
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/cli/action.py", line 83, in wrapper
    return func(self, *args, **kwargs)
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/tools/xcode_project.py", line 506, in run_test
    test_suites, xcresult = self._get_test_suites(
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/tools/xcode_project.py", line 766, in _get_test_suites
    test_suites = XcResultConverter.xcresult_to_junit(xcresult)
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/models/xctests/converter.py", line 56, in xcresult_to_junit
    return cls(xcresult).convert()
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/models/xctests/converter.py", line 333, in convert
    test_suites = [
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/models/xctests/converter.py", line 334, in <listcomp>
    self._get_test_suite(xc_test_suite_node, xc_summary)
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/models/xctests/converter.py", line 320, in _get_test_suite
    time=sum(cls._get_test_node_duration(xc_test_case) for xc_test_case in xc_test_cases),
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/models/xctests/converter.py", line 320, in <genexpr>
    time=sum(cls._get_test_node_duration(xc_test_case) for xc_test_case in xc_test_cases),
  File "/Users/builder/.pyenv/versions/3.8.13/lib/python3.8/site-packages/codemagic/models/xctests/converter.py", line 251, in _get_test_node_duration
    return float(duration)
ValueError: could not convert string to float: '1m 4'
[16:18:13 28-10-2024] DEBUG cli_app.py:286 > [35m------------------------------------------------------------[0m
[16:18:13 28-10-2024] DEBUG cli_app.py:287 > [35mCompleted XcodeProject run-tests in 08:51 with status code 9[0m
[16:18:13 28-10-2024] DEBUG cli_app.py:288 > [35m------------------------------------------------------------[0m
```
Update `xctest` test result parsing for Xcode 16
**This PR is a follow up to issue #423 and its fix in #424.**

Xcode 16.0 introduced changes to `xcresulttool` API which deprecated `xcresult` parsing options that were previously working:
```shell
xcresulttool get object --path <xcresult-path> [--id <object-id>]
```
was replaced with
```shell
xcresulttool get test-results <subcommand>
```
where subcommand is any of `summary`, `tests`, `test-details`, `activities`, `insights`.

Along with the CLI change, the output format was also significantly altered.

This PR adds Python mappings for `xcresulttool get test-results` outputs and uses those to generate JUnit test suites. In most cases the conversion results are identical to what it used to be with legacy options, but for some time already Xcode does not differentiate test cases that errored from those for which some assertions failed. While the legacy implementation kept track of failed results separately, this is not possible with new `xcresulttool` output and now all test cases that failed due to whatever reasons are counted as `errored`.

Examples of `xcresulttool get test-result` JSON outputs can be seen under mocks in `tests/models/xctests/mocks` directory.

Conversion from `xcresult` to JUnit compatible test suite is still done in `XcResultConverter` class whose API remained fully backwards compatible. Now when selected Xcode version is 16+ then new conversion implementation is used, and the legacy one otherwise.

<details>
<summary>Models JSON schemas</summary>

- JSON schema for `xcrun xcresulttool get test-results tests` output

```json
{
    "schemas": {
        "Tests": {
            "type": "object",
            "properties": {
                "testPlanConfigurations": {
                    "type": "array",
                    "items": {
                        "$ref": "#/schemas/Configuration"
                    }
                },
                "devices": {
                    "type": "array",
                    "items": {
                        "$ref": "#/schemas/Device"
                    }
                },
                "testNodes": {
                    "type": "array",
                    "items": {
                        "$ref": "#/schemas/TestNode"
                    }
                }
            },
            "required": [
                "testPlanConfigurations",
                "devices",
                "testNodes"
            ]
        },
        "Configuration": {
            "type": "object",
            "properties": {
                "configurationId": {
                    "type": "string"
                },
                "configurationName": {
                    "type": "string"
                }
            },
            "required": [
                "configurationId",
                "configurationName"
            ]
        },
        "Device": {
            "type": "object",
            "properties": {
                "deviceId": {
                    "type": "string"
                },
                "deviceName": {
                    "type": "string"
                },
                "architecture": {
                    "type": "string"
                },
                "modelName": {
                    "type": "string"
                },
                "platform": {
                    "type": "string"
                },
                "osVersion": {
                    "type": "string"
                }
            },
            "required": [
                "deviceName",
                "architecture",
                "modelName",
                "osVersion"
            ]
        },
        "TestNode": {
            "type": "object",
            "properties": {
                "nodeIdentifier": {
                    "type": "string"
                },
                "nodeType": {
                    "$ref": "#/schemas/TestNodeType"
                },
                "name": {
                    "type": "string"
                },
                "details": {
                    "type": "string"
                },
                "duration": {
                    "type": "string"
                },
                "result": {
                    "$ref": "#/schemas/TestResult"
                },
                "tags": {
                    "type": "array",
                    "items": {
                        "type": "string"
                    }
                },
                "children": {
                    "type": "array",
                    "items": {
                        "$ref": "#/schemas/TestNode"
                    }
                }
            },
            "required": [
                "nodeType",
                "name"
            ]
        },
        "TestResult": {
            "type": "string",
            "enum": [
                "Passed",
                "Failed",
                "Skipped",
                "Expected Failure",
                "unknown"
            ]
        },
        "TestNodeType": {
            "type": "string",
            "enum": [
                "Test Plan",
                "Unit test bundle",
                "UI test bundle",
                "Test Suite",
                "Test Case",
                "Device",
                "Test Plan Configuration",
                "Arguments",
                "Repetition",
                "Test Case Run",
                "Failure Message",
                "Source Code Reference",
                "Attachment",
                "Expression",
                "Test Value"
            ]
        }
    }
}
```

- JSON schema for `xcrun xcresulttool get test-results summary` output

```json
{
    "schemas": {
        "Summary": {
            "type": "object",
            "properties": {
                "title": {
                    "type": "string"
                },
                "startTime": {
                    "type": "number",
                    "format": "double",
                    "description": "Date as a UNIX timestamp (seconds since midnight UTC on January 1, 1970)"
                },
                "finishTime": {
                    "type": "number",
                    "format": "double",
                    "description": "Date as a UNIX timestamp (seconds since midnight UTC on January 1, 1970)"
                },
                "environmentDescription": {
                    "type": "string",
                    "description": "Description of the Test Plan, OS, and environment that was used during testing"
                },
                "topInsights": {
                    "type": "array",
                    "items": {
                        "$ref": "#/schemas/InsightSummary"
                    }
                },
                "result": {
                    "$ref": "#/schemas/TestResult"
                },
                "totalTestCount": {
                    "type": "integer"
                },
                "passedTests": {
                    "type": "integer"
                },
                "failedTests": {
                    "type": "integer"
                },
                "skippedTests": {
                    "type": "integer"
                },
                "expectedFailures": {
                    "type": "integer"
                },
                "statistics": {
                    "type": "array",
                    "items": {
                        "$ref": "#/schemas/Statistic"
                    }
                },
                "devicesAndConfigurations": {
                    "$ref": "#/schemas/DeviceAndConfigurationSummary"
                },
                "testFailures": {
                    "$ref": "#/schemas/TestFailure"
                }
            },
            "required": [
                "title",
                "environmentDescription",
                "topInsights",
                "result",
                "totalTestCount",
                "passedTests",
                "failedTests",
                "skippedTests",
                "expectedFailures",
                "statistics",
                "devicesAndConfigurations",
                "testFailures"
            ],
        },
        "InsightSummary": {
            "type": "object",
            "properties": {
                "impact": {
                    "type": "string"
                },
                "category": {
                    "type" "string"
                },
                "text": {
                    "type" "string"
                }
            },
            "required": [
                "impact",
                "category",
                "text"
            ]
        },
        "TestResult": {
            "type": "string",
            "enum": [
                "Passed",
                "Failed",
                "Skipped",
                "Expected Failure",
                "unknown"
            ]
        },
        "Statistic": {
            "type": "object",
            "properties": {
                "title": {
                    "type": "string"
                },
                "subtitle": {
                    "type": "string"
                }
            },
            "required": [
                "title",
                "subtitle"
            ]
        },
        "DeviceAndConfigurationSummary": {
            "type": "object",
            "properties": {
                "device": {
                    "$ref": "#/schemas/Device"
                },
                "testPlanConfiguration": {
                    "$ref": "#/schemas/Configuration"
                },
                "passedTests": {
                    "type": "integer"
                },
                "failedTests": {
                    "type": "integer"
                },
                "skippedTests": {
                    "type": "integer"
                },
                "expectedFailures": {
                    "type": "integer"
                }
            },
            "required": [
                "device",
                "testPlanConfiguration",
                "passedTests",
                "failedTests",
                "skippedTests",
                "expectedFailures"
            ]
        },
        "Device": {
            "type": "object",
            "properties": {
                "deviceId": {
                    "type": "string"
                },
                "deviceName": {
                    "type": "string"
                },
                "architecture": {
                    "type": "string"
                },
                "modelName": {
                    "type": "string"
                },
                "platform": {
                    "type": "string"
                },
                "osVersion": {
                    "type": "string"
                }
            },
            "required": [
                "deviceName",
                "architecture",
                "modelName",
                "osVersion"
            ]
        },
        "Configuration": {
            "type": "object",
            "properties": {
                "configurationId": {
                    "type": "string"
                },
                "configurationName": {
                    "type": "string"
                }
            },
            "required": [
                "configurationId",
                "configurationName"
            ]
        },
        "TestFailure": {
            "type": "object",
            "properties": {
                "testName": {
                    "type": "string"
                },
                "targetName": {
                    "type": "string"
                },
                "failureText": {
                    "type": "string"
                },
                "testIdentifier": {
                    "type": "integer",
                    "format": "int64"
                }
            },
            "required": [
                "testName",
                "targetName",
                "failureText",
                "testIdentifier"
            ]
        }
    }
}
```
</details>

**QA notes:**

Tested with the following Xcode versions:
- 15.0.1 (15A507)
- 15.1 (15C65)
- 15.2 (15C500b)
- 15.3 (15E204a)
- 15.4 (15F31d)
- 16.0 (16A242d)
- 16.1 Beta 3 (16B5029d)

**Updated actions:**
- `xcode-project run-tests`
- `xcode-project test-summary`
- `xcode-project junit-test-results`


Discussion / hints:
@priitlatt Seems related to this change: https://github.com/codemagic-ci-cd/cli-tools/pull/431#discussion_r1804806834
@sergiymomot thanks for reporting this! I'll look into it tomorrow and let you know once we have a fix in place.
Changelog was updated in [e0fd0ef](https://github.com/codemagic-ci-cd/cli-tools/pull/431/commits/e0fd0ef4ba969df216d68545b97f24a7c9f75bf2) to reflect modifications of JUnit `<testsuites>` element `failures` and `errors` attributes.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 25948999..d91a9544 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,9 @@
+Version 0.54.1
+-------------
+
+**Bugfixes**
+- Fix testcase duration parsing from `XCResult` bundle when using Xcode 16.0+. [PR #433](https://github.com/codemagic-ci-cd/cli-tools/pull/433)
+
 Version 0.54.0
 -------------
 
diff --git a/pyproject.toml b/pyproject.toml
index f9a06334..b5163937 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "codemagic-cli-tools"
-version = "0.54.0"
+version = "0.54.1"
 description = "CLI tools used in Codemagic builds"
 readme = "README.md"
 authors = [
diff --git a/src/codemagic/__version__.py b/src/codemagic/__version__.py
index eeb532ca..631b097c 100644
--- a/src/codemagic/__version__.py
+++ b/src/codemagic/__version__.py
@@ -1,5 +1,5 @@
 __title__ = "codemagic-cli-tools"
 __description__ = "CLI tools used in Codemagic builds"
-__version__ = "0.54.0.dev"
+__version__ = "0.54.1.dev"
 __url__ = "https://github.com/codemagic-ci-cd/cli-tools"
 __licence__ = "GNU General Public License v3.0"


Test patch:
diff --git a/src/codemagic/models/xctests/converter.py b/src/codemagic/models/xctests/converter.py
index 8b516afc..34f53677 100644
--- a/src/codemagic/models/xctests/converter.py
+++ b/src/codemagic/models/xctests/converter.py
@@ -5,6 +5,7 @@
 from abc import ABC
 from abc import abstractmethod
 from datetime import datetime
+from datetime import timedelta
 from typing import Iterator
 from typing import List
 from typing import Optional
@@ -240,15 +241,30 @@ def _get_test_case_skipped(cls, xc_test_case: XcTestNode) -> Optional[Skipped]:
 
         return Skipped(message="\n".join(skipped_messages))
 
+    @classmethod
+    def parse_xcresult_test_node_duration_value(cls, xc_duration: str) -> float:
+        duration = timedelta()
+
+        try:
+            for part in xc_duration.split():
+                part_value = float(part[:-1].replace(",", "."))
+                if part.endswith("s"):
+                    duration += timedelta(seconds=part_value)
+                elif part.endswith("m"):
+                    duration += timedelta(minutes=part_value)
+                else:
+                    raise ValueError("Unknown duration unit")
+        except ValueError as ve:
+            raise ValueError("Invalid duration", xc_duration) from ve
+
+        return duration.total_seconds()
+
     @classmethod
     def _get_test_node_duration(cls, xc_test_case: XcTestNode) -> float:
         if not xc_test_case.duration:
             return 0.0
 
-        duration = xc_test_case.duration.replace(",", ".")
-        if duration.endswith("s"):
-            duration = duration[:-1]
-        return float(duration)
+        return cls.parse_xcresult_test_node_duration_value(xc_test_case.duration)
 
     @classmethod
     def _get_test_case(cls, xc_test_case: XcTestNode, xc_test_suite: XcTestNode) -> TestCase:
diff --git a/src/codemagic/models/xctests/xcresult/xcode_16_xcresult.py b/src/codemagic/models/xctests/xcresult/xcode_16_xcresult.py
index efafd710..fac5724d 100644
--- a/src/codemagic/models/xctests/xcresult/xcode_16_xcresult.py
+++ b/src/codemagic/models/xctests/xcresult/xcode_16_xcresult.py
@@ -164,7 +164,7 @@ def from_dict(cls, d: Dict[str, Any]) -> XcTestInsight:
 class XcTests(XcModel):
     """
     Model definitions for `xcresulttool get test-results tests` output.
-    Check schema with `xcrun xcresulttool help get test-results tests.
+    Check schema with `xcrun xcresulttool help get test-results tests`.
     """
 
     devices: List[XcDevice]
diff --git a/tests/models/xctests/converter/test_xcode_16_converter.py b/tests/models/xctests/converter/test_xcode_16_converter.py
index 1976bdb2..bf213b86 100644
--- a/tests/models/xctests/converter/test_xcode_16_converter.py
+++ b/tests/models/xctests/converter/test_xcode_16_converter.py
@@ -127,3 +127,26 @@ def test_converter(mock_datetime, expected_properties):
         time=3.0,
         error=Error(message="banaanUITests.swift:40: failed - Bad UI", type="Failure"),
     )
+
+
+@pytest.mark.parametrize(
+    ("duration", "expected_value"),
+    (
+        ("0,00045s", 0.00045),
+        ("0,002s", 0.002),
+        ("0,0052s", 0.0052),
+        ("0,26s", 0.26),
+        ("0,2s", 0.2),
+        ("2s", 2.0),
+        ("3s", 3.0),
+        ("1m 3,01s", 63.01),
+        ("1m 3.01s", 63.01),
+        ("1m 4s", 64.0),
+        ("2m 26s", 146.0),
+        ("5m 3s", 303.0),
+        ("6m", 360.0),
+    ),
+)
+def test_parse_xcresult_test_node_duration_value(duration, expected_value):
+    value = Xcode16XcResultConverter.parse_xcresult_test_node_duration_value(duration)
+    assert value == pytest.approx(expected_value)