Repository: swiftlang/swift-testing

Issue #671: Ability to uniquely identify a parameterized test
Created: 2024-09-06T21:59:16Z
Description
Test is identifiable, which is handy to distinguish between tests and avoid letting "global" state to bleed between tests since they can be bucketed by Test.ID. Unfortunately, this breaks when it comes to parameterized tests, because each parameterized case is the same test and thus has the same identity.
Expected behavior
Ideally, Test.ID would incorporate the SPI-internal Test.Case.ID into its identity, or Test.Case.ID would be publicly accessible.
Actual behavior
Currently there's no way to access a parameterized test case's identity because it's SPI.
Steps to reproduce
No response
swift-testing version/commit hash
2e9df4f
Swift & OS version (output of swift --version && uname -a)
n/a

Duplicate issue #1037: Include the location of a parameterized test argument which recorded an issue (if possible) in console output for better traceability
Created: 2025-03-25T14:24:52Z
Motivation
When using @Test(arguments:) to test multiple input patterns in a single test function, it can be difficult to identify which argument caused a failure, especially when several similar test cases are used.
For example, consider the following test setup:
Supporting code:
struct User: Equatable {
    let name: String
    let age: Int
    let email: String
    let username: String
    let password: String
    let isActive: Bool
    let signupDate: Date
    let lastLoginDate: Date
    let phoneNumber: String
    let address: String
    let bio: String
}

extension User {
    static func mock(
        name: String = "John Doe",
        age: Int = 30,
        email: String = "john@example.com",
        username: String = "johndoe",
        password: String = "password123",
        isActive: Bool = true,
        signupDate: Date = Date(),
        lastLoginDate: Date = Date(),
        phoneNumber: String = "000-0000-0000",
        address: String = "123 Apple Street",
        bio: String = "Just a test user."
    ) -> Self {
        .init(
            name: name,
            age: age,
            email: email,
            username: username,
            password: password,
            isActive: isActive,
            signupDate: signupDate,
            lastLoginDate: lastLoginDate,
            phoneNumber: phoneNumber,
            address: address,
            bio: bio
        )
    }
}
Test code:
struct UserTests {
    @Test(arguments: [
        User.mock(),
        .mock(name: "hoge", isActive: false),
        .mock(lastLoginDate: .distantPast, bio: "Hello World"),

        // Multiple mock cases follow

        .mock(email: "fuga@example.com"),
        .mock(phoneNumber: "999-9999-9999")
    ])
    func someTest(_ user: User) async throws {
        let expected: User = .mock() // In practice, this would call some logic
        #expect(user == expected)
    }
}
When a test fails, the output looks like this:
✘ Test someTest(_:) recorded an issue with 1 argument user → User(name: "John Doe", age: 30, email: "john@example.com", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World") at TestTests.swift:72:9: Expectation failed: (user → User(name: "John Doe", age: 30, email: "john@example.com", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World")) == (expected → User(name: "John Doe", age: 30, email: "john@example.com", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 2025-03-25 09:59:04 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Just a test user."))
However, this output alone doesn’t clearly indicate which argument (i.e., which entry in the arguments array) caused the failure. This makes it difficult to trace which case needs to be fixed, especially in large or complex test sets.
Proposed solution
A helpful enhancement would be to include the argument’s index in the failure output. For example, if the second item in the arguments array fails, the output might look like this:
✘ Test someTest(_:) [arguments index: 2] recorded an issue with 1 argument user → User(name: "John Doe", age: 30, email: "john@example.com", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World") at TestTests.swift:72:9: Expectation failed: (user → User(name: "John Doe", age: 30, email: "john@example.com", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 0001-01-01 00:00:00 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Hello World")) == (expected → User(name: "John Doe", age: 30, email: "john@example.com", username: "johndoe", password: "password123", isActive: true, signupDate: 2025-03-25 09:59:04 +0000, lastLoginDate: 2025-03-25 09:59:04 +0000, phoneNumber: "000-0000-0000", address: "123 Apple Street", bio: "Just a test user."))
This small addition would make it significantly easier to identify the failing case and improve the debugging experience, especially when dealing with multiple test arguments.
Currently, developers can work around this by wrapping the argument in a custom struct with a label, like so:
struct UserTestCase {
    let label: String
    let user: User
}

@Test(arguments: [
    UserTestCase(label: "default", user: .mock()),
    UserTestCase(label: "inactive", user: .mock(name: "hoge", isActive: false)),
])
func someTest(_ testCase: UserTestCase) async throws {
    #expect(testCase.user == .mock(), "Failure in case: \(testCase.label)")
}
While this pattern works, it adds boilerplate and isn’t ideal for simpler test cases.
Including the argument index in the default output would be a lightweight and widely useful improvement that benefits many test scenarios.
Alternatives considered
No response
Additional information
No response

Closing comment (2025-03-25T15:35:35Z): Improvements to test case enumeration are currently tracked by #671.

And I'm pleased to report that one subtask of that, #1000, recently landed and provides a mechanism to distinguish test case IDs.
That being said, I think neither #671 nor #1000 will truly address the complaint here. It seems to me that the basic problem is that when a test case for a parameterized test argument records an issue, it can be difficult to trace back to the location in source where that argument was defined. #1000 can help address that issue, but won't solve it on its own.
I think we should find a way to capture more information about the source and location of test arguments so that in the results we could actually note the source location where the argument was passed. That, to me, would be even more powerful than including each arguments' index in the collection; an index is just an indirect way to refer to an element, but if you can jump directly to the location of that element in source code, all the better. (This all assumes that a collection literal is being passed to @Test(arguments:), of course; fully dynamic construction of test arguments could not benefit from this idea since the macro isn't exposed to them.) I'll hold this issue to track this larger idea, since it has come up before.