Repository: swiftlang/swift-testing

Issue #480: Using recording an issue inside of a detached task causes a crash.
Created: 2024-06-13T14:04:54Z
Description
This is something I learned while researching potential ways to address #475. Which, at the time I wrote that issue, I thought would have just resulted in the failure being silently dropped.
The current way of determining what the current reporting tests natively in Swift-Testing is broken when you run an expect inside of a detached task. Technically, you get a failing test result, but it's because the test runner crashed. I haven't yet spelunked enough through the codebase to figure out where that crash is.
This is because Test.current is stored as a task-local value. Which means that when you make an expect inside of a detached task, Swift Testing has lost that context, which means that Test.current is incorrectly (from the semantics of the current running test) as nil.
I don't know the concurrency system well enough to be able to suggest a good solution. The issue is that, because Swift Testing supports task-based parallelization, the value of Test.current strongly depends on the task it's running on. Which makes having a global Test.current like what's in swift-corelibs-xctest the wrong solution.
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
This test should pass. The test runner crashes with a non-zero error instead.
import Testing

@Test func CheckingExpectWithDetachedTasks() async {
    await Task.detached {
        #expect(Bool(false))
    }.value
}

swift-testing version/commit hash
Testing Library Version: 75.7 (arm64e-apple-macos13.0)
Swift & OS version (output of swift --version ; uname -a)
No response

Duplicate issue #804: test with `CheckedContinuation` loses testing context and crashes
Created: 2024-11-04T19:37:58Z
Description
Consider an Observable model that interacts with an actor model:
actor ActorModel {
   func doMyThing() -> Int {
      return 1
   }
}

@Observable final class ObservableModel: @unchecked Sendable {
   private(set) var actorValue: Int?
   private let actorModel = ActorModel()

   @MainActor func buttonPressed() {
      actorValue = nil // reset

      Task.detached {
         let newActorValue = await self.actorModel.doMyThing()
         #expect(newActorValue == 1)
         Task { @MainActor [weak self] in
            self?.actorValue = 2 // *** !! OOPS ***
         }
      }
   }
}

To test that buttonPressed() resets actorValue and then asynchronously assigns actorValue after calling actorModel.doMyThing(), we can construct a test with observation tracking inside a checked continuation:
@Test @MainActor func shouldFail() async throws {
   let observableModel = ObservableModel()
   try #require(observableModel.actorValue == nil)

   try await withCheckedThrowingContinuation { (cont: CheckedContinuation<Void, Error>) -> Void in
      @Sendable func observeActorValue(count: Int) {
         withObservationTracking {
            print("observing observableModel.actorValue=\(String(describing: observableModel.actorValue)) count=\(count)")
         } onChange: {
            Task { @MainActor in
               print("onChange observableModel.actorValue=\(String(describing: observableModel.actorValue))")
               switch count {
               case 0:
                  #expect(observableModel.actorValue == nil)
               case 1:
                  #expect(observableModel.actorValue == 1)
                  cont.resume()
               default:
                  Issue.record("recursed on observeActorModel (\(count) times, expected once")
               }
               observeActorValue(count: count + 1) // recurse
            }
         }
      }
      observeActorValue(count: 0)

      observableModel.buttonPressed()

      do {
         try #require(observableModel.actorValue == nil)
      } catch {
         cont.resume(throwing: error)
      }
   }
}

Because of the OOPS, the #expect in case 1: fails. But we also get a crash:

◇ Test shouldFail() started.
observing observableModel.actorValue=nil count=0
onChange observableModel.actorValue=nil
observing observableModel.actorValue=nil count=1
onChange observableModel.actorValue=Optional(2)
✘ Test «unknown» recorded an issue at CheckedContinuationMultipleEventsShouldFail.swift:24:19: Expectation failed: (observableModel.actorValue → 2) == 1
XCTest/HarnessEventHandler.swift:168: Fatal error: Issue recorded event did not contain a test

The test passes if we set the correct value in buttonPressed() (ie no OOPS), but are we certain the passing #expect is realized by the testing context?
Expected behavior
The test should record an issue and not crash.
Actual behavior
The test crashed.
Steps to reproduce
BugSwiftTestingCheckedContinuation.zip
swift-testing version/commit hash
Version 16.1 (16B40)
Swift & OS version (output of swift --version ; uname -a)
swift-driver version: 1.115 Apple Swift version 6.0.2 (swiftlang-6.0.2.1.2 clang-1600.0.26.4)
Target: arm64-apple-macosx15.0
Darwin BernouliConvergenator.local 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:00:32 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6030 arm64

Closing comment (2024-11-04T20:22:18Z): Duplicate of #480