Change:
<!-- Do Not Delete This! bugreporttemplate --<!-- Please read our Rules of Conduct: --<!-- Read our guide about submitting issues: --<!-- Search existing issues to avoid creating duplicates. --<!-- Test using the latest Insiders build to see if your issue has already been fixed: --<!-- Instead of creating your report here, use 'Report Issue' from the 'Help' menu in VS Code to pre-fill useful information. --<!-- If you answered No above, use 'Help: Start Extension Bisect' from Command Palette to try to identify the cause. --<!-- Issues caused by an extension need to be reported directly to the extension publisher. The 'Help Report Issue' dialog can assist with this. --- VS Code Version: 1.64.0-insider () In the Catch2 C++ test framework it is possible to have multiple test results for the same test in one run. This happens for example for the following code: In this scenario, the section "S" is run twice, and the first time it fails, the second time it succeeds. The C++ TestMate extension will see the two test runs, and call for the first one, and for the second, with the same for both calls. This results in the confusing display that the section is green, even though it shows a failure inside: <img width="653" alt="image" src="https://user-" does it make sense to change and (et al) so that when called multiple times, it sets the item's state to the max of the old and the new state? This seems to be analogous to what happens with the test messages (they are appended, and doesn't clear them). Or would you say it's the extension's responsibility to not set the state more than once for a given TestItem? (Cc Or maybe it is even Catch2's responsibility to accumulate the results for any section that is run more than once?
IMHO, Catch2 reports it and definitely not the responsibility of vscode. It leaves it to me.. (Though I don't know how I can solve this properly)
Why definitely? It doesn't seem so unreasonable to me. As I said, it already does something similar with the test messages.
As I think of it, vscode part is just the UI, the controlling is in my hand. More precisely and . the test provider is the .
I'm not so sure in this case. VS Code is also responsible for setting T's state correctly, you don't do that at all (as far as I'm aware). And note that it gets that one right, so it has some logic already for computing the resulting state of all child runs if there are multiple for the same item.
I would be very interested in your opinion on this.
Yea, I think it would be reasonable to no-op if the extension tries to change the test state to one with a lower priority. For instance, going from passed-skipped or failed-passed.
can you add some steps?
the test sample: this block F5.Verify that, with these changes, test that should pass remain passed in the UI, and tests that failed remain failed.

Discussion:
import { IObservableValue, MutableObservableValue, staticObservableValue } from 'vs/workbench/contrib/testing/common/observableValue'; import { IRichLocation, ISerializedTestResults, ITestItem, ITestMessage, ITestOutputMessage, ITestRunTask, ITestTaskState, ResolvedTestRunRequest, TestItemExpandState, TestMessageType, TestResultItem, TestResultState } from 'vs/workbench/contrib/testing/common/testCollection'; import { TestCoverage } from 'vs/workbench/contrib/testing/common/testCoverage'; <del> import { maxPriority, statesInOrder } from 'vs/workbench/contrib/testing/common/testingStates'; </del> <ins> import { maxPriority, statesInOrder, terminalStatePriorities } from 'vs/workbench/contrib/testing/common/testingStates'; </ins> export interface ITestRunTaskResults extends ITestRunTask { /**

} const index = this.mustGetTaskIndex(taskId); <ins> const oldTerminalStatePrio = terminalStatePriorities[entry.tasks[index].state]; const newTerminalStatePrio = terminalStatePriorities[state]; // Ignore requests to set the state from one terminal state back to a // "lower" one, e.g. from failed back to passed: if (oldTerminalStatePrio !== undefined && (newTerminalStatePrio === undefined || newTerminalStatePrio < oldTerminalStatePrio)) { return; } </ins> this.fireUpdateAndRefresh(entry, index, state, duration); }

export const statesInOrder = Object.keys(statePriority).map(s => Number(s) as TestResultState).sort(cmpPriority); export const isRunningState = (s: TestResultState) => s === TestResultState.Queued || s === TestResultState.Running; <ins> /** * Some states are considered terminal; once these are set for a given test run, they * are not reset back to a non-terminal state, or to a terminal state with lower * priority. */ export const terminalStatePriorities: { [key in TestResultState]?: number } = { [TestResultState.Passed]: 0, [TestResultState.Skipped]: 1, [TestResultState.Failed]: 2, [TestResultState.Errored]: 3, }; </ins>

test('updateState', () => { changed.clear(); <del> r.updateState(new TestId(['ctrlId', 'id-a', 'id-aa']).toString(), 't', TestResultState.Running); </del> <ins> const testId = new TestId(['ctrlId', 'id-a', 'id-aa']).toString(); r.updateState(testId, 't', TestResultState.Running); </ins> assert.deepStrictEqual(r.counts, { ...makeEmptyCounts(), [TestResultState.Unset]: 2, [TestResultState.Running]: 1, [TestResultState.Queued]: 1, }); <del> assert.deepStrictEqual(r.getStateById(new TestId(['ctrlId', 'id-a', 'id-aa']).toString())?.ownComputedState, TestResultState.Running); </del> <ins> assert.deepStrictEqual(r.getStateById(testId)?.ownComputedState, TestResultState.Running); </ins> // update computed state: assert.deepStrictEqual(r.getStateById(tests.root.id)?.computedState, TestResultState.Running); assert.deepStrictEqual(getChangeSummary(), [

{ label: 'aa', reason: TestResultItemChangeReason.OwnStateChange }, { label: 'root', reason: TestResultItemChangeReason.ComputedStateChange }, ]); <ins> r.updateState(testId, 't', TestResultState.Passed); assert.deepStrictEqual(r.getStateById(testId)?.ownComputedState, TestResultState.Passed); r.updateState(testId, 't', TestResultState.Errored); assert.deepStrictEqual(r.getStateById(testId)?.ownComputedState, TestResultState.Errored); r.updateState(testId, 't', TestResultState.Passed); assert.deepStrictEqual(r.getStateById(testId)?.ownComputedState, TestResultState.Errored); </ins> }); test('retire', () => {