opened
cottsay
username_0
If the test run is interrupted, `_LaunchDiedException` triggers a `FailResult` for the entire run. However, some cases may have already been cleaned up by the test run, and `FailResult` calls into `unittest.TestResult.addFailure()` with possibly cleaned-up tests, which are represented by `None`.

This leads to traces like this:
```
Traceback (most recent call last):
  File "/opt/ros/master/bin/ros2", line 11, in <module>
    load_entry_point('ros2cli', 'console_scripts', 'ros2')()
  File "/opt/ros_src/master/build/ros2cli/ros2cli/cli.py", line 67, in main
    rc = extension.main(parser=parser, args=args)
  File "/opt/ros_src/master/build/ros2test/ros2test/command/test.py", line 48, in main
    parser, args, test_runner_cls=launch_testing_ros.LaunchTestRunner
  File "/opt/ros_src/master/build/launch_testing/launch_testing/launch_test.py", line 111, in run
    results = runner.run()
  File "/opt/ros_src/master/build/launch_testing/launch_testing/test_runner.py", line 267, in run
    message='Launch stopped before the active tests finished.'
  File "/opt/ros_src/master/build/launch_testing/launch_testing/test_result.py", line 26, in __init__
    self.addFailure(case, (Exception, Exception(message), None))
  File "/usr/lib64/python3.7/unittest/result.py", line 17, in inner
    return method(self, *args, **kw)
  File "/usr/lib64/python3.7/unittest/result.py", line 122, in addFailure
    self.failures.append((test, self._exc_info_to_string(err, test)))
  File "/usr/lib64/python3.7/unittest/result.py", line 180, in _exc_info_to_string
    if exctype is test.failureException:
AttributeError: 'NoneType' object has no attribute 'failureException'
```

The solution presented here is to change the `_removeTestAtIndex` function to a no-op, since we aren't actually subclassing `unittest.TestSuite` as the documentation suggests.
Disable cleanup of test cases once they have been run
issue
created
cottsay
username_0
The error occurs because `FailResult` is assuming that all of the tests cases can be used when calling `addFailure` in the creation of a `TestResult` instance: https://github.com/ros2/launch/blob/d16628d8abfcdd7d44293c89198a1a64cfc54ee6/launch_testing/launch_testing/test_result.py#L23-L27

In the current implementation, test cases that have already completed have been cleaned up and can not be passed to `addFailure`.

If we just skip `addFailure` for the test cases that have already been completed, we'll lose them entirely, since the current structure of this code doesn't have the capability to report partial success if launch shuts down before all of the tests finish.
comment
created
hidmic
username_1
@username_0 ping.
comment
created
cottsay
username_0
At the point where we're currently calling `addFailure`, the case is already `None` because of the cleanup. Even after suppressing the cleanup, the `TestCase` doesn't store the result so we don't know which tests were successful, failed OR ignored - that is what the `TestResult` class is for. It might be possible to get access to the `TestResult` instance that has the accumulated successful and ignored cases, but I can't see a straightforward way to get access to it. It might not even be possible.
comment
created
hidmic
username_1
Correct, you'd have to propagate it upwards, perhaps along with the `LaunchDiedException`. At any rate, it's a nice to have, if launch dies the test is broken anyways.
comment