Fix for issue #4547

It looks like the `else if` clause isn't really necessary when tearing down the task because having TLS shouldn't mean that you don't necessarily fail the sched loop.

That being said, I'm definitely as familiar with this code as I'm sure someone else is, so this may be breaking something that I'm not seeing. At the very least the test case fails before the patch and succeeds after the patch, so I could re-open pull request with it as an xfail test if the change isn't the right one.