This feels fragile; any sort of timing-based test has the potential to make jenkins fail randomly, which has been a huge issue for us lately. Is there a way you can mock out the async call to return the expected result synchronously and remove this?

According to the docstring this is going all the way from distributor to handler to child importer, but is all of that needed to simulate a missing plugin? The change was basically three lines of except code, it shouldn't be hard to simulate tripping that except block on its own and verifying the change to the error message was made.