Make the new runner extensibility points async (fixes #123)