Avoid deleting providers tests to not mess the other tests

This caused errors in upstream because of removing the providers. This PR should skip some of the provider tests in favour of not messing other tests until referenced BZ is resolved.