Ensure that stuff gets deleted to prevent errors

I noticed that it happened there was some provider left over with same IP which was ruining other tests. I moved all deleting in the finalizer, therefore making it ~100% sure.