Fixes issues when a second class swizzle takes place on an instance

Fixes the issue raised by @erikprice in #1058.