Fixes issues when a second class swizzle takes place on a class

Fixes the issue raised by @erikprice in #1058.