Was our original problem that the Product was actually being added to the `loaded` set more than once or that deserializing associations was keeping extra references around to the `setByRecord` object that was temporarily created?  I'm not sure if there's an easy way to test this but I feel like these are just testing our new implementation and perhaps missing a regression test?