Use sets for the object keys

There's no need to store the object keys in sorted sets since they only exist to provide the remove functionality. Plain sets should be faster.

Note: this is kind of food for thought. I'm not sure what the right way to make this change would be since this would horribly break existing indexes.