If we weren't able to retrieve the object/ghost for a change, dequeue it

I introduced a bug in 593b82866f94439be69836f9ab608c7c13ac2a53 that caused changes to be retried in a loop.