Better handling of this corner case.

This way there aren't a bunch of spurious invalidate() calls.