Tweak the removal of cached heights