Only invalidate if we're touching or just let go.