I think you really want `np.logical_or`. This `not xor` operation will have unexpected results.

Also, you need to specify which label you are examining in the loop with `labeled_im == i`.