Even when indexes are different, zip optimistically with fallback

Also, switch the argument order of diff so it keeps values from the left
side (`self`). This is consistent with all the other operations.