Repository: python/mypy

Issue:
'in' operator rejected with union right operand
Mypy generates a false positive for this fragment:

```py
from typing import Iterable, Container, Union

i: Iterable[str]
c: Container[str]
u: Union[Iterable[str], Container[str]]

'x' in i
'x' in c
'x' in u  # Unsupported right operand type for in ("Union[Iterable[str], Container[str]]")
```

This should be easy to fix.


Discussion:
I think I've run into a similar problem, but given the error message, it's hard to tell.  Here's the code sample:

```
def exclude_disabled(employees: List[Person]) -> List[Person]:
    """Excludes disabled employees."""
    return [employee for employee in employees if 'OU=zDisabled' not in employee.dn]
```

the error I get (with mypy 0.620) is:

`error: Unsupported right operand type for in ("None")`

My guess is that the conditional makes mypy think that `employee` may be `None` when it will never be so.
There must be something else, I just tried a complete version of your sample:
```python
from typing import List

class Person:
    dn: List[str]

def exclude_disabled(employees: List[Person]) -> List[Person]:
    """Excludes disabled employees."""
    return [employee for employee in employees if 'OU=zDisabled' not in employee.dn]
```
and it type-checks cleanly on master. What is the type of `employee.dn`?
Hi,
I have a similar use case involving `Union`s. 

>* Are you reporting a bug, or opening a feature request?

AFAICT this is a bug.

> * Please insert below the code you are checking with mypy,
  or a mock-up repro if the source is private. We would appreciate
  if you try to simplify your case to a minimal repro.

Here it is a minimal code snippets:
```python
from typing import Collection, Union

def function(a: Union[bool, Collection[str]], b: str) -> int:
    if a is True:
        return 1
    if a is False:
        return 2

    if b in a:
        return 3

    return 4
```
> * What is the actual behavior/output?

`example.py:10: error: Unsupported right operand type for in ("Union[bool, Collection[str]]")`

> * What is the behavior/output you expect?

This is a perfectly valid Python code, so mypy shouldn't complain.

> * What are the versions of mypy and Python you are using?

```
$ mypy --version
mypy 0.720
```
>Do you see the same issue after installing mypy from Git master?

Haven't tried

> * What are the mypy flags you are using? (For example --strict-optional)

None

Hope you can help.

@ariciputi Your issue is different. Mypy doesn't understand that the `is True` and `is False` checks together are enough to narrow away the `bool` type from the union. Could you create a new issue about this (you can just copy your comment to a new issue)?
Hi @JukkaL,
I've opened [this](https://github.com/python/mypy/issues/7265) as you suggested.

Thanks.
@JukkaL : This is an interesting case. Did some digging. This works:
```
from typing import Container, Iterable, List, Set, Union

c: Container[str]
i: Iterable[str]
l: List[str]
s: Set[str]
u: Union[Container[str], List[str], Set[str]]

'x' in c
'x' in i
'x' in l
'x' in s
'x' in u
```

As in your example, adding the `Iterable[str]` type to the `Union` kills it.

It seems like it has to do with the fact that the `Iterable` interface doesn't define a `__contains__` method. That's pretty much all that a `Container` is, and in `mypy/typeshed/stdlib/2and3/builtins.pyi`, we see that the `list` stub as well as the the `set` stub both define the `__contains__` method.

If we look in `mypy/typeshed/stdlib/3/typing.pyi` for the `Iterable` stuff, sure enough, no `__contains__`.

But, of course, if you define some dumb iterable, you can use `in` to check for containment successfully at runtime:
```
import collections.abc
from typing import Iterable

class MyIterable(collections.abc.Iterable):
    def __iter__(self) -> int:
        for i in range(10):
            yield i

a = MyIterable()
print(2 in a) # True
print(12 in a) # False
```

`in` is complicated  -- https://stackoverflow.com/questions/12244074/python-source-code-for-built-in-in-operator/12244378#12244378

Simply adding a `__contains__` to the `Iterable` stub fixes the behaviour in your example:
```
@runtime_checkable
class Iterable(Protocol[_T_co]):
    @abstractmethod
    def __iter__(self) -> Iterator[_T_co]: ...
    @abstractmethod
    def __contains__(self, x: object) -> bool: ...
```

I hesitate to make this change -- it seems like messing fundamentally with the protocol. I guess I'll push up a PR on `typeshed` and link to this issue and maybe carry the discussion forward there?

Any direction would be helpful, I've just started trying to understand this codebase.
Yeah, changing `Iterable` to contain `__contains__` breaks all kinds of tests even just in mypy. My naive attempt at resolution is inadvisable.

Would certainly appreciate guidance from someone more experienced with this code.
I think this is also a case of this error (?), but is really odd because there's no need to upcast the value entries of the dict to `object`, as all the values are sets.

```
foo = {'a': set(),
       'b': {'c', 'd'}}

'z' in foo['a']
```

Results in
```
error: Unsupported right operand type for in ("object")
```

If a set literal rather than `set()` is used for the value of the `a` key, type checking is successful.

The workaround is to add an explicit type definition for `foo`.
@MrCreosote that's a different problem, basically that mypy infers `foo` to be `Dict[str, object]`. As you say, an explicit type definition (or use of a TypedDict) solves it.
This still reproduces. The weird part here to me isn't that we reject `x in Iterable | Container`, it's that we accept `x in Iterable`, since the `Iterable` protocol doesn't have `__contains__`. Is there some special casing that happens only at the top level but not inside a union?
The `in` operator works with both `__contains__` and iterables. See https://docs.python.org/3/reference/expressions.html#membership-test-operations for details. So `x in Iterable` should type check without errors, as should `x in Iterable | Container`.
I know that; what's weird to me is that mypy knows about that too, but only if the Iterable is not in a Union. (For completeness, `in` also works with objects that define `__getitem__`.)

Resolving patch:
diff --git a/mypy/checker.py b/mypy/checker.py
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -4499,6 +4499,26 @@ def analyze_iterable_item_type(self, expr: Expression) -> tuple[Type, Type]:
             # Non-tuple iterable.
             return iterator, echk.check_method_call_by_name("__next__", iterator, [], [], expr)[0]
 
+    def analyze_iterable_item_type_without_expression(
+        self, type: Type, context: Context
+    ) -> tuple[Type, Type]:
+        """Analyse iterable type and return iterator and iterator item types."""
+        echk = self.expr_checker
+        iterable = get_proper_type(type)
+        iterator = echk.check_method_call_by_name("__iter__", iterable, [], [], context)[0]
+
+        if isinstance(iterable, TupleType):
+            joined: Type = UninhabitedType()
+            for item in iterable.items:
+                joined = join_types(joined, item)
+            return iterator, joined
+        else:
+            # Non-tuple iterable.
+            return (
+                iterator,
+                echk.check_method_call_by_name("__next__", iterator, [], [], context)[0],
+            )
+
     def analyze_range_native_int_type(self, expr: Expression) -> Type | None:
         """Try to infer native int item type from arguments to range(...).
 
diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py
--- a/mypy/checkexpr.py
+++ b/mypy/checkexpr.py
@@ -2914,68 +2914,108 @@ def visit_comparison_expr(self, e: ComparisonExpr) -> Type:
         That is, 'a < b > c == d' is check as 'a < b and b > c and c == d'
         """
         result: Type | None = None
-        sub_result: Type | None = None
+        sub_result: Type
 
         # Check each consecutive operand pair and their operator
         for left, right, operator in zip(e.operands, e.operands[1:], e.operators):
             left_type = self.accept(left)
 
-            method_type: mypy.types.Type | None = None
-
             if operator == "in" or operator == "not in":
+                # This case covers both iterables and containers, which have different meanings.
+                # For a container, the in operator calls the __contains__ method.
+                # For an iterable, the in operator iterates over the iterable, and compares each item one-by-one.
+                # We allow `in` for a union of containers and iterables as long as at least one of them matches the
+                # type of the left operand, as the operation will simply return False if the union's container/iterator
+                # type doesn't match the left operand.
+
                 # If the right operand has partial type, look it up without triggering
                 # a "Need type annotation ..." message, as it would be noise.
                 right_type = self.find_partial_type_ref_fast_path(right)
                 if right_type is None:
                     right_type = self.accept(right)  # Validate the right operand
 
-                # Keep track of whether we get type check errors (these won't be reported, they
-                # are just to verify whether something is valid typing wise).
-                with self.msg.filter_errors(save_filtered_errors=True) as local_errors:
-                    _, method_type = self.check_method_call_by_name(
-                        method="__contains__",
-                        base_type=right_type,
-                        args=[left],
-                        arg_kinds=[ARG_POS],
-                        context=e,
-                    )
+                right_type = get_proper_type(right_type)
+                item_types: Sequence[Type] = [right_type]
+                if isinstance(right_type, UnionType):
+                    item_types = list(right_type.items)
 
                 sub_result = self.bool_type()
-                # Container item type for strict type overlap checks. Note: we need to only
-                # check for nominal type, because a usual "Unsupported operands for in"
-                # will be reported for types incompatible with __contains__().
-                # See testCustomContainsCheckStrictEquality for an example.
-                cont_type = self.chk.analyze_container_item_type(right_type)
-                if isinstance(right_type, PartialType):
-                    # We don't really know if this is an error or not, so just shut up.
-                    pass
-                elif (
-                    local_errors.has_new_errors()
-                    and
-                    # is_valid_var_arg is True for any Iterable
-                    self.is_valid_var_arg(right_type)
-                ):
-                    _, itertype = self.chk.analyze_iterable_item_type(right)
-                    method_type = CallableType(
-                        [left_type],
-                        [nodes.ARG_POS],
-                        [None],
-                        self.bool_type(),
-                        self.named_type("builtins.function"),
-                    )
-                    if not is_subtype(left_type, itertype):
-                        self.msg.unsupported_operand_types("in", left_type, right_type, e)
-                # Only show dangerous overlap if there are no other errors.
-                elif (
-                    not local_errors.has_new_errors()
-                    and cont_type
-                    and self.dangerous_comparison(
-                        left_type, cont_type, original_container=right_type, prefer_literal=False
-                    )
-                ):
-                    self.msg.dangerous_comparison(left_type, cont_type, "container", e)
-                else:
-                    self.msg.add_errors(local_errors.filtered_errors())
+
+                container_types: list[Type] = []
+                iterable_types: list[Type] = []
+                failed_out = False
+                encountered_partial_type = False
+
+                for item_type in item_types:
+                    # Keep track of whether we get type check errors (these won't be reported, they
+                    # are just to verify whether something is valid typing wise).
+                    with self.msg.filter_errors(save_filtered_errors=True) as container_errors:
+                        _, method_type = self.check_method_call_by_name(
+                            method="__contains__",
+                            base_type=item_type,
+                            args=[left],
+                            arg_kinds=[ARG_POS],
+                            context=e,
+                            original_type=right_type,
+                        )
+                        # Container item type for strict type overlap checks. Note: we need to only
+                        # check for nominal type, because a usual "Unsupported operands for in"
+                        # will be reported for types incompatible with __contains__().
+                        # See testCustomContainsCheckStrictEquality for an example.
+                        cont_type = self.chk.analyze_container_item_type(item_type)
+
+                    if isinstance(item_type, PartialType):
+                        # We don't really know if this is an error or not, so just shut up.
+                        encountered_partial_type = True
+                        pass
+                    elif (
+                        container_errors.has_new_errors()
+                        and
+                        # is_valid_var_arg is True for any Iterable
+                        self.is_valid_var_arg(item_type)
+                    ):
+                        # it's not a container, but it is an iterable
+                        with self.msg.filter_errors(save_filtered_errors=True) as iterable_errors:
+                            _, itertype = self.chk.analyze_iterable_item_type_without_expression(
+                                item_type, e
+                            )
+                        if iterable_errors.has_new_errors():
+                            self.msg.add_errors(iterable_errors.filtered_errors())
+                            failed_out = True
+                        else:
+                            method_type = CallableType(
+                                [left_type],
+                                [nodes.ARG_POS],
+                                [None],
+                                self.bool_type(),
+                                self.named_type("builtins.function"),
+                            )
+                            e.method_types.append(method_type)
+                            iterable_types.append(itertype)
+                    elif not container_errors.has_new_errors() and cont_type:
+                        container_types.append(cont_type)
+                        e.method_types.append(method_type)
+                    else:
+                        self.msg.add_errors(container_errors.filtered_errors())
+                        failed_out = True
+
+                if not encountered_partial_type and not failed_out:
+                    iterable_type = UnionType.make_union(iterable_types)
+                    if not is_subtype(left_type, iterable_type):
+                        if len(container_types) == 0:
+                            self.msg.unsupported_operand_types("in", left_type, right_type, e)
+                        else:
+                            container_type = UnionType.make_union(container_types)
+                            if self.dangerous_comparison(
+                                left_type,
+                                container_type,
+                                original_container=right_type,
+                                prefer_literal=False,
+                            ):
+                                self.msg.dangerous_comparison(
+                                    left_type, container_type, "container", e
+                                )
+
             elif operator in operators.op_methods:
                 method = operators.op_methods[operator]
 
@@ -2983,6 +3023,7 @@ def visit_comparison_expr(self, e: ComparisonExpr) -> Type:
                     sub_result, method_type = self.check_op(
                         method, left_type, right, e, allow_reverse=True
                     )
+                    e.method_types.append(method_type)
 
                 # Only show dangerous overlap if there are no other errors. See
                 # testCustomEqCheckStrictEquality for an example.
@@ -3002,12 +3043,10 @@ def visit_comparison_expr(self, e: ComparisonExpr) -> Type:
                     left_type = try_getting_literal(left_type)
                     right_type = try_getting_literal(right_type)
                     self.msg.dangerous_comparison(left_type, right_type, "identity", e)
-                method_type = None
+                e.method_types.append(None)
             else:
                 raise RuntimeError(f"Unknown comparison operator {operator}")
 
-            e.method_types.append(method_type)
-
             #  Determine type of boolean-and of result and sub_result
             if result is None:
                 result = sub_result


Test patch:
diff --git a/test-data/unit/check-unions.test b/test-data/unit/check-unions.test
--- a/test-data/unit/check-unions.test
+++ b/test-data/unit/check-unions.test
@@ -1202,3 +1202,20 @@ def foo(
         yield i
 foo([1])
 [builtins fixtures/list.pyi]
+
+[case testUnionIterableContainer]
+from typing import Iterable, Container, Union
+
+i: Iterable[str]
+c: Container[str]
+u: Union[Iterable[str], Container[str]]
+ni: Union[Iterable[str], int]
+nc: Union[Container[str], int]
+
+'x' in i
+'x' in c
+'x' in u
+'x' in ni  # E: Unsupported right operand type for in ("Union[Iterable[str], int]")
+'x' in nc  # E: Unsupported right operand type for in ("Union[Container[str], int]")
+[builtins fixtures/tuple.pyi]
+[typing fixtures/typing-full.pyi]