Repository: quora/pyanalyze
Pull request #379 (2021-12-31 10:04:53)

Further type evaluation work
- Change the spec around handling Any
- Spec out more behavior
- Implement most missing functionality


Patch:
diff --git a/docs/changelog.md b/docs/changelog.md
index 7e9a8f8..f2305a2 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -2,12 +2,13 @@
 
 ## Unreleased
 
+- Fix inference for nested `async def` functions (#380)
 - Fix usage of type variables in function parameters
   with defaults (#378)
 - Support the Python 3.10 `match` statement (#376)
 - Support the walrus (`:=`) operator (#375)
 - Initial support for proposed new "type evaluation"
-  mechanism (#374)
+  mechanism (#374, #379)
 - Create command-line options for each config option (#373)
 - Overhaul treatment of function definitions (#372)
   - Support positional-only arguments
diff --git a/docs/type_evaluation.md b/docs/type_evaluation.md
index dbf37c2..f7fd0f3 100644
--- a/docs/type_evaluation.md
+++ b/docs/type_evaluation.md
@@ -14,8 +14,8 @@ definition of `round()` in typeshed:
 With type evaluation, this could instead be written as:
 
     @evaluated
-    def round(number: SupportsRound[_T], ndigits: SupportsIndex | None = ...):
-        if not is_set(ndigits) or ndigits is None:
+    def round(number: SupportsRound[_T], ndigits: SupportsIndex | None = None):
+        if ndigits is None:
             return int
         else:
             return _T
@@ -24,19 +24,66 @@ This makes it easier to see at a glance what the difference is between various o
 
 ## Specification
 
+This section specifies how type evaluation works, without
+commentary. Discussion, with motivating use cases, is
+provided in the "Discussion" section below. The examples
+in this section are meant to clarify the semantics only.
+
 Type-evaluated functions may be declared both at runtime and in stub files, similar to the existing `@overload` mechanism. At runtime, the evaluated function must immediately precede
 the function implementation:
 
-        @evaluated
-        def round(number: SupportsRound[_T], ndigits: SupportsIndex | None = ...):
-            if ...
+    @evaluated
+    def round(number: SupportsRound[_T], ndigits: SupportsIndex | None = None):
+        if ...
 
-        def round(number, ndigits=None):
-            return number.__round__(ndigits)
+    def round(number, ndigits=None):
+        return number.__round__(ndigits)
 
 In stubs, the implementation is omitted.
 
-When a type checker encounters a call to a function for which a type evaluation has been provided, it should symbolically evaluate the body of the type evaluation until it reaches a `return` or `raise` statement. A `raise` statement indicates that the type checker should produce an error, a `return` statement provides the type that the call should return.
+When a type checker encounters a call to a function for which a type evaluation has been provided, it should
+do the following:
+
+- Validate that the arguments to the call are compatible
+  with the type annotations on the parameters to the
+  evaluation function, as with a normal call.
+- Symbolically evaluate the body of the type evaluation
+  until it reaches a `return` statement, which provides the type
+  that the call should return.
+- If execution reached a `return` statement, return the type
+  provided by that statement. Otherwise, return the type
+  set in the evaluation function's return annotation, or
+  `Any` if there is no return annotation.
+
+Type checkers are encouraged to provide a strictness option
+that produces an error if an evaluation function is missing
+a type annotation on a parameter or return type. However, no
+error should be provided if the return annotation is missing
+and all branches (including error branches) return a type.
+
+Simple examples to demonstrate the semantics:
+
+    @evaluated
+    def always_returns(x: int):
+        return str
+
+    always_returns("x")  # error: "x" is not an int
+    always_returns()  # error: not enough arguments
+    reveal_type(always_returns(1))  # str
+
+    @evaluated
+    def always_errors(x: int):
+        show_error("error")
+
+    x = always_errors(1)  # error
+    reveal_type(x)  # Any
+
+    @evaluated
+    def always_errors_with_type(x: int) -> str:
+        show_error("error")
+
+    x = always_errors(1)  # error
+    reveal_type(x)  # str
 
 ### Supported features
 
@@ -45,55 +92,447 @@ The only supported features are:
 
 - `if` statements and `else` blocks. These can only contain conditions of the form specified below.
 - `return` statements with return values that are interpretable as type annotations. This indicates the type that the function returns in a particular condition.
-- `raise` statements of the form `raise Exception(message)`, where `message` is a string literal. When the code takes a branch that ends in a `raise` statement, the type checker should emit an error with the provided message.
+- `pass` statements, which do nothing.
+- Calls to `show_error()`, which cause the type checker
+  to emit an error. These are discussed further below.
 
 Conditions in `if` statements may contain:
-- Calls to the special `is_set()` function, which takes as its argument a variable. This functions returns True if the variable was provided as an argument in the call. For example, given a function `def f(arg=None): ...`, `is_set(arg)` would return True if the function is invoked as `f(None)`, but False if it is invoked as `f()`. A dummy runtime implementation of `is_set()` is provided.
-- Calls to `isinstance()`, but with an extended meaning. `isinstance(arg, type)` will return True if and only if the type checker would accept an assignment `_: type = arg`. For example, `isinstance(arg, Literal["a", "b"])` is valid and returns True if the type of the argument is (for example) `Literal["a"]`.
-- Expressions of the form `arg is (not) <constant>`, where `<constant>` may be True, False, or None.
-- Expressions of the form `arg == <constant>` or `arg != <constant>`, where `<constant>` is any value valid inside `Literal[]` (a bool, int, string, or enum member).
+
+- A call to one of the following functions, which are covered
+  in more detail below:
+  - `is_provided()`, which returns whether a parameter was
+    explicitly provided in a call.
+  - `is_positional()`, which returns whether a parameter
+    was provided through a positional argument.
+  - `is_keyword()`, which returns whether a parameter
+    was provided through a keyword argument.
+  - `is_of_type()`, which returns whether a parameter is of
+    a particular type.
+- Expressions of the form `arg <op> <constant>`, where `<op>`
+  is one of `is`, `is not`, `==`, or `!=`. This is equivalent
+  to `(not) is_of_type(arg, Literal[<constant>], exclude_any=True)`.
+  `<constant>` may be any value that is valid inside `Literal` (`None`, a string, a bool, an int, or an enum
+  member).
 - Version and platform checks that are otherwise valid in stubs, as specified in PEP 484.
 - Multiple conditions combined with `and` or `or`.
 - A negation of another condition with `not`.
 
-### Interaction with Any
+### show_error()
 
-What should `round()` return if the type of the `ndigits`
-argument is `Any`? Existing type checkers do not agree:
-pyright picks the first overload that matches and returns
-`int`, since `Any` is compatible with `None`; mypy and pyanalyze
-see that multiple overloads might match and return `Any`. There
-are good reasons for both choices<!-- insert link to Eric's explanation-->,
-and we allow the same behavior for type evaluations.
-
-Type checkers should pick one of the following two behaviors and 
-document their choice:
-1. All checks (`isinstance`, `is`, `==`) against variables typed 
-   as `Any` in the body of type evaluation succeed. 
-   `round(..., Any)` returns `int`. Note that
-   this means that switching the `if` and `else` blocks may change
-   visible behavior.
-2. Conditions on variables typed as `Any` take both branches of the
-   conditional. If the two branches return different types, `Any`
-   is returned instead. `round(..., Any)` returns `Any`.
+The `show_error()` special function has the following
+signature:
+
+    def show_error(message: str, /, *, argument: Any | None = ...): ...
+
+The `message` parameter must be a string literal.
+Calls to this function cause the type checker to emit an
+error that includes the given message.
+Execution continues past the `show_error()` call as normal.
+
+If the `argument` parameter is provided, it must be one of
+the parameters to the function, indicating the parameter that
+is causing the error. The type checker may use this
+information to produce a more precise error (for example, by
+pointing the error caret at the specified argument in the
+call site).
+
+### is_provided(), is_positional(), and is_keyword()
+
+These special functions have the following signatures:
+
+    def is_provided(arg: Any, /) -> bool: ...
+    def is_positional(arg: Any, /) -> bool: ...
+    def is_keyword(arg: Any, /) -> bool: ...
+
+`arg` must be one of the parameters to the function.
+`is_provided()` returns True if the parameter was explicitly
+provided in the call; that is, the default value was not
+used. Similarly, `is_positional()` returns True if the
+parameter was provided as a positional argument, and
+`is_keyword()` returns True if the parameter was provided
+as a keyword argument.
+
+Parameters in Python can be provided in three ways,
+which we call _argument kinds_ for the purpose of this
+specificatioon:
+
+- `POSITIONAL`: at the call site, either a single
+  positional argument or a variadic one (`*args`)
+- `KEYWORD`: at the call site, either a sinngle keyword
+  argument or a variadic one (`**kwargs`)
+- `DEFAULT`: no value provided at the call site; the
+  default defined in the function is used
+
+Static analyzers must add a fourth kind in the presence
+of calls with `*args` and `**kwargs`:
+
+- `UNKNOWN`: the kind cannot be statically determined.
+  This can happen in the following situations:
+  - A positional-only parameter with a default in a call
+    with `*args` of unknown size.
+  - A keyword-only parameter with a default in a call
+    with `**kwargs` of unknown size.
+  - A positional-or-keyword parameter that matches either
+    of the above conditions.
+  - A positional-or-keyword parameter (with or without a
+    default) in a call with both `*args` and `**kwargs`.
+
+The three special functions map to these kinds as follows:
+
+- `is_provided()`: kind is `POSITIONAL` or `KEYWORD`
+- `is_positional()`: kind is `POSITIONAL`
+- `is_keyword()`: kind is `KEYWORD`
+
+Thus, there is no way to distinguish between `DEFAULT`
+and `UNKNOWN`, and a parameter for which `is_provided()`
+returns False in the type evaluator may actually be
+provided at runtime.
+
+For variadic parameters (`*args` and `**kwargs`), the
+kind is either `DEFAULT` if no arguments are provided
+to the parameter, or either `POSITIONAL` (for `*args`)
+or `KEYWORD` (for `**kwargs`) if arguments may be
+provided. If the type checker can
+prove that a variadic argument is empty, `is_provided()`
+may return False. (For example, given a definition
+`def f(*args)` and a call `f(*())`, `is_provided(args)`
+may return False.)
+
+Examples:
+
+    @evaluated
+    def reject_arg(arg: int = 0) -> None:
+        if is_provided(arg):
+            show_error("error")
+
+    args: Any = ...
+    kwargs: Any = ...
+    reject_arg()  # ok
+    reject_arg(0)  # error
+    reject_arg(arg=0)  # error
+    reject_arg(*args)  # ok
+    reject_arg(**kwargs)  # ok
+
+    @evaluated
+    def reject_star_args(*args: int) -> None:
+        if is_provided(args):
+            show_error("error")
+
+    reject_star_args()  # ok
+    reject_star_args(1)  # error
+    reject_star_args(*(1,))  # error
+    reject_star_args(*())  # may error, depending on type checker
+
+    @evaluated
+    def reject_star_kwargs(**kwargs: int) -> None:
+        if is_provided(kwargs):
+            show_error("error")
+
+    reject_star_kwargs()  # ok
+    reject_star_kwargs(x=1)  # error
+    reject_star_kwargs(**{"x": 1})  # error
+    reject_star_args(**{})  # may error, depending on type checker
+
+    @evaluated
+    def reject_keyword(arg: int = 0) -> None:
+        if is_keyword(arg):
+            show_error("error")
+
+    reject_keyword()  # ok
+    reject_keyword(0)  # ok
+    reject_keyword(arg=0)  # error
+    reject_keyword(*args)  # ok
+    reject_keyword(**kwargs)  # ok
+
+    @evaluated
+    def reject_positional(arg: int = 0)-> None:
+        if is_positional(arg):
+            show_error("error")
+
+    reject_keyword()  # ok
+    reject_keyword(0)  # error
+    reject_keyword(arg=0)  # ok
+    reject_keyword(*args)  # ok
+    reject_keyword(**kwargs)  # ok
+
+    @evaluated
+    def invalid(arg: object) -> None:
+        if is_provided(x):  # error, not a function parameter
+            show_error("error")
+
+### is_of_type()
+
+The special `is_of_type()` function has the following
+signature:
+
+    def is_oF_type(arg: object, type: Any, /, *, exclude_any: bool = True) -> bool: ...
+
+`arg` must be one of the parameters to the function and
+`type` must be a form that the type checker would accept
+in a type annotation.
+
+If `exclude_any` is False, `is_of_type(x, T)` returns true if `x` is
+compatible with `T`; that is, if the type checker would
+accept an assignment `_: T = x`.
+
+If the `exclude_any` parameter is True (the default), normal type checking
+rules are modified so that `Any` is no longer compatible with
+any other type, but only with another `Any`. All other types
+are still compatible with `Any`.
+
+Examples:
+
+    @evaluated
+    def length_or_none(s: str | None = None):
+        if is_of_type(s, str):
+            return int
+        else:
+            return None
+
+    any: Any = ...
+    opt: int | None = ...
+    reveal_type(length_or_none("x"))  # int
+    reveal_type(length_or_none(None))  # None
+    reveal_type(length_or_none(opt))  # int | None
+    reveal_type(length_or_none(any))  # int
+
+    @evaluated
+    def length_or_none2(s: str | None):
+        if is_of_type(s, str, exclude_any=True):
+            return int
+        elif is_of_type(s, None, exclude_any=True):
+            return None
+        else:
+            return Any
+
+    reveal_type(length_or_none2("x"))  # int
+    reveal_type(length_or_none2(None))  # None
+    reveal_type(length_or_none2(opt))  # int | None
+    reveal_type(length_or_none2(any))  # Any
+
+    @evaluated
+    def nested_any(s: Sequence[Any]):
+        if is_of_type(s, str, exclude_any=True):
+            show_error("error")
+        elif is_of_type(s, Sequence[str], exclude_any=True):
+            return str
+        else:
+            return int
+
+    anyseq: Sequence[Any] = ...
+    nested_any("x")  # error
+    reveal_type(nested_any(["x"]))  # str
+    reveal_type(nested_any([1]))  # int
+    reveal_type(nested_any(any))  # int
+    reveal_type(nested_any(anyseq))  # int
 
 ### Interaction with unions
 
 Type checkers should apply normal type narrowing rules to arguments
-that are of Union types. For example, if the `ndigits` argument to
+that are of Union types. If only some members of a Union
+match a condition, both branches of the conditional are
+taken, with the parameter type narrowed appropriately in each
+case. The return type of the function is the union of the two
+branches.
+
+For example, if the `ndigits` argument to
 `round()` is of type `int | None`, the inferred return value should
 be `_T | int`.
 
+### Type compatibility
+
+The type of an evaluated function is compatible with a
+`Callable` with the same arguments and returning the
+`Union` of the possible return types, and with any
+`Callable` for which the evaluation function would
+return a compatible type given the same arguments.
+
+Examples:
+
+    @evaluated
+    def maybe_path(path: str | None):
+        if path is None:
+            return None
+        else:
+            return Path
+
+    _: Callable[[str | None], Path | None] = maybe_path  # ok
+    _: Callable[[None], None] = maybe_path  # ok
+    _: Callable[[str], Path] = maybe_path  # ok
+    _: Callable[[str | None], Path] = maybe_path  # error
+    _: Callable[[str], Path | None] = maybe_path  # ok
+    _: Callable[[Literal["x"]], Path] = maybe_path  # ok
+
+## Discussion
+
+### Interaction with Any
+
+The below is an evaluation function for a simplified
+version of the `open()` builtin:
+
+    @evaluated
+    def open(mode: str):
+        if is_of_type(mode, Literal["r", "w"]):
+            return TextIO
+        elif is_of_type(mode, Literal["rb", "wb"]):
+            return BinaryIO
+        else:
+            return IO[Any]
+
+What should `open()` return if the type of the `mode`
+argument is `Any`? With the equivalent code expressed
+using overloads, existing type checkers do not agree:
+pyright picks the first overload that matches and returns
+`int`, since `Any` is compatible with `None`; mypy and pyanalyze
+see that multiple overloads might match and return `Any`.
+There are good reasons for both choices,
+as discussed [here](https://github.com/microsoft/pyright/issues/2521#issuecomment-956823577)
+by Eric Traut. In particular, mypy's behavior is more sound
+for a type checker, but pyright's behavior helps generate
+better autocompletion suggestions in a language server.
+
+Type evaluation functions potentially have
+the same ambiguity, so in order to provide predictable
+behavior across type checkers, we need to specify a single
+behavior.
+
+As specified above, our choice is to treat `Any` specially
+by default within evaluation functions, making it
+incompatible with other types, both within `is`/`==`
+comparisons and within the `is_of_type` primitive.
+This behavior makes it easiest
+to write evaluation functions that read naturally and
+behave as desired. In particular, this choice makes
+`open(Any)` return `IO[Any]`, which is both the most
+intuitive and the most useful result.
+
+The most natural alternative is to make `is_of_type()` follow
+normal type compatibility rules, where `Any` is compatible
+with everything. But this would create confusing behavior
+for evaluation functions like the one for `open()`:
+
+- `open()` would return `TextIO` if `mode` is `Any`, which
+  is too precise in general.
+- The order of the `BinaryIO` and `TextIO` checks would
+  matter:
+  the function would behave differently if the two checks
+  were flipped. This would be a subtle behavior that is
+  not obvious to readers of the code.
+- There would be no obvious way to provide a customized
+  fallback behavior for `Any`. Technically, a check like
+  `is_of_type(mode, Literal["r"]) and is_of_type(mode, Literal["w"])`
+  could be used to check for `Any` (only `Any` is compatible
+  with both literals), but this would be obscure and
+  unreadable.
+- It would be difficult to show an error for a particular
+  parameter value. For example, a stub for `open()` might
+  want to show a warning if the deprecated `rU` mode is used.
+  The obvious way to do that would be to write
+  `if mode == "rU": show_error(...)`, but if this returned
+  true for `Any`, we would show the error for `mode: Any`.
+
+As an additional example, consider functions that take
+some object or `None` and return either `None` or a
+transformed version of the object, like this:
+
+    @evaluated
+    def maybe_path(path: str | None):
+        if path is None:
+            return None
+        else:
+            return Path
+
+Functions of this form are fairly common, and it is
+natural to write them with the trivial branch (`None`) first,
+both in the implementation and in the evaluation function.
+But if `path is None` would be true for `Any`, the evaluation
+function would return `None`, which is bad both for type
+checkers and for autocomplete suggestions.
+
+One downside of this behavior is that type checkers may
+incorrectly flag `is None` checks after a `maybe_path()`
+call as unreachable. However, such checks are usually only
+enabled in a strict mode, and `Any` should be rare in
+strictly typed code. Type checkers could also provide a
+mechanism that labels types derived from an evaluation
+function that used `Any` to disable diagnostics about
+unreachable code.
+
+Another alternative would be to use a mechanism similar to
+mypy-style overload resolution: conditions that match due to
+`Any` would essentially match neither branch and simply
+return `Any`. This behavior would avoid returning any
+overly precise types, but it would be useless for
+autocompletion suggestions and would remove a lot of
+useful type precision. For example, there would be no way
+for the `open()` evaluation function to produce `IO[Any]`.
+
+### Argument kind functions
+
+The three argument kind functions `is_provided()`,
+`is_positional()`, and `is_keyword()` are useful in various
+ways:
+
+- Functions implemented in C sometimes change behavior
+  depending on the presence of an argument, without a
+  meaningful default. For example, `dict.pop(key)` returns
+  the key's value type (or else it raises an exception), but
+  `dict.pop(key, default)` returns either the value type or
+  the type of `default`. Currently overloads are necessary
+  to represent this behavior, but `is_provided()` provides
+  an alternative.
+- It is common for new versions of Python to add or remove
+  parameters. For example, `zip()` gained a `strict=` keyword
+  argument in Python 3.10. Using `is_provided()` with a
+  `sys.version_info` check, we can provide an error if the
+  parameter is used in an older version, without duplicating
+  the entire function definition.
+- Similarly, new versions of Python often change parameters
+  from positional-or-keyword to positional-only or vice
+  versa. Version checks can be used with `is_positional()` or
+  `is_keyword()` to reflect such changes in the stub.
+- Library authors who want to evolve an API sometimes want
+  to make a function parameter keyword-only. An evaluation
+  function can be used to warn users who pass the parameter
+  positionally without changing the runtime parameter kind,
+  so that users have time to adapt before the runtime code
+  is broken.
+
+As an example, this is the current implemenation of `sum()`
+in typeshed:
+
+    if sys.version_info >= (3, 8):
+        @overload
+        def sum(__iterable: Iterable[_T]) -> _T | Literal[0]: ...
+        @overload
+        def sum(__iterable: Iterable[_T], start: _S) -> _T | _S: ...
+
+    else:
+        @overload
+        def sum(__iterable: Iterable[_T]) -> _T | Literal[0]: ...
+        @overload
+        def sum(__iterable: Iterable[_T], __start: _S) -> _T | _S: ...
+
+This is how it could be implemented using `@evaluated`:
+
+    @evaluated
+    def sum(__iterable: Iterable[_T], start: _S = ...):
+        if not is_provided(start):
+            return _T | Literal[0]
+        if sys.version_info < (3, 8) and is_keyword(start):
+            show_error("start is a positional-only argument in Python <3.8", argument=start)
+        return _T | _S
+
 ## Status
 
 A partial implementation of this feature is available
 in pyanalyze:
 
-    from pyanalyze.extensions import evaluated, is_set
+    from pyanalyze.extensions import evaluated, is_provided
 
     @evaluated
     def simple_evaluated(x: int, y: str = ""):
-        if is_set(y):
+        if is_provided(y):
             return int
         else:
             return str
@@ -105,15 +544,15 @@ in pyanalyze:
             return "x"
 
 Currently unsupported features include:
-- Comparison against enum members
-- Version and platform checks
-- Use of `and` and `or`
+
 - Usage in stubs
 - pyanalyze should provide a way to register
   an evaluation function for a runtime function,
   to replace some impls.
+- Type compatibility for evaluated functions.
 
 Areas that need more thought include:
+
 - Interaction with typevars
 - Interaction with overloads. It should be possible
   to register multiple evaluation functions for a
@@ -133,6 +572,7 @@ Areas that need more thought include:
   to produce deprecation warnings.
 
 Motivations can include:
+
 - Less repetitive overload writing
 - Ability to customize error messages
 - Potential for additional features that work
diff --git a/pyanalyze/__init__.py b/pyanalyze/__init__.py
index f9a0d78..042da44 100644
--- a/pyanalyze/__init__.py
+++ b/pyanalyze/__init__.py
@@ -47,7 +47,11 @@ used(extensions.LiteralOnly)
 used(extensions.NoAny)
 used(extensions.overload)
 used(extensions.evaluated)
-used(extensions.is_set)
+used(extensions.is_provided)
+used(extensions.is_keyword)
+used(extensions.is_positional)
+used(extensions.is_of_type)
+used(extensions.show_error)
 used(value.UNRESOLVED_VALUE)  # keeping it around for now just in case
 used(reexport)
 used(patma)
diff --git a/pyanalyze/analysis_lib.py b/pyanalyze/analysis_lib.py
index a08948e..68889a6 100644
--- a/pyanalyze/analysis_lib.py
+++ b/pyanalyze/analysis_lib.py
@@ -4,6 +4,8 @@ Commonly useful components for static analysis tools.
 
 """
 import ast
+from dataclasses import dataclass
+import linecache
 import os
 import secrets
 import sys
@@ -85,18 +87,35 @@ def get_line_range_for_node(node: ast.AST, lines: List[str]) -> List[int]:
     return list(range(first_lineno, last_lineno))
 
 
+@dataclass
+class _FakeLoader:
+    source: str
+
+    def get_source(self, name: object) -> str:
+        return self.source
+
+
 def make_module(
     code_str: str, extra_scope: Mapping[str, object] = {}
 ) -> types.ModuleType:
     """Creates a Python module with the given code."""
     # Make the name unique to avoid clobbering the overloads dict
     # from pyanalyze.extensions.overload.
+    token = secrets.token_hex()
     module_name = f"<test input {secrets.token_hex()}>"
+    filename = f"{token}.py"
     mod = types.ModuleType(module_name)
     scope = mod.__dict__
     scope["__name__"] = module_name
+    scope["__file__"] = filename
+    scope["__loader__"] = _FakeLoader(code_str)
+
+    # This allows linecache later to retrieve source code
+    # from this module, which helps the type evaluator.
+    linecache.lazycache(filename, scope)
     scope.update(extra_scope)
-    exec(code_str, scope)
+    code = compile(code_str, filename, "exec")
+    exec(code, scope)
     sys.modules[module_name] = mod
     return mod
 
diff --git a/pyanalyze/annotations.py b/pyanalyze/annotations.py
index e064296..fd2f6b4 100644
--- a/pyanalyze/annotations.py
+++ b/pyanalyze/annotations.py
@@ -121,7 +121,10 @@ class Context:
         return qcore.override(self, "should_suppress_undefined_names", True)
 
     def show_error(
-        self, message: str, error_code: ErrorCode = ErrorCode.invalid_annotation
+        self,
+        message: str,
+        error_code: ErrorCode = ErrorCode.invalid_annotation,
+        node: Optional[ast.AST] = None,
     ) -> None:
         """Show an error found while evaluating an annotation."""
         pass
@@ -149,18 +152,52 @@ class Context:
 @dataclass
 class TypeEvaluationContext(Context, type_evaluation.Context):
     variables: type_evaluation.VarMap
-    set_variables: Container[str]
+    positions: Mapping[str, type_evaluation.Position]
     can_assign_context: CanAssignContext
-    globals: Mapping[str, object]
+    globals: Mapping[str, object] = field(repr=False)
 
     def evaluate_type(self, node: ast.AST) -> Value:
         return type_from_ast(node, ctx=self)
 
+    def evaluate_value(self, node: ast.AST) -> Value:
+        return value_from_ast(node, ctx=self, error_on_unrecognized=False)
+
     def get_name(self, node: ast.Name) -> Value:
         """Return the :class:`Value <pyanalyze.value.Value>` corresponding to a name."""
         return self.get_name_from_globals(node.id, self.globals)
 
 
+@dataclass
+class EvaluatorValidationContext(Context, type_evaluation.Context):
+    variables: type_evaluation.VarMap
+    positions: Mapping[str, type_evaluation.Position]
+    can_assign_context: "NameCheckVisitor"
+    node: ast.AST
+
+    def show_error(
+        self,
+        message: str,
+        error_code: ErrorCode = ErrorCode.invalid_annotation,
+        node: Optional[ast.AST] = None,
+    ) -> None:
+        self.can_assign_context.show_error(
+            node or self.node, message, error_code=error_code
+        )
+
+    def evaluate_type(self, node: ast.AST) -> Value:
+        return type_from_ast(node, ctx=self)
+
+    def evaluate_value(self, node: ast.AST) -> Value:
+        return value_from_ast(node, ctx=self, error_on_unrecognized=False)
+
+    def get_name(self, node: ast.Name) -> Value:
+        """Return the :class:`Value <pyanalyze.value.Value>` corresponding to a name."""
+        val, _ = self.can_assign_context.resolve_name(
+            node, suppress_errors=self.should_suppress_undefined_names
+        )
+        return val
+
+
 @used  # part of an API
 def type_from_ast(
     ast_node: ast.AST,
@@ -251,10 +288,13 @@ def type_from_value(
     return _type_from_value(value, ctx, is_typeddict=is_typeddict)
 
 
-def value_from_ast(ast_node: ast.AST, ctx: Context) -> Value:
+def value_from_ast(
+    ast_node: ast.AST, ctx: Context, *, error_on_unrecognized: bool = True
+) -> Value:
     val = _Visitor(ctx).visit(ast_node)
     if val is None:
-        ctx.show_error("Invalid type annotation")
+        if error_on_unrecognized:
+            ctx.show_error("Invalid type annotation", node=ast_node)
         return AnyValue(AnySource.error)
     return val
 
@@ -599,7 +639,8 @@ def _type_from_subscripted_value(
         return GenericValue(root.typ, [_type_from_value(elt, ctx) for elt in members])
 
     if not isinstance(root, KnownValue):
-        ctx.show_error(f"Cannot resolve subscripted annotation: {root}")
+        if root != AnyValue(AnySource.error):
+            ctx.show_error(f"Cannot resolve subscripted annotation: {root}")
         return AnyValue(AnySource.error)
     root = root.val
     if root is typing.Union:
@@ -706,10 +747,15 @@ class _DefaultContext(Context):
         self.globals = globals
 
     def show_error(
-        self, message: str, error_code: ErrorCode = ErrorCode.invalid_annotation
+        self,
+        message: str,
+        error_code: ErrorCode = ErrorCode.invalid_annotation,
+        node: Optional[ast.AST] = None,
     ) -> None:
-        if self.visitor is not None and self.node is not None:
-            self.visitor.show_error(self.node, message, error_code)
+        if node is None:
+            node = self.node
+        if self.visitor is not None and node is not None:
+            self.visitor.show_error(node, message, error_code)
 
     def get_name(self, node: ast.Name) -> Value:
         if self.visitor is not None:
@@ -727,7 +773,9 @@ class _DefaultContext(Context):
         if self.should_suppress_undefined_names:
             return AnyValue(AnySource.inference)
         self.show_error(
-            f"Undefined name {node.id!r} used in annotation", ErrorCode.undefined_name
+            f"Undefined name {node.id!r} used in annotation",
+            ErrorCode.undefined_name,
+            node=node,
         )
         return AnyValue(AnySource.error)
 
@@ -770,11 +818,11 @@ class _Visitor(ast.NodeVisitor):
                 return KnownValue(getattr(root_value.val, node.attr))
             except AttributeError:
                 self.ctx.show_error(
-                    f"{root_value.val!r} has no attribute {node.attr!r}"
+                    f"{root_value.val!r} has no attribute {node.attr!r}", node=node
                 )
                 return AnyValue(AnySource.error)
         elif not isinstance(root_value, AnyValue):
-            self.ctx.show_error(f"Cannot resolve annotation {root_value}")
+            self.ctx.show_error(f"Cannot resolve annotation {root_value}", node=node)
         return AnyValue(AnySource.error)
 
     def visit_Tuple(self, node: ast.Tuple) -> Value:
@@ -861,11 +909,13 @@ class _Visitor(ast.NodeVisitor):
             arg_values = [self.visit(arg) for arg in node.args]
             kwarg_values = [(kw.arg, self.visit(kw.value)) for kw in node.keywords]
             if not arg_values:
-                self.ctx.show_error("TypeVar() requires at least one argument")
+                self.ctx.show_error(
+                    "TypeVar() requires at least one argument", node=node
+                )
                 return None
             name_val = arg_values[0]
             if not isinstance(name_val, KnownValue):
-                self.ctx.show_error("TypeVar name must be a literal")
+                self.ctx.show_error("TypeVar name must be a literal", node=node.args[0])
                 return None
             constraints = []
             for arg_value in arg_values[1:]:
@@ -877,7 +927,7 @@ class _Visitor(ast.NodeVisitor):
                 elif name == "bound":
                     bound = _type_from_value(kwarg_value, self.ctx)
                 else:
-                    self.ctx.show_error(f"Unrecognized TypeVar kwarg {name}")
+                    self.ctx.show_error(f"Unrecognized TypeVar kwarg {name}", node=node)
                     return None
             tv = TypeVar(name_val.val)
             return TypeVarValue(tv, bound, tuple(constraints))
@@ -885,14 +935,18 @@ class _Visitor(ast.NodeVisitor):
             arg_values = [self.visit(arg) for arg in node.args]
             kwarg_values = [(kw.arg, self.visit(kw.value)) for kw in node.keywords]
             if not arg_values:
-                self.ctx.show_error("ParamSpec() requires at least one argument")
+                self.ctx.show_error(
+                    "ParamSpec() requires at least one argument", node=node
+                )
                 return None
             name_val = arg_values[0]
             if not isinstance(name_val, KnownValue):
-                self.ctx.show_error("ParamSpec name must be a literal")
+                self.ctx.show_error(
+                    "ParamSpec name must be a literal", node=node.args[0]
+                )
                 return None
             for name, _ in kwarg_values:
-                self.ctx.show_error(f"Unrecognized ParamSpec kwarg {name}")
+                self.ctx.show_error(f"Unrecognized ParamSpec kwarg {name}", node=node)
                 return None
             tv = ParamSpec(name_val.val)
             return TypeVarValue(tv, is_paramspec=True)
diff --git a/pyanalyze/error_code.py b/pyanalyze/error_code.py
index 2b22a75..943cf79 100644
--- a/pyanalyze/error_code.py
+++ b/pyanalyze/error_code.py
@@ -90,6 +90,7 @@ class ErrorCode(enum.Enum):
     incompatible_override = 73
     impossible_pattern = 74
     bad_match = 75
+    bad_evaluator = 76
 
 
 # Allow testing unannotated functions without too much fuss
@@ -206,6 +207,7 @@ ERROR_DESCRIPTION = {
     ErrorCode.incompatible_override: "Class attribute incompatible with base class",
     ErrorCode.impossible_pattern: "Pattern can never match",
     ErrorCode.bad_match: "Invalid type in match statement",
+    ErrorCode.bad_evaluator: "Invalid code in type evaluator",
 }
 
 
diff --git a/pyanalyze/extensions.py b/pyanalyze/extensions.py
index 5c945e6..7815e53 100644
--- a/pyanalyze/extensions.py
+++ b/pyanalyze/extensions.py
@@ -412,10 +412,56 @@ def evaluated(func: Callable[..., Any]) -> Callable[..., Any]:
     return func
 
 
-def is_set(argument: object) -> bool:
+def is_provided(arg: Any) -> bool:
     """Helper function for type evaluators.
 
     May not be called at runtime.
 
     """
-    raise NotImplementedError("Should only be called in type evaluation functions")
+    raise NotImplementedError(
+        "is_provided() may only be called in type evaluation functions"
+    )
+
+
+def is_positional(arg: Any) -> bool:
+    """Helper function for type evaluators.
+
+    May not be called at runtime.
+
+    """
+    raise NotImplementedError(
+        "is_positional() may only be called in type evaluation functions"
+    )
+
+
+def is_keyword(arg: Any) -> bool:
+    """Helper function for type evaluators.
+
+    May not be called at runtime.
+
+    """
+    raise NotImplementedError(
+        "is_keyword() may only be called in type evaluation functions"
+    )
+
+
+def is_of_type(arg: Any, type: Any, *, exclude_any: bool = False) -> bool:
+    """Helper function for type evaluators.
+
+    May not be called at runtime.
+
+    """
+    raise NotImplementedError(
+        "is_of_type() may only be called in type evaluation functions"
+    )
+
+
+def show_error(message: str, *, argument: Optional[Any] = None) -> bool:
+    """Helper function for type evaluators.
+
+    May not be called at runtime.
+
+    """
+    raise NotImplementedError(
+        "show_error() may only be called in type evaluation functions"
+    )
diff --git a/pyanalyze/functions.py b/pyanalyze/functions.py
index d8331ea..ebaa940 100644
--- a/pyanalyze/functions.py
+++ b/pyanalyze/functions.py
@@ -6,6 +6,7 @@ Code for understanding function definitions.
 from abc import abstractmethod
 import ast
 import asyncio
+import collections.abc
 import types
 import asynq
 import enum
@@ -16,7 +17,7 @@ from typing_extensions import Protocol
 
 from .config import Config
 from .error_code import ErrorCode
-from .extensions import overload, real_overload
+from .extensions import overload, real_overload, evaluated
 from .options import Options, PyObjectSequenceOption
 from .node_visitor import ErrorContext
 from .signature import SigParameter, ParameterKind, Signature
@@ -63,6 +64,7 @@ class FunctionInfo:
     is_staticmethod: bool  # has @staticmethod
     is_decorated_coroutine: bool  # has @asyncio.coroutine
     is_overload: bool  # typing.overload or pyanalyze.extensions.overload
+    is_evaluated: bool  # @pyanalyze.extensions.evaluated
     is_abstractmethod: bool  # has @abstractmethod
     # a list of pairs of (decorator function, applied decorator function). These are different
     # for decorators that take arguments, like @asynq(): the first element will be the asynq
@@ -153,6 +155,7 @@ def compute_function_info(
     is_staticmethod = False
     is_overload = False
     is_abstractmethod = False
+    is_evaluated = False
     decorators = []
     for decorator in [] if isinstance(node, ast.Lambda) else node.decorator_list:
         # We have to descend into the Call node because the result of
@@ -186,6 +189,8 @@ def compute_function_info(
                 is_overload = True
             elif decorator_value == KnownValue(abstractmethod):
                 is_abstractmethod = True
+            elif decorator_value == KnownValue(evaluated):
+                is_evaluated = True
             decorators.append((decorator_value, decorator_value))
     params = compute_parameters(
         node,
@@ -206,6 +211,7 @@ def compute_function_info(
         is_staticmethod=is_staticmethod,
         is_abstractmethod=is_abstractmethod,
         is_overload=is_overload,
+        is_evaluated=is_evaluated,
         decorators=decorators,
         node=node,
         params=params,
@@ -316,6 +322,8 @@ def compute_value_of_function(
         result = info.return_annotation
     if result is None:
         result = AnyValue(AnySource.unannotated)
+    if isinstance(info.node, ast.AsyncFunctionDef):
+        result = GenericValue(collections.abc.Awaitable, [result])
     sig = Signature.make(
         [param_info.param for param_info in info.params],
         result,
diff --git a/pyanalyze/name_check_visitor.py b/pyanalyze/name_check_visitor.py
index 8d66443..2e7b3f0 100644
--- a/pyanalyze/name_check_visitor.py
+++ b/pyanalyze/name_check_visitor.py
@@ -56,6 +56,7 @@ import qcore
 
 from . import attributes, format_strings, node_visitor, importer, method_return_type
 from .annotations import (
+    EvaluatorValidationContext,
     is_instance_of_typing_name,
     type_from_runtime,
     type_from_value,
@@ -174,6 +175,7 @@ from .value import (
     concrete_values_from_iterable,
     unpack_values,
 )
+from pyanalyze import type_evaluation
 
 try:
     from ast import NamedExpr
@@ -993,6 +995,7 @@ class NameCheckVisitor(node_visitor.ReplacingNodeVisitor):
         self._method_cache = {}
         self._statement_types = set()
         self._has_used_any_match = False
+        self._should_exclude_any = False
         self._fill_method_cache()
 
     def make_type_object(self, typ: Union[type, super, str]) -> TypeObject:
@@ -1019,6 +1022,14 @@ class NameCheckVisitor(node_visitor.ReplacingNodeVisitor):
         :meth:`record_any_match`."""
         return qcore.override(self, "_has_used_any_match", False)
 
+    def set_exclude_any(self) -> ContextManager[None]:
+        """Within this context, `Any` is compatible only with itself."""
+        return qcore.override(self, "_should_exclude_any", True)
+
+    def should_exclude_any(self) -> bool:
+        """Whether Any should be compatible only with itself."""
+        return self._should_exclude_any
+
     # The type for typ should be type, but that leads Cython to reject calls that pass
     # an instance of ABCMeta.
     def get_generic_bases(
@@ -1560,7 +1571,7 @@ class NameCheckVisitor(node_visitor.ReplacingNodeVisitor):
             val = compute_value_of_function(info, self)
         else:
             val = KnownValue(potential_function)
-        if not info.is_overload:
+        if not info.is_overload and not info.is_evaluated:
             self._set_name_in_scope(node.name, node, val)
 
         with self.asynq_checker.set_func_name(
@@ -1579,13 +1590,14 @@ class NameCheckVisitor(node_visitor.ReplacingNodeVisitor):
         if (
             not result.has_return
             and not info.is_overload
+            and not info.is_evaluated
+            and not info.is_abstractmethod
             and node.returns is not None
             and info.return_annotation != KnownNone
             and not (
                 isinstance(info.return_annotation, AnnotatedValue)
                 and info.return_annotation.value == KnownNone
             )
-            and not info.is_abstractmethod
         ):
             if info.return_annotation is NO_RETURN_VALUE:
                 self._show_error_if_checking(
@@ -1655,7 +1667,7 @@ class NameCheckVisitor(node_visitor.ReplacingNodeVisitor):
             return_value = GenericValue(collections.abc.Awaitable, [return_value])
 
         sig = self.signature_from_value(val)
-        if sig is None:
+        if sig is None or sig.has_return_value():
             return
         self._argspec_to_retval[id(sig)] = return_value
 
@@ -1712,7 +1724,31 @@ class NameCheckVisitor(node_visitor.ReplacingNodeVisitor):
         if is_collecting and not self.scopes.contains_scope_of_type(
             ScopeType.function_scope
         ):
-            return FunctionResult()
+            return FunctionResult(parameters=params)
+
+        if function_info.is_evaluated:
+            if self._is_collecting():
+                return FunctionResult(parameters=params)
+            with self.scopes.allow_only_module_scope():
+                ctx = EvaluatorValidationContext(
+                    variables={param.name: param.annotation for param in params},
+                    positions={param.name: type_evaluation.DEFAULT for param in params},
+                    can_assign_context=self,
+                    node=node,
+                )
+                for error in type_evaluation.validate(node, ctx):
+                    self.show_error(
+                        error.node, error.message, error_code=ErrorCode.bad_evaluator
+                    )
+                if self.annotate:
+                    with self.catch_errors(), self.scopes.add_scope(
+                        ScopeType.function_scope, scope_node=node
+                    ):
+                        if isinstance(node, ast.Lambda):
+                            self.generic_visit(node.body)
+                        else:
+                            self._generic_visit_list(node.body)
+            return FunctionResult(parameters=params)
 
         # We pass in the node to add_scope() and visit the body once in collecting
         # mode if in a nested function, so that constraints on nonlocals in the outer
diff --git a/pyanalyze/signature.py b/pyanalyze/signature.py
index dad35f1..5be4221 100644
--- a/pyanalyze/signature.py
+++ b/pyanalyze/signature.py
@@ -7,7 +7,7 @@ calls.
 """
 
 from . import annotations
-from .type_evaluation import Evaluator, decompose_union, evaluate
+from .type_evaluation import Evaluator
 from .error_code import ErrorCode
 from .safe import all_of_type
 from .stacked_scopes import (
@@ -20,6 +20,7 @@ from .stacked_scopes import (
     AbstractConstraint,
     VarnameWithOrigin,
 )
+from .type_evaluation import ARGS, KWARGS, DEFAULT, UNKNOWN, Position
 from .value import (
     AnnotatedValue,
     AnySource,
@@ -33,6 +34,7 @@ from .value import (
     HasAttrGuardExtension,
     KVPair,
     KnownValue,
+    MultiValuedValue,
     NoReturnConstraintExtension,
     NoReturnGuardExtension,
     ParameterTypeGuardExtension,
@@ -56,6 +58,7 @@ from .value import (
     unannotate_value,
     unify_typevar_maps,
     unite_values,
+    unannotate,
     NO_RETURN_VALUE,
 )
 
@@ -68,7 +71,6 @@ import enum
 import itertools
 from types import MethodType, FunctionType
 import inspect
-import qcore
 from qcore.helpers import safe_str
 from typing import (
     Any,
@@ -93,9 +95,6 @@ if TYPE_CHECKING:
 
 EMPTY = inspect.Parameter.empty
 UNANNOTATED = AnyValue(AnySource.unannotated)
-ARGS = qcore.MarkerObject("*args")
-KWARGS = qcore.MarkerObject("**kwargs")
-DEFAULT = qcore.MarkerObject("default")
 
 
 @dataclass
@@ -108,12 +107,10 @@ class PossibleKwarg:
 # Representation of a single argument to a call. Second member is
 # None for positional args, str for keyword args,
 # ARGS for *args, KWARGS for **kwargs.
-Argument = Tuple[Composite, Union[None, str, Literal[ARGS], Literal[KWARGS]]]
+Argument = Tuple[Composite, Union[None, str, Literal[ARGS, KWARGS]]]
 
-# Arguments bound to a call. The first member of the tuple is a reference back
-# to the ActualArguments: the value is a str for a kwarg, int for a positional,
-# DEFAULT for something filled from the defaults, or None for something we cannot infer.
-BoundArgs = Dict[str, Tuple[Union[None, Literal[DEFAULT], str, int], Composite]]
+# Arguments bound to a call
+BoundArgs = Dict[str, Tuple[Position, Composite]]
 
 
 @dataclass
@@ -546,7 +543,11 @@ class Signature:
                     )
                     positional_index += 1
                 elif actual_args.star_args is not None:
-                    bound_args[param.name] = None, Composite(actual_args.star_args)
+                    if param.default is None:
+                        position = ARGS  # either that or the call fails
+                    else:
+                        position = UNKNOWN  # default or args
+                    bound_args[param.name] = position, Composite(actual_args.star_args)
                     star_args_consumed = True
                 elif param.default is None:
                     self.show_call_error(
@@ -582,15 +583,20 @@ class Signature:
                         )
                         return None
                     star_args_consumed = True
+                    if param.default is None:
+                        position = ARGS
+                    else:
+                        position = UNKNOWN
                     # It may also come from **kwargs
                     if actual_args.star_kwargs is not None:
                         value = unite_values(
                             actual_args.star_args, actual_args.star_kwargs
                         )
                         star_kwargs_consumed = True
+                        position = UNKNOWN  # could be either args or kwargs
                     else:
                         value = actual_args.star_args
-                    bound_args[param.name] = None, Composite(value)
+                    bound_args[param.name] = position, Composite(value)
                 elif param.name in actual_args.keywords:
                     definitely_provided, composite = actual_args.keywords[param.name]
                     if not definitely_provided and param.default is None:
@@ -604,7 +610,13 @@ class Signature:
                     bound_args[param.name] = param.name, composite
                     keywords_consumed.add(param.name)
                 elif actual_args.star_kwargs is not None:
-                    bound_args[param.name] = None, Composite(actual_args.star_kwargs)
+                    if param.default is None:
+                        position = KWARGS
+                    else:
+                        position = UNKNOWN
+                    bound_args[param.name] = position, Composite(
+                        actual_args.star_kwargs
+                    )
                     star_kwargs_consumed = True
                 elif param.default is None:
                     self.show_call_error(
@@ -627,7 +639,13 @@ class Signature:
                     bound_args[param.name] = param.name, composite
                     keywords_consumed.add(param.name)
                 elif actual_args.star_kwargs is not None:
-                    bound_args[param.name] = None, Composite(actual_args.star_kwargs)
+                    if param.default is None:
+                        position = KWARGS
+                    else:
+                        position = UNKNOWN
+                    bound_args[param.name] = position, Composite(
+                        actual_args.star_kwargs
+                    )
                     star_kwargs_consumed = True
                     keywords_consumed.add(param.name)
                 elif param.default is None:
@@ -643,12 +661,16 @@ class Signature:
                 while positional_index < len(actual_args.positionals):
                     positionals.append(actual_args.positionals[positional_index].value)
                     positional_index += 1
+                position = ARGS
                 if actual_args.star_args is not None:
                     element_value = unite_values(*positionals, actual_args.star_args)
                     star_args_value = GenericValue(tuple, [element_value])
                 else:
                     star_args_value = SequenceIncompleteValue(tuple, positionals)
-                bound_args[param.name] = None, Composite(star_args_value)
+                    if not positionals:
+                        # no *args were actually provided
+                        position = DEFAULT
+                bound_args[param.name] = position, Composite(star_args_value)
             elif param.kind is ParameterKind.VAR_KEYWORD:
                 star_kwargs_consumed = True
                 items = {}
@@ -659,6 +681,7 @@ class Signature:
                     if key in keywords_consumed:
                         continue
                     items[key] = (definitely_provided, composite.value)
+                position = KWARGS
                 if actual_args.star_kwargs is not None:
                     value_value = unite_values(
                         *(val for _, val in items.values()), actual_args.star_kwargs
@@ -668,7 +691,9 @@ class Signature:
                     )
                 else:
                     star_kwargs_value = TypedDictValue(items)
-                bound_args[param.name] = None, Composite(star_kwargs_value)
+                    if not items:
+                        position = DEFAULT
+                bound_args[param.name] = position, Composite(star_kwargs_value)
             else:
                 assert False, f"unhandled param {param.kind}"
 
@@ -837,23 +862,22 @@ class Signature:
                     param: composite.value
                     for param, (_, composite) in bound_args.items()
                 }
-                set_variables = {
-                    param
-                    for param, (position, _) in bound_args.items()
-                    if position is not DEFAULT
+                positions = {
+                    param: position for param, (position, _) in bound_args.items()
                 }
                 ctx = annotations.TypeEvaluationContext(
-                    varmap, set_variables, visitor, self.evaluator.globals
+                    varmap, positions, visitor, self.evaluator.globals
                 )
-                return_value = evaluate(self.evaluator, ctx)
-                if isinstance(return_value, CanAssignError):
+                return_value, errors = self.evaluator.evaluate(ctx)
+                for error in errors:
+                    error_node = node
+                    if error.argument is not None:
+                        composite = bound_args[error.argument][1]
+                        if composite.node is not None:
+                            error_node = node
                     self.show_call_error(
-                        "Error in type evaluator",
-                        node,
-                        visitor,
-                        detail=str(return_value),
+                        error.message, error_node, visitor, detail=error.get_detail()
                     )
-                    return_value = AnyValue(AnySource.error)
 
         if self.allow_call:
             runtime_return = self._maybe_perform_call(preprocessed, visitor, node)
@@ -1344,9 +1368,8 @@ class Signature:
             allow_call=self.allow_call,
         )
 
-    # TODO: do we need these?
     def has_return_value(self) -> bool:
-        return self.has_return_annotation
+        return self.has_return_annotation or self.evaluator is not None
 
 
 ANY_SIGNATURE = Signature.make(
@@ -1991,3 +2014,30 @@ def can_assign_var_keyword(
             )
         tv_maps.append(tv_map)
     return tv_maps
+
+
+def decompose_union(
+    expected_type: Value, parent_value: Value, ctx: CanAssignContext
+) -> Optional[Tuple[TypeVarMap, bool, Value]]:
+    value = unannotate(parent_value)
+    if isinstance(value, MultiValuedValue):
+        tv_maps = []
+        remaining_values = []
+        union_used_any = False
+        for val in value.vals:
+            can_assign, subval_used_any = can_assign_and_used_any(
+                expected_type, val, ctx
+            )
+            if isinstance(can_assign, CanAssignError):
+                remaining_values.append(val)
+            else:
+                if subval_used_any:
+                    union_used_any = True
+                tv_maps.append(can_assign)
+        if tv_maps:
+            tv_map = unify_typevar_maps(tv_maps)
+            assert (
+                remaining_values
+            ), f"all union members matched between {expected_type} and {parent_value}"
+            return tv_map, union_used_any, unite_values(*remaining_values)
+    return None
diff --git a/pyanalyze/stacked_scopes.py b/pyanalyze/stacked_scopes.py
index e44ee55..7aa456c 100644
--- a/pyanalyze/stacked_scopes.py
+++ b/pyanalyze/stacked_scopes.py
@@ -1298,7 +1298,7 @@ class StackedScopes:
             self.scopes.pop()
 
     @contextlib.contextmanager
-    def ignore_topmost_scope(self) -> Iterable[None]:
+    def ignore_topmost_scope(self) -> Iterator[None]:
         """Context manager that temporarily ignores the topmost scope."""
         scope = self.scopes.pop()
         try:
@@ -1306,6 +1306,16 @@ class StackedScopes:
         finally:
             self.scopes.append(scope)
 
+    @contextlib.contextmanager
+    def allow_only_module_scope(self) -> Iterator[None]:
+        """Context manager that allows only lookups in the module and builtin scopes."""
+        rest = self.scopes[2:]
+        del self.scopes[2:]
+        try:
+            yield
+        finally:
+            self.scopes += rest
+
     def get(self, varname: Varname, node: Node, state: VisitorState) -> Value:
         """Gets a variable of the given name from the current scope stack.
 
diff --git a/pyanalyze/type_evaluation.py b/pyanalyze/type_evaluation.py
index fd014c5..ad96cad 100644
--- a/pyanalyze/type_evaluation.py
+++ b/pyanalyze/type_evaluation.py
@@ -6,52 +6,267 @@ Implementation of type evaluation.
 
 import ast
 from contextlib import contextmanager
-from dataclasses import dataclass
+import contextlib
+from dataclasses import dataclass, field
 import inspect
+import operator
+import qcore
+import sys
 import textwrap
+from typing_extensions import Literal
 from typing import (
     Any,
     Callable,
-    Container,
+    Dict,
     Iterator,
+    List,
     Mapping,
     Optional,
     Sequence,
     Tuple,
+    Type,
     Union,
 )
 
 from .extensions import get_type_evaluation
+from .safe import all_of_type
 from .value import (
+    NO_RETURN_VALUE,
     AnySource,
     AnyValue,
+    CanAssign,
     CanAssignContext,
     CanAssignError,
     KnownValue,
     MultiValuedValue,
+    SequenceIncompleteValue,
     Value,
-    can_assign_and_used_any,
+    flatten_values,
     unannotate,
     unite_values,
     unify_typevar_maps,
     TypeVarMap,
 )
 
+ARGS = qcore.MarkerObject("*args")
+KWARGS = qcore.MarkerObject("**kwargs")
+DEFAULT = qcore.MarkerObject("default")
+UNKNOWN = qcore.MarkerObject("unknown")
+
+# How a bound argument was filled: int for a positional arg,
+# str for a keyword arg, DEFAULT for a parameter filled from the
+# default, ARGS for a parameter filled from *args, KWARGS for **kwargs,
+# UNKNOWN if there are multiple possibilities.
+Position = Union[int, str, Literal[DEFAULT, ARGS, KWARGS, UNKNOWN]]
+
 VarMap = Mapping[str, Value]
 
 
-class InvalidEvaluation(Exception):
-    pass
+class Condition:
+    def display(self, negated: bool = False) -> CanAssignError:
+        raise NotImplementedError
+
+
+class NullCondition(Condition):
+    def display(self, negated: bool = False) -> CanAssignError:
+        return CanAssignError()
+
+
+@dataclass
+class ConditionList(Condition):
+    children: Sequence[Condition]
+
+    def display(self, negated: bool = False) -> CanAssignError:
+        return CanAssignError(
+            children=[c.display(negated=negated) for c in self.children]
+        )
+
+
+@dataclass
+class NotCondition(Condition):
+    condition: Condition
+
+    def display(self, negated: bool = False) -> CanAssignError:
+        return self.condition.display(negated=not negated)
+
+
+@dataclass
+class ArgumentKindCondition(Condition):
+    argument: str
+    function: Literal["is_provided", "is_positional", "is_keyword"]
+
+    def display(self, negated: bool = False) -> CanAssignError:
+        maybe_not = " not" if negated else ""
+        text = f"Argument {self.argument} was{maybe_not} provided"
+        if self.function == "is_positional":
+            return CanAssignError(f"{text} as a positional argument")
+        elif self.function == "is_keyword":
+            return CanAssignError(f"{text} as a keyword argument")
+        else:
+            return CanAssignError(text)
+
+
+@dataclass
+class PlatformCondition(Condition):
+    actual: str
+    op: Type[ast.cmpop]
+    expected: object
+
+    def display(self, negated: bool = False) -> CanAssignError:
+        if negated:
+            op = _OP_TO_DATA[self.op].negation
+        else:
+            op = self.op
+        text = _OP_TO_DATA[op].text
+        return CanAssignError(f"Platform ({self.actual}) {text} {self.expected}")
+
+
+@dataclass
+class VersionCondition(Condition):
+    actual: Tuple[int, ...]
+    op: Type[ast.cmpop]
+    expected: object
+
+    def _display_version(self, version: object) -> str:
+        if isinstance(version, tuple):
+            return ".".join(map(str, version))
+        return str(version)
+
+    def display(self, negated: bool = False) -> CanAssignError:
+        if negated:
+            op = _OP_TO_DATA[self.op].negation
+        else:
+            op = self.op
+        text = _OP_TO_DATA[op].text
+        return CanAssignError(
+            f"Platform ({self._display_version(self.actual)})"
+            f" {text} {self._display_version(self.expected)}"
+        )
+
+
+@dataclass
+class IsOfTypeCondition(Condition):
+    arg: str
+    op: "_Operator"
+    original_arg_type: Value
+    remaining_type: Value
+    expected_type: Value
+    exclude_any: bool = True
+
+    def display(self, negated: bool = False) -> CanAssignError:
+        positive_text = _OP_TO_DATA[self.op].text
+        negative_text = _OP_TO_DATA[_OP_TO_DATA[self.op].negation].text
+        if negated:
+            positive_text, negative_text = negative_text, positive_text
+
+        original_arg_type = self.original_arg_type
+        remaining_type = self.remaining_type
+        matched_type = subtract_unions(original_arg_type, remaining_type)
+        if negated:
+            matched_type, remaining_type = remaining_type, matched_type
+
+        if matched_type is NO_RETURN_VALUE:
+            text = negative_text
+            type_to_show = remaining_type
+        elif remaining_type is NO_RETURN_VALUE:
+            text = positive_text
+            type_to_show = matched_type
+        else:
+            text = f"partially {positive_text}"
+            type_to_show = unite_values(matched_type, remaining_type)
+        if self.exclude_any:
+            epilog = ""
+        else:
+            epilog = " (using permissive Any semantics)"
+        if self.op not in (ast.LtE, ast.Gt) and isinstance(
+            self.expected_type, KnownValue
+        ):
+            expected_type = repr(self.expected_type.val)
+        else:
+            expected_type = str(self.expected_type)
+        return CanAssignError(
+            f"Argument {self.arg} (type: {type_to_show}) {text} {expected_type}{epilog}"
+        )
+
+
+def subtract_unions(left: Value, right: Value) -> Value:
+    if right is NO_RETURN_VALUE:
+        return left
+    right_vals = set(flatten_values(right))
+    remaining = [
+        subval
+        for subval in flatten_values(unannotate(left))
+        if subval not in right_vals
+    ]
+    return unite_values(*remaining)
+
+
+_Operator = Union[Type[ast.cmpop], Literal["matches", "does not match"]]
+
+
+@dataclass
+class _Comparator:
+    text: str
+    negation: _Operator
+    impl: Callable[[Any, Any], Any]
+
+
+def _dummy_impl(left: object, right: object) -> object:
+    raise NotImplementedError
+
+
+_OP_TO_DATA: Dict[_Operator, _Comparator] = {
+    ast.Is: _Comparator("is", ast.IsNot, operator.is_),
+    ast.IsNot: _Comparator("is not", ast.Is, operator.is_not),
+    ast.Eq: _Comparator("==", ast.Eq, operator.eq),
+    ast.NotEq: _Comparator("!=", ast.NotEq, operator.ne),
+    ast.Gt: _Comparator(">", ast.LtE, operator.gt),
+    ast.LtE: _Comparator("<=", ast.Gt, operator.le),
+    ast.Lt: _Comparator("<", ast.GtE, operator.lt),
+    ast.GtE: _Comparator(">=", ast.Lt, operator.ge),
+    ast.In: _Comparator("in", ast.NotIn, lambda a, b: a in b),
+    ast.NotIn: _Comparator("not in", ast.In, lambda a, b: a not in b),
+    "matches": _Comparator("matches", "matches", _dummy_impl),
+    "does not match": _Comparator("does not match", "matches", _dummy_impl),
+}
+
+
+@dataclass
+class InvalidEvaluation:
+    message: str
+    node: ast.AST
+
+
+@dataclass
+class UserRaisedError:
+    message: str
+    active_conditions: Sequence[Condition]
+    argument: Optional[str] = None
+
+    def get_detail(self) -> Optional[str]:
+        if self.active_conditions:
+            return str(
+                CanAssignError(
+                    children=[cond.display() for cond in self.active_conditions]
+                )
+            )
+        return None
+
+
+EvaluateError = Union[InvalidEvaluation, UserRaisedError]
 
 
 class Context:
     variables: VarMap
-    set_variables: Container[str]
+    positions: Mapping[str, Position]
     can_assign_context: CanAssignContext
 
     def evaluate_type(self, __node: ast.AST) -> Value:
         raise NotImplementedError
 
+    def evaluate_value(self, __node: ast.AST) -> Value:
+        raise NotImplementedError
+
     @contextmanager
     def narrow_variables(self, varmap: Optional[VarMap]) -> Iterator[None]:
         if varmap is None:
@@ -72,95 +287,171 @@ class Evaluator:
     node: ast.FunctionDef
     globals: Mapping[str, object]
 
-
-@dataclass
-class UnionCombinedReturn:
-    left: "EvalReturn"
-    right: "EvalReturn"
+    def evaluate(self, ctx: Context) -> Tuple[Value, Sequence[UserRaisedError]]:
+        visitor = EvaluateVisitor(ctx)
+        result = visitor.run(self.node)
+        errors = [e for e in visitor.errors if isinstance(e, UserRaisedError)]
+        return result, errors
 
 
 @dataclass
-class AnyCombinedReturn:
-    left: "EvalReturn"
-    right: "EvalReturn"
-
-
-EvalReturn = Union[None, Value, CanAssignError, UnionCombinedReturn, AnyCombinedReturn]
+class CombinedReturn:
+    children: Sequence[Optional[Value]]
+
+    @classmethod
+    def make(cls, *returns: "EvalReturn") -> "EvalReturn":
+        if len(returns) == 1:
+            return returns[0]
+        children = []
+        for child in returns:
+            if isinstance(child, CombinedReturn):
+                children += child.children
+            else:
+                children.append(child)
+        return CombinedReturn(children)
 
 
-def may_be_none(ret: EvalReturn) -> bool:
-    if ret is None:
-        return True
-    elif isinstance(ret, (Value, CanAssignError)):
-        return False
-    else:
-        return may_be_none(ret.left) or may_be_none(ret.right)
+EvalReturn = Union[None, Value, CombinedReturn]
 
 
 @dataclass
 class ConditionReturn:
+    condition: Condition
     # These are None if there is no match, and a (possibly empty)
     # map of new variable values if there is a match.
     left_varmap: Optional[VarMap] = None
     right_varmap: Optional[VarMap] = None
-    is_any_match: bool = False
 
     def reverse(self) -> "ConditionReturn":
         return ConditionReturn(
             left_varmap=self.right_varmap,
             right_varmap=self.left_varmap,
-            is_any_match=self.is_any_match,
+            condition=NotCondition(self.condition),
         )
 
 
 @dataclass
 class ConditionEvaluator(ast.NodeVisitor):
     ctx: Context
+    validation_mode: bool = False
+    errors: List[EvaluateError] = field(default_factory=list, init=False)
+
+    def return_invalid(self, message: str, node: ast.AST) -> ConditionReturn:
+        self.errors.append(InvalidEvaluation(message, node))
+        return ConditionReturn(NullCondition())
 
     def visit_Call(self, node: ast.Call) -> ConditionReturn:
         if not isinstance(node.func, ast.Name):
-            raise InvalidEvaluation("Unexpected call")
+            return self.return_invalid("Unexpected call", node.func)
         name = node.func.id
-        if name == "is_set":
+        if name in ("is_provided", "is_positional", "is_keyword"):
             if node.keywords or len(node.args) != 1:
-                raise InvalidEvaluation("is_set() takes a single argument")
+                return self.return_invalid(f"{name}() takes a single argument", node)
             if not isinstance(node.args[0], ast.Name):
-                raise InvalidEvaluation("Argument to is_set() must be a variable")
+                return self.return_invalid(
+                    f"Argument to {name}() must be a variable", node.args[0]
+                )
             variable = node.args[0].id
-            match = variable in self.ctx.set_variables
+            try:
+                position = self.ctx.positions[variable]
+            except KeyError:
+                return self.return_invalid(
+                    f"{variable} is not a valid variable", node.args[0]
+                )
+            if name == "is_provided":
+                match = position is not DEFAULT and position is not UNKNOWN
+            elif name == "is_positional":
+                match = position is ARGS or isinstance(position, int)
+            elif name == "is_keyword":
+                match = position is KWARGS or isinstance(position, str)
+            else:
+                return self.return_invalid(name, node.func)
+            condition = ArgumentKindCondition(variable, name)
             if match:
-                return ConditionReturn(left_varmap={})
+                return ConditionReturn(left_varmap={}, condition=condition)
             else:
-                return ConditionReturn(right_varmap={})
-        elif name == "isinstance":
-            if node.keywords or len(node.args) != 2:
-                raise InvalidEvaluation("isinstance() takes two positional arguments")
+                return ConditionReturn(
+                    right_varmap={}, condition=NotCondition(condition)
+                )
+        elif name == "is_of_type":
+            if len(node.args) != 2:
+                return self.return_invalid(
+                    "is_of_type() takes two positional arguments", node
+                )
             varname_node = node.args[0]
             typ = self.ctx.evaluate_type(node.args[1])
-            return self.visit_isinstance(varname_node, typ)
+            exclude_any = True
+            for keyword in node.keywords:
+                if keyword.arg == "exclude_any":
+                    if isinstance(
+                        keyword.value, ast.NameConstant
+                    ) and keyword.value.value in (True, False):
+                        exclude_any = keyword.value.value
+                    else:
+                        return self.return_invalid(
+                            "exclude_any argument must be a literal bool", keyword.value
+                        )
+                else:
+                    # Before 3.9 keyword nodes don't have a lineno
+                    if sys.version_info >= (3, 9):
+                        error_node = keyword
+                    else:
+                        error_node = node
+                    return self.return_invalid(
+                        "Invalid keyword argument to is_of_type()", error_node
+                    )
+            return self.visit_is_of_type(
+                varname_node, typ, "matches", exclude_any=exclude_any
+            )
         else:
-            raise InvalidEvaluation(f"Invalid function {name}")
-
-    def visit_isinstance(self, varname_node: ast.AST, typ: Value) -> ConditionReturn:
+            return self.return_invalid(f"Invalid function {name}", node.func)
+
+    def visit_is_of_type(
+        self,
+        varname_node: ast.AST,
+        typ: Value,
+        op: _Operator,
+        *,
+        exclude_any: bool = True,
+    ) -> ConditionReturn:
         if not isinstance(varname_node, ast.Name):
-            raise InvalidEvaluation("First argument to isinstance() must be a name")
+            return self.return_invalid(
+                "First argument to is_of_type() must be a name", varname_node
+            )
         val = self.get_name(varname_node)
-        can_assign, used_any = can_assign_and_used_any(
-            typ, val, self.ctx.can_assign_context
+        if val is None:
+            return ConditionReturn(NullCondition())
+        can_assign = can_assign_maybe_exclude_any(
+            typ, val, self.ctx.can_assign_context, exclude_any
         )
         if isinstance(can_assign, CanAssignError):
-            triple = decompose_union(typ, val, self.ctx.can_assign_context)
-            if triple is not None:
-                _, used_any, remaining = triple
+            pair = decompose_union(typ, val, self.ctx.can_assign_context, exclude_any)
+            if pair is not None:
+                _, remaining = pair
                 return ConditionReturn(
+                    condition=IsOfTypeCondition(
+                        varname_node.id,
+                        op,
+                        val,
+                        remaining,
+                        typ,
+                        exclude_any=exclude_any,
+                    ),
                     left_varmap={varname_node.id: typ},
                     right_varmap={varname_node.id: remaining},
-                    is_any_match=used_any,
                 )
-            return ConditionReturn(right_varmap={})
+            return ConditionReturn(right_varmap={}, condition=NullCondition())
         else:
             return ConditionReturn(
-                left_varmap={varname_node.id: typ}, is_any_match=used_any
+                left_varmap={varname_node.id: typ},
+                condition=IsOfTypeCondition(
+                    varname_node.id,
+                    op,
+                    val,
+                    NO_RETURN_VALUE,
+                    typ,
+                    exclude_any=exclude_any,
+                ),
             )
 
     def visit_UnaryOp(self, node: ast.UnaryOp) -> ConditionReturn:
@@ -168,51 +459,189 @@ class ConditionEvaluator(ast.NodeVisitor):
             ret = self.visit(node.operand)
             return ret.reverse()
         else:
-            raise InvalidEvaluation("Unsupported unary operation")
+            return self.return_invalid("Unsupported unary operation", node)
 
     def visit_Compare(self, node: ast.Compare) -> ConditionReturn:
         if len(node.ops) != 1:
-            raise InvalidEvaluation("Chained comparison is unsupported")
+            return self.return_invalid("Chained comparison is unsupported", node)
         op = node.ops[0]
         right = node.comparators[0]
-        if isinstance(op, (ast.Is, ast.IsNot)):
-            if not isinstance(right, ast.NameConstant):
-                raise InvalidEvaluation(
-                    "is/is not are only supported with True, False, and None"
-                )
-            ret = self.visit_isinstance(node.left, KnownValue(right.value))
-            if isinstance(op, ast.IsNot):
-                return ret.reverse()
-            return ret
-        elif isinstance(op, (ast.Eq, ast.NotEq)):
-            operand = self.evaluate_literal(right)
-            ret = self.visit_isinstance(node.left, operand)
-            if isinstance(op, ast.NotEq):
+        right_operand = self.evaluate_literal(right)
+        if right_operand is None:
+            return ConditionReturn(NullCondition())
+        if isinstance(node.left, ast.Name) and isinstance(
+            op, (ast.Is, ast.IsNot, ast.Eq, ast.NotEq)
+        ):
+            ret = self.visit_is_of_type(node.left, right_operand, type(op))
+            if isinstance(op, (ast.NotEq, ast.IsNot)):
                 return ret.reverse()
             return ret
+
+        if isinstance(node.left, ast.Attribute):
+            mod = self.evaluate_literal(node.left.value)
+            if mod == KnownValue(sys):
+                if node.left.attr == "platform":
+                    left_operand = sys.platform
+                elif node.left.attr == "version_info":
+                    left_operand = sys.version_info
+                else:
+                    return self.return_invalid(
+                        "Only comparisons on sys.platform and sys.version_info are"
+                        " suppoorted",
+                        node.left,
+                    )
+                data = _OP_TO_DATA[type(op)]
+                try:
+                    result = data.impl(left_operand, right_operand.val)
+                except Exception:
+                    return self.return_invalid(
+                        f"Invalid sys.{node.left.attr} comparison", node
+                    )
+                if node.left.attr == "platform":
+                    condition = PlatformCondition(
+                        sys.platform, type(op), right_operand.val
+                    )
+                else:
+                    condition = VersionCondition(
+                        sys.version_info[:2], type(op), right_operand.val
+                    )
+                if result:
+                    return ConditionReturn(condition, left_varmap={})
+                else:
+                    return ConditionReturn(condition, right_varmap={})
+
+        return self.return_invalid("Unsupported comparison operator", node)
+
+    def visit_BoolOp(self, node: ast.BoolOp) -> ConditionReturn:
+        if self.validation_mode:
+            for operand in node.values:
+                self.visit(operand)
+            return ConditionReturn(NullCondition())
+        active = []
+        is_and = isinstance(node.op, ast.And)
+        remaining_varmaps = []
+        narrowed_varmap = {}
+        stack = contextlib.ExitStack()
+        with stack:
+            for operand in node.values:
+                result = self.visit(operand)
+                if is_and:
+                    if result.left_varmap is None:
+                        # Condition returns False
+                        active.append(NotCondition(result.condition))
+                        return ConditionReturn(
+                            right_varmap=result.right_varmap,
+                            condition=ConditionList(active),
+                        )
+                    elif result.right_varmap is None:
+                        # Condition returns True
+                        narrowed_varmap.update(result.left_varmap)
+                        stack.enter_context(
+                            self.ctx.narrow_variables(result.left_varmap)
+                        )
+                        active.append(result.condition)
+                    else:
+                        # Condition matches partially
+                        active.append(result.condition)
+                        narrowed_varmap.update(result.left_varmap)
+                        stack.enter_context(
+                            self.ctx.narrow_variables(result.left_varmap)
+                        )
+                        remaining_varmaps.append(result.right_varmap)
+                else:
+                    if result.left_varmap is None:
+                        # Condition returns False
+                        narrowed_varmap.update(result.right_varmap)
+                        stack.enter_context(
+                            self.ctx.narrow_variables(result.right_varmap)
+                        )
+                        active.append(NotCondition(result.condition))
+                    elif result.right_varmap is None:
+                        # Condition returns True
+                        active.append(result.condition)
+                        return ConditionReturn(
+                            left_varmap=result.left_varmap,
+                            condition=ConditionList(active),
+                        )
+                    else:
+                        # Condition partially matches
+                        active.append(result.condition)
+                        narrowed_varmap.update(result.right_varmap)
+                        stack.enter_context(
+                            self.ctx.narrow_variables(result.right_varmap)
+                        )
+                        remaining_varmaps.append(result.left_varmap)
+
+        # We got only partial matches
+        if is_and:
+            return ConditionReturn(
+                ConditionList(active),
+                left_varmap=narrowed_varmap,
+                right_varmap=unite_varmaps(remaining_varmaps),
+            )
         else:
-            raise InvalidEvaluation("Unsupported comparison operator")
-
-    def evaluate_literal(self, node: ast.expr) -> KnownValue:
-        if isinstance(node, ast.NameConstant):
-            return KnownValue(node.value)
-        elif isinstance(node, ast.Num):
-            return KnownValue(node.n)
-        elif isinstance(node, (ast.Str, ast.Bytes)):
-            return KnownValue(node.s)
-        else:
-            raise InvalidEvaluation("Only literals supported")
+            return ConditionReturn(
+                ConditionList(active),
+                left_varmap=unite_varmaps(remaining_varmaps),
+                right_varmap=narrowed_varmap,
+            )
 
-    def get_name(self, node: ast.Name) -> Value:
+    def evaluate_literal(self, node: ast.expr) -> Optional[KnownValue]:
+        val = self.ctx.evaluate_value(node)
+        if (
+            isinstance(val, SequenceIncompleteValue)
+            and isinstance(val.typ, type)
+            and all_of_type(val.members, KnownValue)
+        ):
+            val = KnownValue(val.typ(elt.val for elt in val.members))
+        if isinstance(val, KnownValue):
+            return val
+        self.errors.append(InvalidEvaluation("Only literals supported", node))
+        return None
+
+    def get_name(self, node: ast.Name) -> Optional[Value]:
         try:
             return self.ctx.variables[node.id]
         except KeyError:
-            raise InvalidEvaluation(f"Invalid variable {node.id}") from None
+            self.errors.append(InvalidEvaluation(f"Invalid variable {node.id}", node))
+            return None
 
 
 @dataclass
-class TypeEvaluator(ast.NodeVisitor):
+class EvaluateVisitor(ast.NodeVisitor):
     ctx: Context
+    errors: List[EvaluateError] = field(default_factory=list)
+    active_conditions: List[Condition] = field(default_factory=list)
+    validation_mode: bool = False
+
+    def run(self, node: ast.AST) -> Value:
+        ret = self.visit(node)
+        return self._evaluate_ret(ret, node)
+
+    def _evaluate_ret(self, ret: EvalReturn, node: ast.AST) -> Value:
+        if ret is None:
+            # TODO return the func's return annotation instead
+            if not self.validation_mode:
+                self.add_invalid("Evaluator failed to return", node)
+            return AnyValue(AnySource.error)
+        elif isinstance(ret, CombinedReturn):
+            children = [self._evaluate_ret(child, node) for child in ret.children]
+            return unite_values(*children)
+        else:
+            return ret
+
+    def add_invalid(self, message: str, node: ast.AST) -> None:
+        error = InvalidEvaluation(message, node)
+        self.errors.append(error)
+
+    @contextmanager
+    def add_active_condition(self, condition: Condition) -> Iterator[None]:
+        self.active_conditions.append(condition)
+        try:
+            yield
+        finally:
+            popped = self.active_conditions.pop()
+            assert popped == condition
 
     def visit_FunctionDef(self, node: ast.FunctionDef) -> EvalReturn:
         return self.visit_block(node.body)
@@ -221,103 +650,113 @@ class TypeEvaluator(ast.NodeVisitor):
         return self.visit_block(node.body)
 
     def visit_block(self, statements: Sequence[ast.stmt]) -> EvalReturn:
+        possible_returns = []
         for stmt in statements:
             result = self.visit(stmt)
-            if not may_be_none(result):
-                return result
+            if result is None:
+                continue
+            if isinstance(result, Value):
+                return CombinedReturn.make(*possible_returns, result)
+            else:
+                if all(res is not None for res in result.children):
+                    return CombinedReturn.make(*possible_returns, *result.children)
+                else:
+                    possible_returns += [
+                        res for res in result.children if res is not None
+                    ]
+        return CombinedReturn.make(*possible_returns, None)
+
+    def visit_Pass(self, node: ast.Pass) -> EvalReturn:
         return None
 
     def visit_Return(self, node: ast.Return) -> EvalReturn:
         if node.value is None:
-            raise InvalidEvaluation("return statement must have a value")
+            self.add_invalid("return statement must have a value", node)
+            return KnownValue(None)
         return self.ctx.evaluate_type(node.value)
 
-    def visit_Raise(self, node: ast.Raise) -> EvalReturn:
+    def visit_Expr(self, node: ast.Expr) -> EvalReturn:
         if (
-            not isinstance(node.exc, ast.Call)
-            or not isinstance(node.exc.func, ast.Name)
-            or node.exc.func.id != "Exception"
-            or node.exc.keywords
-            or len(node.exc.args) != 1
-            or not isinstance(node.exc.args[0], ast.Str)
+            isinstance(node.value, ast.Call)
+            and isinstance(node.value.func, ast.Name)
+            and node.value.func.id == "show_error"
         ):
-            raise InvalidEvaluation(
-                "raise statement must be of the form 'raise Exception(message)'"
+            call = node.value
+            if len(call.args) != 1:
+                self.add_invalid(
+                    "show_error() takes exactly one positional argument", call
+                )
+                return None
+            if not isinstance(call.args[0], ast.Str):
+                self.add_invalid(
+                    "show_error() message must be a string literal", call.args[0]
+                )
+                return None
+            message = call.args[0].s
+            argument = None
+            for keyword in call.keywords:
+                if keyword.arg == "argument":
+                    if not isinstance(keyword.value, ast.Name):
+                        self.add_invalid("argument must be a name", keyword.value)
+                        return None
+                    argument = keyword.value.id
+                    if argument not in self.ctx.variables:
+                        self.add_invalid(
+                            f"{argument} is not a valid argument", keyword.value
+                        )
+                        return None
+                else:
+                    # Before 3.9 keyword nodes don't have a lineno
+                    if sys.version_info >= (3, 9):
+                        error_node = keyword
+                    else:
+                        error_node = node
+                    self.add_invalid(
+                        "Invalid keyword argument to show_error()", error_node
+                    )
+                    return None
+            self.errors.append(
+                UserRaisedError(message, list(self.active_conditions), argument)
             )
-        return CanAssignError(node.exc.args[0].s)
+            return None
+        self.add_invalid("Invalid statement", node)
+        return None
 
     def visit_If(self, node: ast.If) -> EvalReturn:
-        visitor = ConditionEvaluator(self.ctx)
+        visitor = ConditionEvaluator(self.ctx, self.validation_mode)
         condition = visitor.visit(node.test)
-        if condition.is_any_match:
-            with self.ctx.narrow_variables(condition.left_varmap):
+        self.errors += visitor.errors
+        if self.validation_mode:
+            self.visit_block(node.body)
+            self.visit_block(node.orelse)
+            return None
+        if condition.left_varmap is not None:
+            with self.ctx.narrow_variables(
+                condition.left_varmap
+            ), self.add_active_condition(condition.condition):
                 left_result = self.visit_block(node.body)
+        else:
+            left_result = None
+        if condition.right_varmap is not None:
             with self.ctx.narrow_variables(condition.right_varmap):
                 right_result = self.visit_block(node.orelse)
-            return AnyCombinedReturn(left_result, right_result)
         else:
-            if condition.left_varmap is not None:
-                with self.ctx.narrow_variables(condition.left_varmap):
-                    left_result = self.visit_block(node.body)
-            else:
-                left_result = None
+            right_result = None
+        if condition.left_varmap is not None:
             if condition.right_varmap is not None:
-                with self.ctx.narrow_variables(condition.right_varmap):
-                    right_result = self.visit_block(node.orelse)
+                return CombinedReturn.make(left_result, right_result)
             else:
-                right_result = None
-            if condition.left_varmap is not None:
-                if condition.right_varmap is not None:
-                    return UnionCombinedReturn(left_result, right_result)
-                else:
-                    return left_result
+                return left_result
+        else:
+            if condition.right_varmap is not None:
+                return right_result
             else:
-                if condition.right_varmap is not None:
-                    return right_result
-                else:
-                    raise InvalidEvaluation("Condition must either match or not match")
-
-
-def evaluate(evaluator: Evaluator, ctx: Context) -> Union[Value, CanAssignError]:
-    visitor = TypeEvaluator(ctx)
-    try:
-        result = visitor.visit(evaluator.node)
-        return _evaluate_ret(result)
-    except InvalidEvaluation as e:
-        return CanAssignError(
-            "Internal error in type evaluator", [CanAssignError(str(e))]
-        )
-
-
-def _evaluate_ret(ret: EvalReturn) -> Union[Value, CanAssignError]:
-    if ret is None:
-        raise InvalidEvaluation("Evaluator failed to return")
-    elif isinstance(ret, AnyCombinedReturn):
-        left = _evaluate_ret(ret.left)
-        right = _evaluate_ret(ret.right)
-
-        # If one branch is an error, pick the other one
-        if isinstance(left, CanAssignError):
-            if isinstance(right, CanAssignError):
-                return CanAssignError(children=[left, right])
-            return right
-        if isinstance(right, CanAssignError):
-            return left
+                self.add_invalid("Condition must either match or not match", node)
+                return None
 
-        if left == right:
-            return left
-        else:
-            return AnyValue(AnySource.multiple_overload_matches)
-    elif isinstance(ret, UnionCombinedReturn):
-        left = _evaluate_ret(ret.left)
-        if isinstance(left, CanAssignError):
-            return left
-        right = _evaluate_ret(ret.right)
-        if isinstance(right, CanAssignError):
-            return right
-        return unite_values(left, right)
-    else:
-        return ret
+    def generic_visit(self, node: ast.AST) -> Any:
+        self.add_invalid("Invalid code in type evaluator", node)
+        return None
 
 
 def get_evaluator(func: Callable[..., Any]) -> Optional[Evaluator]:
@@ -335,32 +774,55 @@ def get_evaluator(func: Callable[..., Any]) -> Optional[Evaluator]:
         return None
     evaluator = body.body[0]
     if not isinstance(evaluator, ast.FunctionDef):
-        raise InvalidEvaluation(f"Cannot locate {func}")
+        return None
     return Evaluator(evaluation_func, evaluator, evaluation_func.__globals__)
 
 
+def validate(node: ast.AST, ctx: Context) -> List[InvalidEvaluation]:
+    visitor = EvaluateVisitor(ctx, validation_mode=True)
+    visitor.run(node)
+    return [error for error in visitor.errors if isinstance(error, InvalidEvaluation)]
+
+
 def decompose_union(
-    expected_type: Value, parent_value: Value, ctx: CanAssignContext
-) -> Optional[Tuple[TypeVarMap, bool, Value]]:
+    expected_type: Value, parent_value: Value, ctx: CanAssignContext, exclude_any: bool
+) -> Optional[Tuple[TypeVarMap, Value]]:
     value = unannotate(parent_value)
     if isinstance(value, MultiValuedValue):
         tv_maps = []
         remaining_values = []
-        union_used_any = False
         for val in value.vals:
-            can_assign, subval_used_any = can_assign_and_used_any(
-                expected_type, val, ctx
+            can_assign = can_assign_maybe_exclude_any(
+                expected_type, val, ctx, exclude_any
             )
             if isinstance(can_assign, CanAssignError):
                 remaining_values.append(val)
             else:
-                if subval_used_any:
-                    union_used_any = True
                 tv_maps.append(can_assign)
         if tv_maps:
             tv_map = unify_typevar_maps(tv_maps)
             assert (
                 remaining_values
             ), f"all union members matched between {expected_type} and {parent_value}"
-            return tv_map, union_used_any, unite_values(*remaining_values)
+            return tv_map, unite_values(*remaining_values)
     return None
+
+
+def can_assign_maybe_exclude_any(
+    left: Value, right: Value, ctx: CanAssignContext, exclude_any: bool
+) -> CanAssign:
+    if exclude_any:
+        with ctx.set_exclude_any():
+            return left.can_assign(right, ctx)
+    else:
+        return left.can_assign(right, ctx)
+
+
+def unite_varmaps(varmaps: Sequence[VarMap]) -> Optional[VarMap]:
+    if not varmaps:
+        return None
+    keys = set.intersection(*[set(m) for m in varmaps])
+    return {
+        key: unite_values(*[varmap.get(key, NO_RETURN_VALUE) for varmap in varmaps])
+        for key in keys
+    }
diff --git a/pyanalyze/typeshed.py b/pyanalyze/typeshed.py
index 6ea1e5b..3eba0e8 100644
--- a/pyanalyze/typeshed.py
+++ b/pyanalyze/typeshed.py
@@ -82,7 +82,10 @@ class _AnnotationContext(Context):
     module: str
 
     def show_error(
-        self, message: str, error_code: ErrorCode = ErrorCode.invalid_annotation
+        self,
+        message: str,
+        error_code: ErrorCode = ErrorCode.invalid_annotation,
+        node: Optional[ast.AST] = None,
     ) -> None:
         self.finder.log(message, ())
 
diff --git a/pyanalyze/value.py b/pyanalyze/value.py
index 174009a..790e460 100644
--- a/pyanalyze/value.py
+++ b/pyanalyze/value.py
@@ -81,7 +81,7 @@ class Value:
         This is the primary mechanism used for checking type compatibility.
 
         """
-        if isinstance(other, AnyValue):
+        if isinstance(other, AnyValue) and not ctx.should_exclude_any():
             ctx.record_any_used()
             return {}
         elif isinstance(other, MultiValuedValue):
@@ -241,6 +241,14 @@ class CanAssignContext(Protocol):
         :meth:`record_any_match`."""
         return qcore.empty_context
 
+    def set_exclude_any(self) -> ContextManager[None]:
+        """Within this context, `Any` is compatible only with itself."""
+        return qcore.empty_context
+
+    def should_exclude_any(self) -> bool:
+        """Whether Any should be compatible only with itself."""
+        return False
+
 
 @dataclass(frozen=True)
 class CanAssignError:
@@ -602,7 +610,7 @@ class TypedValue(Value):
         return super().can_assign(other, ctx)
 
     def can_assign_thrift_enum(self, other: Value, ctx: CanAssignContext) -> CanAssign:
-        if isinstance(other, AnyValue):
+        if isinstance(other, AnyValue) and not ctx.should_exclude_any():
             ctx.record_any_used()
             return {}
         elif isinstance(other, KnownValue):
@@ -1317,7 +1325,7 @@ class MultiValuedValue(Value):
             if not tv_maps:
                 return CanAssignError(f"Cannot assign {other} to {self}")
             return unify_typevar_maps(tv_maps)
-        elif isinstance(other, AnyValue):
+        elif isinstance(other, AnyValue) and not ctx.should_exclude_any():
             ctx.record_any_used()
             return {}
         else:
@@ -1839,8 +1847,6 @@ class VariableNameValue(AnyValue):
 
     def can_assign(self, other: Value, ctx: CanAssignContext) -> CanAssign:
         if not isinstance(other, VariableNameValue):
-            if isinstance(other, AnyValue):
-                ctx.record_any_used()
             return {}
         if other == self:
             return {}


Test patch:
diff --git a/pyanalyze/test_functions.py b/pyanalyze/test_functions.py
index 71f18b2..3400188 100644
--- a/pyanalyze/test_functions.py
+++ b/pyanalyze/test_functions.py
@@ -1,5 +1,5 @@
 # static analysis: ignore
-from .value import KnownValue, AnyValue, AnySource
+from .value import GenericValue, KnownValue, AnyValue, AnySource, TypedValue
 from .error_code import ErrorCode
 from .implementation import assert_is_value
 from .test_node_visitor import assert_passes, skip_before
@@ -35,7 +35,7 @@ class TestNestedFunction(TestNameCheckVisitorBase):
                     xs = ys
 
     @assert_passes()
-    def test_async(self):
+    def test_asynq(self):
         from asynq import asynq
         from typing_extensions import Literal
 
@@ -49,6 +49,18 @@ class TestNestedFunction(TestNameCheckVisitorBase):
             val = yield nested.asynq()
             assert_is_value(val, KnownValue(3))
 
+    @assert_passes()
+    def test_async_def(self):
+        import collections.abc
+
+        def capybara():
+            async def nested() -> int:
+                return 1
+
+            assert_is_value(
+                nested(), GenericValue(collections.abc.Awaitable, [TypedValue(int)])
+            )
+
     @assert_passes()
     def test_bad_decorator(self):
         def decorator(fn):
diff --git a/pyanalyze/test_type_evaluation.py b/pyanalyze/test_type_evaluation.py
index 3346b8f..33c0a3b 100644
--- a/pyanalyze/test_type_evaluation.py
+++ b/pyanalyze/test_type_evaluation.py
@@ -1,125 +1,88 @@
 # static analysis: ignore
-from typing_extensions import Literal
-
-from .value import AnySource, AnyValue, TypedValue, assert_is_value
+from .value import KnownValue, TypedValue, assert_is_value
 from .test_node_visitor import assert_passes
 from .test_name_check_visitor import TestNameCheckVisitorBase
-from .extensions import evaluated, is_set
-
-from typing import Union
-
-
-# These functions must be defined globally because the
-# mechanism for finding the source code doesn't work inside
-# tests.
-@evaluated
-def simple_evaluated(x: int, y: str = ""):
-    if is_set(y):
-        return int
-    else:
-        return str
-
-
-def simple_evaluated(*args: object) -> Union[int, str]:
-    if len(args) >= 2:
-        return 1
-    else:
-        return "x"
-
-
-@evaluated
-def isinstance_evaluated(x: int):
-    if isinstance(x, Literal[1]):
-        return str
-    else:
-        return int
-
-
-def isinstance_evaluated(x: int) -> Union[int, str]:
-    if x == 1:
-        return ""
-    else:
-        return 0
-
-
-@evaluated
-def not_evaluated(x: int):
-    if not isinstance(x, Literal[1]):
-        return str
-    else:
-        return int
-
-
-def not_evaluated(x: int) -> Union[int, str]:
-    if x != 1:
-        return ""
-    else:
-        return 0
-
-
-@evaluated
-def compare_evaluated(x: object):
-    if x is None:
-        return str
-    elif x == 1:
-        return float
-    else:
-        return int
-
-
-def compare_evaluated(x: object) -> Union[int, str, float]:
-    raise NotImplementedError
-
-
-@evaluated
-def nonempty_please(x: str) -> int:
-    if x == "":
-        raise Exception("Non-empty string expected")
-    else:
-        return int
-
-
-def nonempty_please(x: str) -> int:
-    assert x
-    return len(x)
+from .extensions import is_keyword, is_positional, is_provided, is_of_type, show_error
 
 
 class TestTypeEvaluation(TestNameCheckVisitorBase):
     @assert_passes()
-    def test_is_set(self):
-        from pyanalyze.test_type_evaluation import simple_evaluated
-
-        def capybara():
+    def test_is_provided(self):
+        from pyanalyze.extensions import evaluated
+        from typing import Union
+
+        @evaluated
+        def simple_evaluated(x: int, y: str = ""):
+            if is_provided(y):
+                return int
+            else:
+                return str
+
+        def simple_evaluated(*args: object) -> Union[int, str]:
+            if len(args) >= 2:
+                return 1
+            else:
+                return "x"
+
+        def capybara(args, kwargs):
             assert_is_value(simple_evaluated(1), TypedValue(str))
             assert_is_value(simple_evaluated(1, "1"), TypedValue(int))
+            assert_is_value(simple_evaluated(*args), TypedValue(str))
+            assert_is_value(simple_evaluated(**kwargs), TypedValue(str))
+            assert_is_value(simple_evaluated(1, y="1"), TypedValue(int))
+            assert_is_value(simple_evaluated(1, **{"y": "1"}), TypedValue(int))
 
     @assert_passes()
-    def test_isinstance(self):
-        from pyanalyze.test_type_evaluation import isinstance_evaluated
+    def test_is_of_type(self):
+        from pyanalyze.extensions import evaluated
+        from typing import Union
+        from typing_extensions import Literal
+
+        @evaluated
+        def is_of_type_evaluated(x: int):
+            if is_of_type(x, Literal[1]):
+                return str
+            else:
+                return int
+
+        def is_of_type_evaluated(x: int) -> Union[int, str]:
+            if x == 1:
+                return ""
+            else:
+                return 0
 
         def capybara(unannotated):
-            assert_is_value(isinstance_evaluated(1), TypedValue(str))
-            assert_is_value(isinstance_evaluated(2), TypedValue(int))
+            assert_is_value(is_of_type_evaluated(1), TypedValue(str))
+            assert_is_value(is_of_type_evaluated(2), TypedValue(int))
+            assert_is_value(is_of_type_evaluated(unannotated), TypedValue(int))
             assert_is_value(
-                isinstance_evaluated(unannotated),
-                AnyValue(AnySource.multiple_overload_matches),
-            )
-            assert_is_value(
-                isinstance_evaluated(2 if unannotated else 1),
+                is_of_type_evaluated(2 if unannotated else 1),
                 TypedValue(int) | TypedValue(str),
             )
 
     @assert_passes()
     def test_not(self):
-        from pyanalyze.test_type_evaluation import not_evaluated
+        from pyanalyze.extensions import evaluated
+        from typing import Union
+        from typing_extensions import Literal
+
+        @evaluated
+        def not_evaluated(x: int):
+            if not is_of_type(x, Literal[1]):
+                return str
+            else:
+                return int
+
+        def not_evaluated(x: int) -> Union[int, str]:
+            if x != 1:
+                return ""
+            else:
+                return 0
 
         def capybara(unannotated):
             assert_is_value(not_evaluated(1), TypedValue(int))
             assert_is_value(not_evaluated(2), TypedValue(str))
-            assert_is_value(
-                not_evaluated(unannotated),
-                AnyValue(AnySource.multiple_overload_matches),
-            )
+            assert_is_value(not_evaluated(unannotated), TypedValue(str))
             assert_is_value(
                 not_evaluated(2 if unannotated else 1),
                 TypedValue(int) | TypedValue(str),
@@ -127,7 +90,20 @@ class TestTypeEvaluation(TestNameCheckVisitorBase):
 
     @assert_passes()
     def test_compare(self):
-        from pyanalyze.test_type_evaluation import compare_evaluated
+        from pyanalyze.extensions import evaluated
+        from typing import Union
+
+        @evaluated
+        def compare_evaluated(x: object):
+            if x is None:
+                return str
+            elif x == 1:
+                return float
+            else:
+                return int
+
+        def compare_evaluated(x: object) -> Union[int, str, float]:
+            raise NotImplementedError
 
         def capybara(unannotated):
             assert_is_value(compare_evaluated(None), TypedValue(str))
@@ -140,8 +116,226 @@ class TestTypeEvaluation(TestNameCheckVisitorBase):
 
     @assert_passes()
     def test_error(self):
-        from pyanalyze.test_type_evaluation import nonempty_please
+        from pyanalyze.extensions import evaluated
+        from typing import Any
+
+        @evaluated
+        def nonempty_please(x: str) -> int:
+            if x == "":
+                show_error("Non-empty string expected", argument=x)
+                return Any
+            else:
+                return int
+
+        def nonempty_please(x: str) -> int:
+            assert x
+            return len(x)
 
         def capybara():
             nonempty_please("")  # E: incompatible_call
             assert_is_value(nonempty_please("x"), TypedValue(int))
+
+    @assert_passes()
+    def test_restrict_kind(self):
+        from pyanalyze.extensions import evaluated
+
+        @evaluated
+        def restrict_kind(x: str, y: int):
+            if is_keyword(x):
+                show_error("x must be positional", argument=x)
+            if is_positional(y):
+                show_error("y must be keyword", argument=y)
+            return int
+
+        def restrict_kind(*args, **kwargs):
+            return 0
+
+        def capybara(stuff):
+            restrict_kind("x", y=1)
+            restrict_kind(x="x", y=1)  # E: incompatible_call
+            restrict_kind("x", 1)  # E: incompatible_call
+            restrict_kind(*stuff, **stuff)
+            restrict_kind(**stuff)  # E: incompatible_call
+            restrict_kind(*stuff)  # E: incompatible_call
+
+    @assert_passes()
+    def test_pass(self):
+        from pyanalyze.extensions import evaluated
+
+        @evaluated
+        def only_one(a: int):
+            if a == 1:
+                pass
+            else:
+                show_error("a must be 1", argument=a)
+            return str
+
+        def only_one(a: int) -> str:
+            raise NotImplementedError
+
+        def capybara():
+            assert_is_value(only_one(1), TypedValue(str))
+            assert_is_value(only_one(2), TypedValue(str))  # E: incompatible_call
+
+    @assert_passes()
+    def test_enum(self):
+        import enum
+        from pyanalyze.extensions import evaluated
+
+        class Color(enum.Enum):
+            magenta = 1
+            cyan = 2
+
+        @evaluated
+        def want_enum(color: Color):
+            if color is Color.magenta:
+                return str
+            elif color is Color.cyan:
+                return int
+            else:
+                return bool
+
+        def want_enum(color: Color):
+            raise NotImplementedError
+
+        def capybara(c: Color):
+            assert_is_value(want_enum(Color.magenta), TypedValue(str))
+            assert_is_value(want_enum(Color.cyan), TypedValue(int))
+            assert_is_value(want_enum(c), TypedValue(bool))
+
+    @assert_passes()
+    def test_platform(self):
+        from pyanalyze.extensions import evaluated
+        import sys
+        from typing_extensions import Literal
+
+        @evaluated
+        def where_am_i():
+            if sys.platform == "darwin":
+                return Literal["On a Mac"]
+            else:
+                return Literal["Somewhere else"]
+
+        def where_am_i():
+            raise NotImplementedError
+
+        expected = "On a Mac" if sys.platform == "darwin" else "Somewhere else"
+
+        def capybara():
+            assert_is_value(where_am_i(), KnownValue(expected))
+
+    @assert_passes()
+    def test_version(self):
+        from pyanalyze.extensions import evaluated
+        import sys
+        from typing_extensions import Literal
+
+        @evaluated
+        def is_walrus_available():
+            if sys.version_info >= (3, 8):
+                return Literal[True]
+            return Literal[False]
+
+        def is_walrus_available():
+            return sys.version_info >= (3, 8)
+
+        expected = is_walrus_available()
+
+        def capybara():
+            assert_is_value(is_walrus_available(), KnownValue(expected))
+
+
+class TestBoolOp(TestNameCheckVisitorBase):
+    @assert_passes()
+    def test_and(self):
+        from pyanalyze.extensions import evaluated
+        from typing_extensions import Literal
+
+        @evaluated
+        def use_and(a: int, b: str):
+            if a == 1 and b == "x":
+                return str
+            return int
+
+        def use_and(a: int, b: str) -> object:
+            raise NotImplementedError
+
+        def capybara(
+            a: int, b: str, maybe_a: Literal[1, 2], maybe_b: Literal["x", "y"]
+        ) -> None:
+            assert_is_value(use_and(1, "x"), TypedValue(str))
+            assert_is_value(use_and(a, b), TypedValue(int))
+            assert_is_value(
+                use_and(maybe_a, maybe_b), TypedValue(str) | TypedValue(int)
+            )
+
+    @assert_passes()
+    def test_or(self):
+        from pyanalyze.extensions import evaluated
+        from typing_extensions import Literal
+
+        @evaluated
+        def use_or(b: str):
+            if b == "x" or b == "y":
+                return str
+            return int
+
+        def use_or(b: str) -> object:
+            raise NotImplementedError
+
+        def capybara(
+            b: str, x_or_y: Literal["x", "y"], x_or_z: Literal["x", "z"]
+        ) -> None:
+            assert_is_value(use_or("x"), TypedValue(str))
+            assert_is_value(use_or("y"), TypedValue(str))
+            assert_is_value(use_or(b), TypedValue(int))
+            assert_is_value(use_or(x_or_y), TypedValue(str))
+            assert_is_value(use_or(x_or_z), TypedValue(str) | TypedValue(int))
+
+
+class TestValidation(TestNameCheckVisitorBase):
+    @assert_passes()
+    def test_bad(self):
+        from pyanalyze.extensions import evaluated
+
+        @evaluated
+        def bad_evaluator(a: int):
+            if is_of_type(a, Literal[1]):  # E: undefined_name
+                print("hello")  # E: bad_evaluator
+            if is_of_type():  # E: bad_evaluator
+                return  # E: bad_evaluator
+            if is_of_type(b, int):  # E: bad_evaluator
+                return None
+            if is_of_type(a, int, exclude_any=None):  # E: bad_evaluator
+                return None
+            if is_of_type(a, int, exclude_any=bool(a)):  # E: bad_evaluator
+                return None
+            if is_of_type(a, int, bad_kwarg=True):  # E: bad_evaluator
+                return None
+            if not_a_function():  # E: bad_evaluator
+                return None
+            if ~is_provided(a):  # E: bad_evaluator
+                return None
+            if a == 1 == a:  # E: bad_evaluator
+                return None
+            if a > 1:  # E: bad_evaluator
+                return None
+            if a == len("x"):  # E: bad_evaluator
+                return None
+
+            if is_provided("x"):  # E: bad_evaluator
+                return None
+
+            if is_provided(b):  # E: bad_evaluator
+                show_error()  # E: bad_evaluator
+                show_error(1)  # E: bad_evaluator
+                show_error("message", argument=b)  # E: bad_evaluator
+                show_error("message", arg=a)  # E: bad_evaluator
+                show_error("message", argument=a)
+
+            if (is_provided,)[0](a):  # E: bad_evaluator
+                return None
+            return None
+
+        def bad_evaluator(a: int) -> None:
+            pass