Repository: python/mypy
Pull request: ✨ Added new Error to TypedDict
Fixes #4617

This allows the following code to trigger the error `typeddict-unknown-key`

```python
A = T.TypedDict("A", {"x": int})

def f(x: A) -> None:
    ...

f({"x": 1, "y": "foo"})  # err: typeddict-unknown-key
f({"y": "foo"})  # err: typeddict-unknown-key & typeddict-item
f({"x": 'err', "y": "foo"})  # err: typeddict-unknown-key & typeddict-item

a: A = { 'x': 1 }

# You can set extra attributes
a['extra'] = 'extra' # err: typeddict-unknown-key
# Reading them produces the normal item error
err = a['does not exist'] # err: typeddict-item
```

The user can then safely ignore this specific error at their disgression.

Problem statement:
How to accept dicts with extra keys?
A lot of times I pass data around using dictionaries. When I receive the data, for example as a function argument, what I want is a dict with that **contains** some specifics keys. I don't mind if the dict has more data.

So, I tried typed dicts:

```python
Movie = TypedDict('Movie', {'name': str, 'year': int})
```

This works:

```python
movie = {'name': 'Blade Runner', 'year': 1982}  # type: Movie
```

But this throws the error `Extra key 'has_sequel' for TypedDict "Movie"`:

```python
movie = {'name': 'Blade Runner', 'year': 1982, 'has_sequel': True}  # type: Movie
```

I can understand that you can't replace for the first value, because the result of `keys` or `items` is different.

But if I am only interested in having those keys, not iterating or other stuff, what are my options (if any)?

Hints from issue discussion:
There is a cool new feature called `total=False`, does it work for you?
Here are some docs http://mypy.readthedocs.io/en/latest/kinds_of_types.html#totality
This looks like a feature request for TypedDict.
@gvanrossum Actually I am quite sure `total=False` covers this, or am I missing something?
Hm, I see one problem: potentially, the list of optional keys can be very long (since one still needs to list them).
Sadly no -- `total=False` allows omitting keys, it doesn't allow extra keys.
```py
# ...imports...
A = TypedDict('A', {'x': int}, total=False)
def f(a: A) -> None:
  print(a['x'])
b: A = {'x': 0, 'y': 0}  # E: Extra key 'y' for TypedDict "A"
```
Yes, I think this is a valid feature request, as enumerating keys in the `TypedDict` could be painful.
This is already supported for function arguments:

```py
from mypy_extensions import TypedDict

A = TypedDict('A', {'x': int})
B = TypedDict('B', {'x': int, 'y': str})

def f(x: A) -> None: ...

b: B = {'x': 1, 'y': 'foo'}
f(b)  # Ok
```

The issues seems specific to creating a typed dict -- mypy makes sure that no extra keys are provided (as these could be accidental typos or obsolete key names, for example).

I don't think that changing the semantics of `total=False` is a good idea, since we'd lose a lot of type safety. One option would be to introduce a new flag. Let's call it `allow_extra=True` for now. Here is an attempt to define it:

* Don't complain about extra keys when creating a typed dict object if `allow_extra` is true for the type. Allow arbitrary extra keys and values.
* Allow accessing arbitrary keys, such as `x['whatever']`. The value type would have to be `Any` unless the key is explicitly defined for the typed dict.
* Allow setting arbitrary keys with arbitrary values, such as `x['whatever'] = <anything>`.

These semantics seem kind of arbitrary to me. To seriously consider such as feature I'd like to see more concrete examples where the current behavior is not sufficient. If the structure of a typed dict object is not well specified, the current recommendation is to use `Dict[str, Any]` instead. `TypedDict` is pretty restricted by design.

If we had intersection types, a similar effect to my proposed `allow_extra` flag could perhaps be achieved through `Intersection[MyTypedDict, Dict[str, Any]]`. We don't have any concrete plans to introduce intersection types, however.
Thanks.

I think that I can work with the current behavior.

JukkaL's example is what I am trying to do. Something like:

```python
from typing import Iterable
from mypy_extensions import TypedDict

NamedThing = TypedDict('NamedThing', {'name': str})
Movie = TypedDict('Movie', {'name': str, 'year': int})
Replicant = TypedDict('Replicant', {'name': str, 'model': str})

def slug(x: NamedThing) -> str:
    return x['name'].lower().replace(' ', '-')

blade_runner: Movie = {'name': 'Blade Runner', 'year': 1982}
roy: Replicant = {'name': 'Roy', 'model': 'Nexus-6'}
things: Iterable[NamedThing] = [blade_runner, roy]

for thing in things:
    print(slug(thing))
```

When trying Mypy I was directly assigning the values or using variables without annotations, like:

```python
blade_runner: NamedThing = {'name': 'Blade Runner', 'year': 1982}

slug({'name': 'Blade Runner', 'year': 1982})

blade_runner = {'name': 'Blade Runner', 'year': 1982}
slug(blade_runner)
```


I'm closing this since it seems that the current behavior is mostly good enough, and the solution to the larger issue would be messy.
I would like this issue to be re-opened. I think `extra_keys` is important to have.

The current solution of `Dict[str, Union[str, int, float, bool]]` is significantly less expressive than what other languages offer.

Typescript example:
```ts
// Simple example
interface ElasticsearchDocument {
    _id: string
    _type: string
    _index: string
    @timestamp: number
    [key: string]: string|number|boolean;
}

// Complex example: Nested dict
interface SubObject {
  [key: string]: string|number|boolean|SubObject
}
interface DynamoDBDocument {
  index_id: number
  secondary_index_id: number
  [key: string]: string|number|boolean|SubObject
}
```

The above suggestion of `allow_extra` still has a capability gap where I cannot define the type of extra items (either the key or value).

I don't have a good sense of the restrictions on syntax that MyPy has to deal with. So this proposal might be totally unreasonable. But the syntax could be:

```py
TypedDict("ElasticsearchDocument",
          {"_id": str},
          extra_keys={str: Union[str,int,float,bool]})
```

This says: "this is a dictionary with `_id` as a guaranteed key. There might be other keys, they have string type and value matches `Union[str,int,float,bool]`.

Theoretically you could spec something like this:

```py
TypedDict("ComplexDict",
          {1: str, "a": int},
          extra_keys={int: bool, str: str})
```

Commit message: 🧑‍🔬 Fixed simple error-code tests

We now return two instead of one

Patch to review:
diff --git a/mypy/errorcodes.py b/mypy/errorcodes.py
index 1c15407a955b..234b9bce40f7 100644
--- a/mypy/errorcodes.py
+++ b/mypy/errorcodes.py
@@ -67,6 +67,9 @@ def __str__(self) -> str:
 TYPEDDICT_ITEM: Final = ErrorCode(
     "typeddict-item", "Check items when constructing TypedDict", "General"
 )
+TYPPEDICT_UNKNOWN_KEY: Final = ErrorCode(
+    "typeddict-unknown-key", "Check unknown keys when constructing TypedDict", "General"
+)
 HAS_TYPE: Final = ErrorCode(
     "has-type", "Check that type of reference can be determined", "General"
 )
diff --git a/mypy/messages.py b/mypy/messages.py
index 85fa30512534..731884bb836e 100644
--- a/mypy/messages.py
+++ b/mypy/messages.py
@@ -1612,9 +1612,9 @@ def unexpected_typeddict_keys(
         expected_set = set(expected_keys)
         if not typ.is_anonymous():
             # Generate simpler messages for some common special cases.
-            if actual_set < expected_set:
-                # Use list comprehension instead of set operations to preserve order.
-                missing = [key for key in expected_keys if key not in actual_set]
+            # Use list comprehension instead of set operations to preserve order.
+            missing = [key for key in expected_keys if key not in actual_set]
+            if missing:
                 self.fail(
                     "Missing {} for TypedDict {}".format(
                         format_key_list(missing, short=True), format_type(typ)
@@ -1622,20 +1622,18 @@ def unexpected_typeddict_keys(
                     context,
                     code=codes.TYPEDDICT_ITEM,
                 )
+            extra = [key for key in actual_keys if key not in expected_set]
+            if extra:
+                self.fail(
+                    "Extra {} for TypedDict {}".format(
+                        format_key_list(extra, short=True), format_type(typ)
+                    ),
+                    context,
+                    code=codes.TYPPEDICT_UNKNOWN_KEY,
+                )
+            if missing or extra:
+                # No need to check for further errors
                 return
-            else:
-                extra = [key for key in actual_keys if key not in expected_set]
-                if extra:
-                    # If there are both extra and missing keys, only report extra ones for
-                    # simplicity.
-                    self.fail(
-                        "Extra {} for TypedDict {}".format(
-                            format_key_list(extra, short=True), format_type(typ)
-                        ),
-                        context,
-                        code=codes.TYPEDDICT_ITEM,
-                    )
-                    return
         found = format_key_list(actual_keys, short=True)
         if not expected_keys:
             self.fail(f"Unexpected TypedDict {found}", context)
diff --git a/test-data/unit/check-errorcodes.test b/test-data/unit/check-errorcodes.test
index 798c52629a35..80672fc10c7f 100644
--- a/test-data/unit/check-errorcodes.test
+++ b/test-data/unit/check-errorcodes.test
@@ -455,7 +455,7 @@ class E(TypedDict):
     y: int
 
 a: D = {'x': ''}  # E: Incompatible types (expression has type "str", TypedDict item "x" has type "int")  [typeddict-item]
-b: D = {'y': ''}  # E: Extra key "y" for TypedDict "D"  [typeddict-item]
+b: D = {'y': ''}  # E: Missing key "x" for TypedDict "D"  [typeddict-item] # E: Extra key "y" for TypedDict "D"  [typeddict-unknown-key]
 c = D(x=0) if int() else E(x=0, y=0)
 c = {}  # E: Expected TypedDict key "x" but found no keys  [typeddict-item]
 
diff --git a/test-data/unit/check-typeddict.test b/test-data/unit/check-typeddict.test
index fbef6157087c..dff46e111573 100644
--- a/test-data/unit/check-typeddict.test
+++ b/test-data/unit/check-typeddict.test
@@ -2020,7 +2020,7 @@ v = {union: 2} # E: Expected TypedDict key to be string literal
 num2: Literal['num']
 v = {num2: 2}
 bad2: Literal['bad']
-v = {bad2: 2}  # E: Extra key "bad" for TypedDict "Value"
+v = {bad2: 2}  # E: Missing key "num" for TypedDict "Value" # E: Extra key "bad" for TypedDict "Value"
 
 [builtins fixtures/dict.pyi]
 [typing fixtures/typing-typeddict.pyi]


Code review comments:
diff_hunk: @@ -2020,7 +2020,7 @@ v = {union: 2} # E: Expected TypedDict key to be string literal
 num2: Literal['num']
 v = {num2: 2}
 bad2: Literal['bad']
-v = {bad2: 2}  # E: Extra key "bad" for TypedDict "Value"
+v = {bad2: 2}  # E: Missing key "num" for TypedDict "Value" # E: Extra key "bad" for TypedDict "Value"
path: test-data/unit/check-typeddict.test
text: @user1:
```suggestion
v = {bad2: 2}  # E: Missing key "num" for TypedDict "Value" \
               # E: Extra key "bad" for TypedDict "Value"
```

diff_hunk: @@ -455,7 +455,7 @@ class E(TypedDict):
     y: int
 
 a: D = {'x': ''}  # E: Incompatible types (expression has type "str", TypedDict item "x" has type "int")  [typeddict-item]
-b: D = {'y': ''}  # E: Extra key "y" for TypedDict "D"  [typeddict-item]
+b: D = {'y': ''}  # E: Missing key "x" for TypedDict "D"  [typeddict-item] # E: Extra key "y" for TypedDict "D"  [typeddict-unknown-key]
path: test-data/unit/check-errorcodes.test
text: @user1:
```suggestion
b: D = {'y': ''}  # E: Missing key "x" for TypedDict "D"  [typeddict-item] \
                  # E: Extra key "y" for TypedDict "D"  [typeddict-unknown-key]
```