Repository: pydantic/pydantic

Issue:
Mypy finds errors in the first example from the documentation
<details>
<summary>This example</summary>


```python
from datetime import datetime
from typing import List
from pydantic import BaseModel

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {
    'id': '123',
    'signup_ts': '2019-06-01 12:22',
    'friends': [1, 2, '3']
}
user = User(**external_data)
print(user.id)
print(repr(user.signup_ts))
print(user.friends)
print(user.dict())
```

</details>

```
$ mypy --strict pydantic_example.py 
pydantic_example.py:3: error: Module 'pydantic' has no attribute 'BaseModel'
pydantic_example.py:5: error: Class cannot subclass 'BaseModel' (has type 'Any')
pydantic_example.py:8: error: Incompatible types in assignment (expression has type "None", variable has type "datetime")
```

This is not false positive. Pydantic `__init__.py` uses implicit reexport (this is an error, and I don't want to ignore it using `--no-implicit-reexport`), and this example uses None for a non-optional field (this is also an error).

In general, I can't trust a library that suggests putting None in a non-optional field in the very first example. This violates the mypy type system.


Discussion:
I agree that the implicit reexport is a nuisance; I think it is worth addressing. @samuelcolvin any ideas?

-----

While mypy doesn’t like the following: `signup_ts: datetime = None`, there is an awkward inconsistency with mypy that this *would* be allowed as function arguments, and would imply the argument was optional. Given that it raises a mypy error, if you want to pass mypy, of course you should “correct” it. It is valid pydantic though if you choose not to care.

For what it’s worth, there may be times when you explicitly choose to override the annotations and have mypy assume the type is str, but allow it to be None for pydantic (eg, if you know a validator will always make it non-None). Obviously you would want to add `# type: ignore` but this doesn’t necessarily need to be a part of the documentation.

That said, I’m also not opposed to changing the docs here, PR welcome.

-----

> In general, I can't trust a library that suggests putting None in a non-optional field in the very first example.

Well, if you can’t trust the library, you are absolutely free to not use it.

This kind of language is not appreciated by the people who put much of their spare time into this project, and are offering it up to you for free. If you care enough to create an issue, please be courteous and try to phrase your misgivings in a less incendiary way.
Can we fix the implicit re-export in `__init__` with  `__all__`? It would be a little ugly but very explicit, we could add a test by extending the mypy tests. But very hard to implement a test that makes sure every new thing available to import from `__init__.py` gets added to `__all__`.

I'm less bothered by the Optional/None thing. Every python developer knows what this means, if they don't know what it means, adding optional won't help. Adding this would:
1. Add more fluff to the first example which tries to be as minimal as possible.
2. Perhaps confuse users as to whether `Optional` is required by pydantic. Since pydantic is very unusual in using annotations at run-time we have to be careful about this.

---

Couldn't have said it better myself. 

When I see questions like this I'm always left asking myself the same question: 

Why create the issue? - If you don't like the look of the library (I see lots of libraries that I instantly decide I don't want to use), why create the issue? If you do like it the library, surely you know starting off like that won't help get your problem fixed.

I guess the answer is either, to vent at an unjust world, or to demonstrate some new-found piece of knowledge by pointing out what you reprieve to be an error.

:sleeping: 
>Every python developer knows what this means

Yes, every python developer and every mypy user knows this is an obvious error.

```
print(user.signup_ts.year)
```

Mypy will think this code is correct, but actually it will raise `AttributeError: 'NoneType' object has no attribute 'year'` when `signup_ts` is not set. This is a false negative because of the incorrect type hint.

Using `# type: ignore` is more awful.
python is not mypy. The standard library doesn't use mypy, many widely popular python packages don't use mypy. In fact a tiny percentage of python written or executed in the world everyday has passed mypy.

Using `Optional[datetime]` doesn't prevent the potential bug in python; just allows mypy (or other static analysis libraries) to warn about it more easily. Note, it would in theory be possible for static analysis tools to find this potential bug without the `Optional[]` decoration, as mypy does for keyword arguments.

I still think the example in the documentation is okay. If you'd like to create a PR to add a note linking from the first example to how to use pydantic with mypy I'd be happy to review it.
I understand it, but it's strange that you don't respect the mypy type system. If you think that `print(user.signup_ts.year)` false negative is fine, it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design, and this means that python developers can't use mypy and pydantic together in their projects and libraries (they can use `type:ignore`, but it's equivalent to lack of mypy).
pydantic works absolutely fine with mypy indeed,
* @dmontagu put a massive amount of effort into building a mypy extension just for pydantic, see #708 
* I put a fairly large amount of effort implement full type hint coverage in pydantic in #373
* We have two sections of the docs dedicated to mypy

That still doesn't mean our first example has to be fully type hinted. Just as most examples in django, requests, or the python standard library aren't type hinted.
> it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design

It absolutely does not mean this, and it feels like you are being purposely obtuse for the sake of being contrarian.

**All** it means is that this one documentation example is not valid with mypy. You absolutely do not need to make use of any mypy-violating patterns in your use of pydantic, and we have done our best to make sure that pydantic is completely compatible with complete mypy coverage.

I personally use the same mypy config that pydantic uses in all of my projects, and while it is not `--strict`, it is also far from lenient. (And I think we should absolutely strive toward better supporting `--strict` mode.)

Yes, there are some patterns that work with pydantic that are not valid with mypy. There is also an enormous amount of non-pydantic python code that is valid python and not valid with mypy.
> > it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design
> 
> It absolutely does not mean this, and it feels like you are being purposely obtuse for the sake of being contrarian.

@dmontagu and @samuelcolvin it seems like you're both playing a new troll game :rofl: 

![img](https://i.imgflip.com/3idh9q.jpg)
@euri10 you've confused me and my entire office :confused: :confounded:. Shouldn't it be "is compatible"?
> @euri10 you've confused me and my entire office  . Shouldn't it be "is compatible"?

just for you ;) 

![changemymind](https://i.imgflip.com/3idmap.jpg) 
>there are some patterns that work with pydantic that are not valid with mypy.

This is a problem. ~~And you can't change my mind~~

>django, requests, or the python standard library aren't type hinted.

This is also a problem, but I hope most libraries will be type hinted in the future. Note that typeshed provides type hints for the standard library and some other libraries. Although the standard library itself is not checked with mypy (it's probably also a problem but I'm not sure because the standard library has a lot of unit tests), typeshed allows me to check my code that interacts with the standard library. It doesn't save from bugs in the standard library, but it saves from bugs in my code.

>There is also an enormous amount of non-pydantic python code that is valid python and not valid with mypy.

I prefer to use a python subset that is valid with mypy wherever possible. (Sometimes I use `Any` because I'm very lazy, but this is also a big problem in my discipline that I should fix someday.)
> This is a problem.

All right, well once your PEP to change mypy errors to syntax errors is accepted, we’ll get right on converting the behavior of every edge case in the use of this library.
> django, requests, or the python standard library aren't type hinted.

You've miss quoted me, I said 

> Just as most examples in django, requests, or the python standard library aren't type hinted.

Which is quite different. pydantic **is** type hinted. 

I was talking about examples in documentation where it's common (and I would argue good practice) to partially omit type hints to keep the code as succinct and easy to read as possible.


But the pydantic example is still type hinted. And these type hints are still not valid with mypy.

"missing type hints" ≠ "invalid type hints" (with mypy). ~~Change my mind~~

In fact, many libraries have a bad documentation; sometimes even an API reference does not describe types, and I have to use `print(type(something))` or even read a library source code to determine types. This is useless. I prefer type-hinted examples; they are actually easier to understand even if they less succinct. In general, explicit is better than implicit :)
Just to chime in here, I also found the first example pretty weird for the same `signup_ts: datetime = None` reason.

My background is in Typescript, which has two null-checking modes: one where `null` is implicitly assignable to every type (like most languages), and one where it is not (`--strictNullChecks`, where you have to declare things like `string | null`). The root of the problem seems to be that mypy has a strict mode that people like to use (\*raises hand\*), whereas Python's type system (and, by extension, Pydantic) doesn't take a stance on strict versus lax null checking.

Notably, the example in question does not mention `Optional` or mypy anywhere, and I understand that it is valid Python, as Python operates in the first mode (noted above). So I guess what I'm trying to say is: there is an audience for Pydantic that is coming from other type checked languages (or strict-mypy codebases), rather than from un-type-checked Python, and the first example as written is jarring and unexpected to us.

The second line `name = 'John Doe'` also reads pretty strangely to me, since struct-like field declarations are a unusual place to have type inference (at least, coming from other typed languages). 

I don't know what you want to do with that information, but hopefully it helps clear up why this example is odd to some people.
We could definitely improve the description of that example, explain better about types and mention/link to mypy. PR welcome.

I understand it's tempting to fully type the example, but I'm concerned that might imply to a casual viewer that all pydantic models have to be completely typed, e.g. for example you have to include `Optional[]` on all types with a `None` default, which currently you don't.

Similarly the `name` line is like that (even though I normally add annotations to all fields) to demonstrate type inference. 

Resolving patch:
diff --git a/pydantic/__init__.py b/pydantic/__init__.py
--- a/pydantic/__init__.py
+++ b/pydantic/__init__.py
@@ -11,3 +11,85 @@
 from .tools import *
 from .types import *
 from .version import VERSION
+
+# WARNING __all__ from .errors is not included here, it will be removed as an export here in v2
+# please use "from pydantic.errors import ..." instead
+__all__ = [
+    # dataclasses
+    'dataclasses',
+    'root_validator',
+    'validator',
+    # env_settings
+    'BaseSettings',
+    # error_wrappers
+    'ValidationError',
+    # fields
+    'Field',
+    'Required',
+    'Schema',
+    # main
+    'BaseConfig',
+    'BaseModel',
+    'Extra',
+    'compiled',
+    'create_model',
+    'validate_model',
+    # network
+    'AnyUrl',
+    'AnyHttpUrl',
+    'HttpUrl',
+    'stricturl',
+    'EmailStr',
+    'NameEmail',
+    'IPvAnyAddress',
+    'IPvAnyInterface',
+    'IPvAnyNetwork',
+    'PostgresDsn',
+    'RedisDsn',
+    'validate_email',
+    # parse
+    'Protocol',
+    # tools
+    'parse_file_as',
+    'parse_obj_as',
+    # types
+    'NoneStr',
+    'NoneBytes',
+    'StrBytes',
+    'NoneStrBytes',
+    'StrictStr',
+    'ConstrainedBytes',
+    'conbytes',
+    'ConstrainedList',
+    'conlist',
+    'ConstrainedStr',
+    'constr',
+    'PyObject',
+    'ConstrainedInt',
+    'conint',
+    'PositiveInt',
+    'NegativeInt',
+    'ConstrainedFloat',
+    'confloat',
+    'PositiveFloat',
+    'NegativeFloat',
+    'ConstrainedDecimal',
+    'condecimal',
+    'UUID1',
+    'UUID3',
+    'UUID4',
+    'UUID5',
+    'FilePath',
+    'DirectoryPath',
+    'Json',
+    'JsonWrapper',
+    'SecretStr',
+    'SecretBytes',
+    'StrictBool',
+    'StrictInt',
+    'StrictFloat',
+    'PaymentCardNumber',
+    'ByteSize',
+    # version
+    'VERSION',
+]


Test patch:
diff --git a/tests/mypy/configs/mypy-default.ini b/tests/mypy/configs/mypy-default.ini
--- a/tests/mypy/configs/mypy-default.ini
+++ b/tests/mypy/configs/mypy-default.ini
@@ -5,7 +5,7 @@ warn_redundant_casts = True
 warn_unused_ignores = True
 disallow_any_generics = True
 check_untyped_defs = True
-;no_implicit_reexport = True
+no_implicit_reexport = True
 
 # for strict mypy: (this is the tricky one :-))
 disallow_untyped_defs = True
diff --git a/tests/mypy/configs/mypy-plugin-strict.ini b/tests/mypy/configs/mypy-plugin-strict.ini
--- a/tests/mypy/configs/mypy-plugin-strict.ini
+++ b/tests/mypy/configs/mypy-plugin-strict.ini
@@ -7,9 +7,7 @@ warn_redundant_casts = True
 warn_unused_ignores = True
 disallow_any_generics = True
 check_untyped_defs = True
-;no_implicit_reexport = True
-
-# for strict mypy: (this is the tricky one :-))
+no_implicit_reexport = True
 disallow_untyped_defs = True
 
 [pydantic-mypy]
diff --git a/tests/mypy/configs/mypy-plugin.ini b/tests/mypy/configs/mypy-plugin.ini
--- a/tests/mypy/configs/mypy-plugin.ini
+++ b/tests/mypy/configs/mypy-plugin.ini
@@ -7,7 +7,7 @@ warn_redundant_casts = True
 warn_unused_ignores = True
 disallow_any_generics = True
 check_untyped_defs = True
-;no_implicit_reexport = True
+no_implicit_reexport = True
 
 # for strict mypy: (this is the tricky one :-))
 disallow_untyped_defs = True
diff --git a/tests/mypy/test_mypy.py b/tests/mypy/test_mypy.py
--- a/tests/mypy/test_mypy.py
+++ b/tests/mypy/test_mypy.py
@@ -83,3 +83,15 @@ def test_generation_is_disabled():
     Makes sure we don't accidentally leave generation on
     """
     assert not GENERATE
+
+
+def test_explicit_reexports():
+    from pydantic import __all__ as root_all
+    from pydantic.main import __all__ as main
+    from pydantic.networks import __all__ as networks
+    from pydantic.tools import __all__ as tools
+    from pydantic.types import __all__ as types
+
+    for name, export_all in [('main', main), ('network', networks), ('tools', tools), ('types', types)]:
+        for export in export_all:
+            assert export in root_all, f'{export} is in {name}.__all__ but missing from re-export in __init__.py'