Repository: pydantic/pydantic

Issue:
Make ValidationError unpicklable
# Bug

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
             pydantic version: 1.5.1
            pydantic compiled: False
                 install path: /home/badger/.cache/pypoetry/virtualenvs/antsibull-F3umzYPl-py3.8/lib/python3.8/site-packages/pydantic
               python version: 3.8.3 (default, Feb 26 2020, 00:00:00)  [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
                     platform: Linux-5.6.15-200.fc31.x86_64-x86_64-with-glibc2.2.5
     optional deps. installed: ['typing-extensions']
...
```
<!-- or if you're using pydantic prior to v1.3, manually include: OS, python version and pydantic version -->

<!-- Please read the [docs](https://pydantic-docs.helpmanual.io/) and search through issues to
confirm your bug hasn't already been reported. -->

<!-- Where possible please include a self-contained code snippet describing your bug: -->

## Use case
I'm going to give you two code snippets because it might not be obvious from the simplest case why I would want to do it.

A simple approximation of my use case is here: https://gist.github.com/abadger/bfd55741c281ccb534f7bbc8fe9b6202

I am trying to use pydantic to validate and normalize data from a large number of data sources I need to run each validation separately so that I can know which data sources are providing invalid data. I decided to split it up amongst multiple CPUs by using asyncio's run_in_executor with a ProcessPoolExecutor.  However, when the pydantic.constr validation failed, I would get a BrokenProcessPool error on everything that had been queued but not run rather than a pydantic ValidationError on the specific task which failed.

## Root cause

I was able to workaround the problem by catching the pydantic exception and raising a ValueError with all of the information I needed.  This lead me to the root cause: pydantic errors are not **un**picklable.  Because of that, the exception raised in the worker process is pickled there and sent back to the parent process.  The parent process attempts to unpickle it, encounters the error, and then gives the generic, unhelpful BrokenProcessPool error and cancels the other pending tasks.

Here's a reproducer for the root cause:
```python
import pickle
from pydantic.errors import StrRegexError

p = pickle.dumps(StrRegexError(pattern='test'))
print(pickle.loads(p))

# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: __init__() missing 1 required positional argument: 'pattern'
```

Looking at the python stdlib bugtracker there are many open bugs with interactions between pickle and exceptions.  I didn't see this one so I added this: https://bugs.python.org/issue40917  Some others that might cause different bugs with pydantics exceptions:

* https://bugs.python.org/issue32696
* https://bugs.python.org/issue30005

Given so many potential bugs, I'm not sure if this is solvable in pydantic code or has to wait for pickle fixes.  However, if it's not solvable, adding my workaround and an explanation of what's happening to the docs would be nice.  That way searching for pydantic, ProcessPoolExecutor, pickle, multiprocessing might save the next person some time wondering why only a portion of their data was being converted.


Discussion:
Surely as simple as implementing `__getstate__` and `__setstate__` on `ValidatoinError`?

If it is that simple, PR welcome to implement it.
I don't believe so.  I did spend a few hours trying to do just that and couldn't get it to work.  Iirc, the problem with that approach is that `__setstate__` isn't called until after the object's `__init__` is called.  So the error has already occurred and you don't get a chance to fix things.

Reading the pickle docs, it seemed like `__getnewargs_ex__` might be a way to get that to work but i failed at getting that to work too.  I'm not sure, though, if i just didn't/don't understand getnewargs or if it turned out that exceptions are specialcased already and that specialcasing was interfering (in one of the python bug reports, ncoghlan noted that there were specialcases in exceptions which perhaps no longer worked with the new ways that picked worked)


My python.org bug from earlier was a duplicate.  I've closed it.  This is the older bug: https://bugs.python.org/issue27015  There's a cpython PR which I confirmed would fix at least the mandatory keyword args.  It is currently awaiting another review from a python core developer: https://github.com/python/cpython/pull/11580
Hi @abadger I just had a look at it and I think I have something working.
I'm opening a PR! Please tell me if it works with your whole example

Resolving patch:
diff --git a/pydantic/errors.py b/pydantic/errors.py
--- a/pydantic/errors.py
+++ b/pydantic/errors.py
@@ -1,9 +1,12 @@
 from decimal import Decimal
 from pathlib import Path
-from typing import Any, Set, Type, Union
+from typing import TYPE_CHECKING, Any, Callable, Set, Tuple, Type, Union
 
 from .typing import display_as_type
 
+if TYPE_CHECKING:
+    from .typing import DictStrAny
+
 # explicitly state exports to avoid "from .errors import *" also importing Decimal, Path etc.
 __all__ = (
     'PydanticTypeError',
@@ -91,6 +94,17 @@
 )
 
 
+def cls_kwargs(cls: Type['PydanticErrorMixin'], ctx: 'DictStrAny') -> 'PydanticErrorMixin':
+    """
+    For built-in exceptions like ValueError or TypeError, we need to implement
+    __reduce__ to override the default behaviour (instead of __getstate__/__setstate__)
+    By default pickle protocol 2 calls `cls.__new__(cls, *args)`.
+    Since we only use kwargs, we need a little constructor to change that.
+    Note: the callable can't be a lambda as pickle looks in the namespace to find it
+    """
+    return cls(**ctx)
+
+
 class PydanticErrorMixin:
     code: str
     msg_template: str
@@ -101,6 +115,9 @@ def __init__(self, **ctx: Any) -> None:
     def __str__(self) -> str:
         return self.msg_template.format(**self.__dict__)
 
+    def __reduce__(self) -> Tuple[Callable[..., 'PydanticErrorMixin'], Tuple[Type['PydanticErrorMixin'], 'DictStrAny']]:
+        return cls_kwargs, (self.__class__, self.__dict__)
+
 
 class PydanticTypeError(PydanticErrorMixin, TypeError):
     pass


Test patch:
diff --git a/tests/test_errors.py b/tests/test_errors.py
--- a/tests/test_errors.py
+++ b/tests/test_errors.py
@@ -1,3 +1,4 @@
+import pickle
 import sys
 from typing import Dict, List, Optional, Union
 from uuid import UUID, uuid4
@@ -6,6 +7,7 @@
 
 from pydantic import UUID1, BaseConfig, BaseModel, PydanticTypeError, ValidationError, conint, errors, validator
 from pydantic.error_wrappers import flatten_errors, get_exc_type
+from pydantic.errors import StrRegexError
 from pydantic.typing import Literal
 
 
@@ -22,6 +24,17 @@ def __init__(self, *, test_ctx: int) -> None:
     assert str(exc_info.value) == 'test message template "test_value"'
 
 
+def test_pydantic_error_pickable():
+    """
+    Pydantic errors should be (un)pickable.
+    (this test does not create a custom local error as we can't pickle local objects)
+    """
+    p = pickle.dumps(StrRegexError(pattern='pika'))
+    error = pickle.loads(p)
+    assert isinstance(error, StrRegexError)
+    assert error.pattern == 'pika'
+
+
 @pytest.mark.skipif(not Literal, reason='typing_extensions not installed')
 def test_interval_validation_error():
     class Foo(BaseModel):