Repository: samuelcolvin/pydantic
Pull request #1920 (2020-09-13 11:39:04)

add smart_deepcopy (originaly from #1679)
<!-- Thank you for your contribution! -->
<!-- Unless your change is trivial, please create an issue to discuss the change before creating a PR -->
<!-- See https://pydantic-docs.helpmanual.io/contributing/ for help on Contributing -->

## Change Summary
add `smart_deepcopy()`:
```py
smart_deepcopy(obj: ~Obj) -> ~Obj
    Return type as is for immutable built-in types
    Use obj.copy() for built-in empty collections
    Use copy.deepcopy() only on non-empty collections and unknown objects
```

It's primarily needed for faster copying of default values, since they're often are immutable types or empty collections

Here's benchmark showing the difference in speed between different types of values:
```py
from copy import deepcopy
from timeit import timeit

from pydantic.utils import smart_deepcopy, BUILTIN_COLLECTIONS

cases = {
    "IMMUTABLE":
        (1, 1.0, '1', b'1', int, None, smart_deepcopy, len, smart_deepcopy.__code__, lambda: _, ...),
    "EMPTY_COLLECTIONS":
        map(lambda collection: collection(), BUILTIN_COLLECTIONS),
    "NON_EMPTY_COLLECTIONS":
        map(lambda c: c.fromkeys([1]) if issubclass(c, dict) else c([1]), BUILTIN_COLLECTIONS),
}

for name, values in cases.items():
    deepcopy_results = []
    smart_deepcopy_results = []
    for value in values:
        deepcopy_results.append(timeit(lambda: deepcopy(value), number=100000))
        smart_deepcopy_results.append(timeit(lambda: smart_deepcopy(value), number=100000))

    deepcopy_result = sum(deepcopy_results)
    smart_deepcopy_result = sum(smart_deepcopy_results)
    faster_by = deepcopy_result / smart_deepcopy_result
    print(f"{name}: {deepcopy_result=:.3}s, {smart_deepcopy_result=:.3}s, {faster_by=:.3} times")

```
```py
IMMUTABLE: deepcopy_result=0.658s, smart_deepcopy_result=0.235s, faster_by=2.8 times
EMPTY_COLLECTIONS: deepcopy_result=1.73s, smart_deepcopy_result=0.261s, faster_by=6.62 times
NON_EMPTY_COLLECTIONS: deepcopy_result=2.33s, smart_deepcopy_result=2.5s, faster_by=0.933 times
```


## Related issue number

<!-- Are there any issues opened that will be resolved by merging this change? -->

## Checklist

* [x] Unit tests for the changes exist
* [x] Tests pass on CI and coverage remains at 100%
* [x] Documentation reflects the changes where applicable
* [ ] `changes/<pull request or issue id>-<github username>.md` file added describing change
  (see [changes/README.md](https://github.com/samuelcolvin/pydantic/blob/master/changes/README.md) for details)


codecov[bot]: # [Codecov](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920?src=pr&el=h1) Report
> Merging [#1920](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920?src=pr&el=desc) into [master](https://codecov.io/gh/samuelcolvin/pydantic/commit/bf9cc4a5e7903ada2e819a63fe2da011f292a143?el=desc) will **decrease** coverage by `0.76%`.
> The diff coverage is `100.00%`.


```diff
@@             Coverage Diff             @@
##            master    #1920      +/-   ##
===========================================
- Coverage   100.00%   99.23%   -0.77%     
===========================================
  Files           21       21              
  Lines         3909     3915       +6     
  Branches       788      788              
===========================================
- Hits          3909     3885      -24     
- Misses           0       23      +23     
- Partials         0        7       +7     
```


| [Impacted Files](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pydantic/fields.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvZmllbGRzLnB5) | `100.00% <100.00%> (ø)` | |
| [pydantic/utils.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvdXRpbHMucHk=) | `99.25% <100.00%> (-0.75%)` | :arrow_down: |
| [pydantic/typing.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvdHlwaW5nLnB5) | `79.24% <0.00%> (-20.76%)` | :arrow_down: |
| [pydantic/version.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvdmVyc2lvbi5weQ==) | `88.23% <0.00%> (-11.77%)` | :arrow_down: |
| [pydantic/env\_settings.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvZW52X3NldHRpbmdzLnB5) | `97.43% <0.00%> (-2.57%)` | :arrow_down: |
| [pydantic/networks.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvbmV0d29ya3MucHk=) | `99.22% <0.00%> (-0.78%)` | :arrow_down: |
| [pydantic/decorator.py](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920/diff?src=pr&el=tree#diff-cHlkYW50aWMvZGVjb3JhdG9yLnB5) | `100.00% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920?src=pr&el=footer). Last update [bf9cc4a...85f68ae](https://codecov.io/gh/samuelcolvin/pydantic/pull/1920?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/pydantic/fields.py b/pydantic/fields.py
index 01792b628..6ab039cd5 100644
--- a/pydantic/fields.py
+++ b/pydantic/fields.py
@@ -1,6 +1,5 @@
 import warnings
 from collections.abc import Iterable as CollectionsIterable
-from copy import deepcopy
 from typing import (
     TYPE_CHECKING,
     Any,
@@ -36,7 +35,7 @@ from .typing import (
     is_new_type,
     new_type_supertype,
 )
-from .utils import PyObjectStr, Representation, lenient_issubclass, sequence_like
+from .utils import PyObjectStr, Representation, lenient_issubclass, sequence_like, smart_deepcopy
 from .validators import constant_validator, dict_validator, find_validators, validate_json
 
 Required: Any = Ellipsis
@@ -271,14 +270,7 @@ class ModelField(Representation):
         self.prepare()
 
     def get_default(self) -> Any:
-        if self.default_factory is not None:
-            value = self.default_factory()
-        elif self.default is None:
-            # deepcopy is quite slow on None
-            value = None
-        else:
-            value = deepcopy(self.default)
-        return value
+        return smart_deepcopy(self.default) if self.default_factory is None else self.default_factory()
 
     @classmethod
     def infer(
diff --git a/pydantic/main.py b/pydantic/main.py
index c872f1e3b..c6c486ada 100644
--- a/pydantic/main.py
+++ b/pydantic/main.py
@@ -42,6 +42,7 @@ from .utils import (
     generate_model_signature,
     lenient_issubclass,
     sequence_like,
+    smart_deepcopy,
     unique_list,
     validate_field_name,
 )
@@ -219,7 +220,7 @@ class ModelMetaclass(ABCMeta):
         pre_root_validators, post_root_validators = [], []
         for base in reversed(bases):
             if _is_base_model_class_defined and issubclass(base, BaseModel) and base != BaseModel:
-                fields.update(deepcopy(base.__fields__))
+                fields.update(smart_deepcopy(base.__fields__))
                 config = inherit_config(base.__config__, config)
                 validators = inherit_validators(base.__validators__, validators)
                 pre_root_validators += base.__pre_root_validators__
@@ -527,7 +528,7 @@ class BaseModel(Representation, metaclass=ModelMetaclass):
         Default values are respected, but no other validation is performed.
         """
         m = cls.__new__(cls)
-        object.__setattr__(m, '__dict__', {**deepcopy(cls.__field_defaults__), **values})
+        object.__setattr__(m, '__dict__', {**smart_deepcopy(cls.__field_defaults__), **values})
         if _fields_set is None:
             _fields_set = set(values.keys())
         object.__setattr__(m, '__fields_set__', _fields_set)
@@ -558,6 +559,7 @@ class BaseModel(Representation, metaclass=ModelMetaclass):
         )
 
         if deep:
+            # chances of having empty dict here are quite low for using smart_deepcopy
             v = deepcopy(v)
 
         cls = self.__class__
diff --git a/pydantic/utils.py b/pydantic/utils.py
index d14a2858a..42a4d30e6 100644
--- a/pydantic/utils.py
+++ b/pydantic/utils.py
@@ -1,6 +1,9 @@
 import warnings
+import weakref
+from collections import OrderedDict, defaultdict, deque
+from copy import deepcopy
 from itertools import islice
-from types import GeneratorType
+from types import BuiltinFunctionType, CodeType, FunctionType, GeneratorType, LambdaType, ModuleType
 from typing import (
     TYPE_CHECKING,
     AbstractSet,
@@ -20,7 +23,7 @@ from typing import (
     no_type_check,
 )
 
-from .typing import display_as_type
+from .typing import NoneType, display_as_type
 from .version import version_info
 
 if TYPE_CHECKING:
@@ -50,6 +53,41 @@ __all__ = (
     'ClassAttribute',
 )
 
+# these are types that are returned unchanged by deepcopy
+IMMUTABLE_NON_COLLECTIONS_TYPES: Set[Type[Any]] = {
+    int,
+    float,
+    complex,
+    str,
+    bool,
+    bytes,
+    type,
+    NoneType,
+    FunctionType,
+    BuiltinFunctionType,
+    LambdaType,
+    weakref.ref,
+    CodeType,
+    # note: including ModuleType will differ from behaviour of deepcopy by not producing error.
+    # It might be not a good idea in general, but considering that this function used only internally
+    # against default values of fields, this will allow to actually have a field with module as default value
+    ModuleType,
+    NotImplemented.__class__,
+    Ellipsis.__class__,
+}
+
+# these are types that if empty, might be copied with simple copy() instead of deepcopy()
+BUILTIN_COLLECTIONS: Set[Type[Any]] = {
+    list,
+    set,
+    tuple,
+    frozenset,
+    dict,
+    OrderedDict,
+    defaultdict,
+    deque,
+}
+
 
 def import_string(dotted_path: str) -> Any:
     """
@@ -534,3 +572,22 @@ class ClassAttribute:
         if instance is None:
             return self.value
         raise AttributeError(f'{self.name!r} attribute of {owner.__name__!r} is class-only')
+
+
+Obj = TypeVar('Obj')
+
+
+def smart_deepcopy(obj: Obj) -> Obj:
+    """
+    Return type as is for immutable built-in types
+    Use obj.copy() for built-in empty collections
+    Use copy.deepcopy() for non-empty collections and unknown objects
+    """
+
+    obj_type = obj.__class__
+    if obj_type in IMMUTABLE_NON_COLLECTIONS_TYPES:
+        return obj  # fastest case: obj is immutable and not collection therefore will not be copied anyway
+    elif not obj and obj_type in BUILTIN_COLLECTIONS:
+        # faster way for empty collections, no need to copy its members
+        return obj if obj_type is tuple else obj.copy()  # type: ignore  # tuple doesn't have copy method
+    return deepcopy(obj)  # slowest way when we actually might need a deepcopy


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index d59f2e3ed..935038f04 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,6 +1,7 @@
 import os
 import re
 import string
+from copy import deepcopy
 from distutils.version import StrictVersion
 from enum import Enum
 from typing import NewType, Union
@@ -13,12 +14,14 @@ from pydantic.dataclasses import dataclass
 from pydantic.fields import Undefined
 from pydantic.typing import Literal, all_literal_values, display_as_type, is_new_type, new_type_supertype
 from pydantic.utils import (
+    BUILTIN_COLLECTIONS,
     ClassAttribute,
     ValueItems,
     deep_update,
     get_model,
     import_string,
     lenient_issubclass,
+    smart_deepcopy,
     truncate,
     unique_list,
 )
@@ -347,3 +350,30 @@ def test_all_literal_values():
 
     L312 = Literal['3', Literal[L1, L2]]
     assert sorted(all_literal_values(L312)) == sorted(('1', '2', '3'))
+
+
+@pytest.mark.parametrize(
+    'obj',
+    (1, 1.0, '1', b'1', int, None, test_all_literal_values, len, test_all_literal_values.__code__, lambda: ..., ...),
+)
+def test_smart_deepcopy_immutable_non_sequence(obj, mocker):
+    # make sure deepcopy is not used
+    # (other option will be to use obj.copy(), but this will produce error as none of given objects have this method)
+    mocker.patch('pydantic.utils.deepcopy', side_effect=RuntimeError)
+    assert smart_deepcopy(obj) is deepcopy(obj) is obj
+
+
+@pytest.mark.parametrize('empty_collection', (collection() for collection in BUILTIN_COLLECTIONS))
+def test_smart_deepcopy_empty_collection(empty_collection, mocker):
+    mocker.patch('pydantic.utils.deepcopy', side_effect=RuntimeError)  # make sure deepcopy is not used
+    if not isinstance(empty_collection, (tuple, frozenset)):  # empty tuple or frozenset are always the same object
+        assert smart_deepcopy(empty_collection) is not empty_collection
+
+
+@pytest.mark.parametrize(
+    'collection', (c.fromkeys((1,)) if issubclass(c, dict) else c((1,)) for c in BUILTIN_COLLECTIONS)
+)
+def test_smart_deepcopy_collection(collection, mocker):
+    expected_value = object()
+    mocker.patch('pydantic.utils.deepcopy', return_value=expected_value)
+    assert smart_deepcopy(collection) is expected_value