Repository: pydantic/pydantic
Pull request: Add exclude_none option
## Change Summary

- Add `exclude_none` option to `dict()` and friends.

The original issue talked about `skip_none`, but since all options are now in the `exclude_x` format, that is also what I've done.

## Related issue number

Resolves #587

## Checklist

* [x] Unit tests for the changes exist
* [x] Tests pass on CI and coverage remains at 100%
* [x] Documentation reflects the changes where applicable
* [x] `changes/<pull request or issue id>-<github username>.md` file added describing change
  (see [changes/README.md](https://github.com/samuelcolvin/pydantic/blob/master/changes/README.md) for details)


Problem statement:
add skip_none argument
# Question / Feature Request 

Suppose that I have a class like this:

```
class Jake(BaseModel):
    foo: Optional[str]
```

When I serialize it, I only want to serialize optional values when they are not null.  The `skip_defaults` flag does something _similar_ to what I am looking for but has this weird case:

```
>>> j = Jake()
>>> 
>>> # This is the behavior I expect: 
... j.dict(skip_defaults=True)
{}
>>> 
>>> # Set the foo property: 
>>> j.foo = "baz"
>>> 
>>> # This is also the behaviour I expect: 
... j.dict(skip_defaults=True)
{'foo': 'baz'}
>>> 
>>> 
>>> # Unfortunately, this is not the behavior that I expect:
... j.foo = None
>>> j.dict(skip_defaults=True)
{'foo': None}
```

---

Right now after I serialize to dict, I walk the dict and remove all of the `None` values recursively, but I would rather not have to do this.  Is this built into Pydantic?  Is the behavior that I am seeing a bug?

Hints from issue discussion:
This behavior is actually **not** a bug (I found it a little confusing at first too) -- it is intended so that you can tell precisely which fields were set on the **specific object instance**, rather than taking the default value based on the class definition. (I've included an example below to show why this might be useful.)

Removing `None` from the dicts seems most commonly relevant to serialization logic; based on [this comment](https://github.com/samuelcolvin/pydantic/issues/133#issuecomment-367398040) this seems explicitly outside the scope of `pydantic`.

For what it's worth, FastAPI has a function ([`fastapi.encoders.jsonable_encoder`](https://github.com/tiangolo/fastapi/blob/master/fastapi/encoders.py#L9)) that is specifically intended for converting (nested) pydantic models to dicts, and it has an `include_none` parameter that does what you want. If you want this capability in the context of serialization, you might find this function useful, even if only as a reference implementation (it handles a number of other serialization edge cases as well).

----

Here's a concrete example of how the `skip_defaults=True` logic can be useful in the case of a REST resource with optional properties:

Imagine you want to store a user's username, and optionally an email.
```python
class User(BaseModel):
    user_id: uuid.UUID
    username: str
    email: Optional[str]
```
A clean way to implement updates is to have a corresponding model where all fields are optional. Imagine you have the following update function and model:
```python
def update_user(user_id: uuid.UUID, **update_kwargs):
    for key, value in update_kwargs.items():
        set_user_attribute(user_id, key, value)

class UserUpdate(BaseModel):
    user_id: uuid.UUID
    username: Optional[str]
    email: Optional[str]
```
Then
```python
user_update = UserUpdate(user_id=<user_id>, email="a@b.com")
update_user(**user_update.dict(skip_defaults=True))
```
makes sure that you only update the fields that were specified; in this case, only the email address. (This is a [common pattern](https://fastapi.tiangolo.com/tutorial/body-updates/) for the HTTP PATCH method.)

If the `skip_defaults=True` logic always removed `None`, then it would no longer be possible to specify that you want to change the email attribute from a non-`None` value **back** to `None`.


great answer @dmontagu 
@samuelcolvin - Is this the kind of functionality that you would consider for Pydantic?  Could we leave this open as a feature request?

(Thanks @dmontagu great explanation of that behaviour!)
I'd accept `skip_none` keyword argument to `dict()` and friends.

Commit message: run formatter

Patch to review:
diff --git a/changes/587-niknetniko.md b/changes/587-niknetniko.md
new file mode 100644
index 00000000000..326960bb4f7
--- /dev/null
+++ b/changes/587-niknetniko.md
@@ -0,0 +1 @@
+Add `exclude_none` option to `dict()` and friends
diff --git a/docs/usage/exporting_models.md b/docs/usage/exporting_models.md
index 2fb8e1b0f8e..5d9e1ab2398 100644
--- a/docs/usage/exporting_models.md
+++ b/docs/usage/exporting_models.md
@@ -15,6 +15,8 @@ Arguments:
   Prior to **v1.0**, `exclude_unset` was known as `skip_defaults`; use of `skip_defaults` is now deprecated
 * `exclude_defaults`: whether fields which are equal to their default values (whether set or otherwise) should
   be excluded from the returned dictionary; default `False`
+* `exclude_none`: whether fields which are equal to `None` should be excluded from the returned dictionary; default
+  `False`
 
 Example:
 
@@ -73,6 +75,8 @@ Arguments:
   Prior to **v1.0**, `exclude_unset` was known as `skip_defaults`; use of `skip_defaults` is now deprecated
 * `exclude_defaults`: whether fields which are equal to their default values (whether set or otherwise) should
   be excluded from the returned dictionary; default `False`
+* `exclude_none`: whether fields which are equal to `None` should be excluded from the returned dictionary; default
+  `False`
 * `encoder`: a custom encoder function passed to the `default` argument of `json.dumps()`; defaults to a custom
   encoder designed to take care of all common types
 * `**dumps_kwargs`: any other keyword arguments are passed to `json.dumps()`, e.g. `indent`.
diff --git a/pydantic/main.py b/pydantic/main.py
index b6f0928bb20..5e4c53536fb 100644
--- a/pydantic/main.py
+++ b/pydantic/main.py
@@ -308,6 +308,7 @@ def dict(
         skip_defaults: bool = None,
         exclude_unset: bool = False,
         exclude_defaults: bool = False,
+        exclude_none: bool = False,
     ) -> 'DictStrAny':
         """
         Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
@@ -332,6 +333,7 @@ def dict(
                 exclude=exclude,
                 exclude_unset=exclude_unset,
                 exclude_defaults=exclude_defaults,
+                exclude_none=exclude_none,
             )
         }
 
@@ -350,6 +352,7 @@ def json(
         skip_defaults: bool = None,
         exclude_unset: bool = False,
         exclude_defaults: bool = False,
+        exclude_none: bool = False,
         encoder: Optional[Callable[[Any], Any]] = None,
         **dumps_kwargs: Any,
     ) -> str:
@@ -371,6 +374,7 @@ def json(
             by_alias=by_alias,
             exclude_unset=exclude_unset,
             exclude_defaults=exclude_defaults,
+            exclude_none=exclude_none,
         )
         if self.__custom_root_type__:
             data = data[ROOT_KEY]
@@ -549,6 +553,7 @@ def _get_value(
         exclude: Optional[Union['AbstractSetIntStr', 'DictIntStrAny']],
         exclude_unset: bool,
         exclude_defaults: bool,
+        exclude_none: bool,
     ) -> Any:
 
         if isinstance(v, BaseModel):
@@ -559,6 +564,7 @@ def _get_value(
                     exclude_defaults=exclude_defaults,
                     include=include,
                     exclude=exclude,
+                    exclude_none=exclude_none,
                 )
             else:
                 return v.copy(include=include, exclude=exclude)
@@ -576,6 +582,7 @@ def _get_value(
                     exclude_defaults=exclude_defaults,
                     include=value_include and value_include.for_element(k_),
                     exclude=value_exclude and value_exclude.for_element(k_),
+                    exclude_none=exclude_none,
                 )
                 for k_, v_ in v.items()
                 if (not value_exclude or not value_exclude.is_excluded(k_))
@@ -592,6 +599,7 @@ def _get_value(
                     exclude_defaults=exclude_defaults,
                     include=value_include and value_include.for_element(i),
                     exclude=value_exclude and value_exclude.for_element(i),
+                    exclude_none=exclude_none,
                 )
                 for i, v_ in enumerate(v)
                 if (not value_exclude or not value_exclude.is_excluded(i))
@@ -626,6 +634,7 @@ def _iter(
         exclude: Union['AbstractSetIntStr', 'DictIntStrAny'] = None,
         exclude_unset: bool = False,
         exclude_defaults: bool = False,
+        exclude_none: bool = False,
     ) -> 'TupleGenerator':
 
         value_exclude = ValueItems(self, exclude) if exclude else None
@@ -640,7 +649,7 @@ def _iter(
 
         for k, v in self.__dict__.items():
             if allowed_keys is None or k in allowed_keys:
-                yield k, self._get_value(
+                value = self._get_value(
                     v,
                     to_dict=to_dict,
                     by_alias=by_alias,
@@ -648,7 +657,10 @@ def _iter(
                     exclude=value_exclude and value_exclude.for_element(k),
                     exclude_unset=exclude_unset,
                     exclude_defaults=exclude_defaults,
+                    exclude_none=exclude_none,
                 )
+                if (exclude_none and value is not None) or not exclude_none:
+                    yield k, value
 
     def _calculate_keys(
         self,
diff --git a/tests/test_edge_cases.py b/tests/test_edge_cases.py
index 2db55c3c239..be87d4b6693 100644
--- a/tests/test_edge_cases.py
+++ b/tests/test_edge_cases.py
@@ -1303,3 +1303,89 @@ class MyModel(BaseModel):
 
     m = MyModel(foo={'x': 'a', 'y': None})
     assert m.foo == {'x': 'a', 'y': None}
+
+
+def test_exclude_none_dict():
+    class MyModel(BaseModel):
+        a: Optional[int] = None
+        b: int = 2
+
+    m = MyModel(a=5)
+    assert m.dict(exclude_none=True) == {'a': 5, 'b': 2}
+
+    m = MyModel(b=3)
+    assert m.dict(exclude_none=True) == {'b': 3}
+
+    m = MyModel()
+    assert m.dict(exclude_none=True) == {'b': 2}
+
+
+def test_exclude_none_recursive():
+    class ModelA(BaseModel):
+        a: Optional[int] = None
+        b: int = 1
+
+    class ModelB(BaseModel):
+        c: int
+        d: int = 2
+        e: ModelA
+        f: Optional[str] = None
+
+    m = ModelB(c=5, e={'a': 0})
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+
+    m = ModelB(c=5, e={'b': 20}, f='test')
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'b': 20}, 'f': 'test'}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+
+
+def test_dict_exclude_none_populated_by_alias():
+    class MyModel(BaseModel):
+        a: str = Field('default', alias='alias_a')
+        b: Optional[str] = Field(None, alias='alias_b')
+
+        class Config:
+            allow_population_by_field_name = True
+
+    m = MyModel(alias_a='a')
+
+    assert m.dict(exclude_none=True) == {'a': 'a'}
+    assert m.dict(exclude_none=True, by_alias=True) == {'alias_a': 'a'}
+    assert m.dict(by_alias=True) == {'alias_a': 'a', 'alias_b': None}
+
+
+def test_dict_exclude_none_populated_by_alias_with_extra():
+    class MyModel(BaseModel):
+        a: str = Field('default', alias='alias_a')
+        b: Optional[str] = Field(None, alias='alias_b')
+
+        class Config:
+            extra = 'allow'
+
+    m = MyModel(alias_a='a', c='c')
+
+    assert m.dict(exclude_none=True) == {'a': 'a', 'c': 'c'}
+    assert m.dict(exclude_none=True, by_alias=True) == {'alias_a': 'a', 'c': 'c'}
+    assert m.dict(by_alias=True) == {'alias_a': 'a', 'alias_b': None, 'c': 'c'}
+
+
+def test_dict_exclude_none_populated_with_extra():
+    class MyModel(BaseModel):
+        a: str = 'default'
+        b: Optional[str] = None
+
+        class Config:
+            extra = 'allow'
+
+    m = MyModel(a='a', c='c')
+
+    assert m.dict(exclude_none=True) == {'a': 'a', 'c': 'c'}
+    assert m.dict() == {'a': 'a', 'b': None, 'c': 'c'}
+
+    m = MyModel(a='a', b='b', c=None)
+
+    assert m.dict(exclude_none=True) == {'a': 'a', 'b': 'b'}
+    assert m.dict() == {'a': 'a', 'b': 'b', 'c': None}


Code review comments:
diff_hunk: @@ -1303,3 +1303,89 @@ class MyModel(BaseModel):
 
     m = MyModel(foo={'x': 'a', 'y': None})
     assert m.foo == {'x': 'a', 'y': None}
+
+
+def test_exclude_none_dict():
+    class MyModel(BaseModel):
+        a: Optional[int] = None
+        b: int = 2
+
+    m = MyModel(a=5)
+    assert m.dict(exclude_none=True) == {'a': 5, 'b': 2}
+
+    m = MyModel(b=3)
+    assert m.dict(exclude_none=True) == {'b': 3}
+
+    m = MyModel()
+    assert m.dict(exclude_none=True) == {'b': 2}
+
+
+def test_exclude_none_recursive():
+    class ModelA(BaseModel):
+        a: Optional[int] = None
+        b: int = 1
+
+    class ModelB(BaseModel):
+        c: int
+        d: int = 2
+        e: ModelA
+        f: Optional[str] = None
+
+    m = ModelB(c=5, e={'a': 0})
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+
+    m = ModelB(c=5, e={'b': 20}, f='test')
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'b': 20}, 'f': 'test'}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+
+
+def test_dict_exclude_none_populated_by_alias():
+    class MyModel(BaseModel):
+        a: str = Field('default', alias='alias_a')
+        b: Optional[str] = Field(None, alias='alias_b')
+
+        class Config:
+            allow_population_by_field_name = True
+
+    m = MyModel(alias_a='a')
+
+    assert m.dict(exclude_none=True) == {'a': 'a'}
+    assert m.dict(exclude_none=True, by_alias=True) == {'alias_a': 'a'}
+    assert m.dict(by_alias=True) == {'alias_a': 'a', 'alias_b': None}
+
+
+def test_dict_exclude_none_populated_by_alias_with_extra():
+    class MyModel(BaseModel):
+        a: str = Field('default', alias='alias_a')
+        b: Optional[str] = Field(None, alias='alias_b')
+
+        class Config:
+            extra = 'allow'
+
+    m = MyModel(alias_a='a', c='c')
+
+    assert m.dict(exclude_none=True) == {'a': 'a', 'c': 'c'}
+    assert m.dict(exclude_none=True, by_alias=True) == {'alias_a': 'a', 'c': 'c'}
+    assert m.dict(by_alias=True) == {'alias_a': 'a', 'alias_b': None, 'c': 'c'}
+
+
+def test_dict_exclude_none_populated_with_extra():
path: tests/test_edge_cases.py
text: @user1:
I don't think both this test and the above are required.

diff_hunk: @@ -1303,3 +1303,89 @@ class MyModel(BaseModel):
 
     m = MyModel(foo={'x': 'a', 'y': None})
     assert m.foo == {'x': 'a', 'y': None}
+
+
+def test_exclude_none_dict():
+    class MyModel(BaseModel):
+        a: Optional[int] = None
+        b: int = 2
+
+    m = MyModel(a=5)
+    assert m.dict(exclude_none=True) == {'a': 5, 'b': 2}
+
+    m = MyModel(b=3)
+    assert m.dict(exclude_none=True) == {'b': 3}
+
+    m = MyModel()
+    assert m.dict(exclude_none=True) == {'b': 2}
+
+
+def test_exclude_none_recursive():
+    class ModelA(BaseModel):
+        a: Optional[int] = None
+        b: int = 1
+
+    class ModelB(BaseModel):
+        c: int
+        d: int = 2
+        e: ModelA
+        f: Optional[str] = None
+
+    m = ModelB(c=5, e={'a': 0})
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+
+    m = ModelB(c=5, e={'b': 20}, f='test')
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'b': 20}, 'f': 'test'}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+
+
+def test_dict_exclude_none_populated_by_alias():
path: tests/test_edge_cases.py
text: @user1:
this name is confusing, but also I think the test is unnecessary.

diff_hunk: @@ -640,15 +649,18 @@ def _iter(
 
         for k, v in self.__dict__.items():
             if allowed_keys is None or k in allowed_keys:
-                yield k, self._get_value(
+                value = self._get_value(
                     v,
                     to_dict=to_dict,
                     by_alias=by_alias,
                     include=value_include and value_include.for_element(k),
                     exclude=value_exclude and value_exclude.for_element(k),
                     exclude_unset=exclude_unset,
                     exclude_defaults=exclude_defaults,
+                    exclude_none=exclude_none,
                 )
+                if (exclude_none and value is not None) or not exclude_none:
path: pydantic/main.py
text: @user1:
```suggestion
                if not (exclude_none and value is None):
```
I think slightly clearer and more efficient.

diff_hunk: @@ -1303,3 +1303,89 @@ class MyModel(BaseModel):
 
     m = MyModel(foo={'x': 'a', 'y': None})
     assert m.foo == {'x': 'a', 'y': None}
+
+
+def test_exclude_none_dict():
+    class MyModel(BaseModel):
+        a: Optional[int] = None
+        b: int = 2
+
+    m = MyModel(a=5)
+    assert m.dict(exclude_none=True) == {'a': 5, 'b': 2}
+
+    m = MyModel(b=3)
+    assert m.dict(exclude_none=True) == {'b': 3}
+
+    m = MyModel()
path: tests/test_edge_cases.py
text: @user1:
This case doesn't test anything, can be removed.