Repository: pydantic/pydantic

Issue:
add skip_none argument
# Question / Feature Request 

Suppose that I have a class like this:

```
class Jake(BaseModel):
    foo: Optional[str]
```

When I serialize it, I only want to serialize optional values when they are not null.  The `skip_defaults` flag does something _similar_ to what I am looking for but has this weird case:

```
>>> j = Jake()
>>> 
>>> # This is the behavior I expect: 
... j.dict(skip_defaults=True)
{}
>>> 
>>> # Set the foo property: 
>>> j.foo = "baz"
>>> 
>>> # This is also the behaviour I expect: 
... j.dict(skip_defaults=True)
{'foo': 'baz'}
>>> 
>>> 
>>> # Unfortunately, this is not the behavior that I expect:
... j.foo = None
>>> j.dict(skip_defaults=True)
{'foo': None}
```

---

Right now after I serialize to dict, I walk the dict and remove all of the `None` values recursively, but I would rather not have to do this.  Is this built into Pydantic?  Is the behavior that I am seeing a bug?


Discussion:
This behavior is actually **not** a bug (I found it a little confusing at first too) -- it is intended so that you can tell precisely which fields were set on the **specific object instance**, rather than taking the default value based on the class definition. (I've included an example below to show why this might be useful.)

Removing `None` from the dicts seems most commonly relevant to serialization logic; based on [this comment](https://github.com/samuelcolvin/pydantic/issues/133#issuecomment-367398040) this seems explicitly outside the scope of `pydantic`.

For what it's worth, FastAPI has a function ([`fastapi.encoders.jsonable_encoder`](https://github.com/tiangolo/fastapi/blob/master/fastapi/encoders.py#L9)) that is specifically intended for converting (nested) pydantic models to dicts, and it has an `include_none` parameter that does what you want. If you want this capability in the context of serialization, you might find this function useful, even if only as a reference implementation (it handles a number of other serialization edge cases as well).

----

Here's a concrete example of how the `skip_defaults=True` logic can be useful in the case of a REST resource with optional properties:

Imagine you want to store a user's username, and optionally an email.
```python
class User(BaseModel):
    user_id: uuid.UUID
    username: str
    email: Optional[str]
```
A clean way to implement updates is to have a corresponding model where all fields are optional. Imagine you have the following update function and model:
```python
def update_user(user_id: uuid.UUID, **update_kwargs):
    for key, value in update_kwargs.items():
        set_user_attribute(user_id, key, value)

class UserUpdate(BaseModel):
    user_id: uuid.UUID
    username: Optional[str]
    email: Optional[str]
```
Then
```python
user_update = UserUpdate(user_id=<user_id>, email="a@b.com")
update_user(**user_update.dict(skip_defaults=True))
```
makes sure that you only update the fields that were specified; in this case, only the email address. (This is a [common pattern](https://fastapi.tiangolo.com/tutorial/body-updates/) for the HTTP PATCH method.)

If the `skip_defaults=True` logic always removed `None`, then it would no longer be possible to specify that you want to change the email attribute from a non-`None` value **back** to `None`.


great answer @dmontagu 
@samuelcolvin - Is this the kind of functionality that you would consider for Pydantic?  Could we leave this open as a feature request?

(Thanks @dmontagu great explanation of that behaviour!)
I'd accept `skip_none` keyword argument to `dict()` and friends.

Resolving patch:
diff --git a/pydantic/main.py b/pydantic/main.py
--- a/pydantic/main.py
+++ b/pydantic/main.py
@@ -308,6 +308,7 @@ def dict(
         skip_defaults: bool = None,
         exclude_unset: bool = False,
         exclude_defaults: bool = False,
+        exclude_none: bool = False,
     ) -> 'DictStrAny':
         """
         Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
@@ -332,6 +333,7 @@ def dict(
                 exclude=exclude,
                 exclude_unset=exclude_unset,
                 exclude_defaults=exclude_defaults,
+                exclude_none=exclude_none,
             )
         }
 
@@ -350,6 +352,7 @@ def json(
         skip_defaults: bool = None,
         exclude_unset: bool = False,
         exclude_defaults: bool = False,
+        exclude_none: bool = False,
         encoder: Optional[Callable[[Any], Any]] = None,
         **dumps_kwargs: Any,
     ) -> str:
@@ -371,6 +374,7 @@ def json(
             by_alias=by_alias,
             exclude_unset=exclude_unset,
             exclude_defaults=exclude_defaults,
+            exclude_none=exclude_none,
         )
         if self.__custom_root_type__:
             data = data[ROOT_KEY]
@@ -549,6 +553,7 @@ def _get_value(
         exclude: Optional[Union['AbstractSetIntStr', 'DictIntStrAny']],
         exclude_unset: bool,
         exclude_defaults: bool,
+        exclude_none: bool,
     ) -> Any:
 
         if isinstance(v, BaseModel):
@@ -559,6 +564,7 @@ def _get_value(
                     exclude_defaults=exclude_defaults,
                     include=include,
                     exclude=exclude,
+                    exclude_none=exclude_none,
                 )
             else:
                 return v.copy(include=include, exclude=exclude)
@@ -576,6 +582,7 @@ def _get_value(
                     exclude_defaults=exclude_defaults,
                     include=value_include and value_include.for_element(k_),
                     exclude=value_exclude and value_exclude.for_element(k_),
+                    exclude_none=exclude_none,
                 )
                 for k_, v_ in v.items()
                 if (not value_exclude or not value_exclude.is_excluded(k_))
@@ -592,6 +599,7 @@ def _get_value(
                     exclude_defaults=exclude_defaults,
                     include=value_include and value_include.for_element(i),
                     exclude=value_exclude and value_exclude.for_element(i),
+                    exclude_none=exclude_none,
                 )
                 for i, v_ in enumerate(v)
                 if (not value_exclude or not value_exclude.is_excluded(i))
@@ -626,6 +634,7 @@ def _iter(
         exclude: Union['AbstractSetIntStr', 'DictIntStrAny'] = None,
         exclude_unset: bool = False,
         exclude_defaults: bool = False,
+        exclude_none: bool = False,
     ) -> 'TupleGenerator':
 
         value_exclude = ValueItems(self, exclude) if exclude else None
@@ -640,7 +649,7 @@ def _iter(
 
         for k, v in self.__dict__.items():
             if allowed_keys is None or k in allowed_keys:
-                yield k, self._get_value(
+                value = self._get_value(
                     v,
                     to_dict=to_dict,
                     by_alias=by_alias,
@@ -648,7 +657,10 @@ def _iter(
                     exclude=value_exclude and value_exclude.for_element(k),
                     exclude_unset=exclude_unset,
                     exclude_defaults=exclude_defaults,
+                    exclude_none=exclude_none,
                 )
+                if not (exclude_none and value is None):
+                    yield k, value
 
     def _calculate_keys(
         self,


Test patch:
diff --git a/tests/test_edge_cases.py b/tests/test_edge_cases.py
--- a/tests/test_edge_cases.py
+++ b/tests/test_edge_cases.py
@@ -1303,3 +1303,57 @@ class MyModel(BaseModel):
 
     m = MyModel(foo={'x': 'a', 'y': None})
     assert m.foo == {'x': 'a', 'y': None}
+
+
+def test_exclude_none():
+    class MyModel(BaseModel):
+        a: Optional[int] = None
+        b: int = 2
+
+    m = MyModel(a=5)
+    assert m.dict(exclude_none=True) == {'a': 5, 'b': 2}
+
+    m = MyModel(b=3)
+    assert m.dict(exclude_none=True) == {'b': 3}
+    assert m.json(exclude_none=True) == '{"b": 3}'
+
+
+def test_exclude_none_recursive():
+    class ModelA(BaseModel):
+        a: Optional[int] = None
+        b: int = 1
+
+    class ModelB(BaseModel):
+        c: int
+        d: int = 2
+        e: ModelA
+        f: Optional[str] = None
+
+    m = ModelB(c=5, e={'a': 0})
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': 0, 'b': 1}, 'f': None}
+
+    m = ModelB(c=5, e={'b': 20}, f='test')
+    assert m.dict() == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+    assert m.dict(exclude_none=True) == {'c': 5, 'd': 2, 'e': {'b': 20}, 'f': 'test'}
+    assert dict(m) == {'c': 5, 'd': 2, 'e': {'a': None, 'b': 20}, 'f': 'test'}
+
+
+def test_exclude_none_with_extra():
+    class MyModel(BaseModel):
+        a: str = 'default'
+        b: Optional[str] = None
+
+        class Config:
+            extra = 'allow'
+
+    m = MyModel(a='a', c='c')
+
+    assert m.dict(exclude_none=True) == {'a': 'a', 'c': 'c'}
+    assert m.dict() == {'a': 'a', 'b': None, 'c': 'c'}
+
+    m = MyModel(a='a', b='b', c=None)
+
+    assert m.dict(exclude_none=True) == {'a': 'a', 'b': 'b'}
+    assert m.dict() == {'a': 'a', 'b': 'b', 'c': None}