Repository: tiangolo/fastapi

Issue:
`jsonable_encoder` function not working as expected with different type of variables
### First check

* [X] I added a very descriptive title to this issue.
* [X] I used the GitHub search to find a similar issue and didn't find it.
* [X] I searched the FastAPI documentation, with the integrated search.
* [X] I already searched in Google "How to X in FastAPI" and didn't find any information.
* [X] I already read and followed all the tutorial in the docs and didn't find an answer.
* [X] I already checked if it is not related to FastAPI but to [Pydantic](https://github.com/samuelcolvin/pydantic).
* [X] I already checked if it is not related to FastAPI but to [Swagger UI](https://github.com/swagger-api/swagger-ui).
* [X] I already checked if it is not related to FastAPI but to [ReDoc](https://github.com/Redocly/redoc).
* [X] After submitting this, I commit to one of:
    * Read open issues with questions until I find 2 issues where I can help someone and add a comment to help there.
    * I already hit the "watch" button in this repository to receive notifications and I commit to help at least 2 people that ask questions in the future.
    * Implement a Pull Request for a confirmed bug.

<!-- 

I'm asking all this because answering questions and solving problems in GitHub issues consumes a lot of time. I end up not being able to add new features, fix bugs, review Pull Requests, etc. as fast as I wish because I have to spend too much time handling issues.

All that, on top of all the incredible help provided by a bunch of community members that give a lot of their time to come here and help others.

That's a lot of work they are doing, but if more FastAPI users came to help others like them just a little bit more, it would be much less effort for them (and you and me 😅).

-->

### Example

Here's a self-contained, [minimal, reproducible, example](https://stackoverflow.com/help/minimal-reproducible-example) with my use case:

#### Example 1 (BaseModel)

```python
import pytest
from attr import dataclass
from fastapi.encoders import jsonable_encoder
from pydantic.main import BaseModel

class MyBaseModel(BaseModel):
    foo: int
    bar: int


@pytest.mark.parametrize(
    "obj, include, exclude, expected",
    [
        (MyBaseModel(foo=1, bar=2), None, None, {"foo": 1, "bar": 2}),  # Passed
        (MyBaseModel(foo=1, bar=2), {}, {}, {}),  # Passed
        (MyBaseModel(foo=1, bar=2), {"foo"}, {}, {"foo": 1}),  # Passed
        (MyBaseModel(foo=1, bar=2), {}, {"foo"}, {}),  # Passed
        (MyBaseModel(foo=1, bar=2), {"foo"}, None, {"foo": 1}),  # Passed
        (MyBaseModel(foo=1, bar=2), None, {"foo"}, {"bar": 2}),  # Passed
    ],
)
def test_jsonable_encoder_include_exclude_base_model(obj, include, exclude, expected):
    assert jsonable_encoder(obj, include=include, exclude=exclude) == expected
```

#### Example 2 (Python dictionary)
```python
import pytest
from fastapi.encoders import jsonable_encoder


@pytest.mark.parametrize(
    "obj, include, exclude, expected",
    [
        ({"foo": 1, "bar": 2}, None, None, {"foo": 1, "bar": 2}),  # Passed
        ({"foo": 1, "bar": 2}, {}, {}, {}),  # Failed
        ({"foo": 1, "bar": 2}, {"foo"}, {}, {"foo": 1}),  # Failed
        ({"foo": 1, "bar": 2}, {}, {"foo"}, {}),  # Failed
        ({"foo": 1, "bar": 2}, {"foo"}, None, {"foo": 1}),  # Failed
        ({"foo": 1, "bar": 2}, None, {"foo"}, {"bar": 2}),  # Passed
    ],
)
def test_jsonable_encoder_include_exclude_dict(obj, include, exclude, expected):
    assert jsonable_encoder(obj, include=include, exclude=exclude) == expected
```

#### Example 3 (Python object with dataclass)

```python
@dataclass
class MyModel:
    foo: int
    bar: int


@pytest.mark.parametrize(
    "obj, include, exclude, expected",
    [
        (MyModel(foo=1, bar=2), None, None, {"foo": 1, "bar": 2}),  # Passed
        (MyModel(foo=1, bar=2), {}, {}, {}),  # Failed
        (MyModel(foo=1, bar=2), {"foo"}, {}, {"foo": 1}),  # Failed
        (MyModel(foo=1, bar=2), {}, {"foo"}, {}),  # Failed
        (MyModel(foo=1, bar=2), {"foo"}, None, {"foo": 1}),  # Failed
        (MyModel(foo=1, bar=2), None, {"foo"}, {"bar": 2}),  # Failed
    ],
)
def test_jsonable_encoder_include_exclude_obj(obj, include, exclude, expected):
    assert jsonable_encoder(obj, include=include, exclude=exclude) == expected
```

### Description

<!-- Replace the content below with your own problem, question, or error -->

I was trying to understand how the `include` and `exclude` parameters worked. I looked in the documentation ([here](https://fastapi.tiangolo.com/tutorial/encoder/)) and the the test suite ([here](https://github.com/tiangolo/fastapi/blob/fdb6c9ccc504f90afd0fbcec53f3ea0bfebc261a/tests/test_jsonable_encoder.py), latest commit of this file in master branch) and I did not find anything related to `include` and `exclude` in both places.

I did see that these 2 parameters came from Pydantic (from the [dict function](https://github.com/samuelcolvin/pydantic/blob/43308d1b24c5d1163c44b8fa786d1b27e000c64e/pydantic/main.py#L440)) and it is being used in the [_calculate_keys method](https://github.com/samuelcolvin/pydantic/blob/43308d1b24c5d1163c44b8fa786d1b27e000c64e/pydantic/main.py#L817).

The first thing I did was to check the `jsonable_encoder` using a BaseModel object, which uses the dict function from Pydantic (Example 1), so I created a set of test with a BaseModel instance. All those tests are passing. Then I created the same tests with a Python dictionary (Example 2) and a Python object (Example 3).

I expect that the same tests with a different type of variable would work the same. In the case of Example 3, it is failing the most because it is missing the `include` and `exclude` parameters at the end of the function ([here](https://github.com/tiangolo/fastapi/blob/fdb6c9ccc504f90afd0fbcec53f3ea0bfebc261a/fastapi/encoders.py#L142)).

### Environment

* OS: Linux
* Python: 3.8.5
* FastAPI Version: 0.63.0

To know the FastAPI version use:

```bash
python -c "import fastapi; print(fastapi.__version__)"
```

* Python version:

To know the Python version use:

```bash
python --version
```

### Additional context

To reproduce the Example snippets, you will also need `pytest` installed (6.2.1 in my case)
I am happy to help by creating a PR for this bug once it is confirmed.

<details>
  <summary>All examples together</summary>
  
  ```python
import pytest
from attr import dataclass
from fastapi.encoders import jsonable_encoder
from pydantic.main import BaseModel


class MyBaseModel(BaseModel):
    foo: int
    bar: int


@dataclass
class MyModel:
    foo: int
    bar: int


@pytest.mark.parametrize(
    "obj, include, exclude, expected",
    [
        (MyBaseModel(foo=1, bar=2), None, None, {"foo": 1, "bar": 2}),  # Passed
        (MyBaseModel(foo=1, bar=2), {}, {}, {}),  # Passed
        (MyBaseModel(foo=1, bar=2), {"foo"}, {}, {"foo": 1}),  # Passed
        (MyBaseModel(foo=1, bar=2), {}, {"foo"}, {}),  # Passed
        (MyBaseModel(foo=1, bar=2), {"foo"}, None, {"foo": 1}),  # Passed
        (MyBaseModel(foo=1, bar=2), None, {"foo"}, {"bar": 2}),  # Passed
        ({"foo": 1, "bar": 2}, None, None, {"foo": 1, "bar": 2}),  # Passed
        ({"foo": 1, "bar": 2}, {}, {}, {}),  # Failed
        ({"foo": 1, "bar": 2}, {"foo"}, {}, {"foo": 1}),  # Failed
        ({"foo": 1, "bar": 2}, {}, {"foo"}, {}),  # Failed
        ({"foo": 1, "bar": 2}, {"foo"}, None, {"foo": 1}),  # Failed
        ({"foo": 1, "bar": 2}, None, {"foo"}, {"bar": 2}),  # Passed
        (MyModel(foo=1, bar=2), None, None, {"foo": 1, "bar": 2}),  # Passed
        (MyModel(foo=1, bar=2), {}, {}, {}),  # Failed
        (MyModel(foo=1, bar=2), {"foo"}, {}, {"foo": 1}),  # Failed
        (MyModel(foo=1, bar=2), {}, {"foo"}, {}),  # Failed
        (MyModel(foo=1, bar=2), {"foo"}, None, {"foo": 1}),  # Failed
        (MyModel(foo=1, bar=2), None, {"foo"}, {"bar": 2}),  # Failed
    ],
)
def test_jsonable_encoder_include_exclude(obj, include, exclude, expected):
    assert jsonable_encoder(obj, include=include, exclude=exclude) == expected
  ```
</details>




Discussion:
The pydantic docs for include/exclude are [here](https://pydantic-docs.helpmanual.io/usage/exporting_models/#advanced-include-and-exclude), for anyone curious regarding this issue.

@xaviml How would you expect this to work for objects that are not pydantic models? Seems like we'd have to make a lot of assumptions about the structure of the objects.

Perhaps docs could be added saying include/exclude are only valid for pydantic models? 

Trying to get some more information: If you really want to use include/exclude, is there anything preventing you from using a pydantic model.
Hi @falkben,

Thank you for your reply.

> How would you expect this to work for objects that are not pydantic models?

I would expect to be consistent, so to work the same as Pydantic models. For example, if we do:
```python
jsonable_encoder({"foo": 1, "bar": 2}, include={"foo"})
```
I would expect to return `{"foo": 1}`. However, this returns: `{"foo": 1, "bar": 2}`. The same way that if we do:
```python
jsonable_encoder({"foo": 1, "bar": 2}, exclude={"foo"})
```
It returns `{"bar": 2}` as expected.

> Seems like we'd have to make a lot of assumptions about the structure of the objects.

I don't think is about the assumption of the structure of the object. They are all converted to dictionaries with `dict` ([here](https://github.com/tiangolo/fastapi/blob/fdb6c9ccc504f90afd0fbcec53f3ea0bfebc261a/fastapi/encoders.py#L134)) or `vars`([here](https://github.com/tiangolo/fastapi/blob/fdb6c9ccc504f90afd0fbcec53f3ea0bfebc261a/fastapi/encoders.py#L138)). What needs to change is the way of including and excluding ([here](https://github.com/tiangolo/fastapi/blob/fdb6c9ccc504f90afd0fbcec53f3ea0bfebc261a/fastapi/encoders.py#L80)).

> Perhaps docs could be added saying include/exclude are only valid for pydantic models?

You can see in [this `if` branch](https://github.com/tiangolo/fastapi/blob/fdb6c9ccc504f90afd0fbcec53f3ea0bfebc261a/fastapi/encoders.py#L70) that `include` and `exclude` is also used when the object is a dictionary.

> Trying to get some more information: If you really want to use include/exclude, is there anything preventing you from using a pydantic model.

I am using the `jsonable_encoder` with a SQLAlchemy object, the same way it is done in [here](https://github.com/tiangolo/full-stack-fastapi-postgresql/blob/490c554e23343eec0736b06e59b2108fdd057fdc/%7B%7Bcookiecutter.project_slug%7D%7D/backend/app/app/crud/base.py#L49). This is what is preventing me to use a Pydantic model, because I use a SQLAlchemy model. I will find a workaround to this, but this is another matter not related to this issue.

When I created this issue I tried to understand what is the expected use of include and exclude parameters from `jsonable_encoder`. This is why I went to the tests and see what is the expected behaviour of this function with the use of include and exclude. What I proposed is how I think it should be (and how it works with Pydantic), but I am happy to hear and discuss the right implementation for this function with these parameters.

Regards,
Xavi M.
Possibly related? https://github.com/tiangolo/fastapi/pull/2016
Hi @falkben,

I did see that issue before creating this one. The #2016 PR adds support to dictionaries for the include and exclude, but it does.not change the way include and exclude are being used. Although both of these issues (#2016 and this one) are about jsonable_encoder, they are not related.

Regards,
Xavi M.

Resolving patch:
diff --git a/fastapi/encoders.py b/fastapi/encoders.py
--- a/fastapi/encoders.py
+++ b/fastapi/encoders.py
@@ -80,6 +80,11 @@ def jsonable_encoder(
         return obj
     if isinstance(obj, dict):
         encoded_dict = {}
+        allowed_keys = set(obj.keys())
+        if include is not None:
+            allowed_keys &= set(include)
+        if exclude is not None:
+            allowed_keys -= set(exclude)
         for key, value in obj.items():
             if (
                 (
@@ -88,7 +93,7 @@ def jsonable_encoder(
                     or (not key.startswith("_sa"))
                 )
                 and (value is not None or not exclude_none)
-                and ((include and key in include) or not exclude or key not in exclude)
+                and key in allowed_keys
             ):
                 encoded_key = jsonable_encoder(
                     key,
@@ -144,6 +149,8 @@ def jsonable_encoder(
             raise ValueError(errors)
     return jsonable_encoder(
         data,
+        include=include,
+        exclude=exclude,
         by_alias=by_alias,
         exclude_unset=exclude_unset,
         exclude_defaults=exclude_defaults,


Test patch:
diff --git a/tests/test_jsonable_encoder.py b/tests/test_jsonable_encoder.py
--- a/tests/test_jsonable_encoder.py
+++ b/tests/test_jsonable_encoder.py
@@ -93,16 +93,42 @@ class Config:
     return ModelWithPath(path=request.param("/foo", "bar"))
 
 
+def test_encode_dict():
+    pet = {"name": "Firulais", "owner": {"name": "Foo"}}
+    assert jsonable_encoder(pet) == {"name": "Firulais", "owner": {"name": "Foo"}}
+    assert jsonable_encoder(pet, include={"name"}) == {"name": "Firulais"}
+    assert jsonable_encoder(pet, exclude={"owner"}) == {"name": "Firulais"}
+    assert jsonable_encoder(pet, include={}) == {}
+    assert jsonable_encoder(pet, exclude={}) == {
+        "name": "Firulais",
+        "owner": {"name": "Foo"},
+    }
+
+
 def test_encode_class():
     person = Person(name="Foo")
     pet = Pet(owner=person, name="Firulais")
     assert jsonable_encoder(pet) == {"name": "Firulais", "owner": {"name": "Foo"}}
+    assert jsonable_encoder(pet, include={"name"}) == {"name": "Firulais"}
+    assert jsonable_encoder(pet, exclude={"owner"}) == {"name": "Firulais"}
+    assert jsonable_encoder(pet, include={}) == {}
+    assert jsonable_encoder(pet, exclude={}) == {
+        "name": "Firulais",
+        "owner": {"name": "Foo"},
+    }
 
 
 def test_encode_dictable():
     person = DictablePerson(name="Foo")
     pet = DictablePet(owner=person, name="Firulais")
     assert jsonable_encoder(pet) == {"name": "Firulais", "owner": {"name": "Foo"}}
+    assert jsonable_encoder(pet, include={"name"}) == {"name": "Firulais"}
+    assert jsonable_encoder(pet, exclude={"owner"}) == {"name": "Firulais"}
+    assert jsonable_encoder(pet, include={}) == {}
+    assert jsonable_encoder(pet, exclude={}) == {
+        "name": "Firulais",
+        "owner": {"name": "Foo"},
+    }
 
 
 def test_encode_unsupported():
@@ -144,6 +170,14 @@ def test_encode_model_with_default():
     assert jsonable_encoder(model, exclude_unset=True, exclude_defaults=True) == {
         "foo": "foo"
     }
+    assert jsonable_encoder(model, include={"foo"}) == {"foo": "foo"}
+    assert jsonable_encoder(model, exclude={"bla"}) == {"foo": "foo", "bar": "bar"}
+    assert jsonable_encoder(model, include={}) == {}
+    assert jsonable_encoder(model, exclude={}) == {
+        "foo": "foo",
+        "bar": "bar",
+        "bla": "bla",
+    }
 
 
 def test_custom_encoders():