Package: drf-eager-fields
DRF Eager FieldsThis library provides a dynamically fields selection and prefetching at the same time, in order to improve the response performance in Django REST Framework.OverviewThis library has the primary scope to handle the eager loading while providing a dynamic, nested, run-time selection of the fields to display.
It's similiar todrf-dynamic-fieldsanddrf-flex-fieldsgreat libraries, with the main aim of improving the poor database perfomances achieved when the fields to return are evaluted runtime and thus it's not possible to hardcode a prefetchable queryset.
It uses the capabaility of the "Prefetch" class of Django ORM to build the queryset on the fly.I'd liked to name this librarydfr-extra-fieldsbut the name had been already stolen :disappointed: :smile:Features:dynamical nested fields selectiondynamical nested fields exclusionoptional Json to represent the wanted response structureworks with model serializers and standard onessetup the response inside the view for a complete serializers reusabilityprefetching of dynamically selected fields to decouple the number of queries (fix the N+1 problem)custom prefetch queryset to filter the related dataset in the serializer definitionInstallpip install drf-eager-fieldsQuick exampleImagine that you have a customer API that returns all its orders and, inside each order, the related article. Our customer View queryset isCustomer.objects.all()GET /api/customers/1(number of queries: hundreds)

"id": 1,  # unnecessary
"name" : "Mario",
"countries": [1],  # unnecessary
"orders": [
    {   
        "id": 1,  # unnecessary
        "code": "DFGHJ",
        "created_at: '2020-11-29 20:04:53'
        "article": {
            "id": 1,  # unnecessary
            "code": "PIZZA"
            "customer": 1  # unnecessary
        }
    },
    {
        "id": 2,  # unnecessary
        "code": "BNMVC",
        created_at: '2020-12-01 21:12:07'
        "article": {
            ...
        }
    },
    ... many other orders (and articles) here ...
    {
        "id": 199,  # unnecessary
        "code": "QSDER",
        created_at: '2021-02-01 19:48:43'
        "article": {...}
    }
]This response has two problems: unwanted fields (we don't want ids, any customer's data except for the name) and speed, because we have camed across the N+1 problems.Standard solutionWe can fix those issues by modifying the queryset in the view, adding all the prefetch stuff and creating another custom serializer to remove to annoing unnecessary fields. We must do this every time we want to return an even sligthly different response. In this case we want the article inside order, but in another case we just want the plain order's data without any other nested field, such as article.Drf-Eager-FieldsUsing thedrf-eager-fieldsyou can achieve this result without modifying the view or the serializer.Default responseGET /api/customers/1"id": 1
"name": "Mario"Custom responseGET /api/customers/1?extra=orders.article,customer&fields=name,orders.created_at,orders.customer.name,orders.article.code(number of queries: 3)

"name" : "Mario",
"orders": [
    {   
        "code": "DFGHJ",
        "created_at: '2020-11-29 20:04:53'
        "article": {
            "code": "PIZZA"
        }
    },
    {
        "code": "BNMVC",
        created_at: '2020-12-01 21:12:07'
        "article": {...}
    },
    ... other orders here ...
    {
        "code": "QSDER",
        created_at: '2021-02-01 19:48:43'
        "article": {
            ...
        }
    }
]UsageFirst of all, you need to have your serializer extend the eager mixing class and, if you want to use the extra fields, you must set theextraclass property.extradictionary is defined as a function with the@classpropertydecorator, to allow the import of nested serializers and avoiding the circular import issue. Until you face that problem, you can handle theextraproperty as a normal class attribute.This is an example of the eager serializers:# article_serializer.py
class ArticleSerializer(EagerFieldsMixin, serializers.ModelSerializer):

    class Meta:
        models = models.Article
        fields = ('id', 'code')
    
        @classproperty
        def extra(self):
            from .customer_serializers import CustomerSerializer
            return {
                "customer": {
                    "field": CustomerSerializer(),
                    "prefetch": True # see below
                }
            }

# customer_serializer.py
class CustomerSerializer(EagerFieldsMixin, serializers.ModelSerializer):

    class Meta:
        models = models.Customer
        fields = ('id', 'name')
    
        @classproperty
        def extra(self):
            from .countries_serializers import CountrySerializer
            return {
                "countries": {
                    "field": CountrySerializer(many=True),
                    "prefetch": True # see below
                }
            }GET /api/articles/the standard response is, as usual:[
    {
        "id": 1,
        "code": "TNT"
    },
    {
        "id": 2,
        "code": "PIZZA"
    }
]if you just want the article's code, you can request it explictly with thefieldsparameterGET /api/articles/?fields=codeor excluding the id with theexcludeparameterGET /api/articles/?exclude=id[
    {
        "code": "TNT"
    },
    {
        "code": "PIZZA"
    }
]If you want to add an extra fields, which is defined in theextraproperty, do thisGET /api/articles/?extra=customer[
    {
        "id": 1,
        "code": "TNT",
        "customer": {
            "id": 1,
            "name": "Willy"
        }
    },
    {
        "id": 2,
        "code": "PIZZA",
        "customer": {
            "id": 1,
            "name": "Mario"
        }
    }
]Withdottednotation, you can add the any nested extra fields previously defined in the serializersextradict:GET /api/articles/?extra=customer.countries[
    {
        "id": 1,
        "code": "TNT",
        "customer": {
            "id": 1,
            "name": "Willy",
            "countries": [
                {
                    "id": 1,
                    "name": "USA"
                }
            ]
        }
    },
    {
        "id": 2,
        "code": "PIZZA",
        "customer": {
            "id": 1,
            "name": "Mario",
            "countries": [
                {
                    "id": 1,
                    "name": "USA"
                },
                {
                    "id": 2,
                    "name": "Italy"
                },
            ]
        }
    }
]Advice: if you want the take the best from this library, put all your related fields inside theextradictionary
and leave only the model's flat properties in the serializer.Meta 'fields' attribute.Of course, you can combine all the parameters to get your custom responseGET /api/articles/?extra=customer.countries,fields=code,customer.name,customer.countries.name[
    {
        "code": "TNT",
        "customer": {
            "name": "Willy",
            "countries": [
                {
                    "name": "USA"
                }
            ]
        }
    },
    {
        "code": "PIZZA",
        "customer": {
            "name": "Mario",
            "countries": [
                {
                    "name": "USA"
                },
                {
                    "name": "Italy"
                },
            ]
        }
    }
]Once you are done with the eager serializer, make your view to extend theEaderFieldsViewMixinlike this:Class ArticleView(EagerFieldsViewMixin, ListAPIView):
        queryset = Article.objects.all()  # one level queryset, no prefetching
        serializer_class = ArticleSerializer
        serializer_extra = 'customer.countries'
        serializer_fields = 'code,customer.name,customer.countries.name'
        serializer_exclude = ...As you can see, you can set the serializer params directly from the view.
There are two ways more to do that:inside the request, as query params. This overrides the default parameters set in the view:GET /api/articles/?extra=customer,fields=code,customer.name,exclude=...inside the GET request, asJSON in the body. This JSON represent the wanted response structure and overrides other paramas set in view or in the query params string:{
      "fields" : {
          "code": null,
          "customer": {
              "name": null
          }
      }
  }PrefetchWith great power comes great responsibility. [Peter Parker]The problem with the dynamic response, is how to eager loading (prefetching)
all the necessary data without leading ourself to the famous N+1 nightmare.
This library takes the capability of the DjangoPrefetchclass to deal with it.
If you set the property "prefetch" in yourextradictionary, the queryset will be extended runtime to prefetch them."countries": {
    "field": CountrySerializer(many=True),
    "prefetch": True
        # is equal to
    "prefetch": Prefetch("countries", queryset=CountrySerializer.Meta.model.objects.all())
}If you set theprefetchattribute in every serializer, this library will compose the prefetched queryset for you, even for the nested fields. :bomb:Prefetch classorprefetch_relatedis mandatory in Django to prefetching the related objects from the "many" side of the relationship (many-to-one and many-to-many). On the "one" side, you should useselect_related. However I chose to use thePrefetch classfor both, as recommended by Django documentation in cases whereyou want to use a QuerySet that performs further prefetching on related modelsoryou want to prefetch only a subset of the related objects.One step further..The prefetch attribute can accept a bool or a Prefetch object where you
can declare your queryset. Let me show you an example with the article API: image that you just want to show the related orders, but only the last 10 orders, and of course, prefetch them.@classproperty
    def extra(self):
        from .order_serializer import OrderSerializer

        return {
            "orders": {"field": OrderSerializer(many=True), "prefetch": True},
            "last_10_orders": {
                "field": OrderSerializer(source="orders", many=True),
                "prefetch": Prefetch(
                    "orders",
                    queryset=OrderSerializer.Meta.model.objects.filter(
                        id__in=Subquery(
                            OrderSerializer.Meta.model.objects.filter(
                                article_id=OuterRef("article_id")
                            )
                            .order_by("-created_at")
                            .values_list("id", flat=True)[:10]
                        )
                    ).order_by("-created_at"),
                ),
            },
        }Now, if you callGET /api/articles/?extra=last_10_ordersyou will get only the last 10 prefetch orders, ordered by the date. :bomb: :bomb:Sorry for the complex queryset but, as far as I know, in Django you cannot limit the queryset directly if you want to keep it as queryset for further extension, so you have to nest a subquery to get the latest ids.Example and testingYou can run an example Django DRF application insideexamplefolder$ python manage.py runserver 0.0.0.0:8000In that folder, to run the tests$ python manage.py testCreditsdrf-dynamic-fieldsdrf-flex-fieldsLicenseMIT license, seeLICENSEfile.