Package: django-data-fetcher
django-data-fetcherWhat is this?This library contains 3 utilities:a utility to access the currently-executing django request from anywhereThis can help with a wide variety of fringe use-cases, e.g. wanting to access the current request from a decoupled log helper functionThere are 'leaner' tools that provide this, such asdjango-middleware-global-request, but it relies on thread-locals (not async-safe) and hasn't been updated in a very long timeA caching abstraction to cache a function's return value on the requestThis is probably the most useful thing in this package. It helps withrepeatedqueries or general cachingAdata-fetcherabstraction to ease batching,In addition to caching, this is forbatching, i.e. N+1 queriesThis is the least important, most complicated and most-specific use case. For historical reasons, the entire library is named after the util. Naming things is hard :(Installationpipinstalldjango-data-fetcherAfter installing, you'll need to add our middleware:# settings.py# ...MIDDLEWARE=[# ..."data_fetcher.middleware.GlobalRequestMiddleware",# ...]UsageAccessing the global request objectOnce the middleware is added, accessing the global request is simple:fromdata_fetcher.utilimportget_requestdefsome_random_util_function():request=get_request()do_something_with_request(request)CachingIf you'd just like to cache a function so you don't repeat it in different places, you can just use thecache_within_requestdecorator:fromdata_fetcherimportcache_within_request@cache_within_requestdefget_most_recent_order(user_id):returnOrder.objects.filter(user_id=user_id).order_by('-created_at').first()Now you can callget_most_recent_orderas many times as you want within a request, e.g. in template helpers and in views, and it will only hit the database once (assuming you use the same user_id). This is a wrapper aroundfunctools.cache, so it will also cache across calls to the same function with the same arguments.BatchingThis library also supportsbatchingfetching logic. You need to subclass ourDataFetcherclass and implement abatch_load(orbatch_load_dict) method with the batching logic. Then you can use its factory method to get an instance of your fetcher class, and call itsget(),get_many(), orprefetch_keys()methods.For example, it's usually pretty difficult to efficiently fetch permissions for a list of objects without coupling your view to your templates/helpers. With this library, we can offload the work to a data-fetcher instance and have a re-usable template-helper that checks permissions. When we notice performance problems, we simply add aprefetch_keyscall to our view to pre-populate the cache.# my_app/fetchers.pyfromdata_fetcherimportDataFetcherclassArticlePermissionFetcher(DataFetcher):defbatch_load_dict(self,article_ids):permissions=ArticlePermission.objects.filter(article_id__in=article_ids)return{p.article_id:pforpinpermissions}# my_app/template_helpers.pyfrommy_app.fetchersimportArticlePermissionFetcher@register.simple_tag(takes_context=True)defcan_read_article(context,article):"""called in a loop in article_list.html, e.g. to condtionally render a link"""fetcher=ArticlePermissionFetcher.get_instance()permission=fetcher.get(article.id)returnpermission.can_read# my_app/views.pyfrommy_app.fetchersimportArticlePermissionFetcherdefarticle_list(request):articles=Article.objects.all()fetcher=ArticlePermissionFetcher.get_instance()fetcher.prefetch_keys([a.idforainarticles])returnrender(request,'article_list.html',{'articles':articles})Behind the scenes, fetchers'get_instancewill use the global-request middleware's request object to always return the same instance of the fetcher for the same request. This allows the fetcher to call your batch function once, when the view callsprefetch_keys, and then use the cached results for all subsequent calls togetorget_many.Fetchers also cache values that were called withgetorget_many. If you request a key that isn't cached, it will call your batch method again for that single key. It's recommended to monitor your queries while developing with a tool likedjango-debug-toolbar.Fetcher APIPublic method:get(key): fetch a single resource by keyget_many(keys): fetch multiple resources by key, returns a listget_many_as_dict(keys): like get_many, but returns a dict indexed by your requested keysprefetch_keys(keys): Like get-many but returns nothing. Pre-populates the cache with a list of keys. This is useful when you know you're going to need a lot of objects, and you want to avoid N+1 queries.prime(key,value)manually set a value in the cache. This isn't recommended, but it can be useful for performance in certain casesSubclass-API:You can implementbatch_load(keys)ORbatch_load_dict(keys).batch_load(keys)needs to return a list of resources in the same order (and length) as the keys. If a resource is missing, you need an explicit None in the returned list.batch_load_dict(keys)should return a dict of resources, indexed by the keys. If a value is missing,Nonewill be returned when that key is requested (it tolerates missing keys).ShortcutsIt's extremely common to want to fetch a single object by id, or by a parent's foreign key. We provide a few baseclasses for this:fromdata_fetcherimportAbstractModelByIdFetcher,AbstractChildModelByAttrFetcherclassArticleByIdFetcher(AbstractModelByIdFetcher):model=ArticleclassArticleByAuthorIdFetcher(AbstractChildModelByAttrFetcher):model=Articleparent_attr='author_id'In fact, the ID fetcher was so common we have a factory for it. This factory returns the same class every time, so you can use it in multiple places without worrying about creating multiple classes with distinct caches.fromdata_fetcherimportPrimaryKeyFetcherFactoryArticleByIdFetcher=PrimaryKeyFetcherFactory.get_model_by_id_fetcher(Article)ArticleByIdFetcher2=PrimaryKeyFetcherFactory.get_model_by_id_fetcher(Article)assertArticleByIdFetcher==ArticleByIdFetcher2article_1=ArticleByIdFetcher.get_instance().get(1)Testing data-fetchersBatch logic is often complex and error-prone. We recommend writing tests for your fetchers.  provides a mock request object that you can use to test your fetchers. Without this context-manager, your fetchers won't be able to cache anything and might raise errors. Here's an example in pytest:fromdata_fetcher.utilimportGlobalRequestdeftest_article_permission_fetcher(django_assert_num_queries):withGlobalRequest():withdjango_assert_num_queries(1):fetcher=ArticlePermissionFetcher.get_instance(request)fetcher.prefetch_keys([1,2,3])assertfetcher.get(1).can_readassertnotfetcher.get(2).can_readassertnotfetcher.get(3).can_readHow to provide non-key data to fetchersData-fetcher's main feature is not performance, but enabling decoupling. The view layer no longer has to be responsible for passing data to downstream consumers (e.g. utils, template-helpers, service-objects, etc.).This paradigm shift can be a challenging adjustment. For instance, ourArticlePermissionFetcherabove was naïve. Permission records should be fetched with respect to a user. How can we provide the user's ID to the fetcher?It's tempting to subclass DataFetcher and add a user argument to itsget_instance()method. Unfortunately, extending the factory pattern is rather complex. There are broadly 3 different ways to solve this problem:Use the global request to get the user. This is the simplest solution, but it limits your data-fetcher to the current user. You couldn't, for example, build a view that shows a list of articles available tootherusers.Create composite-keys: instead of loading permissions by article id, you load them by(user_id, article_id)pairs. This is a good solution, but is often complex to implement and you usually don't need this flexibility.Dynamically create a data-fetcherclassthat has a reference to the userThe 3rd solution fulfills the OOP temptation of adding a user argument to the constructor, but it's a "higher-order" solution. Rather than attaching the user to the fetcher-class, we would dynamically create a class that has a reference to the user, and then use a factory to ensure we recycle the same class for the same user.There's a builtin shortcut for this pattern, too, calledValueBoundDataFetcher. ValueBoundDataFetcher classes have abound_valueattribute available inside their batch-load methods.# my_app/fetchers.pyfromdata_fetcherimportValueBoundDataFetcherclassUserBoundArticlePermissionFetcher(ValueBoundDataFetcher):defbatch_load_dict(self,article_ids):user_id=self.bound_valuepermissions=ArticlePermission.objects.filter(user_id=user_id,article_id__in=article_ids)return{p.article_id:pforpinpermissions}# my_app/views.pyfrommy_app.fetchersimportUserBoundArticlePermissionFetcherdefarticle_list(request):# generate a class that has a reference to the userUserBoundArticlePermissionFetcher=ValueBoundDataFetcher.get_value_bound_class(UserBoundArticlePermissionFetcher,request.user.id)fetcher=UserBoundArticlePermissionFetcher.get_instance()articles=Article.objects.all()fetcher.prefetch_keys([a.idforainarticles])returnrender(request,'article_list.html',{'articles':articles})With this solution, we're still able to create fetchers for multiple users. However, it won't be as efficient as the composite-key solution (e.g. one query per user vs. one query for all users).Note thatbound_valuecan be anything, so you can use this pattern to provide more than a single piece of data to your fetcher, just make sure it's hashable so it can be used as a key (otherwise, you'll want to pass separate value and key kwargs toget_value_bound_class.Recipe: Caching a single data-structure with complex dataBatching logic often has a high-cognitive load, it may not be worth it to batch everything. Fortunately, the@cache_within_requestdecorator can cache anything, there's no need to restrict ourselves to a single resource. For instance, let's say we have a complex home-feed page that needs to fetch a lot of data for a particular user. We can use thecache_within_requestdecorator to cache the entire data-structure.@cache_within_requestdefget_home_feed_data(user_id):user=User.objects.filter(id=user_id).prefetch_related(Prefetch('articles',queryset=Article.objects.filter(deleted=False),to_attr='article_list'),Prefetch('articles__comments',queryset=Comment.objects.filter(deleted=False),to_attr='comment_list'),Prefetch('articles__author',queryset=User.objects.all(),to_attr='author_list'),# ...)more_data=get_more_data(user)# assemble a rich data structure with convenient APIreturn{'user':user,'articles':user.article_list,'comments':flatten([article.comment_listforarticleinuser.article_list]),'articles_by_id':# ...'comments_by_article_id':# ...'comments_by_id':# ...# ...}Now any function can request the entire structure and use its rich API. We can isolate the ugly fetching logic and don't need to pass data around (e.g. view -> template -> helpers) to remain efficient.This is not a perfect approach, as it couples our consumers (e.g. views, helpers) to this data-structure. This makes it difficult to re-use those helpers, or parts of the data-structure. However, in a pinch, it may be preferable to setting up fetchers (e.g. article-by-id, comments-by-article-id) for every atomic piece of data. A neat compromise might be to split this up into multiple cache functions, or a class that that executes other cached functions lazily.AsyncLike most ORM-consuming code, data-fetcher is synchronous. You'll need to usesync_to_asyncto use it inside async views. Behind the scenes, the global-request middleware uses context-vars, which are both thread-safe and async-safe.