Package: queryish
queryishA Python library for constructing queries on arbitrary data sources following Django's QuerySet API.MotivationDjango's QuerySet API is a powerful tool for constructing queries on a database. It allows you to compose queries incrementally, with the query only being executed when the results are needed:books=Book.objects.all()python_books=books.filter(topic='python')latest_python_books=python_books.order_by('-publication_date')[:5]print(latest_python_books)# Query is executed hereThis pattern is a good fit for building web interfaces for listing data, as it allows filtering, ordering and pagination to be handled as separate steps.We may often be required to implement similar interfaces for data taken from sources other than a database, such as a REST API or a search engine. In these cases, we would like to have a similarly rich API for constructing queries to these data sources. Even better would be to follow the QuerySet API as closely as possible, so that we can take advantage of ready-made tools such asDjango's generic class-based viewsthat are designed to work with this API.queryishis a library for building wrappers around data sources that replicate the QuerySet API, allowing you to work with the data in the same way that you would with querysets and models.InstallationInstall using pip:pipinstallqueryishUsage - REST APIsqueryishprovides a base classqueryish.rest.APIModelfor wrapping REST APIs. By default, this follows the out-of-the-box structure served byDjango REST Framework, but various options are available to customise this.fromqueryish.restimportAPIModelclassParty(APIModel):classMeta:base_url="https://demozoo.org/api/v1/parties/"fields=["id","name","start_date","end_date","location","country_code"]pagination_style="page-number"page_size=100def__str__(self):returnself.nameThe resulting class has anobjectsproperty that supports the usual filtering, ordering and slicing operations familiar from Django's QuerySet API, although these may be limited by the capabilities of the REST API being accessed.>>>Party.objects.count()4623>>>Party.objects.filter(country_code="GB")[:10]<PartyQuerySet[<Party:16BitShow1991>,<Party:AcornUserShow1991>,<Party:AnarchyEasterParty1992>,<Party:AnarchyWinterConference1991>,<Party:AtariPreservationParty2007>,<Party:CommodoreComputerClubUK1stMeet>,<Party:CommodoreShow1987>,<Party:CommodoreShow1988>,<Party:DejaVu1998>,<Party:DejaVu1999>]>>>>Party.objects.get(name="Nova 2023")<Party:Nova2023>Methods supported includeall,count,filter,order_by,get,first, andin_bulk. The result set can be sliced at arbitrary indices - these do not have to match the pagination supported by the underlying API.APIModelwill automatically make multiple API requests as required.The following attributes are available onAPIModel.Meta:base_url: The base URL of the API from where results can be fetched.pk_field_name: The name of the primary key field. Defaults to"id". Lookups on the field name"pk"will be mapped to this field.detail_url: A string template for the URL of a single object, such as"https://demozoo.org/api/v1/parties/%s/". If this is specified, lookups on the primary key and no other fields will be directed to this URL rather thanbase_url.fields: A list of field names defined in the API response that will be copied to attributes of the returned object.pagination_style: The style of pagination used by the API. Recognised values are"page-number"and"offset-limit"; all others (including the default ofNone) indicate no pagination.page_size: Required ifpagination_styleis"page-number"- the number of results per page returned by the API.page_query_param: The name of the URL query parameter used to specify the page number. Defaults to"page".offset_query_param: The name of the URL query parameter used to specify the offset. Defaults to"offset".limit_query_param: The name of the URL query parameter used to specify the limit. Defaults to"limit".ordering_query_param: The name of the URL query parameter used to specify the ordering. Defaults to"ordering".To accommodate APIs where the returned JSON does not map cleanly to the intended set of model attributes, the class methodsfrom_query_dataandfrom_individual_dataonAPIModelcan be overridden:classPokemon(APIModel):classMeta:base_url="https://pokeapi.co/api/v2/pokemon/"detail_url="https://pokeapi.co/api/v2/pokemon/%s/"fields=["id","name"]pagination_style="offset-limit"verbose_name_plural="pokemon"@classmethoddeffrom_query_data(cls,data):"""Given a record returned from the listing endpoint (base_url), return an instance of the model."""# Records within the listing endpoint return a `url` field, from which we want to extract the IDreturncls(id=int(re.match(r'https://pokeapi.co/api/v2/pokemon/(\d+)/',data['url']).group(1)),name=data['name'],)@classmethoddeffrom_individual_data(cls,data):"""Given a record returned from the detail endpoint (detail_url), return an instance of the model."""returncls(id=data['id'],name=data['name'],)def__str__(self):returnself.nameCustomising the REST API queryset classTheobjectsattribute of anAPIModelsubclass is an instance ofqueryish.rest.APIQuerySetwhich initially consists of the complete set of records. As with Django's QuerySet, methods such asfilterreturn a new instance.It may be necessary to subclassAPIQuerySetand override methods in order to support certain API responses. For example, the base implementation expects unpaginated API endpoints to return a list as the top-level JSON object, and paginated API endpoints to return a dict with aresultsitem. If the API you are working with returns a different structure, you can override theget_results_from_responsemethod to extract the list of results from the response:fromqueryish.restimportAPIQuerySetclassTreeQuerySet(APIQuerySet):base_url="https://api.data.amsterdam.nl/v1/bomen/stamgegevens/"pagination_style="page-number"page_size=20http_headers={"Accept":"application/hal+json"}defget_results_from_response(self,response):returnresponse["_embedded"]["stamgegevens"]APIQuerySetsubclasses can be instantiated independently of anAPIModel, but results will be returned as plain JSON values:>>>TreeQuerySet().filter(jaarVanAanleg=1986).first(){'_links':{'schema':'https://schemas.data.amsterdam.nl/datasets/bomen/dataset#stamgegevens','self':{'href':'https://api.data.amsterdam.nl/v1/bomen/stamgegevens/1101570/','title':'1101570','id':1101570},'gbdBuurt':{'href':'https://api.data.amsterdam.nl/v1/gebieden/buurten/03630980000211/','title':'03630980000211','identificatie':'03630980000211'}},'id':1101570,'gbdBuurtId':'03630980000211','geometrie':{'type':'Point','coordinates':[115162.72,485972.68]},'boomhoogteklasseActueel':'c. 9 tot 12 m.','jaarVanAanleg':1986,'soortnaam':"Salix alba 'Chermesina'",'stamdiameterklasse':'0,5 tot 1 m.','typeObject':'Gekandelaberde boom','typeSoortnaam':'Bomen','soortnaamKort':'Salix','soortnaamTop':'Wilg (Salix)'}This can be overridden by defining amodelattribute on the queryset, or overriding theget_instance/get_individual_instancemethods. To use a customised queryset with anAPIModel, define thebase_query_classattribute on the model class:classTree(APIModel):base_query_class=TreeQuerySetclassMeta:fields=["id","geometrie","boomhoogteklasseActueel","jaarVanAanleg","soortnaam","soortnaamKort"]# >>> Tree.objects.filter(jaarVanAanleg=1986).first()# <Tree: Tree object (1101570)>Other data sourcesqueryishis not limited to REST APIs - the base classqueryish.Queryishcan be used to build a QuerySet-like API around any data source. At minimum, this requires defining arun_querymethod that returns an iterable of records that is filtered, ordered and sliced according to the queryset's attributes. For example, a queryset implementation that works from a simple in-memory list of objects might look like this:fromqueryishimportQueryishclassCountryQuerySet(Queryish):defrun_query(self):countries=[{"code":"nl","name":"Netherlands"},{"code":"de","name":"Germany"},{"code":"fr","name":"France"},{"code":"gb","name":"United Kingdom"},{"code":"us","name":"United States"},]# Filter the list of countries by `self.filters` - a list of (key, value) tuplesfor(key,val)inself.filters:countries=[cforcincountriesifc[key]==val]# Sort the list of countries by `self.ordering` - a tuple of field namescountries.sort(key=lambdac:[c.get(field,None)forfieldinself.ordering])# Slice the list of countries by `self.offset` and `self.limit`. `offset` is always numeric# and defaults to 0 for an unsliced list; `limit` is either numeric or None (denoting no limit).returncountries[self.offset:self.offset+self.limitifself.limitelseNone]Subclasses will also typically override the methodrun_count, which returns the number of records in the queryset accounting for any filtering and slicing. If this is not overridden, the default implementation will callrun_queryand count the results.