Package: ckanext-collection
ckanext-collectionBase classes for viewing data series from CKAN.ContentRequirementsInstallationUsageOverviewCollection intializationServicesCommon logicData servicePager serviceSerializer serviceColumns serviceFilters serviceService implementations and usage examplesDataStaticDataBaseModelDataStatementModelDataModelDataUnionModelDataApiDataApiSearchDataApiListDataPagerClassicPagerColumnsFiltersSerializerCsvSerializerJsonlSerializerJsonSerializerHtmlSerializerTableSerializerHtmxTableSerializerConfig settingsLicenseRequirementsCompatibility with core CKAN versions:CKAN versionCompatible?2.9no2.10yesmasteryesInstallationTo install ckanext-collection:Install the extension:pipinstallckanext-collectionAddcollectionto theckan.pluginssetting in your CKAN
config file .UsageOverviewThe goal of this plugin is to supply you with generic classes for processing
collections of data. As result, it doesn't do much out of the box and you have
to write some code to see a result.Majority of useful classes are available insideckanext.collection.utilsmodule and all examples bellow require the following line in the beginning of
the script:from ckanext.collection.utils import *.Let's start with the basics.ckanext-collectiondefines a few collections for
different puproses. The most basic collection isCollection, but it has no
value without customization, so we'll start fromStaticCollection:col=StaticCollection("name",{})Constructor of any collection has two mandatory arguments: name and
parameters. Name is mostly used internally and consists of any combination of
letters, digits, hyphens and underscores. Parameters are passed inside the
dictionary and they change the content of the collection.In the most basic scenario, collection represents a number of similar items:
datasets, users, organizations, dictionaries, numbers, etc. As result, it can
be transformed into list or iterated over:list(col)foritemincol:print(item)Our test collection is empty at the moment, so you will not see anything just
yet. Usually,StaticCollectioncontains static data, specified when
collection is created. But because we haven't specified any data, collection
contains nothing.To fix this problem, we have to configure a part of the collection responsible
for data production usingsettings. Collection divides its internal logic
between a number ofservices, and service that we need is calleddataservice. To modify it, we can pass a named argument calleddata_settingsto
the collection's constructor:col=StaticCollection("name",{},data_settings={"data":[1,2,3]})Now try again iterating over the collection and now you'll see the result:foritemincol:print(item)It's not very impressive, but you didn't expect much fromstaticcollection, right? There are other collections that are more smart, but we have
to learn more concepts of this extension to use them, so for now we'll only
take a brief look on them.Note: collections have certain restrictions when it comes to amount of
data. By default, you'll see only around 10 records, even if you have more. The
same is true forStaticCollection- you can see it if you setdataattribute of its data-service torange(1, 100). We'll learn how to control
these restrictions later.StaticCollectionworks with static data. It can be used for tests or as a
placeholder for a collection that is not yet implemented. In rare cases it can
be used with arbitrary iterable to create a standard interface for data
interaction.ModelCollectionworks with SQLAlchemy models. We are going to use two
attributes of its data-service:modelandis_scalar. The former sets actual
model that collection processes, while the latter controls, how we work with
every individual record. By default,ModelCollectionreturns every record as
a number of columns, but we'll set its value toTrueand receive model
instance for every record instead:col=ModelCollection("",{},data_settings={"is_scalar":True,"model":model.User})foruserincol:assertisinstance(user,model.User)print(f"{user.name},{user.email}")ApiSearchCollectionworks with API actions similar topackage_search. They
have to userowsandstartparameters for pagination and their result must
containcountandresultskeys. Its data-service acceptsactionattribute
with the name of API action that produces the data:col=ApiSearchCollection("",{},data_settings={"action":"package_search"})forpkgincol:print(f"{pkg['id']}:{pkg['title']}")ApiListCollectionworks with API actions similar topackage_list. They have
to uselimitandoffsetparameters for pagination and their result must be
represented by a list.col=ApiListCollection("",{},data_settings={"action":"package_list"})fornameincol:print(name)ApiCollectionworks with API actions similar touser_list. They have to
return all records at once, as list.col=ApiCollection("",{},data_settings={"action":"user_list"})foruserincol:print(user["name"])Collection intializationCollection constructor has two mandatory arguments: name and parameters.Name is used as collection identifier and it's better to keep this value unique
accross collections. For example, name is used for computing HTML tableidattribute when serializing collection as an HTML table. If you render two
collections with the same name, you'll get two identical IDs on the page.Params are usually used by data and pager service for searching, sorting,
etc. Collection does not keep all the params. Instead, it stores only items
with key prefixed by<name>:. I.e, if collection has namehello, and you
pass{"hello:a": 1, "b": 2, "world:c": 3}, collection will removebandworld:cmembers. As forhello:a, collection strips<name>:prefix from
it. So, in the end, collection stores{"a": 1}.  You can check params of the
collection usingparamsattribute:col=Collection("hello",{"hello:a":1,"b":2,"world:c":3})assertcol.params=={"a":1}col=Collection("world",{"hello:a":1,"b":2,"world:c":3})assertcol.params=={"c":3}It allows you rendering and processing multiple collections simultaneously on
the same page. Imagine that you have collectionusersand collectionpackages. You want to see second page ofusersand fifth ofpackages. Submit the query string?users:page=2&packages:page=5and
initialize collections using the following code:fromckan.logicimportparse_paramsfromckan.pluginsimporttoolkitastkparams=parse_params(tk.request.args)users=ModelCollection("users",params,data_settings={"model":model.User})packages=ModelCollection("packages",params,data_settings={"model":model.Package})assertusers.pager.page==2assertpackages.pager.page==5ServicesCollection itself contains just a bare minimum of logic, and all the
heavy-lifting is delegated toservices. Collection knows how to initialize
services and usually the only difference between all your collections, is the
way all their services are configured.Collection contains the following services:data: controls the exact data that can be received from
collection. Contains logic for searching, filters, sorting, etc.pager: defines restrictions for data iteration. Exactly this service shows
only 10 records when you iterating over static collectionserializer: specifies how collection can be transformed into desired
form. Using correct serializer you'll be able to dump the whole collection as
CSV, JSON, YAML or render it as HTML table.columns: contains configuration of specific data columns used by other
services. It may define model attributes that are dumped into CSV, names of
the transformation functions that are applied to the certain attribute, names
of the columns that are available for sorting in HTML representation of data.filters: contains configuration of additional widgets produced during data
serialization. For example, when data is serialized into an HTML table,
filters can define configuration of dropdowns and input fields from the data
search form.Note: You can define more services in custom collections. The list above enumerates
all the services that are available in the base collection and in all
collections shipped with the current extension.When a collection is created, it creates an instance of each service using
service factories and service settings. Base collection and all collections
that extend it already have all details for initializing every service:col=Collection("name",{})print(f"""{col.data=},{col.pager=},{col.serializer=},{col.columns=},{col.filters=}""")assertlist(col)==[]This collection has no data. We can initialize an instance ofStaticDataand
replace the data service of the collection with it. Every service has one
required argument: collection that will use the service. All other arguments
are used as a service settings and must be passed by name. Remember, all the
classes used in this manual are available insideckanext.collection.utils:static_data=StaticData(col,data=[1,2,3])col.replace_service(static_data)assertlist(col)==[1,2,3]Look atColletion.replace_service. It accepts only service instance. There is
no need to pass the name of the service that must be replaced - collection can
understand it without help. And pay attention to the first argument of service
constructor. It must be the collection that is going to use the service. Some
services may work even if you pass a random value as the first argument, but
it's an exceptional situation and one shouldn't rely on it.If existing collection is no longer and you are going to create a new one, you
probably want to reuse a service from an existing collection. Just to avoid
creating the service and callingCollection.replace_service, which will save
you two lines of code. In this case, use<service>_instanceparameter of the
collection constructor:another_col=Collection("another-name",{},data_instance=col.data)assertlist(another_col)==[1,2,3]If you do such thing, make sure you are not using old collection anymore. You
just transfered one of its services to another collection, so there is no
guarantees it will function properly. If you want to use a custom factory for
the service, instead of transfering a service instance, it's better to
customize service factory. You can tell which class to use for making an
instance of a service using<service>_factoryparameter of the collection
contstructor:col=Collection("name",{},data_factory=StaticData)assertlist(col)==[]But in this way we cannot specify thedataattribute of thedatafactory!
No worries, there are multiple ways to overcome this problem. First of all, all
the settings of the service are available as its attributes. It means thatdatasetting is the same asdataattribute of the service. If you can doStaticData(..., data=...), you can as well doservice = StaticData(...); service.data = ...:col=Collection("name",{},data_factory=StaticData)col.data.data=[1,2,3]assertlist(col)==[1,2,3]But there is a better way. You can pass<service>_settingsdictionary to the
collection constructor and it will be passed down into corresponding service
factory:col=Collection("name",{},data_factory=StaticData,data_settings={"data":[1,2,3]})assertlist(col)==[1,2,3]It works well for individual scenarios, but when you are creating a lot of
collections with the static data, you want to omit some standard parameters. In
this case you should define a new class that extends Collection and declares<Service>Factoryattribute:classMyCollection(Collection):DataFactory=StaticDatacol=MyCollection("name",{},data_settings={"data":[1,2,3]})assertlist(col)==[1,2,3]You still can passdata_factoryintoMyCollectionconstructor to override
data service factory. But now, by default,StaticDatais used when it's not
specified explicitly.Finally, if you want to create a subclass of service, that has a specific value
of certain attributes, i.e something like this:classOneTwoThreeData(StaticData):data=[1,2,3]you can useService.with_attributes(attr_name=attr_value)factory method. It
produce a new service class(factory) with specified attributes bound to a
static value. For example, that's how we can define a collection, that always
contains[1, 2, 3]:classMyCollection(Collection):DataFactory=StaticData.with_attributes(data=[1,2,3])col=MyCollection("name",{})assertlist(col)==[1,2,3]Now you don't have to specifydata_factoryordata_settingswhen creating a
collection. It will always useStaticDatawithdataset to[1, 2, 3]. Make sure you mean it, because now you cannot override the data usingdata_settings.Common logicAll services share a few common features. First of all, all services contain a
reference to the collection that users/owns the service. Only one collection
can own the service. If you move service from one collection to another, you
must never use the old collection, that no longer owns the service. Depending
on internal implementation of the service, it may work without changes, but we
recommend removing such collections. At any point you can get the collection
that owns the service viaattachedattribute of the service:col=Collection("name",{})assertcol.data.attachediscolassertcol.pager.attachediscolassertcol.columns.attachediscolanother_col=Collection("another-name",{},data_instance=col.data)assertcol.data.attachedisnotcolassertcol.data.attachedisanother_colassertcol.dataisanother_col.dataSecond common point of services issettings. Let's useStaticDatafor
tests. It has one configurable attribute(setting) -data. We can specify it
directly when creating data service instance:StaticData(..., data=DATA). Or
we can specify it viadata_settingswhen creating a collection:StaticCollection("name", {}, data_settings={"data": DATA}). In both casesDATAwill be available as adataattribute of the data service. But it
doesn't mean that we can pass just any attribute in this way:data=StaticData(col,data=[],not_real=True)asserthasattr(data,"data")assertnothasattr(data,"not_real")To allow overriding the value of attribute via settings, we have to define this
attribute as aconfigurable attribute. For this we needconfigurable_attributefunction fromckanext.collection.shared:classMyData(StaticData):i_am_real=configurable_attribute(False)data=MyData(col,data=[],i_am_real=True)asserthasattr(data,"data")asserthasattr(data,"i_am_real")assertdata.i_am_realisTrueconfigurable_attributeaccepts either positional default value of the
attribute, or nameddefault_factoryfunction that generated default value
every time new instance of the service is created.default_factorymust
accept a single argument - a new service that is instantiated at the moment:classMyData(StaticData):ref=42i_am_real=configurable_attribute(default_factory=lambdaself:self.ref*10)data=MyData(col,data=[])assertdata.i_am_real==420Never use another configurable attributes in thedefault_factory- order in
which configurable attributes are initialized is not strictly defined. At the
moment of writing this manual, configurable attributes were initialized in
alphabetical order. But this implementation detail may change in future without
notice.TODO: with_attributesData serviceThis service produces the data for collection. Every data service must:be Iterable and iterate over all available records by defaultdefinetotalproperty, that reflects number of available records so thatlen(list(data)) == data.totaldefinerange(start: Any, end: Any)method that returns slice of the dataBase class for data services -Data- already contains a simple version of
this logic. You have to define only one methods to make you custom
implementations:compute_data(). When data if accessed for the first time,compute_datais called. Its result cached and used for iteration in
for-loops, slicing viarangemethod and size measurement viatotalproperty.classCustomData(Data):defcompute_data(self)->Any:return"abcdefghijklmnopqrstuvwxyz"col=Collection("name",{},data_factory=CustomData)assertlist(col)==["a","b","c","d","e","f","g","h","i","j"]assertcol.data.total==26assertcol.data.range(-3,None)=="xyz"If you need more complex data source, make sure you defined__iter__,total, andrange:classCustomData(Data):names=configurable_attribute(default_factory=["Anna","Henry","Mary"])@propertydeftotal(self):returnlen(self.names)def__iter__(self):yield fromself.namesdefrange(self,start:Any,end:Any):ifnotisinstance(start,str)ornotisinstance(end,str):return[]fornameinnames:ifname<start:continueifname>end:breakyieldnamePager servicePager service sets the upper and lower bounds on data used by
collection. Default pager used by collection relies on numericstart/endvalues. But it's possible to define custom pager that uses alphabetical or
temporal bounds, as long asrangemethod of your custom data service supports
these bounds.Standard pager(ClassicPager) has two configurable attributes:page(default:androws_per_page(default: 10).col=StaticCollection("name",{})assertcol.pager.page==1assertcol.pager.rows_per_page==10Because of these values you see only first 10 records from data when iterating
the collection. Let's change pager settings:col=StaticCollection("name",{},data_settings={"data":range(1,100)},pager_settings={"page":3,"rows_per_page":6})assertlist(col)==[13,14,15,16,17,18]Pagination details are often passed with search parameters and have huge
implact on the required data frame. Because of it, ifpager_settingsare
missing,ClassicPagerwill look for settings inside collection
parameters(second argument of the collection constructor). But in this case,
pager will use only items that has<collection name>:prefix:col=StaticCollection("xxx",{"xxx:page":3,"xxx:rows_per_page":6},data_settings={"data":range(1,100)})assertlist(col)==[13,14,15,16,17,18]col=StaticCollection("xxx",{"page":3,"rows_per_page":6},data_settings={"data":range(1,100)})assertlist(col)==[1,2,3,4,5,6,7,8,9,10]Serializer serviceSerializer converts data into textual or binary representation. Serializers are
main users of columns service, because it contains details about specific data
columns. And serializers often iterate data service directly(ignoringrangemethod), to serialize all available records. The only required method for
serializer isstream. This method must return an iterable ofstrorbytefragments of the serialized data. And there must be at least one fragment. Even
if there is no data, serializer must return at least''orb'fromstream.classNewLineSerializer(Serializer):defstream(self):foriteminself.attached.data:yieldstr(item)+"\n"col=StaticCollection("name",{},serializer_factory=NewLineSerializer,data_settings={"data":[1,2,3]})assert"".join(col.serializer.stream())=="1\n2\n3\n"Serializer also hasrendermethod, that combines all fragments fromstreaminto a single sequence:assertcol.serializer.render()=="1\n2\n3\n"Userenderonly when you are sure that you have enough memory for all the
serialized records. In most cases it will be wiser to send streaming response
to client usingstreamgenerator.Columns serviceThis service contains additional information about separate columns of data
records. It defines following settings:names: all available column names. Used by other settings of columns servicehidden: columns that should not be shown by serializer. Used by serializer
servicesvisible: columns that must be shown by serializer. Used by serializer
servicessortable: columns that support sorting. Used by data servicesfilterable: columns that support filtration/facetting. Used by data servicessearchable: columns that support search by partial match. Used by data
serviceslabels: human readable labels for columns. Used by serializer servicesFilters serviceThis service used only by HTML table serializers at the moment. It has two
configurable attributesstatic_filtersandstatic_actions.static_filtersare used for building search form for the data table.static_actionsare not
used, but you can put into it details about batch or record-level actions and
use these details to extend one of standard serializers. For example,
ckanext-admin-panel defines allowed actions (remove, restore, hide) for content
and creates custom templates that are referring these actions.Service implementations and usage examplesTBADataTBAStaticDataTBABaseModelDataTBAStatementModelDataTBAModelDataTBAUnionModelDataTBAApiDataTBAApiSearchDataTBAApiListDataTBAPagerTBAClassicPagerTBAColumnsTBAFiltersTBASerializerTBACsvSerializerTBAJsonlSerializerTBAJsonSerializerTBAHtmlSerializerTBATableSerializerTBAHtmxTableSerializerTBAConfig settings# Names of registered collections that are viewable by any visitor, including# anonymous.# (optional, default: )ckanext.collection.auth.anonymous_collections=# Names of registered collections that are viewable by any authenticated# user.# (optional, default: )ckanext.collection.auth.authenticated_collections=# Add HTMX asset to pages. Enable this option if you are using CKAN v2.10# (optional, default: false)ckanext.collection.include_htmx_asset=false# Initialize CKAN JS modules every time HTMX fetches HTML from the server.# (optional, default: false)ckanext.collection.htmx_init_modules=false# Import path for serializer used by CSV export endpoint.# (optional, default: ckanext.collection.utils.serialize:CsvSerializer)ckanext.collection.export.csv.serializer=ckanext.collection.utils.serialize:CsvSerializer# Import path for serializer used by JSON export endpoint.# (optional, default: ckanext.collection.utils.serialize:JsonSerializer)ckanext.collection.export.json.serializer=ckanext.collection.utils.serialize:JsonSerializer# Import path for serializer used by JSONl export endpoint.# (optional, default: ckanext.collection.utils.serialize:JsonlSerializer)ckanext.collection.export.jsonl.serializer=ckanext.collection.utils.serialize:JsonlSerializer# Import path for serializer used by `format`-export endpoint.# (optional, default: )ckanext.collection.export.<format>.serializer=LicenseAGPL