Package: dalec
Django Aggregator of a Lot of External Contents aka DALEC 🤖Django Aggregator of a Lot of External Contents (DALEC) is a generic app to aggregate contents from various
external sources. Purposes are to manage (retrieve, clean, display…) those contents in a
generic way independent of the source.It's designed to be customized / extended to fit your needs.Tests, QA, consistency and compatibilityThis app is tested to runs with:Django 2.2, 3.2, 4.2Python 3.7, 3.9, 3.11To ensure code quality and consistency:Python formatted withblackPython validated withflake8.Python static types checked withmypyPython tests coverage checked withcoverage(100% tested)JS tested withjestJS formatted withprettierTests runned in local viatoxand on github viagithub actions workflowversionned withsemverlogicConceptsContents are categorized via :app: the app which is requested to retrieve contents
(eg.gitlab)content_type: the type of contents we want to retrieve from this app
(eg.issue,activity,commit,merge requests…)channel: some apps can be requested to get a more or less filtered contents.
For example, in Gitlab, it's called "scope". You can retrieve issues from:the whole site (channel=Noneandchannel_object=Nonefor us)a specific group (channel="group"andchannel_object="<gitlab_group_id>")a specific project (channel="project"andchannel_object="<gitlab_project_id>")a specific author (channel="author"andchannel_object="<gitlab_user_id>")a specific assignee (channel="assignee"andchannel_object="<gitlab_user_id>")etc.channel_object: the ID (for the app requested) of the channel's related objectchannel_objects: the list of IDs (for the app requested) of the channel's related objects (eg."['42','443']")Before you ask: yes, some contents can be duplicated (eg. an issue from the "group" channel
could be duplicated in the "project" channel). It's normal andwanted. Remember : the purpose
of this app is to retrieve and display thelast N contents of something. If the issue we are
talking about is in thelast N issues of project "cybermen", it does not mean it's also inthe last N issues of group "dr-who-best-friends". To manage different timelines for each channel,
and keep a KISS Model, we need those duplicates.External sources supported🦝dalec-gitlab:
get issues and activities from a gitlab instance☁dalec-nextcloud:
get events and activities from a nextcloud instance🗣dalec-discourse:
get last messages or topics from a discourse instance🔗dalec-openproject:
get tasks from an openproject instance📅dalec-caldav: events and tasks from a caldav instance (eg: nextcloud agenda)External sources which could be nice to support🌍 dalec-mediawiki: get last pages from a mediawiki instance📂 dalec-webdav: get lastmodified files from a webdav instance (eg: nextcloud files)📰 dalec-feedparser: get contents from atom and rss feeds📫 dalec-imap: get emails from imap instance🐥 dalec-mastodon: get toots from a mastodon instance📺 dalec-peertube: get last uploaded videos from a peertube instance🌵 dalec-git: get last commits, branches, tags from a git repository🐱 dalec-github: get issues, pull-requests, activity from github🐦 dalec-twitter: get tweets from twitter🎞 dalec-youtube: get last uploaded videos from youtube🎥 dalec-imdb: get last movies from imdbInstallationIf you are using Django older than 3.1, the JSONField is provided fromdjango-jsonfield-backport.
For 3.1+, youMUSTuse a DB supporting the officialdjango'sJSONField.YouSHOULD NOTinstall this app but youSHOULDinstall one (or more) of it's children
(see). eg:pip install dalec-gitlab dalec-nextcloudThen, adddalec,dalec_primeand dalec's children toINSTALLED_APPSinsettings.py:# settings.pyINSTALLED_APPS=[# …"dalec","dalec_prime",# if you want to use your own Content model, don't add it"dalec_gitlab","dalec_nextcloud",# …]And in yoururls.py:# urls.pyurl_patterns=[...re_path("^dalec/",include("dalec.urls")),...]UsageEach dalec's child app will probably need some specific configuration to retrieve external contents
(eg: token or login/password). Please refer to this dalec's child app configuration section first.Now your dalec's child app is configured, you can display it's X last contents somewhere in a
template by using the templatetagdalec:{%loaddalec%}{%dalecappcontent_type[channel=None][channel_object=None][template=None][ordered_by=None]%}real examples:Retrieves last gitlab issues for a specific user:{%dalec"gitlab""issue"channel="user"channel_object="doctor-who"%}Retrieves recent gitlab event for a group:{%dalec"gitlab""event"channel="group"channel_object='42'%}Retrieves recent gitlab event for a project:{%dalec"gitlab""event"channel="project"channel_object='443'%}Retrieves recent gitlab issues for a project:{%dalec"gitlab""issue"channel="project"channel_object='404'%}Retrieves recent gitlab issues for multiple projects:{%dalec"gitlab""issue"channel="project"channel_objects='["42","443"]'%}Retrieves recent gitlab issues for multiple projects and order them by descendingissue internal ID (default is `last_update_dt`):{%dalec"gitlab""issue"channel="project"channel_object='42'ordered_by="-iid"%}dalec_exampleAn example app is packaged to get a working example which does not require any extra configuration.adddalec_example,dalec_primeanddalectoINSTALLED_APPSrun migrationsinclude dalec.urls inside your project's urlsadd those fragments of code inside the template where you want to display the example:{%loaddalec%}<h1>Last quarters (very usefull, isn't it ?)</h1>{%dalec"example""quarter"%}<h1>Last updated establishments of french national education</h1>{%dalec"example""french_educ"%}<h1>Last updated establishments of french national education depending of the Academy of Grenoble</h1>{%dalec"example""french_educ"channel="academy"channel_object="Grenoble"%}ConfigurationThis app have general settings which can be erased for all of it's children and sometimes by
content type.General setting format :DALEC_SOMETHINGoverride child version (it's app name, like gitlab for example):DALEC_GITLAB_SOMETHINGoverride content type version (gitlab's issues for example):DALEC_GITLAB_ISSUE_SOMETHINGDALEC_NB_CONTENTS_KEPTdefault:10per child app setting: yesper child app's content type setting: yesSet the number of contents to keep by type. Oldest will be purged to keep only the last X contents
of each channel and type.0means "no limit".DALEC_AJAX_REFRESHdefault:Trueper child app setting: yesper child app's content type setting: yesIfTrue, when an user display a channel contents, an ajax requests is sent to refresh content.
It's usefull if you do not want to use a cron task and/or need to get always the last contents.DALEC_TTLdefault:900per child app setting: yesper child app's content type setting: yesNumber of seconds before an ajax request sends a new query to the instance providing instance.DALEC_CONTENT_MODELdefault:"dalec_prime.Content"per child app setting: noper child app's content type setting: noConcrete model to use to store contents. If you do not want to use the default one,
you should not adddalec.primeinINSTALLED_APPSto avoid to load a useless model
and have an empty table in your data base.DALEC_FETCH_HISTORY_MODELdefault:"dalec_prime.FetchHistory"per child app setting: noper child app's content type setting: noSame asDALEC_CONTENT_MODELbut for theFetchHistorymodel.DALEC_CSS_FRAMEWORKdefault:Noneper child app setting: yesper child app's content type setting: noName of the (S)CSS framework you use on your website. It changes the default templates used to
display contents. Templates are priorized in this order (css_frameworkversions only used if
you set a value toDALEC_CSS_FRAMEWORK):dalec/%(app)s/%(tpl)s-list.html: only if you use a specific template, see templatetagdalecdalec/%(app)s/%(css_framework)s/%(content_type)s-%(channel)s-list.htmldalec/%(app)s/%(content_type)s-%(channel)s-list.htmldalec/%(app)s/%(css_framework)s/%(content_type)s-list.htmldalec/%(app)s/%(content_type)s-list.htmldalec/%(app)s/%(css_framework)s/list.htmldalec/%(app)s/list.htmldalec/default/%(css_framework)s/list.htmldalec/default/list.htmlSupported valued are:None: only dalec classes and data will be added. HTML elements are sementics. Templates are the default ones.Future supported values could be:materializebulmabootstrapsemantic-uiCustomizationStylesNo styles are included inside dalec who must remains pure with no feelings.
But some SCSS framework may be supported. Please refer toDALEC_CSS_FRAMEWORKsetting.TemplatesYou should always inherit from the default templates and use defined blocks to customise it.ListEach app can have it's own "list" template but if it doesn't, a default one will be used.
In priority order:dalec/<childappname>/<contenttype>-<channel>-list.htmldalec/<childappname>/<contenttype>-list.htmldalec/<childappname>/list.htmldalec/default/list.htmlItemEach content types can have it's own template. Those filenames will be tried:dalec/<childappname>/<contenttype>-<channel>-item.htmldalec/<childappname>/<contenttype>-item.htmldalec/<childappname>/item.htmldalec/default/item.htmlFor "gitlab" app, "issue" content type and "project" channel we will try :dalec/gitlab/issue-project-item.htmldalec/gitlab/issue-item.htmldalec/gitlab/item.htmldalec/default/item.htmlModelsModel used to store contents is defined by the settingDALEC_CONTENT_MODELwhich has the
default value"dalec_prime.Content".
If you want to use your own model, in yoursettings.py:removedalec.primefromINSTALLED_APPSset the settingDALEC_CONTENT_MODELwith<yourapp>.<yourModel>Manage a new external sourceIf you want to add a specific external source, you just have to extendsdalec.proxy.Proxyand override it's_fetchmethod.To create a dalec child (a proper way), you should create a new django app with the name patterndalec_<yourExternalSourceUname>NAQ (Never Asked Questions)Why this logo is so ugly ?Because I'm a developperAren't you afraid to be accused of blasphemy by the Daleks ?A Dalek which have a concept of Blasphemy can not be a sane Dalek!https://youtu.be/6ThpkjDgdvY?t=162What's the dalec origin ?The Daleks are a fictional extraterrestrial race of mutants principally portrayed in the
British science fiction television programme Doctor Who. Name (and logo) of this django app is
directly related to them.This project was made possible thanks to theOpen Space Maker Federationwhose goal is to open
up the world of space infrastructure to as many people as possible.They need boards to aggregate contents from different sources (gitlab, discourse…).
We (Webu) didn't find any applications which fit our needs,
so we create a new one and released it under the MIT licence.