Package: django-geosource
Django GeoSourceThis django application provide a Rest Framework API that allow to manage many geo data
sources and integrate that data wherever you need, like a Django model or any output
pipeline. Its provided with necessary celery jobs that do the job.SettingsYou can define the settingGEOSOURCE_MAX_TASK_RUNTIMEthat allow to define the max run time of a task before it can be launched one more
time. It allow to prevent when a task is stuck and disallow launching one more.Configure and run CeleryYou must define in your project settings the variables CELERY_BROKER_URL and CELERY_RESULT_BACKEND as specified in Celery documentation.
You also need to create the celery app following thisdocumentation.Then to run the celery worker:$ celery worker -A django_geosource -l infoTo run periodic tasks, use celery beat:fromceleryimportCeleryfromcelery.schedulesimportcrontabapp=Celery()@app.after_finalize.connectdefsetup_periodic_tasks(sender,**kwargs):# Calls refresh check every 30 minutessender.add_periodic_task(60.0*30,run_auto_refresh_source.s())@app.taskdefrun_auto_refresh_source():fromdjango_geosource.periodicsimportauto_refresh_sourceauto_refresh_source()Then run celery beat worker that allow to synchronize periodically sources, launch this command:$ celery beat -A django_geosource -l infoConfigure data destinationNow, you must set the callback methods that are used to insert data in your destination database.If you use django-geostore, we provide a set of callback in thegeostore_callbacksmodule, else you can define your
own callbacks.GEOSOURCE_LAYER_CALLBACKThe callback signature receive as first argument the SourceModel object, and must return your Layer object.
Example:deflayer_callback(geosource):returnLayer.objects.get_or_create(name=geosource.name)[0]GEOSOURCE_FEATURE_CALLBACKThis one, define a feature creation callback method.
Example:deffeature_callback(geosource,layer,identifier,geometry,attributes):returnFeature.objects.get_or_create(layer=layer,identifier=identifier,geom=geometry,properties=attributes)[0]GEOSOURCE_CLEAN_FEATURE_CALLBACKThis callback is called when the refresh is done, to clear old features that are not anymore present in the database.
It receives as parametter the geosource, layer and begin update date, so you can advise what to do depending of your
models.
Example:defclear_features(geosource,layer,begin_date):returnlayer.features.filter(updated_at__lt=begin_date).delete()GEOSOURCE_DELETE_LAYER_CALLBACKThis is called when a Source is deleted, so you are able to do what you want with the loaded content in database, when
the source doesn't exist anymore. It's executed before real deletion.
Example:defdelete_layer(geosource,layer):iflayer.features.count()>0:layer.features.delete()returnlayer.delete()To start a dev instanceDefine settings you wants intest_geosourcedjango project.docker-composebuild
docker-composeupFirst start should failed as the database need to be initialized. Just launch the same command twice.Then initialize the database:docker-composerunweb./manage.pymigrateYou can now edit your code. A django runserver is launched internally so the
this is an autoreload server.You can access to the api onhttp://localhost:8000/api/Test and coverageTo run test suite, just launch:docker-composerun--rmweb./manage.pytestTo run test suite with coverage, execute this commande:docker-composerunweb/code/src/coverage.sh0.5.3 / 2022-03-04add ci job to check pending migrations (#42)Add missing migrations0.5.2 / 2022-02-02Simplify WMTS source serializer validation0.5.1 / 2021-10-13Add hack to handle mondrian exception with logging proxy0.5.0 / 2021-09-14Change way periodic tasks are launched0.4.12 / 2021-09-08Fix scheduler0.4.11 / 2021-02-04Disable record validation in CSVSourceSerialiser0.4.10 / 2020-12-01Add property values list endpoint[Bug] Force order update & add it to default0.4.9 / 2020-10-15Add error reporting on source0.4.8 / 2020-10-07Fields order is kept from the sourceAdd credit field on SourceAdd date as a field type0.4.7 / 2020-07-01Csv empty cell are recorded as None valueUpdate test, CSVSource settings are not write only anymoreOnly update csvsource settings and make it readable0.4.6 / 2020-05-14Add refresh_data assertion in get_records testsSerializer do not return None value to representationRecords name are string even with no header0.4.5 / 2020-05-13Ignored columns are removed from records0.4.4 / 2020-05-11let pyexcel handle file type0.4.3 / 2020-05-11Correctly extract srid from datafix typo in separators nameEnsure correct csv decoding0.4.2 / 2020-05-07Add CSVSource source0.4.1 / 2020-03-24Fix wmts geom_type mandatory0.4.0 / 2020-03-19BREAKING CHANGE : change way celery is working to allow using celery in another app0.3.7 / 2020-03-17Enhance tests to valide search and filterAdd option to sync sources to have more controlAdd zipfile shapefilesource0.3.6 / 2019-12-19Fix bug with FileSourceSerializer0.3.5 / 2019-12-18Add ordering and filtering for sourcesAdd flake8 linting to CI0.3.4 / 2019-12-16ImprovesImprove documentationFix python3.8, django 3.0 and DRF 3.11 compatibility0.3.3 / 2019-11-06ImprovesDefine MANIFEST.in0.3.1 / 2019-11-06ImprovesImprove error message when identifier field is not found in the sourceImprove error message when geojson features has bad geometriesUse black for linting in pipelines0.3.0 / 2019-10-18ReleaseFirst release