Package: grout
GroutGrout is a flexible-schema framework for geospatial data, powered by Django and PostgreSQL. Think: NoSQL database server, but with schema validation and PostGIS support.ContentsIntroductionGetting startedDjangoRequirementsInstallationConfigurationMore examplesNon-Django applicationsConceptsData modelVersioned schemasAPI documentationRequest and response formatsPaginationResourcesRecordTypesRecordSchemasRecordsBoundariesBoundaryPolygonsDevelopingRequirementsInstallationRunning testsCleaning upMaking migrationsResourcesGrout suiteHistorical documentsRoadmapIntroductionGrout combines the flexibility of NoSQL databases with the geospatial muscle ofPostGIS, allowing you to make migration-free edits to
your database schema while still having access to powerful geospatial queries.Grout will help you:Define, edit, and validate schemasfor records in your applicationKeep track of changes to schemasusing a built-in versioning systemPerformfast filtering of user-defined fieldsRuncomplex geospatial queries, even on records stored with unstructured dataGrout is the core library of theGrout suite, a toolkit for easily building
flexible-schema apps on top of Grout. You can use Grout by installing it as an app
in a Django project, or you can deploy it as astandalone API
serverwith an optionaladmin
backend.Ready for more? To get started using Grout with Django, seeGetting
started. To get started using Grout with another
stack, seeNon-Django applications. For more
background on how Grout works, seeConcepts.Getting startedDjangoIf you're developing a Django project, you can install Grout as a Django app and
use it in your project.RequirementsGrout supports the following versions of Python and Django:Python: 2.7, 3.4, 3.5, 3.6, 3.7Django: 1.11, 2.0Certain versions of Django only support certain versions of Python. To ensure
that your Python and Django versions work together, see the Django FAQ:What
Python version can I use with
Django?InstallationInstall the Grout library from PyPi usingpip.$pipinstallgroutTo use the development version of Grout, install it from GitHub.$gitclonegit@github.com:azavea/grout.gitMake sure Grout is included inINSTALLED_APPSin your project'ssettings.py.# settings.pyINSTALLED_APPS=(...'grout',)To use Grout as an API server, you need to incorporate the API views into yoururls.pyfile. The following example will include Grout views under the/groutendpoint.# urls.pyurlpatterns=[url(r'^grout/',include('grout.urls'))]Note that Grout automatically nests views under the/api/endpoint, meaning
that the setting above would create URLs likehostname.com/grout/api/records.
If you'd prefer Grout views to live under a top-level/api/endpoint (likehostname.com/api/records), you can import the Grouturlpatternsdirectly.# urls.pyfromgroutimporturlpatternsasgrout_urlpatternsurlpatterns=grout_urlpatternsConfigurationGrout requires that theGROUTconfiguration variable be defined in yoursettings.pyfile
in order to work properly. TheGROUTvariable is a dictionary of configuration
directives for the app.Currently,'SRID'is the only required key in theGROUTdictionary.'SRID'is an integer
corresponding to thespatial reference
identifierthat Grout should use to store geometries.4326is the most common SRID, and
is a good default for projects.Here's an example configuration for a development project:# settings.py# The projection for geometries stored in Grout.GROUT={'SRID':4326}Note that Grout usesDjango REST Frameworkunder the hood to provide API endpoints. To configure DRF-specific settings like
authentication, see theDRF docs.More examplesGrout Serveris a simple deployment
of a Grout API server designed to be used as a standalone app. It also serves
as a good example of how to incorporate Grout into a Django project, and
includes a preconfigured authentication module to boot. If you're
having trouble installing or configuring Grout in your project,Grout
Serveris a good resource for troubleshooting.Non-Django applicationsIf you're not a Django developer, you can still use Grout as a standalone
API server using theGrout Serverproject. See theGrout Server docsfor details on how to install a Grout Server instance.ConceptsData model.Grout is centered aroundRecords, which are justentities in your database.
A Record can be any type of thing or event in the world, although Grout is most
useful when your Records have some geospatial and temporal component.Every Record contains a reference to aRecordSchema, which catalogs theversioned schemaof the Record that points to it. This schema is stored asJSONSchema, a specification for describing data models in JSON.Finally, each RecordSchema contains a reference to aRecordType, which is
asimple container for organizing Records. The RecordType exposes a way to
reliably access a set of Records that represent the same type of thing, even if
they have different schemas. As we’ll see shortly, RecordTypes are useful access
points to Records because RecordSchemas can change at any moment.Versioned schemasIn Grout, RecordSchemas are append-only, meaning that they cannot be deleted.
Instead, when you want to change the schema of a Record, you create a new
RecordSchema and update theversionattribute.For a quick example, say that we have a RecordSchema describing data stored on
acatRecordType. The RecordSchema might look something like this:{"version":1,"next_version":null,"schema":{"type":"object","title":"Initial Schema","$schema":"http://json-schema.org/draft-04/schema#","properties":{"catDetails":{"$ref":"#/definitions/driverPosterDetails"},"definitions":{"catDetails":{"type":"object","title":"Cat Details","properties":{"Name":{"type":"string","fieldType":"text","isSearchable":true,"propertyOrder":1},"Age":{"type":"integer","fieldType":"integer","minimum":0,"maximum":100,"isSearchable":true,"propertyOrder":2},"Color":{"type":"string","fieldType":"text","isSearchable":true,"propertyOrder":3},"Breed":{"type":"select","fieldType":"selectlist","enum":["Tabby","Bobtail","Abyssinian"],"isSearchable":true,"propertyOrder":4}}}}}}A few things to note about this RecordSchema object:This is the first version of the schema (itsversionis1)There is no more recent version than this one (itsnext_versionisnull)The schema definition itself is stored as a JSONSchema object on theschemaattributeAll of the available fields are namespaced by thecatDetailsattribute, which
we sometimes refer to as aformorrelated contentNow say we want to change theAgefield to aDate of Birthfield. Instead of
changing the schema directly, we'll create a new schema. Grout will automatically
setversion: 2andnext_version: nullfor this updated schema:{"version":2,"next_version":null,"schema":{"type":"object","title":"Initial Schema","$schema":"http://json-schema.org/draft-04/schema#","properties":{"catDetails":{"$ref":"#/definitions/driverPosterDetails"},"definitions":{"catDetails":{"type":"object","title":"Cat Details","properties":{"Name":{"type":"string","fieldType":"text","isSearchable":true,"propertyOrder":1},"Age":{"type":"integer","fieldType":"integer","minimum":0,"maximum":100,"isSearchable":true,"propertyOrder":2},"Date of Birth":{"type":"string","format":"datetime","fieldType":"text","isSearchable":true,"propertyOrder":3},"Color":{"type":"string","fieldType":"text","isSearchable":true,"propertyOrder":4},"Breed":{"type":"select","fieldType":"selectlist","enum":["Tabby","Bobtail","Abyssinian"],"isSearchable":true,"propertyOrder":5}}}}}}}In addition, Grout will update the initial schema to setnext_version: 2:{
  "version": 1,
  "next_version": 2,
  "schema": {
    ...
  }
}Now, when a user searches for Records in thecatRecordType, Grout can find
the most recent schema by looking for the RecordSchema wherenext_version: null.
This preserves a full audit trail of the RecordSchema, allowing us to
inspect how the schema has changed over time.For a closer look at the Grout data model, see themodels.pyfile in the Grout
library.API documentationRequest and response formatsCommunication with the API generally follows the principles of RESTful API design.
API paths correspond to resources,GETrequests are used to retrieve objects,POSTrequests are used to create new objects, andPATCHrequests are used to update
existing objects. This pattern is followed in nearly all cases; any exceptions
will be noted in the documentation.Responses from the API are exclusively JSON.Endpoint behavior can be configured using query parameters forGETrequests,
whilePOSTrequests require a payload in JSON format.PaginationAll API endpoints that return lists of resources are paginated. The pagination takes the following format:{
    "count": 57624,
    "next": "http://localhost:8000/api/records/?offset=20",
    "previous": "http://localhost:7000/api/records/",
    "results": [
        ...
    ]
}In a real response, the domain and port for thenextandpreviousfields
will be that of the server responding to the request.This format applies to the API endpoints below and will not be repeated in the
documentation for each individual endpoint.ResourcesRecordTypesBecause the RecordSchema for a set of Records can change at any time, the RecordType API
endpoint provides a consistent access point for retrieving a set of Records.
Use the RecordType endpoints to discover the most recent RecordSchema for the Records
you are interested in before performing further queries.Paths:List:/api/recordtypes/Detail:/api/recordtypes/{uuid}/Query parameters:active: BooleanFilter for only RecordTypes with anactivevalue of True.
Generally, you will want to limit yourself to active RecordTypes.Results fields:Field nameTypeDescriptionuuidUUIDUnique identifier for this RecordType.current_schemaUUIDThe most recent RecordSchema for this RecordType.createdTimestampThe date and time when this RecordType was created.modifiedTimestampThe date and time when this RecordType was last modified.labelStringThe name of this RecordType.plural_labelStringThe plural version of the name of this RecordType.descriptionStringA short description of this RecordType.activeBooleanWhether or not this RecordType is active. This field allows RecordTypes to be deactivated rather than deleted.geometry_typeStringThe geometry type supported for Records of this RecordType. One ofpoint,polygon,multipolygon,linestring, ornone.temporalBooleanWhether or not Records of this RecordType should store datetime data in theoccurred_fromandoccurred_tofields.RecordSchemasThe RecordSchema API endpoint can help you discover the fields that
should be available on a given Record. This can be useful for automatically generating filters
based on a Record's fields, or for running custom validation on a Record's
schema.Paths:List:/api/recordschemas/Detail:/api/recordschemas/{uuid}/Results fields:Field nameTypeDescriptionuuidUUIDUnique identifier for this RecordSchema.createdTimestampThe date and time when this RecordSchema was created.modifiedTimestampThe date and time when this RecordSchema was last modified.versionIntegerA sequential number indicating what version of the RecordType's schema this is. Starts at 1.next_versionUUIDUnique identifier of the RecordSchema with the next-highest version number for this schema's RecordType. If this is the most recent version of the schema, this field will benull.record_typeUUIDUnique identifier of the RecordType that this RecordSchema refers to.schemaObjectAJSONSchemaobject that should validate Records that refer to this RecordSchema.RecordsRecords are the heart of a Grout project: the entities in your database. The
Records API endpoint provides a way of retrieving these objects for analysis
or display to an end user.Paths:List:/api/records/Detail:/api/records/{uuid}/Query Parameters:archived: BooleanRecords can be "archived" to denote that they are no longer current, as an
alternative to deletion. PassTrue(case-sensitive) to this parameter to return archived
Records only, and passFalse(case-sensitive) to return current Records only.
Omitting this parameter returns both types.details_only: BooleanIn theGrout Schema Editor,
every Record is automatically generated with a<record_type>Detailsform which is intended to contain a basic summary of information about the Record.
PassingTrue(case-sensitive) to this parameter will omit any other
forms which may exist on the Record. This is useful for limiting the size
of the payload returned when only a summary view is needed.record_type: UUIDLimit the response to Records matching the passed RecordType UUID.
This is optional in theory, but for most applications it is a good idea
to include this parameter by default. It is considered rare that it will
be useful to return two different types of Records in a single request.
It is usually a better idea to make a separate request for each RecordType.jsonb: ObjectQuery the data fields of the object and filter on the result.Keys in this object mimic the search paths to filter on a particular object
field. However, in place of values, a filter rule definition is used. Example:{ "accidentDetails": {   "Main+cause": { "_rule_type": "containment", "contains": [ "Vehicle+defect", "Road+defect", ["Vehicle+defect"], ["Road+defect"] ]   },   "Num+driver+casualties": { "_rule_type": "intrange", "min": 1, "max": 3   } }}. This query defines the following two filters:accidentDetails -> "Main cause" == "Vehicle defect" OR accidentDetails -> "Main cause" == "Road defect"accidentDetails -> "Num driver casualties" >= 1 AND accidentDetails -> "Num driver casualties" <= 3There is a third filter rule type available:containment_multiple.
This is used when searching a form of which there can be several on a single Record.
Here's an example:{"person":{"Injury":{"_rule_type":"containment_multiple","contains":["Fatal"]}}}occurred_min: TimestampFilter to Records occurring after this date.occurred_max: TimestampFilter to Records occurring before this date.polygon_id: UUIDFilter to Records which occurred within the Polygon identified by the
UUID. The value must refer to aBoundaryin the database.polygon: GeoJSONFilter to Records which occurred within the bounds of a valid GeoJSON
object.Results fields:Field nameTypeDescriptionuuidUUIDUnique identifier for this Record.createdTimestampThe date and time when this Record was created.modifiedTimestampThe date and time when this Record was last modified.occurred_fromTimestampThe earliest time at which this Record might have occurred.occurred_toTimestampThe latest time at which this Record might have occurred. Note that this field is mandatory for temporal Records: if a Record only occurred at one moment in time, theoccurred_fromfield and theoccurred_tofield will have the same value.geomGeoJSONGeometry representing the location associated with this Record.location_textStringA description of the location where this Record occurred, typically an address.archivedBooleanA way of hiding records without deleting them completely.Trueindicates the Record is archived.schemaUUIDReferences the RecordSchema which was used to create this Record.dataObjectA JSON object representing the flexible data fields associated with this Record. It is always true that the object stored indataconforms to the RecordSchema referenced by theschemaUUID.BoundariesBoundaries provide a quick way of storing Shapefile data in Grout without
having to create separate RecordTypes. Using a Boundary, you can upload
and retrieve Shapefile data for things like administrative borders and focus
areas in your application.Paths:List:/api/boundaries/Detail:/api/boundaries/{uuid}/Results fields:Field nameTypeDescriptionuuidUUIDUnique identifier for this Boundary.createdTimestampThe date and time when this Boundary was created.modifiedTimestampThe date and time when this Boundary was last modified.labelStringLabel of this Boundary, for display.colorStringColor preference to use for rendering this Boundary.display_fieldStringWhich field of the imported Shapefile to use for display.data_fieldsArrayList of the names of the fields contained in the imported Shapefile.errorsArrayA possible list of errors raised when importing the Shapefile.statusStringImport status of the Shapefile.source_fileStringURI of the Shapefile that was originally used to generate this Boundary.Notes:Creating a new Boundary and itsBoundaryPolygoncorrectly is a two-step process.POSTto/api/boundaries/with a zipped Shapefile attached; you will need
to include the label as form data. You should receive a 201 response which
contains a fully-fledged Boundary object, including a list of available
data fields indata_fields.The response from the previous request will have a blankdisplay_field.
Select one of the fields indata_fieldsand make aPATCHrequest to/api/boundaries/{uuid}/with that value indisplay_field.
You are now ready to use this Boundary and its associated BoundaryPolygon.BoundaryPolygonsBoundaryPolygons store the Shapefile data associated with aBoundary,
including geometry and metadata.Paths:List:/api/boundarypolygons/Detail:/api/boundarypolygons/{uuid}/Query Parameters:boundary: UUIDFilter to Polygons associated with this parent Boundary.nogeom: BooleanWhen passed with any value, causes the geometry field to be replaced with
a bbox field. This reduces the response size and is sufficient for many purposes.Results fields:Field nameTypeDescriptionuuidUUIDUnique identifier for this BoundaryPolygon.createdTimestampThe date and time when this BoundaryPolygon was created.modifiedTimestampThe date and time when this BoundaryPolygon was last modified.dataObjectEach key in this Object will correspond to one of thedata_fieldsin the parent Boundary, and will store the value for that field for this Polygon.boundaryUUIDUnique identifier of the parent Boundary for this BoundaryPolygon.bboxArrayMinimum bounding box containing this Polygon's geometry, as an Array of lat/lon points. This field is optional -- see thenogeomparameter above for more details.geometryGeoJSONGeoJSON representation of this Polygon. This field is optional -- see thenogeomparameter above for more details.DevelopingThese instructions will help you set up a development version of Grout and
contribute changes back upstream.RequirementsThe Grout development environment is containerized with Docker to ensure similar
environments across platforms. In order to develop with Docker, you need the
following dependencies:Docker CE Engine>= 1.13.0 (must be
compatible withDocker Compose file v3
syntax)Docker ComposeInstallationClone the repo with git.$gitclonegit@github.com:azavea/grout.git
$cdgroutRun theupdatescript to set up your development environment.$./scripts/updateRunning testsOnce your environment is up to date, you can use thescripts/testscript to
run the Grout unit test suite.$./scripts/testThis command will run a matrix of tests forevery supported version of Python and
Djangoin the project. If you're developing locally and you just want to run
a subset of the tests, you can specify the version of Python
that you want to use to run tests:# Only run tests for Python 2.7 (this will test Django 1.8).$./scripts/testapppy27# Only run tests for Python 3.7 (this will test Django 2.0).$./scripts/testapppy37For a list of available Python versions, see theenvlistdirective in thetox.inifile.Cleaning upTox creates a new virtualenv for every combination of Python and Django versions
used by the test suite. In order to clean up stopped containers and
remove these virtualenvs, use thecleanscript:$./scripts/cleanNote thatcleanwill removeall dangling images, stopped containers, and
unused volumeson your machine. If you don't want to remove these artifacts,view thecleanscriptand run only the command that
interests you.Making migrationsIf you edit the data model ingrout/models.py, you'll need to create a new
migration for the app. You can use thedjango-adminscript in thescriptsdirectory to automatically generate the migration:$./scripts/django-adminmakemigrationsMake sure to register the new migrations file with Git:$gitaddgrout/migrationsResourcesThe following resources provide helpful tips for deploying and using Grout.Grout suiteGrout Server: An easily-deployable
standalone instance of a Grout API server.Grout Schema Editor: A
purely static app that can read and write flexible schemas from a Grout API.Demo app: A demo project
providing an example of incorporating the Grout suite into a Vue.js app.Historical documentsConcept map: An early description of the Grout suite
(formerly known as Ashlar) from an Open Source Fellow working on it during the summer
of 2018. Describes the conceptual architecture of the suite, and summarizes
ideas for future directions.Renaming the package to Grout: An ADR documenting
the decision to rename the package from "Ashlar" to "Grout".Evaluating Record-to-Record references: An ADR
documenting the reasons and requirements for implementing a Record-to-Record
foreign key field. See alsothe pull request
threadfor further
discussion.Evaluating alternate backends: An ADR presenting
research into possible NoSQL backends and service providers for Grout.Grout 2018 Fellowship: A
project management repo for working on Grout during Azavea's Summer 2018Open Source Fellowship. Useful for
documentation around the motivation and trajectory of the project.RoadmapWant to know where Grout is headed? See theRoadmapto
get a picture of future development.