Package: vera
verais the reference implementation of the Entity-Record-Attribute-Value (ERAV) data model. ERAV is an extension to Entity-Attribute-Value (EAV) that adds support for maintaining multi-faceted provenance metadata for an entity[1].The implementation of ERAV provided by vera is optimized for storing and tracking changes totime series dataas it is exchanged between disparate technical platforms (e.g. mobile devices, Excel spreadsheets, and third-party databases). In this context, ERAV can be interpreted to mean Event-Report-Attribute-Value, as it represents a series ofeventsbeing described by thereportssubmitted about them by various contributors in e.g. an environmental monitoring or citizen science project.Getting Started# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activatepipinstallveravera is an extension to thewq framework. Seehttps://github.com/wq/verato report any issues.ModelsThe core of vera is a collection ofDjango modelsthat describe the various components of the ERAV data model.vera ModelsThere are four primary models (ERAV) and three auxilary models, for a total of seven models. The mapping from vera models to their ERAV conceptual equivalents is below:ERAV equivalentmodelmoduleSitevera.paramsEntity``Event``vera.seriesReportStatusvera.paramsRecord``Report``vera.seriesAttribute``Parameter``vera.paramsValue``Result``vera.resultsEventResultvera.resultsThe vera models are allswappable, which means they can be subclassed and extended without breaking the foreign key relationships needed by the ERAV model. The base models are technically split between three modules within vera, but can all be imported fromvera.base_models. For example, to customize theEventmodel, subclassBaseEventand update yoursettings.py:# myapp/models.pyfromdjango.dbimportmodelsfromvera.base_modelsimportBaseEventclassEvent(BaseEvent):date=models.DateTimeField()type=models.CharField()# settings.pyWQ_EVENT_MODEL="myapp.Event"Note that as with anyswappablemodel, Django migrations do not expect swappable settings to change after the initial migration. Thus, it is best to leave these settings alone once there is data in the database.Each of the seven models are described in detail below.SiteTheSitemodel represents the location where an event occured. It is not strictly a part of the original ERAV definition but is a natural extension. In the default implementation,Siteis anIdentifiedModelwithlatitudeandlongitudefields.# myapp/models.pyfromdjango.dbimportmodelsfromvera.base_modelsimportBaseSiteclassSite(BaseSite):description=models.TextField()# settings.pyWQ_SITE_MODEL="myapp.Site"All site instances have avalid_eventsproperty that returns all of the event instances that contain at least one valid report.EventTheEventmodel corresponds to theEntityin the ERAV data model.Eventrepresents a time series of monitoring events. For example, each visit a volunteer makes to an observation site could be called anEvent. TheEventmodel does not contain any metadata about the digital record describing the event. This information is in theReportmodel, discussed below.At a minimum, an Event instance has asitereference (see below) and an eventdate, which might be either a date or a full date and time, depending on project needs. The default implementation assumes a date without time. A customdatefield and additional attributes can be configured by extendingBaseEventand swapping outEventvia theWQ_EVENT_MODELsetting. Note that ifEventis swapped out,EventResultshould be as well.ReportStatusTo support custom workflows, the list of report statuses is maintained as a separate model,ReportStatus.ReportStatusextendsIdentifiedModelwith anis_validboolean indicating whether reports with that status should be considered valid. Additional attributes can be added by extendingBaseReportStatusand swapping outReportStatusvia theWQ_REPORTSTATUS_MODELsetting.In a typical project, theReportStatusmodel might contain the following instances:slugnameis_validunverifiedUnverifiedFalseverifiedVerifiedTruedeletedDeletedFalseReportTheReportmodel corresponds to theRecordin the ERAV data model.Reporttracks the provenance metadata about theEvent, e.g. who entered it, when it was entered, etc. Depending on when and how data is entered, there can be multipleReportsdescribing the same event. The status of each of these reports is tracked separately.At a minimum,Reportinstances have aneventattribute, astatusattribute (see below), auserattribute, and anenteredtimestamp.userandenteredare set automatically when a report is created via theREST API. Additional attributes can be added by extendingBaseReportand swapping outReportvia theWQ_REPORT_MODELsetting. Note that theReportmodel contains only provenance metadata and no information about the event itself - theEventmodel should contain that information.In addition to the default manager (objects),Reportalso has a custom manager,valid_objectsthat includes only reports with valid statuses.Reportinstances have avalsproperty that can be used to retrieve (and set) adictmapping of parameter names to result values (see below).In cases where there are more than one valid report for an event, there may be an ambiguity if reports contain contradicting data. In this case theWQ_VALID_REPORT_ORDERsetting can be used control which reports are given priority. The default setting is("-entered",), which gives priority to the most recently entered reports. (See theCSCW paperfor an in depth discussion of conflicting reports).ParameterTheParametermodel corresponds to theAttributein the ERAV data model.Parametermanages the definitions of the data “attributes” (or “characteristics”, or “fields”) being tracked by the project. By keeping these definitions in a separate table, the project can adapt to new task definitions without needing a developer add columns to the database.BaseParameterextendsIdentifiedModelwithis_numericboolean, and aunitsdefinition (which usually only applies to numeric parameters). Additional attributes can be added by extendingBaseParameterand swapping outParametervia theWQ_PARAMETER_MODELsetting.ResultTheResultmodel corresponds to theValuein the ERAV data model.Resultmanages the definitions of the data attributes (or characteristics, or fields) being tracked by the project.Resultis effectively a many-to-many relationship linkingReportandParameterwith a value: e.g. “Report #123 has a Temperature value of 15”. Note thatResultdoes not have a foreign key pointing toEventdirectly - this is a core distinction of theERAVmodel.At a minimum,Resultinstances have atype(which referencesParameter), areport, andvalue_textandvalue_numericfields - usually only one of which is set for a givenResult, depending on theis_numericproperty of theParameter.Resultinstances also contain anemptyproperty to facilitate fast filtering during analysis (see below). Additional attributes and custom behavior can be added by extendingBaseResultand swapping outResultvia theWQ_RESULT_MODELsetting. Note that ifResultis swapped out,EventResultshould be as well.Resultinstances have a settablevalueattribute which is internally mapped to thevalue_textorvalue_numericproperties depending on theParameter.Resultinstances also have anis_empty(val)method which is used to set theemptyproperty. The default implementation countsNone, empty strings, and strings containing only whitespace as empty.EventResultTheEventResultmodel is adenormalizedtable containing data from the “active” results for all valid events. A valid event is simply an event with at least one report with anis_validReportStatus. To determine which results are active:First, all of the results are collected from all of the valid reports for each event. Only non-empty results are included.Next, results are grouped by parameter. There can only be one active result per parameter.Within each parameter group, the results are sorted byReport, using theWQ_VALID_REPORT_ORDERsetting. The first result in each group is the “active” result for that group.(This is not exactly how the algorithm is implemented, but gives an idea of how it works)In the simple case, where there is only one validReportfor an event, all of theResultinstances from thatReportwill be counted as active. In more complex situations, someResultinstances might be occluded.Since this algorithm can be computationally expensive, the results are stored in theEventResultmodel for fast retrieval. TheEventResultmodel should never be modified directly, as it is updated automatically whenever anEvent,Report, orResultis updated.TheEventResultmodel contains aneventattribute, aresultattribute, and all of the fields from bothEventandResult(prefixed with the source model name). The full set of fields for the defaultEventResultmodel isevent,result,event_site,event_date,result_type,result_report,result_value_numeric,result_value_text, andresult_empty.WheneverEventorResultare swapped out,EventResultshould be swapped as well. Thecreate_eventresult_model()function can be used to generate anEventResultclass without needing to manually duplicate all of the field definitions.# myapp/models.pyfromdjango.dbimportmodelsfromvera.base_modelsimportBaseEvent,ResultclassEvent(BaseEvent):date=models.DateTimeField()type=models.CharField()EventResult=create_eventresult_model(Event,Result)# settings.pyWQ_EVENT_MODEL="myapp.Event"WQ_EVENTRESULT_MODEL="myapp.EventResult"Data ManagementData Entryvera is designed for use with thewq framework, which can automatically generate offline-capable data entry forms for theSite,Parameter,ReportStatus, andReportmodels. TheEvent,Result, andEventResultmodels are not meant to be edited directly, as they are populated when aReportform is submitted. The defaultreport_edittemplate can be customized for a more compact layout. For example, see theTry WQ report_edit templateand thewqxwq report_edit template.Bulk Data Importvera includes built-in support for importing data from Excel and other spreadsheet formats via theDjango Data Wizard. Four default wizard templates (serializers) are provided, as shown in the screenshot below.Serializer Choices: Site Metadata, Parameter Metadata, Report Series, and Result SeriesBoth the Report Series and Result Series serializers are used to import timeseries data (simultaniously populating the Event, Report, and Result tables). The difference between Report Series and Result Series is that the former assumes parameter names are listed as columns across the top of the spreadsheet (as in the screenshot below), while the latter assumes each row lists a single parameter and a single result.Report Series column mappingBulk Export and Interactive Chartingvera also ships with anEventResultSerializerand views that leverageDjango REST Pandas’ charting serializers. This makes it possible to quickly generate d3.js charts from theEventResulttable viawq/chartapp.jsor the underlying modules (wq/chart.jsandwq/pandas.js). The providedTimeSeriesView,ScatterView, andBoxPlotViewimplementidentifyURL filtering, meaning you can filter bySiteand/orParameterby adding additional slugs to the URL.For example, with the following URL configuration:# myproject/urls.pyfromvera.results.viewsimportTimeSeriesViewurlpatterns=[url(r'^data/(?P<ids>[^\.]+)/timeseries$',cls.as_view())]The following requests would be possible:URL PathOutput/data/st ream1/temp /timeserie sHTML table and
interactivewq/chartapp.j
schart showing
EventResult
values for the
Parameter"temp"at
the Site"stream1"/data/st ream1/temp /timeserie s.csvCSV export of
the same/data/st ream1/lake 2/timeseri es.csvCSV export for
all values
from Sites"stream1"and"lake2"[1]Sheppard, S. Andrew, Andrea Wiggins, and Loren Terveen. “Capturing Quality: Retaining Provenance for Curated Volunteer Monitoring Data.” In Proceedings of the 17th ACM conference on Computer Supported Cooperative Work & Social Computing (CSCW 2014), pp. 1234-1245. ACM, 2014.