Package: pyniva
Python wrapper for NIVA's Data Platformpynivais a Python library and API wrapper(s) for programatic access to data
in NIVA's data platform and services.Currently the following parts/APIs are supported:NIVA Flow (metaflowfor meta data access)tsbfor access to time series data (including FerryBox)Getting accessThe API is protected with API tokens. Please contactcloud@niva.noto request access.When provided with a token, all pyniva invokations need to include a header object:from pyniva import token2header

header = token2header("/some/folder/containing/token/niva-service-account.json")Please make sure that the token is not shared. In case of data breach or lost token, please contact us and we will invalidate the token and generate a new one.InstallationThe package and all it's dependencies can be installed usingpippip install pynivaContactIf you have any questions or feedback, please contact us atcloud@niva.noGeneral informationThe external APIs usesjwtfor user authentication and secure
data transfer. In order to use these you'll need to get a JWT access token (JSON file),
which contains account information and a private ssh key.Note that the APIs are intended for interactive use, where the user fetches and
search for meta data and available data sets, and then query the actual data.Also note that for time series, the API (tsb) is built to support interactive
use and visualization by doing server side  aggregating of the data (including
aggregation of data on (asynchronous) GPS tracks).
This means that the consumer should in general avoid using the API to download all raw data
for client side aggregation etc.Internally, all objects, including time-series (signals), are identified and queried usingUUIDs.
Thepynivalibrary wraps and hides this from the end user, allowing access to
and querying of data objects through Python object instances.All timestamps returned from the APIs are in UTC.
All the endpoints return data in JSON format.metaflow - meta data and "Thing" universemetaflowis NIVA's service and API for meta data. This service
allows for searching meta data, and retrieving all stored meta data
for data objects ("Things").Themetaflowservice stores all meta-data as JSON documents with
a set of specific reserved fields, includingttypewhich is
used to identify the object type of the meta data. These
object types is mapped directly to the Thing objects exposed
in thepynivaAPI wrapper.Inmetaflowand in thepynivaAPI-wrapper all data objects are
represented asThingclass instances or a subclass ofThing.
Using thepynivawrapper search and detailed meta data is also
available throughThingclasses (see examples bellow).The "Thing universe" is a simple hierarchy of classes,
with the generic "Thing" as base class. Inpynivaall objects
and data are represented as "Thing" instances. Access to meta-data
and data is provided through class and instance methods of "Things"
(including query and retrieval of time-series).Currently the following classes are implemented and supported by
themetaflowback-end:Thing (ttype = 'thing')
  |
  |-> Platform (ttype = 'platform')
  |   |
  |   |-> Vessel (ttype = 'vessel')
  |
  |-> Component (ttype = 'component')
  |  |
  |  |-> Sensor (ttype = 'sensor')
  |
  |-> TimeSeriers (ttype = 'tseries')
     |
     |-> FlagTimeSeries (ttype = 'qctseries')
     |
     |-> GPSTrack (ttype = 'gpstrack')The type of an object is defined by thettypeattribute of
an instance (i.e. thettypeattribute of the underlying
JSON document), and it is straightforward to extend the data model
with new types and functionality.All public methods inpynivahas informative docstrings.pynivaalso exposes/includes URLs to publicmetaflowendpoints.PUB_META (public endpoints to get meta data)Getting Things from 'metaflow'The following class methods will search and/or fetch meta-data
from themetaflowand return the data as a Thing instance or
a list of Thing instances.Thing.get_thing(meta_endpoint, header=header, params=params, **kwargs)Thing.list(meta_endpoint, header=header, params=params, **kwargs)Arguments can be passed as a parameter dictionary and/or as keyword arguments.frompynivaimportVessel,PUB_METAvessel=Vessel.get_thing(meta_host=PUB_META,header=header,params={"path":"FA"})is equivalent tofrompynivaimportVessel,PUB_METAvessel=Vessel.get_thing(PUB_META,header=header,path="FA")Getting meta data and domain modelTo get the full domain model of a Thing instance use thething.get_tree()instance
method:frompynivaimportThing,PUB_METAthing=Thing.get_thing(PUB_META,header=header,path="FA")print(thing.path)thing_with_children=thing.get_tree(PUB_META,header=header)forpartinthing_with_children.parts:# access each part/children of the thing:print(part.path)This will print the following:FAFA/ferrybox
FA/PCO2
FA/gpstrack
FA/PH
FA/GPSNote that thethinginstance have to be present inmetaflow.Examples# Get list of available vessels in metaflow, print their names# and the number of avaliable time series for each vesselfrompynivaimportVessel,PUB_META,token2headerheader=token2header("path/to/my/tokenfile.json")# Get list of all available vesselsvessel_list=Vessel.list(PUB_META,header=header)forvinvessel_list:time_series=v.get_all_tseries(PUB_META,header=header)print(v.name,len(time_series))# Get the full domain model for the first vessel in the listvessel=vessel_list[0]vessel_full=vessel.get_tree(PUB_META,header=header)To get all available meta data for aThing(or subclass) instance
you can call theto_dict()method which will return all meta data
as a Python dictionary.print(v.as_dict())Objects in theThinghierarchy will have different attributes,
depending on type, etc. For more information take a look at the
doc-string of theThinginstances you are interested in or use the
Pythondir()method to examine the object data. In general, any
key not starting with an_in the dictionary returned byto_dict()is also available as instance attributes.
AllThingor subclass ofThinginstances
persisted bymetaflowwill at least have anuuidand attypeattribute.tsb - time series dataTime series data is stored in designated time series database(s) and
the actual data can be accessed throughtsbservice.pynivaexposes/includes URLs to publictsbendpoints.PUB_TSB (public endpoint for time series data)ThetsbAPI is intended for interactive use, visualization, data
"drill in" and merging of asynchronous-heterogenous time series data.
Including merging of data on GPS tracks.
This means that the typical use of the API isnotto download all
avaliable raw data (which can be huge), instead the user will query
and fetches aggregated data for a given time interval (default is
approximately 1000 data-points pr. time series).
The default number of data points can be overridden by setting
thenordtparameter in the query.dt=0returns raw data
(not reccomended for large datasets).Using thepynivalibrary you can access and query data throughTimeSeriesclass (or subclasses) orTimeSeriesinstances.
This allows direct access to the data
while hiding the details of the underlyingtsbservice.
When querying throughpynivadata is returned as time indexedPandaswhich is convenient for further
analyses, plotting, data export, etc.Thetsbsystem holds and handles three kinds of asynchron time series:"normal" time series (TimeSeriesclass),
which is a time indexes sequence of single numerical
(floating point) values, i.e. one numerical value for each time stamp,
for this datatype there can also be a quality flag for each measurement.
This flag will typically be -1 for "bad quality",
0 for quality flag not set, or +1 for "good quality". When querying data
you can filter on the flag (but the actual flags are not returned).Flags and/or event data time series (FlagTimeSeriesclass), implemented as a
time indexed sequence of integers. This datatype is also used for
individual data quality flags. For this datatype the standard aggregation
type ismodewhich returns the most frequent value in the interval.GPS tracks (GPSTrackclass), which is a time indexed sequence of
longitude and latitude values (WGS84). GPS tracks can be used for geo-fencing
and they are aggregaetd by keeping actual data at (near) wanted time intervals.
Note that if aGPSTrackis in the query list data will be merged with the track
and the aggregation intervals will be dictated by the data in theGPSTrackTheTimeSeriesclass has two methods for quering time series data:get_tseries()(instance method) to queries and fetch data corresponding to
the instance in question.get_timeseries_list()(class method) which takes a list ofTimeSeriesinstances and return a joined dataset for the time series.Query parametersThese interfaces takes and requiers the same set of parameters. The following
parameters must be included:ts_host, url for thetsbservice (in practice this will bePUB_SIGNAL)headers, aJWTheader must also be included
(for documentation seetoken2header()documentation above)a time rangefor the queryThere are two ways to specify the time span The parameters used to set the time range in a query are:start_timeandend_time(start and end time of query)ts(time span of query)All timestamps and time spans are assumed to be ISO8601 formatted string, with
one exception:end_time=nowwhich will force end-time intodatetime.utcnow()Time intervals can be expressed in several ways with a combination of the three parameters:As an ISO8601 time interval ("ts") parameter with start and end time.
Examples:ts=2007-03-01T13:00:00Z/2008-05-11T15:30:00Zts=P1Y2M10DT2H30M/2008-05-11T15:30:00ZAs explicit start and end parameters (ISO8601 formatted)
Example:start_time=2017-01-01T00:10:10.82812end_time=2017-02-01T10:21:33.15As a time interval parameter ("ts") and either a corresponding
"start" or "end" parameter or implicit end=now by omitting
start/end parameters.
Example:ts=PT1H10M10.03Send=2013-10-12T10Example:ts=P1M(one month ending now)Also note that the API has the following default behavior:If start and end parameters are both given any given "ts" parameter will be ignoredIf no parameters are given the function will return one week ending nowIf only a time span (without start or end) are given end time is set to nowOptional parametersIn addition the API support the following additional parameters:n(integer): approximate number of data-points to return from the querydt, time interval in aggregation. Must be either number of seconds or
isodate durationISO8601(note: "P1M" would refer to 1 month interval while "PT1M" would refer to 1 minute interval).
Also note that the API don't guarantee that the returned time spans will match the requested string, it will just try to
match it as close as possible with a valid Timescaletime aggregation string.agg_type, aggregation type, possible values:
"avg" (default), "min", "max", "sum", "count", "stddev", "mode", "median" and "percentile"percentileif agg_type is percentile the API also requires this parameter to be
set, floating point number between 0 and 1noqc(flag, true if included): flag to ignore the Data Quality flag in the query.
If not included only data which has passed the data quality check will be returned.region(WKT string): Only return data from inside a given geographical region.
The argument must be a region (polygon) defined as aWKTstring where the
coordinates are assumed to be inWGS84format.
Also note: if a region is supplied the querymust also includea uuid for an existing GPS track.Notes and chevatsIf data is completely missing for a signal the returned DataFrame will not
include the corresponding column.If the all data is missing in a query, an empty DataFrame will be returned.Time aggregation: You are not guarantied to get the exact time spans asked for
the server will try to match the requested time windows with 1, 2, 5, 10 or 30
multiples of seconds, minutes, hours and days. And return the nearest match.
Raw data is returned ifdt=0is set.Geo fencing: Since only data from withing a particular geographical region is
returned in the query, you must include a GPSTrack (i.e. the vessel's GPS-track)
in order to receive data. Also the time spans in the time aggregation will
match the actual time stamps in the GPS track signal.For normal time series queries arefiltered on the data qualityflag,
meaning that only data points which has passed QC is included in the
returned result. This behavior can be overridden using thenoqcquery flag.If a GPS-track is included in the querydata is merged with the track,
and the GPS-track data is returned aslongitudeandlatitude.Only one GPS-track can be submitted at the timeAggregation level is forced to the GPS-track, with actual
GPS-track time stamps.