Package: onavdata
Organic Navigation Reference DataTheonavdatamodule provides easy access to bothreference datasetsanduser-specified datasetsfor the design, development, and testing of navigation algorithms.  Additionally, usefulutilitiesare included for working with common navigation sensors.Reference datasetsare well understood, documented, and purged of data logging artifacts.  A valuable set of reference datasets are included withonavdata.  They are well-understood yet realistic datasets, making them useful for testing a data processing script or candidate algorithm.User-specified datasetsare managed and made available via the sameonavdatainterface as the reference datasets, with the exception that these datasets are added by the user.  Thus, the transition from reference to experimental datasets is relatively smooth.Utilitiesare included for facilitating common unit conversions or getting data columns for common navigation sensors.The primary data structure used byonavdatais thePandas DataFrame.  A basic familiarity with DataFrames is recommended (e.g. one-liners to down-sample, average, plot, or save to CSV).  All configuration or meta data files use the easily understoodTOMLformat.Quick StartList Data AvailableAll data sets are referenced using their shortname.>>importonavdata>>onavdata.print_shortnames()2011UMN-UAVGPSFASER12012UMN-UAVFASER52012UMN-UAVGPSFASER32012UMN-UAVTHOR602012UMN-UAVTHOR752012UMN-UAVTHOR772012UMN-UAVTHOR792021-04-27HGuiden380CarDriveBetweenTrafficLights2021-04-27HGuiden380CarHillDrivingSIM-MEASURED-CARNORTHFIXED-SPEEDFIXED-HEADINGSIM-MEASURED-CARNORTHFIXED-SPEEDVARY-HEADINGSIM-MEASURED-CARNORTHVARY-SPEEDFIXED-HEADINGSIM-MEASURED-CARNORTHVARY-SPEEDVARY-HEADINGSIM-PERFECT-CARNORTHFIXED-SPEEDFIXED-HEADINGSIM-PERFECT-CARNORTHFIXED-SPEEDVARY-HEADINGSIM-PERFECT-CARNORTHVARY-SPEEDFIXED-HEADINGSIM-PERFECT-CARNORTHVARY-SPEEDVARY-HEADINGLoad a Reference Data SetAn arbitrary data set can be loaded if no shortname is specified.  In this case the data from2014 UMN-UAV THOR77was returned.>>importonavdata>>df=onavdata.get_data()get_data():Noshortnamespecifiedsochoicewillbearbitrary.Returning:2014UMN-UAVTHOR77>>df.head()AccelX(m/s^2)AccelY(m/s^2)AccelZ(m/s^2)...AngleHeading(rad)AnglePitch(rad)AngleRoll(rad)TimeFromStart(s)...0.001.862795-0.392167-9.673463...1.0047000.1928060.0061300.021.895476-0.424848-9.640783...1.0042250.1928560.0061300.041.895476-0.359487-9.706145...1.0037860.1928880.0062590.061.895476-0.392167-9.640783...1.0032580.1929900.0063540.081.895476-0.392167-9.640783...1.0029070.1930560.006482[5rowsx12columns]Load User-Specified DatasetsAdd Dataset FolderTheexamples/simple_data_load.pyfile shows an example of adding a user-specified dataset using theadd_dataset()method.  There are two main requirements:Path to a file namedshortnames.txt, populated with a suitable entries for each dataset. Namely:Ashortnamefor the datasetPath to dataset file(s)Path to an associated meta data file for the dataset (more on this below)Informingonavdataof the user-specified datasets by using theonavdata.add_dataset()function.Multipleshortnames.txtcan be used (e.g. using one per project).  Assuming no conflicting shortnames, there is no issue with adding multiple user-specified datasets.Directly Load DatasetTheexamples/simple_data_load_direct.pyfile shows how it is possible to directly load a user-specified dataset.  The two requirements are:The path (or list of paths) to the datasetA meta data dictionaryThis is advantages for programmatically usingonavdata.Apply Meta to DataFrameTheexamples/apply_meta_to_dataframe.pyfile shows how a dataset meta (see description below) can be applied to an existing dataframe.  This is useful for programmatically transforming a loaded dataset using meta parameters.Dataset Meta DataEach dataset must have an associated meta (data) file.  The meta file contains parameters necessary to pre-process and load the dataset in acleanstate.  Steps like adding a time-index, renaming columns, rotating axes, dropping rows where data is unreliable... all of these and more are made possibly by parameter specifications in the meta file.  Hence, the data loading process handles the steps to clean a raw dataset.  The goal is for experimental user-specified datasets to have similar quality (e.g. remove unreliable transients) and attributes (e.g. columns names) as reference datasets.Note that more than one dataset can point to the same meta file.  This is commonly done when multiple datasets are collected using the same device or setup.Sample Meta Data File FormatBelow is an example meta file with all possible parameters demonstrated.  Comment lines begin with the#symbol.  In practice, only the parameters needed should be included.sample_meta.toml:# Define Sampling Time.# Checks for a `TimeFromStart (s)` column (after column renaming) or uses the# specified sampling freq. The latter is only used if the former doesn't exist.# 'SampleFreq (Hz)' = 200# Rows to skip before loading dataset (e.g. multi-line header).'skip-lines'=0# Option to drop unnamed columns.'drop-unnamed'=True# Specify list of columns to load (or list of columns to drop `columns-drop`).'columns-use'=['Time (s)','Ax','Ay','Az','GNSSNumSV','GNSSFix']# 'columns-drop' = ['Empty Channel 1']# Drop rows if column has a matching entry with a list of values.[drop-rows-where]'GNSSNumSV'=[0]# Only keep rows if column has a matching entry with a list of values.[keep-rows-where]'GNSSFix'=[1]# Only keep rows where column has entries GREATER than, LESS than, or BETWEEN.[keep-rows-gt]'Time (s)'=5[keep-rows-lt]'Time (s)'=65[keep-rows-between]'Az'=[-1,1]# Override default dtypes# The dtype names must be acceptable to pd.DataFrame.astype() function.[columns-dtype]'GNSSNumSV'='int32'# Round columns to decimals specified.[columns-round-decimals]'Ax'=3'Ay'=3'Az'=3# Subtract offset for specific columns (e.g. remove sensor bias).[columns-subtract-offset]'Ax'=0.23'Ay'=0.01'Az'=-0.1# Scale data for specific columns (e.g. change units).[columns-scale]'Ax'=9.81'Ay'=9.81'Az'=9.81# Rename columns: 'old name' = 'new name'[columns-rename]'Time (s)'='TimeFromStart (s)''Ax'='AccelX (m/s^2)''Ay'='AccelY (m/s^2)''Az'='AccelZ (m/s^2)'# Apply rotations to groups of columns which start with key.# Notes:#   Columns found are sorted so as to get order: X->Y->Z#   Key entries should refer to the renamed column names[Rsensor2body]'Accel'=[[0,-1,0],[-1,0,0],[ 0,  0, -1]]# The columns with the `Accel` prefix will be found and the defined rotation# matrix applied.# Add additional columns with constant values, or by copying another column.[add-columns]'AnglePitch (rad)'=0.0'Temperature (C)'=28.0'No. of Satellites'='GNSSNumSV'