Package: niimpy
NiimpyWhatNiimpy is a Python package for analyzing and quantifying behavioral data. It uses pandas to read data from disk, perform basic manipulations, provides explorative data analysis functions, offers many high-level preprocessing functions for various types of data, and has functions for behavioral data analysis.For WhoNiimpy is intended for researchers and data scientists analyzing digital digital behavioral data. Its purpose is to facilitate data analysis by providing a standardized replicable workflow.WhyDigital behavioral studies using personal digital devices typically produce rich multi-sensor longitudinal datasets of mixed data types. Analyzing such data requires multidisciplinary expertise and software designed for the purpose. Currently, no standardized workflow or tools exist to analyze such data sets. The analysis requires domain knowledge in multiple fields and programming expertise. Niimpy package is specifically designed to analyze longitudinal, multimodal behavioral data. Niimpy is a user-friendly open-source package that can be easily expanded and adapted to specific research requirements. The toolbox facilitates the analysis phase by providing tools for data management, preprocessing, feature extraction, and visualization. The more advanced analysis methods will be incorporated into the toolbox in the future.HowThe toolbox is divided into four layers by functionality: 1) reading, 2) preprocessing, 3) exploration, and 4) analysis. For more information about the layers, refer the toolbox architecture chapter :doc:architecture. Quickstart guide would be a good place to start :doc:quick-start. More detailed demo Jupyter notebooks are provided in user guide chapter :doc:demo_notebooks/Exploration. Instructions for individual functions can be found under API chapter :doc:api/niimpy.InstallationOnly supports Python 3 (tested on 3.5 and above)This is a normal Python package to install.pip install niimpyIt can also be installed manually:pip install https://github.com/digitraceslab/niimpy/archive/master.zipGetting started with location dataAll of the functions for reading, preprocessing, and feature extraction for location data is inlocation.py. Currently implemented features are:dist_total: total distance a person traveled in meter.variance,log_variance: variance is defined as sum of variance in latitudes and longitudes.speed_average,speed_variance, andspeed_max: statistics of speed (m/s). Speed, if not given, can be calculated by dividing the distance between two consequitive bins by their time difference.n_bins: number of location bins that a user recorded in dataset.n_static: number of static points. Static points are defined as bins whose speed is lower than a threshold.n_moving: number of moving points. Equivalent ton_bins - n_static.n_home: number of static bins which are close to the person's home. Home is defined the place most visited during nights. More formally, all the locations recorded during 12 Am and 6 AM are clusterd and the center of largest cluster is assumed to be home.max_dist_home: maximum distance from home.n_sps: number of significant places. All of the static bins are clusterd using DBSCAN algorithm. Each cluster represents a Signicant Place (SP) for a user.n_rare: number of rarely visited (referred as outliers in DBSCAN).n_transitions: number of transitions between significant places.n_top1,n_top2,n_top3,n_top4,n_top5: number of bins in the topNcluster. In other words,n_top1shows the number of times the person has visited the most freqently visited place.entropy,normalized_entropy: entropy of time spent in clusters. Normalized entropy is the entropy divided by the number of clusters.Usage:importpandasaspdimportniimpyimportniimpy.locationasniloCONTROL_PATH="PATH/TO/CONTROL/DATA"PATIENT_PATH="PATH/TO/PATIENT/DATA"# Read data of control and patients from databaselocation_control=niimpy.read_sqlite(CONTROL_PATH,table='AwareLocation',add_group='control',tz='Europe/Helsinki')location_patient=niimpy.read_sqlite(PATIENT_PATH,table='AwareLocation',add_group='patient',tz='Europe/Helsinki')# Concatenate the two dataframes to have one dataframelocation=pd.concat([location_control,location_patient])# Remove low-quality and outlier locationslocation=nilo.filter_location(location)# Downsample locations (median filter). Bin size is 10 minute.location=niimpy.util.aggregate(location,freq='10T',method_numerical='median')location=location.reset_index(0).dropna()# Feature extractionfeatures=nilo.extract_features(lats=location['double_latitude'],lons=location['double_longitude'],users=location['user'],groups=location['group'],times=location.index,speeds=location['double_speed'])DocumentationNiimpy documentation is hosted at [readthedocs]https://niimpy.readthedocs.io/en/latest/.DevelopmentThis is a pretty typical Python project with code and documentation as
you might expect.requirements-dev.txtcontains some basic dev requirements, which
includes a editable dev install of niimpy itself (pip install -e).Run tests with:pytest .Documentation is built with Sphinx:cd docs
make html
# output in _build/html/Enable nbdime Jupyter notebook diff and merge via git with:nbdime config-git --enableSee alsoTo learn about pandas, see its documentation.  It isnotthe most
clearly written documentation you will find, but you should try
starting with the "Package overview" and "10 minutes to pandas"
sections.Matplotlibis the standard Python
plotting package, butSeabornwill
produce nicer graphics by default.  Hint: look for examples and copy
them.