Package: deepethogram
DeepEthogramWritten by Jim Bohnslav, except where as notedJBohnslav@gmail.comDeepEthogram is an open-source package for automatically classifying each frame of a video into a set of pre-defined
behaviors. Designed for neuroscience research, it could be used in any scenario where you need to detect actions from
each frame of a video.Example use cases:Measuring itching or scratching behaviors to assess the differences between wild-type and mutant animalsMeasuring the amount of time animals spend courting, and comparing between experimental conditionsCounting licks from video for appetite measurementMeasuring reach onset times for alignment with neural activityDeepEthogram uses state-of-the-art algorithms fortemporal action detection. We build on the following previous machine
learning research into action detection:Hidden Two-Stream Convolutional Networks for Action RecognitionTemporal Gaussian Mixture Layer for VideosInstallationFor full installation instructions, seethis readme file.In brief:Install PyTorchpip install deepethogramDataNEW!All datasets collected and annotated by the DeepEthogram authors are now available from this DropBox link:https://www.dropbox.com/sh/3lilfob0sz21och/AABv8o8KhhRQhYCMNu0ilR8wa?dl=0If you have issues downloading the data, please raise an issue on Github.COLABI've written a Colab notebook that shows how to upload your data and train models. You can also use this if you don't
have access to a decent GPU.To use it, pleaseclick this link to the Colab notebook.
Then, clickcopy to Driveat the top. You won't be able to save your changes to the notebook as-is.NewsWe now support docker! Docker is a way to rundeepethogramin completely reproducible environments, without interacting
with other system dependencies.See docs/Docker for more informationPretrained modelsRather than start from scratch, we will start with model weights pretrained on the Kinetics700 dataset. Go to
To download the pretrained weights, please usethis Google Drive link.
Unzip the files in yourproject/modelsdirectory. Make sure that you don't add an extra directory when unzipping! The path should beyour_project/models/pretrained_models/{models 1:6}, notyour_project/models/pretrained_models/pretrained_models/{models1:6}.LicensingCopyright (c) 2020 - President and Fellows of Harvard College. All rights reserved.This software is free for academic use. For commercial use, please contact the Harvard Office of Technology
Development (hms_otd@harvard.edu) with cc to Dr. Chris Harvey. For details, seelicense.txt.UsageTo use the GUI, clickTo use the command line interface, clickDependenciesThe major dependencies for DeepEthogram are as follows:pytorch, torchvision: all the neural networks, training, and inference pipelines were written in PyTorchpytorch-lightning: for nice model training base classeskornia: for GPU-based image augmentationspyside2: for the GUIopencv: for video and image reading and writingopencv_transforms: for fast image augmentationscikit-learn, scipy: for binary classification metricsmatplotlib: plotting metrics and neural network outputspandas: reading and writing CSVsh5py: saving inference outputs as HDF5 filesomegaconf: for smoothly integrating configuration files and command line inputstqdm: for nice progress barsHardware requirementsFor GUI usage, we expect that the users will be working on a local workstation with a good NVIDIA graphics card. For training via a cluster, you can use the command line interface.CPU: 4 cores or more for parallel data loadingHard Drive: SSD at minimum, NVMe drive is better.GPU: DeepEthogram speed is directly related to GPU performance. An NVIDIA GPU is absolutely required, as PyTorch uses
CUDA, while AMD does not.
The more VRAM you have, the more data you can fit in one batch, which generally increases performance. a
I'd recommend 6GB VRAM at absolute minimum. 8GB is better, with 10+ GB preferred.
Recommended GPUs:RTX 3090,RTX 3080,Titan RTX,2080 Ti,2080 super,2080,1080 Ti,2070 super,2070Some older ones might also be fine, like a1080or even1070 Ti/1070.testingTest coverage is still low, but in the future we will be expanding our unit tests.First, download a copy oftesting_deepethogram_archive.zipMake a directory in tests calledDATA. Unzip this and move it to thedeepethogram/tests/DATAdirectory, so that the path isdeepethogram/tests/DATA/testing_deepethogram_archive/{DATA,models,project_config.yaml}. Then runpytest tests/to run.
thezz_commandlinetest module will take a few minutes, as it is an end-to-end test that performs model training
and inference. Its name reflects the fact that it should come last in testing.Changelog0.1.4: bugfixes for dependencies; added docker0.1.2/3: fixes for multiclass (not multilabel) training0.1.1.post1/2: batch prediction0.1.1.post0: flow generator metric bug fix0.1.1: bug fixes0.1: deepethogram beta! See above for details.0.0.1.post1: bug fixes and video conversion scripts added0.0.1: initial version