Package: openlabcluster
OpenLabClusterActive Learning Based Clustering and Classification of Animal Behaviors in Videos Based on Automatically Extracted Kinematic Body KeypointsUsageInstallationInstall from pipCreate a new environment with conda and install the packageForLinuxstart the environment withspec-file.txtconda create --name OpenLabCluster --file spec-file.txt
conda activate OpenLabCluster
pip install openlabclusterforMac OSandWindowsconda create --name OpenLabCluster python=3.7
conda activate OpenLabCluster
pip install 'openlabcluster[gui]'ExecutionRun the following forLinuxpython -m openlabclusterRun the following forMac-OSconda install python.app
pythonw -m openlabclusterRun the following forWindowspythonw.exe -m openlabclusterRun a DemoCreate the Demo ProjectDownload theexamplefolder fromhereincluding the extracted keypoints of Home-Cage Mouse dataset (fromAutomated home-cage behavioural phenotyping of miceHueihan Jhuang et al., 2010) and follow the instructions in the README file in the example folder.Go toyour_download-dir/examplefolder runpython3 prepare_video_list.pywhich generates video_segments_names.text file.Launch OpenLabCluster GUI (see Execution above)Set Project Name: e.g., demoClickLoad Preprocessed Keypoints, choose datafile: your_download_dir/openlabcluster_example/demo_data.h5ClickLoad Video Segments Name List, choose the file: your_download_dir/openlabcluster_example/video_segments_names.textUncheckCheck to use GPU, if GPU is not available.Set Feature Length = 16ClickOKto create the projectStart the Demo Project:Go toManage Projectpanel,ChooseLoad ProjectSelect the config file asfiledirectory/project_name/config.yamlClickOK, then go toCluster MapCluster Map:ClickStart Clusteringbutton to start unsupervised clustering.ClickGo To Classificationwhen unsupervised clustering is finished, then go toBehavior Classification Mappanel.Behavior Classification Map:The scatter plot indicating sample clusters is initialized on the bottom left, with suggested samples for annotation.Label samples on the bottom right panel.ClickRun Classification, and start classification.Manage Project (Start a New Project or Load Earlier Project) Detailed DescriptionStart a New ProjectProject Name - the name of the projectIf you have only videos, use markerless pose estimators (e.g., DeepLabCut) to extract keypoints for each video. If you already have DeepLabCut-like formatted files, select them with "Load Keypoints Data."List video names in the "Load Video Segments Names List" file. There should be one video for each keypoint file. Make sure the videos and keypoints files are in the same order.Optional: Set a directory for the project (the default is the working directory of this project).Keep the GPU box checked if you have a GPU on your computer and you would like to use it for training.Enter the features length (number of body parts * number of dimensions per body part. For example, 5 keypoints in 2D would be 5*2 =10).Choose "OK" to create the project.If you want to edit the config (i.e., change class names, change video cropping), press "Edit Config File."Loading a ProjectSelect the config.yaml file generated when you created that project.Press "OK."Cluster Map (Unsupervised Learning)Set the Training ParametersUpdate Cluster Map Every (Epochs): The frequency to update the Cluster Map (bottom left panel in the figure), e.g., set 1 to update Cluster Map every training epoch, or 5 to update every 5 epochs.Save Cluster Map Every (Epochs): The frequency to save the Cluster Map.Maximum Epochs: The maximum number of epochs to be performed unsupervised training.Cluster Map Dimension: Possible choices are 2d or 3d. For 2d, the Cluster Map will be shown in two dimensions; otherwise, it will be three dimensions.Dimension Reduction Methods: Possible choices are PCA, tSNE, and UMAP. The GUI will use the chosen method to perform dimension reduction and show results in Cluster Map.ButtonsAfter setting the parameters, you can perform the analysis:Start Clustering: Perform an unsupervised sequence regeneration task.Stop Clustering: Usually, the clustering will stop when it reaches the maximum epochs, but if you want to stop at an intermediate stage, click this button.Continue Clustering: If you stopped the clustering at some stage and want to perform clustering with earlier clustering results, click this button.Go to Classification: After the unsupervised clustering, we go to the next step, which includes: i) annotation suggestion, ii) sample annotating, and iii) semi-supervised action classification with labeled samples.Behavior Classification MapSet the Training ParametersSelection Method: In this part, your selection will decide which method GUI uses to select samples for annotation. There are four possible choices: Marginal Index (MI), Core Set (CS), Cluster Center (TOP), Cluster Random (Rand), and Uniform.# Samples per Selection: the number of samples you want to label in the current selection stage. You can select or deselect samples in the behavior classification map.Maximum Epochs: The maximum epoch the network will be trained when performing the action recognition.Cluster Map Dimension: You can choose 2d or 3d. If it is 2d, the Cluster Map will be shown in two-dimensional space; otherwise, it is three-dimensional space.Dimension Reduction Method: Possible choices are PCA, tSNE, and UMAP. The GUI will use the chosen reduction method to perform dimension reduction and show results in the Cluster Map.Buttons:Run Classification: Save annotation results and train the action recognition model.Stop Classification: Stop training.Next Selection: Suggest a new set of samples based on with indicated active learning method.Notice: If you change Cluster Map Dimension or Dimension Reduction Method, click theNext Selectionto show suggested samples.Get Results: Get the Behavior Classification Map (predicted class label) from the trained model on unlabeled samples.PlotsBehavior Classification Plot:Visualization mode: The points representing each action segment is shown in black for visualization. It can be visualized in 2D or 3D with different dimension reduction methods.Annotation mode: In this mode, dots for each action segment in a different color are shown in the Behavior Classification plot (only in 2D).
Red: current sample for annotating, and the video segment is shown on the right.
Blue: the suggested samples for annotating in this iteration (deselect them by clicking the point).
Green: Samples have been annotated.Buttons:Zoom: Zoom in or zoom out the plot.Pan: Move the plot around.Videos Panel:Left panel: The corresponding video of the action segment, which is shown in the Behavior Classification Map in red.Right panel: The class name and class id. According to the video, one can label the action segment.Buttons:Previous: Load the previous video.Play: Play the video.Next: Go to the next video.AttributionOpenLabCluster code was developed in University of Washington UW NeuroAI Lab by Jingyuan Li and Moishe Keselman. OpenLabCluster interface is inspired byDeepLabCut, which code is used as a backbone for user interface panels, interaction with the back-end, logging, and visualization. For specific usage please seethird_party/deeplabcutfolder. OpenLabCluster also usesGoogle Active Learning Playgroundcode for implementation of the K-center selection method in Core-Set active learning option. For specific usage please seethird_party/kcenterfolder.Related projects and publications:OpenLabCluster@article{li2022openlabcluster,
  title={OpenLabCluster: Active Learning Based Clustering and Classification of Animal Behaviors in Videos Based on Automatically Extracted Kinematic Body Keypoints},
  author={Li, Jingyuan and Keselman, Moishe and Shlizerman, Eli},
  journal={bioRxiv},
  pages={2022--10},
  year={2022},
  publisher={Cold Spring Harbor Laboratory}
}DeepLabCut@article{Mathisetal2018,
    title = {DeepLabCut: markerless pose estimation of user-defined body parts with deep learning},
    author = {Alexander Mathis and Pranav Mamidanna and Kevin M. Cury and Taiga Abe  and Venkatesh N. Murthy and Mackenzie W. Mathis and Matthias Bethge},
    journal = {Nature Neuroscience},
    year = {2018},
    url = {https://www.nature.com/articles/s41593-018-0209-y}}DeepLabCut2.x@article{NathMathisetal2019,
    title = {Using DeepLabCut for 3D markerless pose estimation across species and behaviors},
    author = {Nath*, Tanmay and Mathis*, Alexander and Chen, An Chi and Patel, Amir and Bethge, Matthias and Mathis, Mackenzie W},
    journal = {Nature Protocols},
    year = {2019},
    url = {https://doi.org/10.1038/s41596-019-0176-0}}DeeperCut@article{insafutdinov2016eccv,
    title = {DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model},
    author = {Eldar Insafutdinov and Leonid Pishchulin and Bjoern Andres and Mykhaylo Andriluka and Bernt Schiele},
    booktitle = {ECCV'16},
    url = {http://arxiv.org/abs/1605.03170}}Google Active Learning Playground@misc{activelearningplayground,
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/google/active-learning}}}