Mouse Lockbox Dataset: Behavior Recognition of Mice Solving Mechanical Puzzles

Authors: Patrik Reiske, Marcus Nicolaas Boon, Niek Andresen, Sole Traverso, Katharina Hohlbaum, Lars Lewejohann, Christa Thöne-Reineke, Olaf Hellwich, Henning Sprekeler
Primary area: datasets and benchmarks
Keywords: dataset, video, action classification, behavior, mice, machine learning, computer vision, neuroscience, computational neuroscience

Abstract:
Machine learning and computer vision have a major impact on the study of natural animal behavior, as they enable automated action classification of large bodies of videos. Mice are the standard mammalian model system in many fields of research, but the open datasets that are currently available to refine machine learning methods mostly focus on either simple or social behaviors. In this work, we present a large video dataset of individual mice solving complex mechanical puzzles, so-called lockboxes. The dataset consists of a total of well over 110 hours of animal behavior, recorded with three cameras from different perspectives. As a benchmark for frame-level action classification methods, we provide human-annotated labels for all videos of two different mice, that equal 13% of our dataset. The used keypoint (pose) tracking-based action classification framework illustrates the challenges of automated labeling of fine-grained behaviors, such as the manipulation of objects. We hope that our work will help accelerate the advancement of automated action and behavior classification in the computational neuroscience community. An anonymized preview of our dataset is available for the reviewers of this manuscript at https://www.dropbox.com/scl/fo/h7nkai8574h23qfq9m1b2/AP4gNZOpDJJ7z0yGtbWQiOc?rlkey=w36jzxqjkghg0j0xva5zsxy2v&st=5r9msqjw&dl=0

Decision: reject
Reviewer scores: 5, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730106184647

thread_type:
official_review

<thread_object>
creator: reviewer_fe9j

<content>
summary:
This paper collected a multi-perspective video dataset of individual mice solving complex mechanical puzzles. This paper also provided the human-annotated labels, i.e., the proximity between a mouse and a mechanism. The authors give detailed statistics of playtime shares per lockbox.

strengths:
1. This paper collected a large-scale video dataset of individual mice solving complex mechanical puzzles. 
2. This paper is well-organized and easy to read.
3. This paper provides a comprehensive review of relevant work.

weaknesses:
1. This paper only collected a new dataset and ignored benchmark methods for recognizing mouse actions.
2. The comparison between the dataset collected in this paper and previous datasets is unclear. The author can provide a summary table to illustrate the differences from previous work.
3. There is no baseline or proposed method regarding the behavior recognition of mice.

questions:
1. What pose keypoint definitions were used in this work? Will different key point definitions affect recognition?
2. Are all the mice in the dataset presented in this paper under normal light conditions?
3. Does the hunger state of mice affect their ability to solve mechanical puzzles?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470764175

thread_type:
official_review

<thread_object>
creator: reviewer_jr7d

<content>
summary:
The paper introduces a new dataset in the field of computational ethology. It focuses on creating various scenarios where mouse behaviour and actions are recorded while it tries to solve complex mechanical lockbox opening problems in return for treats. The novelty of the paper lies with the fact that existing datasets in this topic focuses only on problems which are not sufficiently complex or are concerned with studying social behaviour of animals.

strengths:
•	The dataset has over 117 hours of videos of mice behaviour doing complex tasks. This multi-perspective video data is novel and crucial to enable the development of advance methods in the field of neuroscience. 
•	Action based labelling is very detailed, providing annotations at sub-second intervals.
•	The authors are very transparent in how they gathered the data and in compliance of local laws regarding to animal research. 
•	Human annotation is provided by the authors with special care taken to ensure that there is no leakage between the labelled and unlabelled set by keeping the mouses separate for labelled set. 
•	The human annotators followed a specific set of ethograms which makes the process objective in nature and lowers the possibility of disagreement.
•	The authors provide detailed statistics of their dataset

weaknesses:
•	Labelled data provided is very limited and represents only 13% of the total available data. More annotated data would be immensely helpful to researchers.
•	The dataset focuses exclusively on female mice and the reasoning behind this is not explained and the authors do not cite any paper which disproves this. This choice could limit generalizability. 
•	There is an uneven distribution between playtime action labels. If this cannot be made closer to each other, the authors should mention why it is so.
•	The dataset is provided in grayscale. While this is not a major weakness, RGB images could contain more information that is lost in the grayscale conversion.
•	The authors do not provide camera homography information of the cameras involved.

questions:
•	It would be great if the authors provided a baseline method on a task with their dataset.
•	A data dictionary of the labels as well as a getting started notebook was not provided. These would greatly increase the ease of use of the dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535457098

thread_type:
official_review

<thread_object>
creator: reviewer_562x

<content>
summary:
The paper introduces a novel video dataset of individual mice interacting with complex mechanical lockbox puzzles, labeled as the "Mouse Lockbox Dataset." It provides a significant contribution to computational neuroethology, presenting over 110 hours of video data capturing intricate, non-social behaviors. The dataset uniquely includes multi-perspective recordings, human-annotated action labels for a subset of videos, and benchmark results utilizing a keypoint-based action classification method. This work aims to support machine learning advancements in behavioral action classification, particularly in fine-grained tasks such as object manipulation.

strengths:
1. This dataset addresses a crucial gap in animal behavior research by focusing on complex, single-agent interactions in mice—particularly rare for capturing intricate, non-social behaviors.
2. Its scale and detail are impressive, with over 110 hours of footage from multiple angles. The multi-perspective setup and extensive annotations make it a highly valuable resource for training and evaluating behavior recognition models.
3. The inclusion of a keypoint-based action classification benchmark is a well-considered addition. It establishes a standard for future comparisons and highlights the strengths of current methods (e.g., proximity detection) as well as their limitations (e.g., recognizing fine-grained actions).
4. The authors provide a detailed methodology for creating the dataset, which supports reproducibility and helps ensure the reliability of the process.

weaknesses:
1. Although this dataset is a valuable resource for behavior recognition, the paper lacks clear examples of specific use cases and potential research problems it could address. The absence of concrete applications or suggested research directions may limit its utility as an inspiration for future work. Providing examples of problems or experimental scenarios where this dataset could be useful would strengthen its research impact and relevance.
2. The pseudo-synchronization of the multi-camera setup presents a minor challenge for users needing precise 3D reconstructions. While the misalignment is slight, it may still affect analyses requiring high precision.
3. Certain actions, such as biting, have lower annotator agreement, indicating challenges in reliably annotating these behaviors. This suggests that alternative or more standardized annotation methods may be necessary to improve reliability.

questions:
1. Could you clarify whether alternative pose-tracking methods were considered, particularly to improve the accuracy of detecting fine-grained actions like biting?
2. Is there a plan to expand the dataset with additional annotated videos, or are any tools being developed to help automate or verify annotation accuracy for challenging actions?
3. Given the temporal desynchronization in the multi-perspective recordings, have you considered any post-processing or alignment correction methods to reduce this effect?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714446049

thread_type:
official_review

<thread_object>
creator: reviewer_trhr

<content>
summary:
The authors propose a novel dataset, the _Mouse Lockbox Dataset_, which includes videos of individual mice dealing with mechanical puzzles named lockboxes. While existing datasets mainly target mice's social or common behaviors, the aim of the proposed dataset is to provide a useful resource for studying single-agent intelligent behaviors.
The paper describes with great detail both the setup used by the authors to record it and the procedure they established to annotate the (13% of the) dataset. The human-annotated labels are also compared with a benchmark automatic method, which relies on the poses of the mice as extracted with DeepLabCut. In particular, the authors compare the human-human agreement on the labels with the one obtained by comparing the human annotator and the automatic method performance.

strengths:
1. The paper is clear and well-written, reporting several important details on the dataset generation process. 
2. The dataset represents an interesting source for computational ethology to study the mice's behavior in controlled environments when dealing with mechanical problem-solving.
3. The ~13% of the dataset is annotated with several features that can be leveraged to effectively delve into the mice's behavior.
4. The related work section clearly reports and compares the proposed dataset with the most important datasets in rodent computational ethology literature.

weaknesses:
1. While the proposed dataset represents a novel source for studying mice behavior, the authors lack a proper experiment to showcase the relevance of their contribution. Benchmarking an automatic method against the human annotators reinforces the idea of a high-quality dataset but does not deliver to a non-ethologist reader the idea of how to use the  _Mouse Lockbox Dataset_. 
2. To complete section 2, the authors could report a table comparing their proposed dataset with the available literature described therein.
3. The method used for benchmarking is not reported in the paper, although it is cited in the introduction. To help the reader understand the importance of this baseline, the authors should introduce it in its dedicated section, reporting at least some high-level details about the structure of this baseline.
4. The introduction is fragmented (with several paragraphs) and does not focus enough on the paper's contributions. In particular, the section describes recent advancements and methods in computational ethology well, but only the last two paragraphs describe the content of the proposed paper.
5. In general, the paper would greatly benefit from introducing and evaluating some methods that can be used on the _Mouse Lockbox Dataset_ to provide a reference for future users and encourage research with it.

questions:
1. Will the key-point tracking results be released with the _Mouse Lockbox Dataset_? In the conclusive section, the authors claim, "(...) we are convinced that approaches beyond keypoint (pose) tracking, e.g., representations learnt without any or under self-supervision, are crucial to future advancements in neuroscience.", however when dealing with metric tasks, e.g., estimating the distance of the mouse from the lockbox, keypoint data represent an important piece of information. Also, to qualitatively assess the baseline performance, adding an image of the key points extracted by the baseline would be interesting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717183102

thread_type:
official_review

<thread_object>
creator: reviewer_mh4b

<content>
summary:
The authors present a new, large dataset of lab mice solving mechanical lockbox puzzles. Unlike existing animal behavior datasets, their lockboxes capture more cognitive / learning / problem solving abilities of the animals. In the future, this dataset, or the use of the lockboxes morie generally, could be used to better understand motor and cognitive learning and complex neural computations and processes. At the moment, the dataset's benchmark includes recognizing a set of actions or action.puzzle states.

strengths:
The author present an interesting and novel new dataset of mice solving puzzles. It is not 100% clear to me the extent of the research questions one could use such a dataset for, and I encourage the reviewers to expound on this in their revision.

weaknesses:
As a dataset paper, the authors should be providing additional technical details, data quality metrics, and an expanded set of methods for benchmarking. I've left specific comments on these points in the questions section.

questions:
A few things in the introduction that would be good to clarify in the text.

L55 -- you say that it is a disadvantage that poses are low dimensional. But there are many scenarios where this is advantage.
L58 -- I don't understand what you mean when you write, "It impedes mapping of keypoint locations from different perspectives into a common coordinate system."
L59-60 -- there is actually an ideal set of keypoint locations -- skeletal joints 
L107 -- "the more recent works" are actually older than some of the ones you cite in the previous sentence

Relatedly, you spend a lot of the intro saying how bad it is to use keypoints for behavioral analysis, but then go on to use keypoints for your benchmark action recognition results. What am I missing?

Please provide a clear breakdown of how much of your dataset video time is the animals engaged with the puzzle. What is your definition of "playtime"? I would prefer if these data were in the table and not in a ring/pie chart that is harder to read. 

How precise is your keypoint tracking (3D error in mm)? How many keypoints were tracked exactly? What tracking method did you use? What method/model did you use to go from keypoints to frame labels in your benchmarking? There is a lot of missing detail that is required of a datasets / benchmarks paper that the authors must provide for the work to be useful to others.

Similarly, as a dataset/benchmark paper, it is standard to show results from several different methods, or at least some internal comparisons over different model configurations/hyperparamters.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new dataset consisting of mice solving puzzles (lockboxes), captured over 40 hours from 3 camera views. There are 12 total mice in the dataset, and 2/12 has human-annotated behavior labels (6 total labels). Each video has 1 mouse, and videos have been manually trimmed to remove experimenter hands coming in/out. The authors test a baseline method on their dataset, which is to use keypoint proximity to compute behavior labels. While reviewers appreciate the introduction of a new dataset, there are concerns remaining on the experimental design, lack of benchmark models on the dataset, and some missing details on the baseline. It appears that many standard baselines and analysis are missing (non-exhaustive list of example experiments to conside: clustering using keypoints, different keypoint definitions, different keypoint estimators, different hand-crafted features, image/video embeddings -- standard ones like CLIP or DINO would work). The AC encourage the authors to refer to other similar datasets published at ML conferences for the expectations on using experiments to illustrate the contribution of the dataset (see CalMS21, PAIR-24M, MABe22 (ICML 2023), ChimpACT (NeurIPS dataset and benchmarks 2023)). Separately, the AC would like to note that it is unusual in the field to multiply the hour of recorded videos by the # of views -- the goal is not to have the largest amount of hours, but to provide a dataset that can provide the community with interesting insights. Reviewers have left valuable feedback for this paper, and taking those into account could help the authors make this dataset a strong contribution to the community.