SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jieming Cui  $^{1,2,\star}$
cuijieming@stu.pku.edu.cn
Ziren Gong $^{3,\star}$
ziren.gong@outlook.com
Baoxiong Jia $^{2,\star}$
jiabaoxiong@bigai.ai
Siyuan Huang
syhuang@bigai.ai
Zilong Zheng2,
Jianzhu Ma  $^{3,4\boxtimes}$
Yixin Zhu1,2,5,
* J. Cui, Z. Gong, and B. Jia contributed equally. corresponding authors
$^{2}$  National Key Laboratory of General Artificial Intelligence
<sup>3</sup> Institute for AI Industry Research, Tsinghua University
$^{4}$  Department of Electronic Engineering, Tsinghua University
$^{5}$  PKU-WUHAN Institute for Artificial Intelligence
https://probio-dataset.github.io
The challenge of replicating research results has posed a significant impediment to the field of molecular biology. The advent of modern intelligent systems has led to notable progress in various domains. Consequently, we embarked on an investigation of intelligent monitoring systems as a means of tackling the issue of the reproducibility crisis. Specifically, we first curate a comprehensive multimodal dataset, named ProBio, as an initial step towards this objective. This dataset comprises fine-grained hierarchical annotations intended for studying activity understanding in Molecular Biology Lab (BioLab). Next, we devise two challenging benchmarks, transparent solution tracking, and multimodal action recognition, to emphasize the unique characteristics and difficulties associated with activity understanding in BioLab settings. Finally, we provide a thorough experimental evaluation of contemporary video understanding models and highlight their limitations in this specialized domain to identify potential avenues for future research. We hope ProBio with associated benchmarks may garner increased focus on modern AI techniques in the realm of molecular biology.
Despite notable progress in scientific research, the challenge of reproducing research findings has surfaced as a significant obstacle. Baker (2016) suggests that a significant proportion of researchers, exceeding  $70\%$ , have reported unsuccessful attempts to replicate experiments carried out by their colleagues, primarily attributed to inadequate clarity of protocols (Ioannidis, 2005; Begley and Ellis,
Figure 1: An overview of two challenging tasks identified and presented in ProBio. denotes a set of cameras, and denotes intelligent monitoring models with access to BioLab protocols. Task (a): track transparent solutions. Task (b): action understanding guided by protocols. Object IDs share the same color across frames and in the texts.
2012). These protocols (i.e., procedural instructions) frequently exclude crucial factors, such as temperature or pH, that can substantially impact the results. As a result, researchers rely heavily on mentorship from seasoned experts when conducting experiments. The management of such experiments necessitates a significant investment of time and resources, often unattainable for many laboratories, particularly in Molecular Biology Labs (BioLabs), wherein replicating experiments is more time-consuming and costly (Calne, 2016). Since modern intelligent systems have brought significant advancements in various fields (Grosan and Abraham, 2011; Gretzel, 2011; Stephanopoulos and Han, 1996; Tavakoli et al., 2020), there is a growing need to develop an AI assistant to tackle this reproducibility crisis. Toward building such an AI assistant, we set off to curate a multimodal dataset recorded in BioLabs with benchmarks of modern AI methods.
Due to the inherent characteristics of BioLab, constructing this multimodal dataset faces two grand challenges. The first one is the lack of readily available protocols with sufficient details; existing ones typically only provide high-level guidance (Latour, 1987; Cetina, 1999; López-Rubio and Ratti, 2021; Peterson and Panofsky, 2021), lacking details necessary for reproducing results step by step. An example of process failure in cell culturing occurs when the culture medium is not inverted correctly after the addition of yeast. Nevertheless, such crucial information is frequently regarded as a standard practice in research and disregarded in written protocols. The heterogeneity of experimental instructions exacerbates the complexity of this scenario, as different labs document identical tasks in diverse manners contingent upon their respective resource availability (Braybrook, 2017). Therefore, curating standardized protocols with sufficient details, coupled with videos of each instruction's execution, is essential for building an AI assistant for BioLab.
The second challenge pertains to comprehending domain-specific actions and objects at the intricate level of granularity. From the computer vision perspective, this poses a challenging task for achieving fine-grained understanding in contrast to typical scenarios like sports (Shao et al., 2020; Xu et al., 2022) or instructional videos (Zhang et al., 2023; Tang et al., 2019; Miech et al., 2019; Das et al., 2013; Zhou et al., 2018; Zhukov et al., 2019). The complexity of event understanding in BioLab is primarily attributed to the specialized instruments and the ambiguity of actions involved. For instance, experiments commonly involve liquid transfer between visually similar and transparent containers (Liang et al., 2016, 2018), posing additional challenges in object detection and event parsing (Jia et al., 2020; Huang et al., 2023). Moreover, actions that are perceptually similar may have divergent semantic meanings across various experiments due to the strong dependence between actions and experimental contexts (Stacy et al., 2022; Jiang et al., 2022, 2021; Chen et al., 2021). These visual ambiguities (Fan et al., 2022; Zhu et al., 2020; Zhu, 2018) have been mostly left untouched in prior arts (Murray et al., 2012; Shao et al., 2020; Goyal et al., 2017; Kay et al., 2017; Zhu et al., 2022; Panda et al., 2017; Kanehira et al., 2018) and present an ideal and unique testbed for multimodal video understanding (Wang et al., 2022b; Huang et al., 2023).
We present ProBio, the first protocol-guided multimodal dataset in BioLab to tackle the above challenges. ProBio provides (i) a meticulously curated set of detailed and standardized protocols
Table 1: A comparison between ProBio and existing activity understanding datasets. We use segments to denote the number of object segmentation maps annotated, activity.cls to denote the number of protocols, activity.num to denote the number of clips that align with protocols.
Domain	Dataset	Duration	Segments	Procedure	Instruction	HOI pairs	Multi-view	Hierarchy	Activity
cls	num
kitchen	EPIC-Kitchen (2020)	100h	90,000	✓	✓	X	X	X	20,000	39,596
YouCook2 (2018)	175.6h	13,829	✓	✓	X	X	X	89	2,000
LEMMA (2020)	10.1h	11,781	X	✓	X	✓	✓	15	324
daily	COIN (2019)	476.63h	46,354	✓	✓	X	X	✓	180	11,287
HowTo100M (2019)	134,472h	136M	X	✓	X	X	X	12	23,611
HOMAGE (2021)	25.4h	1752	X	✓	✓	✓	✓	453	24,600
IAW (2023)	183h	48,850	✓	✓	X	✓	X	14	420
sport	FineGYM (2020)	708h	-	✓	✓	X	✓	✓	15	4,883
FineDiving (2022)	57.9h	-	✓	X	X	✓	✓	52	3,000
BioLab	ProBio	180.6h	213,361	✓	✓	✓	✓	✓	79	3,724
with corresponding video recordings for each experiment and (ii) a natural and systematic evaluation framework for fine-grained multimodal activity understanding; see Fig. 1. We construct ProBio by selecting a set of 13 frequently conducted experiments and augmenting existing protocols by incorporating three-level hierarchical annotations. This configuration yields 3,724 practical-experiment instructions and 37,537 Human-Object Interaction (HOI) annotations with an overall length of 180.6 hours; see Sec. 3. We design two tasks in ProBio: transparent solution tracking and multimodal action recognition, assessing models' capability to leverage both visual observations and protocols to discern unique environmental states and actions. In light of the significant disparities observed between human and model performance in action recognition, we devise diagnostic splits that stratify experimental instruction into three categories based on difficulties (i.e., easy, medium, hard). We hope ProBio and associated benchmarks will foster new insights to mitigate the reproducibility crisis in BioLab and promote fine-grained multimodal video understanding in computer vision.
This paper makes three primary contributions:
- We introduce ProBio, the first protocol-guided dataset with dense hierarchical annotations in BioLab to facilitate the standardization of protocols and the development of intelligent monitoring systems for reducing the reproducibility crisis.
- We propose two challenging benchmarking tasks to measure models' capability in leveraging both visual observations and language protocols for fine-grained multimodal video understanding, especially for ambiguous actions and environment states.
- We provide an extensive experimental analysis of the proposed tasks to highlight the limitations of existing multimodal video understanding models and point out future research directions.
Fine-grained activity datasets Action understanding has been a long-standing problem in computer vision with successful attempts in data curation (Murray et al., 2012; Kay et al., 2017; Soomro et al., 2012; Caba Heilbron et al., 2015; Monfort et al., 2019). To provide fine-grained activity annotations, datasets (Goyal et al., 2017; Stein and McKenna, 2013; Damen et al., 2020; Jia et al., 2020; Rai et al., 2021; Luo et al., 2022b; Grauman et al., 2022) come with HOI labels, object bounding boxes, hand masks, etc. Tab. 1 compares ProBio with existing datasets.
The task of delineating intricate action hierarchies for daily activities is challenging. One line of work justifies action hierarchy design by examining activities in sports (Shao et al., 2020; Xu et al., 2022) and kitchens (Kuehne et al., 2014; Li et al., 2018; Damen et al., 2020). The annotations provided, while detailed, may lack strong contextual information and may not be suitable for complex tasks that demand nuanced multimodal understanding. Another line of work leverages furniture assembly (Ben-Shabat et al., 2021; Sener et al., 2022; Zhang et al., 2023) as a means to highlight action dependencies. Nonetheless, the practical applications of these tasks are limited.
Multimodal video understanding Complex video understanding tasks require leveraging context in addition to direct visual inputs. Instructional videos (Zhou et al., 2018; Tang et al., 2019; Zhukov et al., 2019; Miech et al., 2019), as the most readily available multimodal video learning source, have been frequently utilized for various multimodal video understanding tasks, such as retrieval
(Anne Hendricks et al., 2017; Wang et al., 2019), captioning (Zhou et al., 2018; Xu et al., 2016; Yu et al., 2019), and question answering (Li et al., 2016; Lei et al., 2018; Grunde-McLaughlin et al., 2021; Xiao et al., 2021; Yang et al., 2021; Jia et al., 2022). These datasets are oftentimes large in scale and curated from internet videos with language primarily sourced from online encyclopedia platforms or transcribed from subtitles. The quality of the language modality is considerably impeded by the substantial human effort required to refine insufficient and inaccurate language descriptions (Miech et al., 2019). In addition, the extensive scale of data necessitates the frequent utilization of pre-trained models from other modalities, such as images, for the purpose of multimodal video understanding (Wang et al., 2021; Zellers et al., 2021; Xu et al., 2021; Luo et al., 2022a). Nevertheless, adapting such pre-trained models to specialized domains, such as BioLabs, presents a formidable challenge due to the distinctive nature of objects and actions involved. To tackle these issues, ProBio provides aligned video-protocol pairs, accompanied by detailed experimental instructions for every procedure. Benchmarks on ProBio offer a comprehensive examination of existing models for multimodal video comprehension in specific domains.
3 The ProBio Dataset
$\mathbb{O}$  ProBio consists of 180.6 hours of multi-view recordings that encompass 13 common biology experiments conducted in BioLab. This section introduces the collection (Sec. 3.1) and annotation (Sec. 3.2) process of  $\mathbb{O}$  ProBio and describes the associated benchmarking tasks (Sec. 3.3).
Biology protocol We have assembled a collection of standard protocols along with comprehensive instructions, and videos guided by these protocols, all included in ProBio. We collect our protocol database by first crawling publicly available protocols published in top-tier journals and conferences. As these protocols often contain only high-level instructions, commonly referred to as brief experiments (brf_exp), we construct an online annotation tool for seasoned researchers to augment them with additional experimental instructions, commonly referred to as practical experiments (prc_exp); of note, this augmentation could be different from experiments to experiments, resulting in a one-to-many mapping from brief to practical experiments. After this augmentation, we select 13 brief experiments with multiple practical experiments and instruct seasoned researchers in BioLab to perform. Please refer to Appx. A.2 for additional details.
Monitoring video The video data is recorded in a laboratory that adheres to the international standard for molecular biology (Nest.Bio Labs, 2023). A total of ten cameras are installed to oversee all experimental procedures. To ensure the quality and clarity of the collected videos, we consult with experienced biological researchers regarding the cameras' viewpoints and positions. A total of eight high-resolution RGB cameras are affixed above the operation tables and instruments. To capture intricate HOIs in detail, two supplementary RGB-D cameras have been positioned in close proximity to the primary operating table and sterility chamber. Over 700 hours of video are collected through a 24-hour monitoring process, which minimizes disruption to the researchers' regular activities. The raw videos undergo additional processing through two steps: (i) automatically filtering of no-action frames using OpenPose (Cao et al., 2017) and YOLOv5 (Ultralytics, 2022), and (ii) manual removal of frames depicting actions unrelated to the intended focus, such as conversing, note-taking, or texting. A total of 180.6 hours hours video pertaining to the 13 selected brief experiments has been obtained.
In molecular biology experiments, it is common for routine operations to occur periodically. To facilitate the annotation process, a representative and distinct subset of video clips is selected. This subset consists of a total of 9.64 hours top-down view videos and 1.05 hours nearby-view videos, marked with detailed action labels to provide clear, fine-grained information. During the process of annotation, we consider (i) detailed HOIs in the form of HOI pairs for each frame and (ii) object segmentation masks for each interacted object. To establish a connection between the fine-grained annotations and the underlying biological experiment, supplementary annotations are furnished to denote the precise location of each action within the brief and practical experiments. This process establishes a hierarchical structure consisting of three levels, encompassing fine-grained action data for multimodal video understanding and categorical information for future research on intelligent
Figure 2: An example of BioLab protocols, data, and tasks. We show activities recorded (right) and their corresponding protocols (left). HOI annotations are visualized in the top row. The bottom row gives an example of how knowledge in protocols guides (i) the recognition of actions (in red) given matched actions (green) and (ii) tracking the transparent solution status (blue).
monitoring systems. When exporting annotations, we translate them to a list of indexes to collect the relations between humans and objects (e.g., ["human_1", "object_2"], "inject"] indicates "human_1" and "object_2" has a relation of type "inject"). To improve the precision of our annotations, we organized our dataset into 16 separate batches, with each batch comprising between 1,000 and 5,000 frames. After the annotation group finished labeling each batch, we engaged in 2 to 3 rounds of thorough reviews with biological experts. These specialists helped us detect and rectify any errors in the labels and their corresponding relationships. Any sections that were identified as inaccurate underwent a process of correction and re-annotation.
We annotate a list of 48 objects and 21 action verbs in HOIs based on their significance confirmed by seasoned biology researchers. A team of crowd-sourced labeling workers was recruited to annotate the selected videos, following appropriate training in the annotation process. In total, we obtained 37,537 HOI annotations and 213,361 object segmentation maps for all object categories. To further enhance comprehension of alterations of object status, supplementary labels for object status were furnished over segmentation maps in the nearby-view video recordings. As transparent solutions and containers are commonly employed in molecular biology experiments, additional solution annotations primarily pertain to the transparent solution types inside containers such as tubes or pipettes, resulting in additional 40,443 additional labels (e.g., ["tube_1", "LB_solution"] for test tube with LB Solution). Fig. 2 shows an example of annotations. The task of annotation entails establishing a correspondence between the present state of videos and practical experiments (i.e., prc_exp) through the allocation of action labels, thereby enabling the subsequent annotation of more detailed actions. Please refer to Appx. A.2 for additional details on the annotation process.
3.3 Benchmark design
We devise two benchmarking tasks associated with ProBio, aimed at enhancing multimodal video understanding. These two tasks are referred to as transparent solution status tracking (TransST) and multimodal action recognition (MultiAR). We present the statistics of data and annotations utilized in each task in Tab. 2 and explicate the settings of each task as outlined below.
Transparent solution tracking (TransST) Tracking and understanding object status in BioLabs is challenging due to their visual ambiguity as visualized in Fig. 2. Multiple factors contribute to this challenge: (i) most objects in biology experiments are small and difficult to detect and track, (ii) containment relationships obscure visibility frequently, and (iii) most containers and liquid solutions lack appearance cues, therefore determining status changes relies heavily on the accurate understanding of protocols; tracking in BioLabs is a multimodal video understanding challenge that demands fine-grained comprehension of both modalities.
Table 2: Statistics of data and annotations used for TransST and MultiAR. segmap denotes segmentation maps in each data split. brf_exp and prc_exp denote the brief and practical experiments. sol denotes solutions. We use the suffix .cls to indicate the number of annotation categories for certain data categories and the suffix .num to indicate the number of annotated instances in that data category.
ambiguity	hours	frame	segmap	brf_exp.cls	prc_exp.cls	hoi.cls	hoi.num	obj.cls	obj.num	action.cls	action.Num
MultiAR	easy	5.1	17485	75371	11	52	155	22965	36	22965	20	22965
medium	2.81	7890	33205	13	19	63	10651	13	10651	16	10651
hard	1.74	2492	8561	9	8	57	5866	11	5866	17	5866
total	9.64	26259	112937	13	79	245	37537	48	37537	21	37537
TransST	hours	frame	segmap	brf_exp.cls	brf_exp.num	prc_exp.cls	prc_exp.num	obj. cls	obj. num	sol.cls	sol. num
1.05	41725	100424	6	31	17	34	14	90888	12	40443
(a)
(b)
Figure 3: (a) Despite the discrepancy between video lengths in TransST and MultiAR, we provide a comparable number of segmentation maps as ground truths in TransST for solution tracking. (b) We split the videos in MultiAR based on the ambiguity level of protocols, resulting in a 6:3:1 easy/medium/hard split. (c) One protocol is defined as hard when its ambiguity score surpasses 0.7, easy when below 0.45, and medium otherwise.
(c) Protocol-level ambiguity scores of all practical experiments.
We evaluate the models' capability via TransST, leveraging all nearby-view videos with liquid solution labels. Each tracking problem includes the bounding box of the target object (e.g., a tube) and the category label of the liquid solution inside (e.g., double-distilled water). We further consider two diagnostic settings, pure visual and protocol-guided, to confirm the significance of protocols. Protocol-guided tracking leverages practical experiments as additional input to equip models with information w.r.t. invisible solution status changes. Please refer to Sec. 4.1 and Appx. B.1 for details.
Multimodal action recognition (MultiAR) An intelligent monitoring system in BioLabs must recognize actions and identify the corresponding protocol to track the experimental progress. However, establishing such a capability is challenging in BioLab: perceptually similar motions may have divergent semantic interpretations, and the same sub-experiment protocols across different experiments may refer to different meanings. However, current datasets have neglected the ambiguity present within fine-grained actions (Murray et al., 2012; Shao et al., 2020; Goyal et al., 2017; Kay et al., 2017; Zhu et al., 2022; Panda et al., 2017; Kanehira et al., 2018). Currently, there is no universally recognized standard for quantifying the ambiguity present in various actions. Our experiment indicates that the straightforward approach of using the similarity of human-object interactions hoi (e.g. Jaccard coefficient) is insufficient for adequately capturing both object ambiguity and procedural ambiguity. To address this, we propose a method for defining the ambiguity between two actions by employing the bidirectional Levenshtein distance ratio, as illustrated in Equation (1). In this equation,  $P(A)$  and  $P(B)$  signify the power sets of the given sets  $A$  or  $B$  of hoi. ratio here refers to the Levenshtein distance ratio. Notably, the ambiguity (labeled as amb) between two practical experiments can exceed a value of 1, indicating a significant similarity between the two procedures or experiments, referred to as prc_exp. To measure the average ambiguity of each action, we then introduce a method for
Table 3: Tracking results of all models in TransST. We visualize the best results in bold.
Categories	Method	PRE ↑	NPRE ↑	CLS ↑	FPS ↑	Param ↓
Vision-only	TransATOM (2021a)	27.54	32.20	29.36	26.0	7.54M
YOLOv5 (2022) + StrongSORT (2022)	47.71	49.49	42.43	27.9	86.19M
YOLOv7 (2022a) + StrongSORT (2022)	59.27	66.41	57.22	35.8	6.22M
SAM (2023) + DeAOT (2022)	91.07	96.94	45.83	2.3	641.27M
Protocol-guided	YOLOv7 (2022a) + StrongSORT (2022)	60.25	67.11	61.94	35.4	6.22M
SAM (2023) + DeAOT (2022)	92.40	97.46	62.43	2.1	641.27M
calculating the average ambiguity for each action, expressed mathematically as  $\frac{1}{N}\sum_{amb\in N}amb_i$ :
$$
a m b = \frac {1}{P (A)} * \sum_ {x \in P (A)} \max  _ {y \in P (B)} \left(r a t i o (x, y)\right) + \frac {1}{P (B)} * \sum_ {y \in P (B)} \max  _ {x \in P (A)} \left(r a t i o (y, x)\right). \tag {1}
$$