SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To ensure the precision and comprehensiveness of biological protocol data, the initial step involves the retrieval of a substantial number of protocols from highly regarded journals and conferences such as Cells (MDPI, 2011), Jove (JOVE, 2006), and Protocol Exchange (NATURE, 2000) for the period spanning 2022 and prior years. The aforementioned protocols represent the forefront of experimental guidelines within the realm of biology and serve as a highly appropriate foundation for establishing a standardized protocol for biological experimentation. The microscopic realm is the setting for certain biological experiments, including brain neuroscience and genetic sequencing, which are not discernible to the unaided eye. In light of this, we have identified 12,381 experiments that are amenable to oversight via a monitoring system.
The experimental protocols procured from high-ranking academic journals are notably succinct, with most protocols offering mere guidance without practical operational steps (Ioannidis, 2005; Begley and Ellis, 2012). Hence, they are denoted as brief experiments, commonly abbreviated as brf_exp. To render these succinct and theoretical procedures feasible, it is imperative to deconstruct them and augment them with comprehensive instructions. For instance, we can expand the "PCR preparation" to [“adding dd water into the solution,” “placing the PCR in an ice bath”, etc.] by breaking it down into a series of steps. These expanded protocols are commonly referred to as practical experiments and can be denoted as prc_exp. Doctoral students in biology from renowned institutions, including Harvard, Peking University, and Tsinghua University, were employed to perform annotation tasks. The annotation results were thoroughly verified through multiple rounds of mutual checks to ensure accuracy and completeness. As a result, the protocols that were previously only instructive can now be executed.
An online annotation tool has been developed to streamline the annotation process for annotators across the globe and facilitate real-time multiple rounds of mutual checks. We track the information of annotators and modifiers through IDs, aiming to improve the efficiency and standardization of the annotation process. The instructions for using the annotation interface and tools are shown in Fig. A1.
A.1.2 Monitoring video
To gather a comprehensive video collection, we have partnered with an internationally recognized biological laboratory that adheres to standard protocols Nest.Bio Labs (2023). This collaboration enables us to capture the various activities involved in conducting biology experiments. This category of laboratory adheres to an international standard that mandates uniformity in both the interior and exterior appearance and design across laboratories worldwide. Unified regulations dictate the number, color, and size of workstations, the height of the ceiling, and the dimensions of the rooms. This offers a superb opportunity to broaden the global impact and augment the applicability of our ProBio.
Under the supervision of experienced researchers, we conducted the process of laboratory selection and camera setup. The selected molecular biology laboratory comprises seven primary experimental stations, a refrigeration unit, and a sterile enclosure. To ensure comprehensive coverage of all operations and instruments, we deployed ten high-resolution cameras strategically positioned from a top-down perspective to minimize occlusion. Every experimental table, refrigerator, and chamber is furnished with a specialized camera for documentation. An additional camera has been installed with
a specific focus on the frequently utilized water bath during experimental procedures, to guarantee that no procedural details are impeded or overlooked during the water bath process. Furthermore, we positioned a single RGB-D camera in proximity to the experimental table and sterile chamber to record operations with a higher level of detail and a closer perspective. Following the completion of the setup, a continuous and uninterrupted silent recording plan was implemented for the ongoing experimental operations, to minimize any potential impact on the experimenters. The raw video footage collected for this study exceeded a total of 700 hours. Subsequently, the dataset was generated via post-processing techniques and annotation procedures.
A.2 Data annotation
Before annotation, we use the semi-automated method to remove irrelevant video clips, such as clips with no human, clips with unrelated actions, etc. In the semi-automated filtering process, we apply YOLOv5 (Ultralytics, 2022) and OpenPose (Cao et al., 2017) to crop key video clips with related experiment instructions and operations. We then manually remove frames depicting actions unrelated to the intended focus, such as conversing, note-taking, or texting. To ensure the efficiency of pre-processing, we carefully check each clip of our filtered videos. Finally, we obtain a total of 180.6h videos.
A.2.1 Alignment
In the process of data collection, a total of 12,381 brief experiments were acquired, along with their respective practical experiments, following necessary adjustments and completion. We also obtained a collection of raw videos spanning 180.6h; however, no connection was established between this dataset and the aforementioned data type. To establish the correlation between the aforementioned modalities, a team of master's and doctoral students from prestigious academic institutions such as Peking University, Tsinghua University, and Peking Union Medical College Hospital were recruited to conduct alignment annotation. The task of annotation entails establishing a correspondence between the present state of videos and practical experiments (i.e., prc_exp) through the allocation of action labels, thereby enabling the subsequent annotation of more detailed actions. An offline video action annotation tool has been developed to enhance the annotation process for annotators located in various regions. The tool, depicted in Fig. A2, enables the application of diverse labels through the use of keyboard shortcuts, thereby enhancing the efficiency of the annotation process. In the course of annotating alignments, we have ascertained that the periodic occurrence of routine operations is a common phenomenon. Consequently, we opted to engage in a collaboration with expert experimenters to carefully choose a subset of video frames from the existing footage for further detailed annotations.
A.2.2 Fine-grained annotation
Then, we employ a team of annotators and provide a two-day professional training on all BioLab instruments, solutions, and operations. After the training, we divide the current video into multiple batches of 30-50 minutes each and deliver them iteratively to the annotation team. Before each batch delivery, we provide corresponding annotation guidelines, including the IDs of the experimental personnel, the items involved in the operation, and their respective labels. We create the dataset through real-time acceptance of online annotations. After completing 12 batches of annotations, we have annotated 213,361 segmentation maps for 10.69h and summarized two characteristics in our dataset: (i) Many operations involve the combination of multiple transparent solutions to yield a new transparent solution. In experimental settings, it is customary to employ transparent and uncolored apparatus and solutions. (ii) Similar movements represent entirely different jobs and lead to divergent purposes, which is called ambiguity.
Solution status Given the two main characteristics of this dataset, while also considering the huge number of segmentation maps, we divide the dataset into two major parts. We first annotate 1.05h videos to learn more about transparent objects and solutions. Following consultation with experienced experimenters, we collect 48 object categories and 12 solution categories. Instance masks and bounding boxes are employed in video annotation to denote the positions and identities of objects. We further track the location of solutions used throughout the experiments to track the status and progress of experiments. This information is annotated by providing additional labels over container object annotations (e.g., ["tube_1", "LB_solution"] for test tube with LB Solution). While exporting
annotations, we use a list of labels to represent the relations between the reagent and objects (e.g., ["tube_1", "LB_solution"]).
Hierarchical structure As for the second part, we focus on the ambiguity in the rest of 9.64h videos. There will be a high similarity between current practical experiments. To differentiate these ambiguous actions, we have decided to further refine them at the granularity of human-object interaction pairs in the prc_exp. We have divided our ProBio dataset into a three-level hierarchical structure, as shown in Fig. A3. At the top level, we use brief experiment (bf_exp) to define the overall goal of an experiment, which is only documented in the paper and works in theory, e.g., "yeast transformation" and "PCR preparation." Next, we use practical experiment (prc_exp) to represent practical experiments in protocols which are composed of several HOIs, e.g., "measure OD" and "add YPD_medium into vector." Finally, we use HOI pairs to define atomic operations (act) in experiments. In total, we obtain 13 bf_exp, 3,724 prc_exp, and 37,537 act categories. We use a triplet for HOI annotation (e.g., ["human_1", ["tube_2", "hold)]) to represent the human subject id, interacting object, and the action verb. While exporting annotations, we translate this annotation to a list of indexes to collect the relations between humans and objects (e.g., ["human_1", "object_2"], "inject ]). Finally, We instruct experimenters to conduct an additional round of verification to ensure the accuracy of labels, and the relationship of prc_exp and hoi are shown in Fig. A4.
B.1 Transparent solution tracking (TansST)
Typically, the solution observed in BioLab exhibits characteristics of being both transparent and colorless. Since the liquid can be transferred between different containers such as beakers, petri dishes, and test tubes, the geometric shape of the liquid changes according to the shape of the container it is housed. Hence, the monitoring of the solution is an arduous and potentially unattainable undertaking. The successful execution of experiments in biology laboratories is largely dependent on the transfer and fusion of solutions, making the tracking of solutions a crucial and fundamental task in the development of a monitoring system. In our ProBio dataset, we obtained pairs of containers and solutions based on the experiment's protocol and annotated them, facilitating the tracking of the solutions. During the process of using various baselines for solution tracking, we have also discovered that narrowing down the category of liquid solution types to only categories mentioned in the protocols is more effective than learning-based designs (e.g., fusing protocol features with tracking features).
B.1.1 Implementation details
In this section, we provide details on model implementation, hyperparameters selection, and environment setup. We present the details for each selected model as follows:
Vision-only
- TransATOM Following the TransATOM (Fan et al., 2021a) benchmark, we first train the transparent solution segmentation network (Xie et al., 2020) with the TransST subset of our ProBio and the easy subset of Trans10K (Fan et al., 2021a) dataset on 1 NVIDIA 3090 GPU for 40 epochs. We set the initial learning rate to 0.02, batch size to 8, and extracted visual features using ResNet18. In order to remain consistent with the original text, we also choose the ATOM (Danelljan et al., 2019) as the tracker.
- YOLOv5 + StrongSORT Based on StrongSORT (Broström, 2022; Wang et al., 2022a), we change different detection backbones and gain final tracking results. We first finetune the yolov5n model with the TransST subset of our ProBio on 1 NVIDIA 3090 GPU for 20 epochs, we have set the initial learning rate to  $1 \times 10^{-5}$ , batch size to 128, and the IOU threshold as 0.45. Then, we track the detected object-solution pairs with a confidence threshold of 0.25.
- YOLOv7 + StrongSORT Similar to the baseline  $YOLOv5 + \text{StrongSORT}$ , we first finetune yolov7-tiny model with the TransST subset of our ProBio on 1 NVIDIA 3090 GPU for 20 epochs, we have set the initial learning rate to  $1 \times 10^{-5}$ , batch size to 128, and the IOU threshold as 0.45. Then, we track the detected object-solution pairs with a confidence threshold of 0.25.
- SAM + DeAOT Inspired by Chen et al. (2023), we train a SAM-adapter based on vit_h pre-trained weights and AdamW optimizer with the TransST subset of our ProBio dataset. We have set the learning rate to  $2 \times 10^{-4}$ , batch size to 2. The adapter consists of two MLPs and an activate function GELU (Hendrycks and Gimpel, 2016) within two MLPs (Liu et al., 2023). We further passed the output of the adapter through a classification network, which has five Conv2d layers with input patch sizes of 24. We set the patch_size as 16, window_size as 14, input image resolution as  $1024 \times 1024$ , and train on 4 NVIDIA A100 GPUs for 20 epochs. For models with large parameter sizes like this, training adapters have shown good performance on our ProBio dataset. Then, we track the detected object-solution pairs with DeAOT (Yang and Yang, 2022), choosing the model R50-DeAOT-L.
Protocol-guided
- YOLOv7 + StrongSORT Similar to the vision-only method, we first select object-solution pairs that have occurred based on the protocol of this experiment, including prf_exp and prc_exp, and compile them into a list. Then, we finetune the yolov7-tiny model with the filtered list on 1 NVIDIA 3090 GPU for 15 epochs, we have set the initial learning rate to  $1 \times 10^{-5}$ , batch size to 128, and the IOU threshold as 0.45. Then, we track the detected object-solution pairs with a confidence threshold of 0.25.
- SAM + DeAOT Using the same approach as protocol-guided baseline YOLOv7 + StrongSORT, we first filter the desired object-solution pairs through a protocol and compile them into a list. Afterward, we perform model finetuning and subsequent tracking as baseline SAM + DeAOT.
B.2 Multimodal action recognition (MultiAR)
As evidenced in Appx. A.2.2, motions that are perceptually similar may possess distinct semantic interpretations, and practical experiments conducted across varying protocols may pertain to dissimilar meanings. To demonstrate the protocol-level ambiguity between two protocols in an intuitive manner, we perform a calculation of the overlap of all downstream HOI annotations. Based on the computed ambiguity metric, the complete dataset has been categorized into three distinct levels of complexity: easy, medium, and hard. Given that each level encompasses distinct practical experiments prc_exp, we conducted separate experiments at each level and subsequently derived conclusions. Subsequently, each of them will be explicated individually.
B.2.1 Ambiguity
With the increased granularity of action refinement, the inherent ambiguity of actions becomes apparent. However, current datasets have neglected the ambiguity present within fine-grained actions (Murray et al., 2012; Shao et al., 2020; Goyal et al., 2017; Kay et al., 2017; Zhu et al., 2022; Panda et al., 2017; Kanehira et al., 2018). Furthermore, there is currently no widely accepted metric for measuring ambiguity in actions. We find that the simplicity of using the similarity of human-object interactions hoi (e.g., Jaccard coefficient) to describe both the object ambiguity and procedure ambiguity is inadequate. Therefore, we define ambiguity between two actions with the bidirectional Levenshtein distance ratio, as shown in Equation (1). In Equation (1), P(A) and P(B) represent the power set of the given A or B set of hoi, while ratio denotes the Levenshtein distance ratio. The ambiguity (i.e., amb) between two practical experiments can exceed 1, which represents a high similarity between the two prc_exp (shown in Fig. A5). Afterward, to measure the average ambiguity of each action, we define it by taking the average value (i.e.,  $\frac{1}{N}\sum_{amb\in N}amb_i$ ).
$$
a m b = \frac {1}{P (A)} * \sum_ {x \in P (A)} \max  _ {y \in P (B)} \left(r a t i o (x, y)\right) + \frac {1}{P (B)} * \sum_ {y \in P (B)} \max  _ {x \in P (A)} \left(r a t i o (y, x)\right) \tag {A1}
$$
B.2.2 Model Structure
To enhance the proficiency of the model, it is imperative to employ the technique of variable manipulation to isolate the specific components that necessitate refinement. Initially, a comparison is made between the conversion of human-object interactions into descriptive text and pure vision. It is concluded that the visual modality presents a greater potential for enhancement. Subsequently, the model is enhanced through the incorporation of an alignment module and an object-centric mask
module, resulting in a notable enhancement of the multimodal model's performance. Ultimately, we substitute the concise instructions with hands-on experiments that furnish extensive insights for more intricate guidance. Fig. A6 depicts the particular operations, whereby spatial information about objects is incorporated via graph neural network (GNN) (Scarselli et al., 2008), and practical experimental information is incorporated via SentenceBERT (Reimers and Gurevych, 2019). The calculation of similarity is performed consistently, and subsequently, the ultimate prediction outcome is generated.
B.2.3 Implementation details
human study To assess the viability of the two proposed benchmarks and establish the maximum attainable experimental performance, a human study was conducted with the participation of ten master's students hailing from UC Berkeley, Peking University, and Tsinghua University. The study was bifurcated into two parts: with protocol and without protocol. The study involved the extraction of data from video recordings at varying levels of difficulty, namely easy, medium, and hard. The amount of data extracted was equivalent to 0.05 times the total of each level, and a list of 79 practical experiments was provided for the participants to choose from. The experimental data about the section labeled as without protocol had already been prepared. For the with protocol part, additional information about the brief experiment to which the video belonged was provided to the participants to provide direction. All participants in the experiment were remunerated according to the criteria mentioned in Appx. A.1.
Protocol-only First, we process the detection results of human-object interaction in the video into textual form as input for subsequent steps. We then use protocol-guided techniques to predict the actions in the target video. This method helps reduce the influence of detection errors in the video and achieve the highest performance achievable at the current stage.
- BERT We use the pre-trained BERT model and implementation provided by Hugging Face (Devlin, 2018). We use the Adam optimizer Kingma and Ba (2014) and apply cross-entropy loss. We set the initial learning rate to 0.02, dropout as 0.5, batch size to 8, and train with our descriptive text on 1 NVIDIA 3090 GPU for 20 epochs.
- SBERT Similar to BERT, we use the pre-trained SentenceBERT model and implementation provided by Hugging Face (Chiusano, 2019). Based on the current descriptive text, we connect the hoi using prompts to create a practical experiment with a sequence of operations. For example, "First, we open the tube. Second, we take the pipette, etc." The generated sentences are then used as training inputs for the model. We use the Adam optimizer Kingma and Ba (2014) and apply cosine similarity loss. We set the initial learning rate to  $2 \times 10^{-5}$ , batch size to 8, and train with our descriptive text on 1 NVIDIA 3090 GPU for 20 epochs.
- I3D Follow (Carreira and Zisserman, 2017), ResNet50 is selected as the backbone and the frames and sampling rate are set to 8. The input video undergoes a resizing process to achieve dimensions of  $224 \times 224$ . The Adam optimizer Kingma and Ba (2014) is employed with a weight decay of  $1 \times 10^{-4}$  and a uniform batch size of 64. The present model exhibits uniform settings across three distinct categories and undergoes training through the utilization of a single NVIDIA A100 GPU, throughout 100 epochs.
- SlowFast Follow (Feichtenhofer et al., 2019), we also choose ResNet50 as the backbone and both the frames and sampling rate are set to 8. The input video undergoes a resizing process to achieve dimensions of  $224 \times 224$ . The Adam optimizer Kingma and Ba (2014) is employed with a weight decay of  $1 \times 10^{-4}$  and a uniform batch size of 64. The present model exhibits uniform settings across three distinct categories and undergoes training through the utilization of a single NVIDIA A100 GPU, throughout 100 epochs.
- MVIT Follow (Fan et al., 2021b), we choose MViT as the backbone and set the frames as 16, and the sampling rate as 4. The input video undergoes a resizing process to achieve dimensions of  $224 \times 224$ . The AdamW optimizer Loshchilov and Hutter (2019) is employed with a weight decay of  $5 \times 10^{-2}$  and a uniform batch size of 16. We apply soft cross entropy as the loss function.
The present model exhibits uniform settings across three distinct categories and undergoes training through the utilization of a single NVIDIA A100 GPU, throughout 100 epochs.
- MViTv2 Follow (Li et al., 2022), we choose MViT as the backbone and set the frames as 16, and the sampling rate as 4. The input video undergoes a resizing process to achieve dimensions of  $224 \times 224$ . The AdamW optimizer Loshchilov and Hutter (2019) is employed with a weight decay of  $5 \times 10^{-2}$  and a uniform batch size of 4. We apply soft cross entropy as the loss function. The present model exhibits uniform settings across three distinct categories and undergoes training through the utilization of a single NVIDIA A100 GPU, throughout 100 epochs.
Protocol-guided (brief)
- Vita-CLIP Follow (Wasim et al., 2023), we finetune the pretrained CLIP model with our ProBio dataset on 4 NVIDIA A100 GPUs for 50 epochs. The Adam optimizer Kingma and Ba (2014) is employed with a weight decay of  $5 \times 10^{-2}$  and a uniform batch size of 64. We set the initial learning rate to  $4 \times 10^{-4}$ , and the frames and sampling rate as 8.
- EVL Follow (Lin et al., 2022), we finetune the pretrained CLIP model with our ProBio dataset on 4 NVIDIA A100 GPUs for 50 epochs. The Adam optimizer Kingma and Ba (2014) is employed with a weight decay of  $5 \times 10^{-2}$  and a uniform batch size of 64. We set the initial learning rate to  $4 \times 10^{-4}$ , the frames as 32, and the sampling rate as 8.
- ActionCLIP Follow (Lin et al., 2022), we finetune the pretrained ViT-B model with our ProBio dataset on 1 NVIDIA 3090 GPU for 40 epochs. The AdamW optimizer Loshchilov and Hutter (2019) is employed with a weight decay of  $2 \times 10^{-1}$  and a uniform batch size of 4. We set the initial learning rate to  $5 \times 10^{-6}$ , the frames as 32, and the sampling rate as 8.
- ActionCLIP + SAM We have the same vision branch and similarity calculation module as baseline ActionCLIP. Furthermore, we encode the object information with the graph neural network (GNN). The encoder contains two parts: temporal and spatial, each composed of MLPs with different layers, and ultimately outputs object features of 256 dimensions. After that, it is concatenated with the image feature and inputted into the subsequent loss calculation and backpropagation module.
Protocol-guided (detailed) The input caption of the model was modified by replacing its text modality with a practical experiment (prc_exp) connected by prompts. This modified input was then passed to the encoder as a text sequence. Subsequently, the text encoder in the model was substituted with SentenceBERT. The training input and associated particulars about this segment of the model have been expounded upon in great detail within this passage (refer to Appx. B.2.3). The following is a list solely comprised of hyperparameters: