SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As depicted in Fig. 3, there is considerable overlap among most practical experiments, leading to ambiguity when trying to distinguish them based solely on sequences of HOI. More comprehensive visual results of this phenomenon are presented in Appx. B.2. Considering the common occurrence of overlapping atomic actions, we focus on protocol-level ambiguity in MultiAR and leave perceptual-level action ambiguity as a natural intermediary challenge for models. The MultiAR benchmark is a protocol-level action recognition task with all annotated top-down view videos in ProBio. We split all videos into three folds (i.e., easy, medium, and hard) to evaluate protocol-level ambiguity. Over these splits, we devise four benchmarking settings: protocol-only, vision-only, vision with brief experiment guidance, and vision with detailed protocol guidance. In the protocol-only setting, we provide ground-truth HOI annotations (i.e., perfect perception) to models as a performance upper bound. We add protocols of varied granularity to multimodal learning training in protocol-guided scenarios. Vision-only models are tasked to recognize protocol-level activities during testing.
In this section, we evaluate and analyze the performance of models on tasks associated with ProBio. Particularly, we provide details of the experimental setup, evaluation metrics, and result analysis for TransST and MultiAR. Fundamentally, we aim to address the following questions:
- How challenging is the fine-grained understanding of objects and actions in BioLab?
- How crucial are the protocols in tasks associated with ProBio?
- What is missing in existing models when adapted to the specialized BioLab environment?
4.1 TransST
Setup As mentioned in Sec. 3.3, we consider two settings in TransST: visual tracking and protocol-guided tracking. The training, validation, and testing sets are divided in a 6:3:1 ratio, respectively, across all videos captured from nearby perspectives. In visual tracking, we select a number of leading-edge models to serve as our baseline comparisons, including TransATOM (Xie et al., 2020), StrongSORT with different detection backbones (Broström, 2022; Wang et al., 2022a), and Segment-and-Track-Anything (Cheng et al., 2023) based on SAM (Kirillov et al., 2023). Since SAM (Sequential Attention Model) is initially trained on general images, we adopt the strategy suggested in Chen et al. (2023) and integrate a five-layer convolutional SAM adapter. This approach is intended to adapt the SAM weights for effective application within the BioLab setting. Regarding protocol-guided tracking, our preliminary experiments indicate that narrowing down the category of liquid solution types to only categories mentioned in the protocols is more effective than learning-based designs (e.g., fusing protocol features with tracking features). Please refer to Appx. B.1 for details.
Evaluation metrics Following Fan et al. (2019, 2021a), we measure the tracking quality by the precision (PRE) and normalized precision (NPRE) with an intersection-over-union (IoU) over 0.45. In addition, we evaluate the prediction of the solution status within the tracked bounding box with classification accuracy (CLS). To provide a comprehensive analysis of models, we report the memory and time overhead of all methods in transparent solution tracking.
Figure 4: Examples of success and failure cases in two tasks. Top: Tracking results of SAM-adapter+DeAoT with protocol-guidance in TransRT. We visualize correct tracking predictions in green boxes and failure cases in red boxes. We observe that most failure cases could be attributed to the perceptual difficulty of transparent objects or occlusion. Bottom: Protocol-level action recognition results of ActionClip+SAM with protocol guidance in MultiAR. We visualize correct predictions in green and wrong ones in red. Of note, most incorrectly identified actions have similar HOIs.
Results and analysis We report transparent solution tracking results in Tab. 3 and visualize qualitative results in Fig. 4. Specifically, we summarize our major findings as follows:
- Visual tracking in TransST is challenging. As shown in Tab. 3, the performance of traditional (e.g., StrongSORT) tracking models with only visual inputs is significantly lower than the near-perfect performance they present in common tracking scenarios (e.g., driving). We have achieved higher detection efficiency while maintaining computational speed, resulting in an optimal trade-off. In TransST, solutions and containers often have transparent appearances and similar shapes that are visually difficult to distinguish. The frequent occlusion further complicates this because of containment relationship changes in biology experiments. All these facts add difficulty to the visual tracking problem in BioLab environments.
- More robust object detectors benefit tracking in TransST. We observe a consistent performance improvement when adopting more robust object detectors (e.g., SAM). As these models are often pre-trained on large-scale object detection and segmentation datasets, we believe they are beneficial for mitigating the discrepancy between objects in daily life and specialized domains. However, limited by their memory and computation overhead, these models are still unsuitable for real-time monitoring. This urges the need for lightweight adaptations of existing pre-trained models for specialized downstream domains.
- Understanding protocols is crucial in TransST. As explained in Sec. 3.3, tracking the status of transparent liquid within containers is difficult as there are no direct visual features indicating the transition of liquid status. This makes label prediction for the tracked solution extremely challenging without protocol information. Our results on the classification accuracy reflect this fact, showing that event with the simplest heuristic of answer filtering, adding experimental protocols can significantly improve label prediction for all models. However, this improvement is still marginal. This implies that current models still fall short of reasoning about solution types. This promotes future research on multimodal methods for inferring visually unobservable object status changes.
4.2 MultiAR
Setup As discussed in Sec. 3.3, we evaluate performance under four distinct settings: protocol-only, vision-only, vision with brief experiment guidance, and vision with detailed protocol guidance. Similar to experiments in Sec. 4.1, we randomly split video data at each ambiguity level into train/Validation/test with a 6:3:1 ratio. We evaluate the performance of BERT (Devlin et al., 2018) and SBERT (Reimers and Gurevych, 2019) in the protocol-only setting. For the vision-only scenario, we propose finetuning state-of-the-art (SOTA) video recognition models for the MultiAR task. This includes I3D (Carreira and Zisserman, 2017), SlowFast (Feichtenhofer et al., 2019), and Multiscale Vision Transformers (MViT) (Fan et al., 2021b; Li et al., 2022). For settings with detailed protocol guidance, we select strong baselines, including ActionCLIP (Wang et al., 2021), EVL (Lin et al., 2022), and Vita-CLIP (Wasim et al., 2023). We feed additional protocol information into these models
Table 4: Experiment results of MultiAR in ProBio. We highlight the best results in bold.
categories	method	easy	mid	hard
top1	top5	avg.	Δ	top1	top5	avg.	Δ	top1	top5	avg.	Δ
Human oracle	w protocol	98.99	-	-	0	94.34	-	-	0	93.94	-	-	0
w/o protocol	64.64	-	-	-34.35	56.60	-	-	-37.74	51.51	-	-	-42.43
Protocol-only	BERT 2018	56.19	67.00	59.14	-42.8	46.11	71.33	54.72	-48.23	33.72	47.55	37.23	-60.22
SBERT 2019	82.39	98.14	89.97	-16.6	69.28	98.07	86.28	-25.06	65.03	87.7	77.21	-28.91
Vision-only	I3D 2017	41.16	78.78	67.23	-57.83	29.76	63.93	58.88	-64.58	11.16	33.79	23.79	-82.78
SlowFast 2019	50.03	89.97	70.07	-48.96	42.16	67.64	59.72	-52.18	15.00	42.22	33.08	-78.94
MViT 2021b	47.72	89.02	69.92	-51.27	39.92	64.84	54.29	-54.42	13.34	38.01	28.14	-80.6
MViTv2 2022	55.28	91.35	79.92	-43.71	45.25	69.74	61.24	-49.09	21.37	46.77	38.94	-72.57
Protocol-guided (brief)	Vita-CLIP 2023	69.54	73.65	70.22	-29.45	50.30	75.44	71.92	-44.04	21.75	37.43	25.66	-72.19
EVL 2022	72.64	89.74	80.74	-26.35	55.23	90.74	81.76	-39.11	36.62	47.75	39.97	-57.32
ActionCLIP 2021	71.79	88.26	81.17	-27.2	53.75	86.22	77.21	-40.59	37.55	70.29	57.64	-56.39
ActionCLIP 2021 + SAM 2023	75.27	95.67	86.05	-23.72	61.77	92.45	84.24	-32.57	44.54	76.62	69.22	-49.4
Protocol-guided (detailed)	Vita-CLIP 2023	67.25	74.48	70.57	-31.74	51.76	79.02	66.49	-42.58	41.25	67.82	53.33	-52.69
EVL 2022	73.44	88.75	81.46	-25.55	53.34	91.27	69.74	-41	39.64	63.34	52.05	-54.3
ActionCLIP 2021	73.93	93.67	82.23	-25.06	59.42	84.27	80.01	-34.92	40.7	71.11	57.67	-53.24
ActionCLIP 2021 + SAM 2023	76.75	97.94	85.79	-22.24	62.79	91.24	84.40	-31.55	46.75	79.97	67.64	-47.19
during training. Considering that the text encoders utilized in these models are typically trained on text from general domains, we substitute them with SentenceBERT, which has been specifically fine-tuned in the protocol-only setting. In light of experimental findings in Sec. 4.1, we also explore the use of strong object segmentation models (e.g., SAM) for the multimodal understanding problem in MultiAR by adding an additional object branch into current models. We provide more model design and implementation details in Appx. B.2.
Evaluation metrics In all experiments, we report the recognition performance with the top-1, top-5, and mean accuracy. Additionally, we conduct human evaluations and provide the average performance of 10 experienced biology researchers with and without protocols. We report the difference  $(\Delta)$  between the performance of each method and the human oracle to visualize the gap on all ambiguity levels in MultiAR.
Results and analysis We present model performance results of the MultiAR task in Tab. 4 and provide qualitative results in Fig. 4. In summary, we identify the following major findings:
- Actions are visually ambiguous in MultiAR. As shown in Tab. 4, both human and vision-only models suffer from perceptual-level ambiguity in actions. Without protocol information, we observe a  $40\%$  human performance drop in recognizing the correct protocol being executed. This indicates that humans largely depend on protocols to distinguish visually similar actions. This is also reflected by the low performance of SOTA video recognition models on the hard split with pure vision input.
- Recognition in MultiAR demands a detailed understanding of protocols. In contrast to other multimodal video understanding benchmarks, leveraging pre-trained language models is insufficient for MultiAR due to the specialized domain. As shown in the protocol-only setting of Tab. 4, fine-tuning a pre-trained BERT model results in low overall performance. Meanwhile, we observe a significant improvement in SBERT with better modeling of protocol contexts. This suggests potential improvements from more powerful language models that can capture fine-grained experimental contexts illustrated within protocols.
- Contextual information is crucial for multimodal understanding in MultiAR. As shown in Tab. 4, the stark contrast between protocol-guided models and pure vision-based models verifies the importance of contextual information in video action recognition, regardless of the protocol granularity. Although we only provide protocols during training, this suggests that it is critical for models to align visual perceptions with fine-grained protocols. Meanwhile, improving the granularity of protocol guidance generally improves model performance, especially on the hard split of MultiAR. This reveals the potential of more fine-grained multimodal interaction designs in models for improving protocol-level action recognition in MultiAR.
- Solving protocol-level ambiguities is a bottleneck for multimodal video understanding in MultiAR. Across the three ambiguity levels, we observe a significantly lower performance of most models in the MultiAR hard split. Intuitively, with perceptual-level ambiguity in recognizing actions, identifying protocols depends on matching typical and recognizable actions between visual
inputs and protocols. Protocol-level ambiguities aggravate this condition by adding variety in the protocols that could be matched. Nonetheless, as humans can accurately identify the protocol being executed in videos when provided with protocols, we argue that it is essential to mitigate this performance gap with better multimodal video understanding and reasoning designs.
What is ProBio? We curate ProBio, the first protocol-guided dataset, which includes comprehensive hierarchical annotations in BioLab. Our dataset aims to promote protocol standardization and the development of intelligent monitoring systems to address the reproducibility crisis in biology science. We've collected 180.6h of videos within an internationally recognized molecular biology laboratory and meticulously annotated all the instruments and solutions as 213,361 segmentation maps. Additionally, we provided annotations for the conditions and enclosures of 48 transparent objects and the state of 12 solutions in 1.05h nearby-view videos. We further restructure all 9.64h videos from a top-down view into three hierarchical levels: i.e., 13 brf_exp, 3,724 prc_exp, and 37,537 hoi. This arrangement allows for fine-grained multimodal data and categorical information insight for future research on intelligent monitoring systems.
What can we do with ProBio? Based on the fine-grained multimodal dataset, we devise two benchmarking tasks associated with ProBio, aimed at enhancing multimodal video understanding. These two tasks are referred to as transparent solution status tracking (TransST) and multimodal action recognition (MultiAR). In TransST, we provide the transparent object's bounding box and the <liquid category, object id> paired labels. We further discuss the difference between pure-vision tracking and protocol-guided tracking to explore the contribution of procedure information in fundamental tasks. In MultiAR, perceptually similar motions may have divergent semantic interpretations, and the same sub-experiment protocols across different experiments may refer to different meanings. We quantify the definition of ambiguity and establish multimodal and protocol-guided action recognition to enhance the importance of fine-grained contextual information. Our experiments consistently highlight the value of such contextual information in both TransST and MultiAR tasks. Furthermore, this dataset paves the way for other tasks like video segmentation, task prediction, and procedural reasoning.
What will ProBio contribute? ProBio is the pioneering multimodal dataset captured within a molecular biology lab, aiming to enhance video comprehension by providing contextual information for visual data. The newly introduced protocol not only enhances the model's ability to process sequences over time but also incorporates the idea of "procedure." This helps our model to effectively handle action interpretations that may be ambiguous. For building a proficient monitoring system, a model skilled in understanding multimodal videos is crucial. Especially in a sterile biology lab, such a system becomes a superior tool to ensure that researchers follow the prescribed standards and procedures. The failure to reproduce experiments is frequently attributed to unintentional mistakes committed by experimenters. Within the 13 experimental categories in ProBio, a notable quantity of operational actions are not executed as intended, and errors have been identified. The implementation of a monitoring system that possesses advanced video comprehension skills holds the promise of rapidly notifying experimenters of their anomalies, thereby enhancing the overall reproducibility of experiments. Consequently, this serves as a deterrent against the squandering of several months' worth of work and significant financial resources amounting to tens of thousands of dollars.
Limitation and future work Currently, we considered two distinct tasks associated with ProBio. However, these two tasks do not adequately reflect the myriad challenges present in today's biological laboratory environments, nor do they showcase the broad applicability and potential of our dataset. In an upcoming version of ProBio, we plan to add new tasks like video-text retrieval, video segmentation, task prediction, and procedural reasoning, along with more comprehensive annotations. Regarding the model structure, we have not closely aligned the procedure information with the video feature, indicating the potential for further improvement in action recognition accuracy. Our efforts will concentrate on models, aiming to extend their capabilities to recognize and interpret actions with higher levels of detail and complexity.
Acknowledgement The authors would like to thank Xuan Zhang (Helixon Inc.) and Ningwan Sun (Helixon Inc.) for professional data annotation, Ms. Zhen Chen (BIGAI) for designing the figures, Tao Pu (SYSU) for assisting the experiments, and NVIDIA for their generous support of GPUs and hardware. This work is supported in part by the National Key R&D Program of China (2022ZD0114900), an NSFC fund (62376009), the Beijing Nova Program, and the National Comprehensive Experimental Base for Governance of Intelligent Society, Wuhan East Lake High-Tech Development Zone.
Anne Hendricks, L., Wang, O., Shechtman, E., Sivic, J., Darrell, T., and Russell, B. (2017). Localizing moments in video with natural language. In International Conference on Computer Vision (ICCV). 4
Baker, M. (2016). Reproducibility crisis. Nature, 533(26):353-66. 1
Begley, C. G. and Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391):531-533. I, A1
Ben-Shabat, Y., Yu, X., Saleh, F., Campbell, D., Rodriguez-Opazo, C., Li, H., and Gould, S. (2021). The iea asm dataset: Understanding people assembling furniture through actions, objects and pose. In Proceedings of Winter Conference on Applications of Computer Vision (WACV). 3
Braybrook, J. (2017). Reproducibility: Developing standard measures for biology. Nature, 551(7679):168-168.
Broström, M. (2022). Real-time multi-camera multi-object tracker using yolov5 and strongsort with osnet. https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet.7,A3
Caba Heilbron, F., Escorcia, V., Ghanem, B., and Carlos Niebles, J. (2015). Activitynet: A large-scale video benchmark for human activity understanding. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Calne, R. (2016). Vet reproducibility of biology preprints. Nature, 535(7613):493-493. 2
Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. (2017). Realtime multi-person 2d pose estimation using part affinity fields. In Conference on Computer Vision and Pattern Recognition (CVPR). 4, A2
Carreira, J. and Zisserman, A. (2017). Quo vadis, action recognition? a new model and the kinetics dataset. In Conference on Computer Vision and Pattern Recognition (CVPR). 8, 9, A5
Cetina, K. K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University Press. 2
Chen, T., Zhu, L., Ding, C., Cao, R., Zhang, S., Wang, Y., Li, Z., Sun, L., Mao, P., and Zang, Y. (2023). Sam fails to segment anything?—sam-adapter: Adapting sam in underperformed scenes: Camouflage, shadow, and more. In International Conference on Computer Vision (ICCV). 7, A4
Chen, Y., Li, Q., Kong, D., Kei, Y. L., Gao, T., Zhu, Y., and Huang, S. (2021). Yourefit: Embodied reference understanding with language and gesture. In International Conference on Computer Vision (ICCV). 2
Cheng, Y., Li, L., Xu, Y., Li, X., Yang, Z., Wang, W., and Yang, Y. (2023). Segment and track anything. arXiv preprint arXiv:2305.06558. 7
Chiusano, F. (2019). https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.A5
Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2020). Rescaling egocentric vision. arXiv preprint arXiv:2006.13256. 3
Danelljan, M., Bhat, G., Khan, F. S., and Felsberg, M. (2019). Atom: Accurate tracking by overlap maximization. In Conference on Computer Vision and Pattern Recognition (CVPR). A3
Das, P., Xu, C., Doell, R. F., and Corso, J. J. (2013). A thousand frames in just a few words: Linguual description of videos through latent topics and sparse object stitching. In Conference on Computer Vision and Pattern Recognition (CVPR). 2
Devlin, J. (2018). https://huggingface.co/bert-base-uncased. A5
Xu, J., Mei, T., Yao, T., and Rui, Y. (2016). Msr-vtt: A large video description dataset for bridging video and language. In Conference on Computer Vision and Pattern Recognition (CVPR). 4
Xu, J., Rao, Y., Yu, X., Chen, G., Zhou, J., and Lu, J. (2022). Finediving: A fine-grained dataset for procedure-aware action quality assessment. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3
Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C. (2021). Just ask: Learning to answer questions from millions of narrated videos. In International Conference on Computer Vision (ICCV). 4
Yang, Z. and Yang, Y. (2022). Decoupling features in hierarchical propagation for video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS). 7, A4
Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., and Tao, D. (2019). Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI Conference on Artificial Intelligence (AAAI). 4
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., and Choi, Y. (2021). Merlot: Multimodal neural script knowledge models. In Advances in Neural Information Processing Systems (NeurIPS). 4
Zhang, J., Cherian, A., Liu, Y., Ben-Shabat, Y., Rodriguez, C., and Gould, S. (2023). Aligning step-by-step instructional diagrams to video demonstrations. Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3
Zhou, L., Xu, C., and Corso, J. (2018). Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence (AAAI). 2, 3, 4
Zhu, W., Lu, J., Han, Y., and Zhou, J. (2022). Learning multiscale hierarchical attention for video summarization. Pattern Recognition, 122:108312. 2, 6, A4
Zhu, Y. (2018). Visual commonsense reasoning: Functionality, physics, causality, and utility. PhD thesis, University of California, Los Angeles. 2
Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi, S., Wu, Y. N., Tenenbaum, J., and Zhu, S.-C. (2020). Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310-345. 2
Zhukov, D., Alayrac, J.-B., Cinbis, R. G., Fouhey, D., Laptev, I., and Sivic, J. (2019). Cross-task weakly supervised learning from instructional videos. In Conference on Computer Vision and Pattern Recognition (CVPR). 2, 3
In this section, we introduce our dataset construction process, covering both data collection and annotation. We will provide insights into our data sources, collection methods, and annotation tools. We present in detail as follows:
A.1 Data collection
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? Yes, we did. Before the annotation and human study process, compensation was prearranged and discussed with the participating individuals. A labor fee of 100 RMB per 30 minutes will be remunerated to them, with any duration less than 30 minutes being considered as half an hour. The aggregate labor charges for all individuals involved sum up to 5,000 RMB.
A.1.1 Biology protocol