SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Vita-CLIP The Adam optimizer Kingma and Ba (2014) is employed with a weight decay of 5  $\times 10^{-2}$  and a uniform batch size of 64. We set the initial learning rate to  $4 \times 10^{-4}$ , and the frames and sampling rate as 8.
- EVL The Adam optimizer Kingma and Ba (2014) is employed with a weight decay of  $5 \times 10^{-2}$  and a uniform batch size of 64. We set the initial learning rate to  $4 \times 10^{-4}$ , the frames as 32, and the sampling rate as 8.
- ActionCLIP The AdamW optimizer Loshchilov and Hutter (2019) is employed with a weight decay of  $2 \times 10^{-1}$  and a uniform batch size of 4. We set the initial learning rate to  $5 \times 10^{-6}$ , the frames as 32, and the sampling rate as 8.
- ActionCLIP + SAM The AdamW optimizer Loshchilov and Hutter (2019) is employed with a weight decay of  $2 \times 10^{-1}$  and a uniform batch size of 4. We set the initial learning rate to  $5 \times 10^{-6}$ , the frames as 32, and the sampling rate as 8.
C Ethical review
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? Yes, we did. We captured the daily experimental operations of the researchers through ten cameras fixed on the ceiling, filming in a 24-hour uninterrupted silent mode. We obtained consent from all personnel involved in the experiment and applied blur to the recorded faces to ensure the confidentiality of personal information. During the data recording period, no specific actions were required from the participants, and we submitted a complete set of materials
to the Institutional Review Board (IRB), including the list of subjects, experimental details, duration, and all relevant materials.
C.1 Responsibility & data license
D Future work
Currently, regarding the two benchmarks proposed in this article, we have demonstrated the effectiveness of detailed protocol-guided for complex video understanding through experiments. Our plans for model structure, data annotation, and task enhancement are outlined. Furthermore, expanding the applicability of our dataset is a priority for us. To this end, we aim to develop a monitoring system using our current multimodal dataset. This system is designed to reduce the occurrence of experimental errors by experimenters, improve the repeatability and correctness of experiments, curtail expenses, and augment efficacy.
Figure A1: (a) The home page. The logged user needs to choose the target protocols and whether to view or edit. (b) The annotation page. Protocol details are shown on the top of the page, and annotators need to complete the annotation process through multiple clicks, dragging, and input operations.
(c)
Figure A2: (a) The main page of our tool. (b) The main interface for playing videos at variable speeds. (c) List of prc_exp of the chosen brf_exp.
Figure A3: The three-level hierarchical structure.
Figure A4: Relationship between prc_exp and hoi.
Figure A5: Visualization of the ambiguity between any two actions.
(a) Previous multi-modal model
Figure A6: Structure of our action recognition module
Figure A7: Visualization of the TransST results.
Figure A8: Visualization of the MultiAR results.