SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conduct experiments on image-level activity understanding benchmarks with diverse tasks: HICO [5] (Human-Object Interaction (HOI) recognition), Stanford40 [33] (action recognition), HAKE [21]-Verb (verb recognition), HAKE [21]-PaSta (conditional PaSta Q-A). HAKE [21]-Verb and HAKE [21]-PaSta is the newly constructed benchmark based on HAKE [21] data. We adopt mAP metric for HICO [5], HAKE [21]-Verb and Stanford40 [33], and top-1 accuracy metric for HAKE [21]-PaSta. For more details, please refer to the supplementary.
4.2 Symbolic System Experiment
An example. The LLM used is OpenAI GPT3.5. To demonstrate the instantiation of our symbolic system, we take the activity "human board an airplane" as an example. Tab. 1 shows the symbols and rules generated, and Fig. 4 illustrates the detailed process. In the 1st round, 5 initial symbols are generated. Then in the 2nd round, they are extended into rules satisfying entailment. Later, the new symbols are used in the next round. Finally, 17 symbols and 12 rules are generated within 5
Figure 5: Statistics of the symbolic system. a) Accumulated symbols & rules count in each round. The data is from 50 randomly sampled activities. The average value and fluctuation range are shown. b) Increased entailment scores as the 10 randomly sampled rules extended with more premise symbols.
rounds. In practice, at most 15 symbols are used for rule extension to avoid semantic redundancy. For example, in Fig. 4, symbols 16 and 17 are not used for rule extension, leading to a stop in the 5th round. From Fig. 4 and Tab. 1, we can find that diverse symbols are mined, including corner cases such as "adjust seatbelt" "reach for the airplane door handle", verifying the effectiveness of the proposed symbolic system. Some special cases are shown as gray text. The rule  $6 \rightarrow 1 \rightarrow 7 \rightarrow 8$  is equivalent to the existing rule  $1 \rightarrow 6 \rightarrow 7 \rightarrow 8$ , thus omitted. Extension from the symbols 10 and 14 stops early because adding premises nevertheless leads to a drop in entailment scores. In practice, multiple rules are extended from a known symbol to increase diversity.
Entailment. Fig. 5 shows statistics of the symbolic system. In Fig. 5a, we randomly sample 50 activities and show the accumulated symbols and rules count in each round, verifying the diversity of the generated symbols and rules. In Fig. 5b, we randomly sample 10 rules and show the entailment scores change as the rule extends with more premise symbols. For example, for rule 0:  $m_1 \wedge m_2 \wedge m_3 \wedge m_4 \models c$ , the entailment score for  $m_1 \models c$ ,  $m_1 \wedge m_2 \models c$ ,  $m_1 \wedge m_2 \wedge m_3 \models c$ ,  $m_1 \wedge m_2 \wedge m_3 \wedge m_4 \models c$  are 0.28, 0.86, 0.87, and 0.93, respectively. The score is calculated by querying 5 times and averaged. We find a climb up in the entailment score, implying the effectiveness of the entailment check. Once the entailment score  $e$  surpasses the threshold  $e_h = 0.9$ , it is regarded as a rule equipped with logical entailment and updated into the symbolic system.
Performance Evaluation. To evaluate the symbolic system, we construct a SymAct (Symbol Activity) test set. It is a small subset of the HICO test set (120 images for 10 activity classes). For each image and each of its ground-truth activities, we list all of the symbols related to the activity in the symbolic system and annotate whether a symbol happens  $(0/1)$  in the image via human judgment. We find the proposed symbolic system has: 1) broader semantic coverage than HAKE. For quantitative measurement, we count the happening symbols for each image-activity pair under different symbolic systems. The average number is 10.8 for ours and 1.8 for HAKE. 2) more rational rules than HAKE. We evaluate confusion by counting different image-activity pairs which share the same symbols. For the HAKE symbolic system, the confusion problem is severe, with 261 confusion pairs over  $C_{120}^2 = 7140$  pairs (accounting for  $3.7\%$ ). For our symbolic system, there are no confusion pairs on the test set because of the presentation ability of symbols and entailment check.
Robustness. We analyze the robustness of the generated symbolic system. We find it satisfies: 1) Convergence. After a certain stopping criterion, a new round will bring few or even no new symbols to the sub-system. In practice, the stopping criterion is: At most 15 symbols are used for rule extension to avoid semantic redundancy. After that, new symbols are not used for rule extension, leading to a stop of the loop. 2) Low sensitivity to initial conditions. The symbol-rule loop provides a reliable and stable symbolic sub-system over a range of initial conditions. 3) Low sensitivity to different prompts. Experiment results are less sensitive to prompts with minor differences, i.e., slightly paraphrasing the prompts. For more details, please refer to the supplementary.
4.3 Visual Reasoning Experiment
Implementation and Settings. For visual reasoning, we use BLIP2 [19] ViT-g FlanT5-XL model to extract visual symbols. BLIP2 [19] is a visual-language pre-trained model with a frozen large language model, thus well equipped with question-answering abilities to effectively extract symbols.
Figure 6: Visualization results. Symbols & activity predictions for "human buy orange" are shown.
Symbol probability	I	II	III	IV
m1: talk with seller	0.92	0.87	0.07	0.16
m2: reach for an orange	0.14	0.68	0.95	0.12
m3: seller hand over orange	0.94	0.82	0.09	0.13
m4: stand in front of fruit stand	0.57	0.85	0.04	0.77
m5: place orange in a bag	0.67	0.90	0.06	0.45
m6: pick orange from a basket	0.68	0.83	0.03	0.74
m7: hold a bag of oranges	0.47	0.88	0.05	0.75
m8: reach for a wallet	0.86	0.83	0.04	0.43
m9: seller put the orange in bag	0.83	0.86	0.04	0.45
m10: give money to seller	0.82	0.87	0.04	0.41
Method	HICO mAP	HAKE-verb mAP	Stanford-40 zero-shot mAP	HAKE-PaSta zero-shot Acc(%)
fine-tuned	zero-shot	fine-tuned	zero-shot
CLIP [26]	67.12	37.08	73.82	43.92	75.68	39.36
CLIP [26]+Reasoning	69.73	43.21	75.27	48.95	82.22	40.47
BLIP2 [19]	-	50.61	-	49.47	91.85	43.81
BLIP2 [19]+Reasoning	-	53.15	-	51.37	92.59	44.65
Table 2: Results on activity benchmarks. More HICO [5] baselines are listed in supplementary.
As a plug-and-play, the reasoning is compatible with existing System-1-like methods. We follow the setting of HAKE [20] to integrate the reasoning result with the prediction of System-1-like methods. We mainly choose 2 typical System-1-like methods: 1) CLIP [26]: a visual-language model pre-trained with large-scale image-text pairs and a contrastive objective, where we use its ViT-L/14 model with a resolution of 336 pixels; 2) BLIP2 [19]: with its image-to-text generation ability, it functions either as a System-1-like baseline, or as the symbol extractor for reasoning. To fully test the ability of the models, we adopt both "fine-tuned" and "zero-shot" settings, where the former model is fine-tuned on the training set of the dataset.
Results. The results of visual reasoning are shown in Tab. 2. Comparing the two baselines, we find that with a frozen language model, BLIP2 [19] is more capable of understanding activity semantics and outperforms CLIP [26] in various zero-shot benchmarks. Under the fine-tuned setting, our method achieves 69.73 and  $75.27\mathrm{mAP}$  respectively on HICO [5] and HAKE [21]-Verb, which outperforms the baseline CLIP [26] with 2.61 and  $1.45\mathrm{mAP}$  respectively, verifying its effectiveness. Under the zero-shot setting, our method also outperforms existing System-1-like methods. The improvement compared with CLIP [26] is: 6.13 mAP (HICO [5]), 5.03 mAP (HAKE [20]-Verb), 6.54 mAP (Stanford40 [33]),  $1.11\%$  acc (HAKE [20]-PaSta). The improvement compared with BLIP2 [19] is: 2.54 mAP (HICO [5]), 1.90 mAP (HAKE [20]-Verb), 0.74 mAP (Stanford40 [33]),  $0.84\%$  acc (HAKE [20]-PaSta). In conclusion, our method brings performance improvement and shows significant zero-shot generalization abilities in various activity understanding tasks. Notably, the improvement is achieved by exploiting knowledge instead of adding visual training samples. The exploited knowledge can be reused for future study. Thus, our method is superior in data efficiency. Further, we visualize the results in Fig. 6. The rules and the logic calculation add explainability to the inference process. Also, they help to correct false negative (I, II in Fig. 6) and false positive (III, IV) cases, thus improving compositional generalization.
We conduct ablation studies on HAKE [20]-Verb zero-shot setting and list the results in Tab. 3.
Reasoning. We first verify the effectiveness of integrating System-2 reasoning. Results of CLIP [26] baseline can be either combined with reasoning results or trivially combined with other baseline results (CLIP [26] + BLIP2 [19]). We find that the former is superior (48.95 mAP) to the latter (44.76 mAP), though both outperform the baseline CLIP [26] (43.92 mAP). It verifies the necessity of System-2 reasoning other than trivially combining predictions from two models.
Symbolic System. We then verify the effects of the symbolic system on visual reasoning. We adopt the symbolic system in HAKE [20] and find a degradation from 48.95 to 44.98 mAP,
as its defects in semantic coverage and rule rationality, and the shortage is amplified in the fuzzy logic calculation without learnable parameters. Next, we construct the symbolic system without logical entailment, where each rule has 3 premise symbols without checking its entailment score. It leads to a dropped performance because of rules with errors or incomplete semantics. Then, the symbol-rule loop is removed, with only rules in the 1st and 2nd rounds updated to the symbolic system. As the diversity of rules decreases, the model suffers from a decreased performance. The importance of diversity is further verified by the performance drop when only  $80\%$ ,  $50\%$ , and  $20\%$  randomly sampled rules are used.
Symbols Extractor/Checker. Next, we replace the symbol extractor BLIP2[19] with CLIP [26] and find a performance fall with a weaker ability to extract visual information. Also, reasoning without a symbol
Method	mAP
CLIP [26]+Reasoning	48.95
CLIP [26]+BLIP2 [19]	44.76
CLIP [26]	43.92
rules from HAKE [20]	44.98
w/o entailment check	46.51
w/o symbol-rule loop	47.53
80% rules	48.57
50% rules	47.61
20% rules	46.22
CLIP [26] as extractor	46.09
w/o checking symbols	48.17
checker suffers from degradation due to the negative effect of inaccurate symbol predictions.
4.5 Bottleneck Analysis
Based on the SymAct test set, we make further analysis of the bottleneck of System-2 reasoning. We investigate the impact of symbol prediction and symbolic system separately. With perfect symbol predictions, we assume the probabilities of symbols are definitely known as  $0/1$  instead of  $p \in [0,1]$ . The ground-truth symbols are annotated as explained in Sec. 4.2. With a perfect symbolic system, we assume that the generated rules cover all samples (image-activity pairs).
The performance drop (100.00→60.79 mAP, 83.11→41.52 mAP) indicates the symbolic system errors, i.e., the generated rules have not covered all samples (image-activity pairs). The drop (100.00 → 83.11 mAP, 60.79 → 41.52 mAP) indicates the errors of symbol predictions, i.e., symbol probabilities cannot be predicted accurately. In the supplementary, we also provide two failure cases that emerge from symbol prediction errors and symbolic system errors respectively.
Table 3: Ablation studies on zero-shot HAKE [20]-Verb.
Symbol Prediction	Symbolic System	mAP
perfect	perfect	100.00
imperfect	perfect	83.11
perfect	imperfect	60.79
imperfect	imperfect	41.52
Table 4: System-1/2 analysis on SymAct test set.
In this work, we rethink the symbolic system in activity reasoning and propose a new one with broad-coverage symbols and rational rules. Thus, enhanced System-2 reasoning is integrated into System-1. We demonstrate how to instantiate it and how it helps to reason with visual inputs. Our method shows superiority in explainability, generalization, and efficiency in extensive experiments.
Computation Cost. Symbol predictions will increase the computational cost as a trade-off for explainability and generalization. It can be eased by discovering the hierarchical and reusable nature of the symbols, which is detailed in the supplementary.
Broader Application. The method could be more general and facilitate real-world applications. We choose activity understanding as a good and important initial test bed because it is a difficult task with complex visual patterns and a compositional nature. To verify the broader application, we provide some initial results on object recognition and VCR [36] tasks in the supplementary.
Limitation. Rules generated from language models mostly boost System-1, but sometimes bring language bias, e.g., the person only decides to buy an orange despite apples in the background. Further, the approximation is mainly based on pre-trained LLMs and can be improved with more elaborate designs, e.g., human-in-the-loop, customized LLMs with higher-quality knowledge. Besides, System-2 reasoning may be boosted if integrated into the System-1 training process. Also, instance-level activities and temporal encoding are beyond our scope since our main focus is on an overall framework to formulate and construct a novel symbolic system. We leave them to future work.
Some of the drawing materials in Fig. 1,2 are designed by Layerace / Freepik.
This work is supported in part by the National Natural Science Foundation of China under Grants 62306175, National Key R&D Program of China (No.2021ZD0110704), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), Shanghai Qi Zhi Institute, Shanghai Science and Technology Commission (21511101200).
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[2] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016.
[3] Claude Berge. Hypergraphs: combinatorics of finite sets, volume 45. Elsevier, 1984.
[4] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
[5] Yu Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recognizing human-object interactions in images. In ICCV, 2015.
[6] Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. arXiv preprint arXiv:1904.11694, 2019.
[7] Yong Du, Wei Wang, and Liang Wang. Hierarchical recurrent neural network for skeleton based action recognition. In CVPR, 2015.
[8] Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. Can neural networks understand logical entailment? arXiv preprint arXiv:1802.08535, 2018.
[9] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time. TPAMI, 2023.
[10] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
[11] Daan Fierens, Guy Van den Broeck, Joris Renkens, Dimitar Shterionov, Bernd Gutmann, Ingo Thon, Gerda Janssens, and Luc De Raedt. Inference and learning in probabilistic logic programs using weighted boolean formulas. Theory and Practice of Logic Programming, 15(3):358-401, 2015.
[12] Giorgio Gallo, Giustino Longo, Stefano Pallottino, and Sang Nguyen. Directed hypergraphs and applications. Discrete applied mathematics, 42(2-3):177-201, 1993.
[13] Artur d'Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv:1905.06088, 2019.
[14] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018.
[15] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint arXiv:1505.04474, 2015.
[16] Drew Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. NeurIPS, 2019.
[17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[18] Sumith Kulal, Jiayuan Mao, Alex Aiken, and Jiajun Wu. Hierarchical motion understanding via motion programs. In CVPR, 2021.
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
[20] Yong-Lu Li, Xinpeng Liu, Xiaogian Wu, Yizhuo Li, Zuoyu Qiu, Liang Xu, Yue Xu, Hao-Shu Fang, and Cewu Lu. Hake: a knowledge engine foundation for human activity understanding. TPAMI, 2022.
[21] Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, and Cewu Lu. Pastanet: Toward human activity knowledge engine. In CVPR, 2020.
[22] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
[23] Yunchao Liu and Zheng Wu. Learning to describe scenes with programs. In ICLR, 2019.
[24] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In CVPR, 2021.
[25] Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. NeurIPS, 2021.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[27] Luc De Raedt, Kristian Kersting, Siraam Natarajan, and David Poole. Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis lectures on artificial intelligence and machine learning, 10(2):1-189, 2016.
[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[29] Botian Shi, Lei Ji, Pan Lu, Zhendong Niu, and Nan Duan. Knowledge aware semantic concept expansion for image-text matching. In IJCAI, volume 1, page 2, 2019.
[30] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
[31] Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3d skeletons as points in a lie group. In CVPR, 2014.
[32] Yue Xu, Yong-Lu Li, Jiefeng Li, and Cewu Lu. Constructing balance from imbalance for long-tailed image recognition. In ECCV, 2022.
[33] Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. Human action recognition by learning bases of action attributes and parts. In ICCV, 2011.
[34] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. NeurIPS, 2018.
[35] L Zadeh. Fuzzy sets. Inform Control, 8:338-353, 1965.
[36] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019.
[37] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.00685, 2023.