SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoqian Wu Yong-Lu Li* Jianhua Sun Cewu Lu*
{enlighten, yonglu_li, gothic, lucewu}@sjtu.edu.cn
Human reasoning can be understood as a cooperation between the intuitive, associative "System-1" and the deliberative, logical "System-2". For existing System-1-like methods in visual activity understanding, it is crucial to integrate System-2 processing to improve explainability, generalization, and data efficiency. One possible path of activity reasoning is building a symbolic system composed of symbols and rules, where one rule connects multiple symbols, implying human knowledge and reasoning abilities. Previous methods have made progress, but are defective with limited symbols from handcraft and limited rules from visual-based annotations, failing to cover the complex patterns of activities and lacking compositional generalization. To overcome the defects, we propose a new symbolic system with two ideal important properties: broad coverage symbols and rational rules. Collecting massive human knowledge via manual annotations is expensive to instantiate this symbolic system. Instead, we leverage the recent advancement of LLMs (Large Language Models) as an approximation of the two ideal properties, i.e., Symbols from Large Language Models (Symbol-LLM). Then, given an image, visual contents from the images are extracted and checked as symbols and activity semantics are reasoned out based on rules via fuzzy logic calculation. Our method shows superiority in extensive activity understanding tasks. Code and data are available at https://mvig-rhos.com/symbol_1lm.
Human reasoning can be understood as a cooperation between two cognitive systems: the intuitive, associative "System-1" and the deliberative, logical "System-2" [25]. An example of System-2 reasoning is illustrated in Fig. 1a. A feather and an iron ball are dropped at the same height in a vacuum, which one will land first? Human intelligence derives the answer from the Universal Gravitation Law, i.e., reasoning in a physical symbolic system to avoid intuitive errors. As a critical component in building AI systems, visual activity understanding urgently needs System-2 reasoning. Previous works [26, 17, 30, 19] typically establish the mapping between visual inputs and natural language, where knowledge is embedded into some non-symbolic form such as the weights of language models [24]. Though effective, these System-1-like methods depend on large-scale visual data, thus suffering from diminishing marginal utility. Also, it fails in explainability and generalization due to black-box computing. Thus, it is important to integrate System-2 reasoning.
With one glance at an image, we can effortlessly ground visual inputs to symbols/concepts and apply commonsense reasoning to imagine the world beyond the pixels, a process similar to the gravity case. Fig. 1b shows a possible path for System-2 visual reasoning for human activity: a symbolic system composed of symbols and rules. One rule "Hip seated in a boat  $\wedge$  Hands control an engine  $\wedge$  Hands hold onto the side of a boat  $=$  Human ride a boat" connects symbols
Figure 1: System-2 reasoning in a) solving physical questions and b) understanding human activities. They both involve a symbolic system implying human commonsense.
(e.g., "Hands control an engine" "Human ride a boat") and implies human commonsense. Then, given an image, symbols are first extracted from visual inputs and then activity semantics are deduced based on rules. In this paper, we focus on this line of System-2 reasoning and zoom in from general visual understanding to human activities in various complex contexts, which is a challenging problem in visual understanding.
Current activity reasoning methods are not well equipped with System-2 reasoning. For instance, HAKE [21, 20] adopts a similar path as Fig. 1b from the perspective of human body part states and makes progress toward activity reasoning, but is defective with limited symbols from handcraft (93 primitives) and limited rules from visual-based annotations (100 K+ images). For example, Fig. 1b shows several rules related to the activity "Human ride a boat", e.g.,
(R1,green)Hip seated in a boat  $\wedge$  Hand hold onto the side of the boat  $\wedge$  Hand operate the steering wheel  $=$  Human ride a boat
(R2, orange) A rope is attached to a boat  $\wedge$  Hand hold the rope  $\wedge$  Hand adjust a sail  $\wedge$  Feet hold onto the rope  $\wedge$  Leg stretched out  $=$  Human ride a boat
However, the rules summarized by [20] are “(R3) Hip sit in ∧ Hand hold |= Human ride a boat” (“(R4) Hand hold ∧ Feet close with |= Human ride a boat”, which fail to cover the complex patterns of human activities. Thus, it lacks compositional generalization, i.e., a generalized understanding of visual concepts, their relationships, and their novel combinations.
In view of this problem, we propose a novel symbolic system that implies abundant human knowledge and organizes diverse reasonable rules as in Fig. 1b. To achieve this, the symbols should be with broad semantic coverage to express various activity contexts. Moreover, the rules should be rational and unambiguous. R4 is a failure case, where 'Hand hold' and "Feet close with" do not necessarily lead to "Human ride a boat", and "Human tie a boat""Human exit a boat" are other possible conclusions. Ambiguous semantics should be corrected in rules to avoid confusion. Collecting massive human knowledge via manual annotation is expensive to instantiate this symbolic system. Our main insight is Symbols from Large Language Models (Symbol-LLM), i.e., leveraging the recent advancement of Large Language Models (LLMs) to approximate the above two important properties. This is achieved by the symbol-rule loop and entailment check strategy. Then, the symbolic system can be used to reason with visual inputs, where visual contents from the images are extracted and checked as symbols and activity semantics are reasoned out via fuzzy logic calculation.
In conclusion, our main contributions are: 1) We point out that current System-2 reasoning [20] suffers from a defective symbolic system with hand-crafted symbols and limited, ambiguous rules. 2) To overcome the defects, we propose a novel symbolic system with broad-coverage symbols and rational rules. Further, we propose Symbol-LLM to instantiate it and show how it helps visual reasoning. 3) In extensive activity understanding tasks, our method shows superiority in explainability, generalization, and data efficiency.
Activity Understanding [22, 10] is still a difficult task compared with object detection [28] or pose estimation [9]. This is caused by its complex visual patterns and long-tail data distribution [32].
There are mainly image-based [5, 15], video-based [14, 4] and skeleton-based [31, 7] methods. In this paper, we focus on image-based methods.
Vision-Language Models (VLMs) have been intensively studied recently. Trained with web-scale image-text pairs, they can make zero-shot predictions on various recognition tasks [37] including human activities. Though effective, previous System-1-like methods [26, 17, 30, 19] need to be improved in explainability, generalization, and data efficiency. Thus, integrating System-2 reasoning, which is our main focus, is important.
Probabilistic Programming [27, 11] unifies probabilistic modeling and traditional programming. We follow the principles of the standard first-order logic (FoL) in probabilistic programming, e.g., logical connectives  $\land$ ,  $\lor$ . The standard FoL typically solves simplified relational tasks, where symbols are definitely True/False. However, when adapting FoL in vision reasoning tasks, the symbols are not known to be True/False. Instead, the models are required to predict the symbol probabilities (usually via a neural network). Thus, a hybrid "neuro-symbolic" method is adopted.
Neuro-Symbolic Reasoning provides a possible way of combining System-1 and System-2 learning, where knowledge is represented in symbolic form and learning and reasoning are computed by a neural network. [13]. Recently, researchers utilize neural-symbolic reasoning in tasks including visual question answering [2, 16, 34, 24, 29], decision-making [6], motion programming [18], scene understanding [23]. Focused on activity reasoning, HAKE [21, 20] propose an intermediate space spanned by human body part states as primitives. To overcome its inherent defect in compositional generalization, we propose a novel symbolic system with broad semantics and rational rules.
3.1 Formulating the Symbolic System
3.1.1 Structure
We first analyze the structure of the symbolic system. As aforementioned, the activity symbolic system is composed of rules that connect symbols, which implies human knowledge and reasoning abilities. Formally, one rule involves several symbols, where one of them is the conclusion and the others are premises. Each rule takes the form  $r: m_1 \wedge m_2 \wedge \ldots \wedge m_{n_r} \models c$ , where  $M = \{m_i\}_{i=1}^{n_r}$  is the premise set and  $c$  is the conclusion. Each conclusion corresponds to more than one rule with varied combinations of premise symbols because one activity typically has different visual patterns. Multiple rules for one conclusion are logically connected with  $\vee$ , i.e., one conclusion holds if at least one rule is satisfied. A hyper-graph is a proper data structure to handle the complexity of the connection. Therefore, we define the structure of the activity symbolic system as a  $\mathcal{B}$ -Graph [12].
Definition 1. (Directed Hypergraph [3]) A hypergraph is a generalization of a graph in which an edge can join any number of vertices. A directed hypergraph is a pair  $(\mathcal{X},\mathcal{E})$ , where  $\mathcal{X}$  is a set of vertices, and  $\mathcal{E}$  is a set of pairs of subsets of  $\mathcal{X}$ . Each of these pairs  $(D,C)\in \mathcal{E}$  is a hyperedge; the vertex subset  $D$  is its domain, and  $C$  is its codomain.
Definition 2. (B-Graph [12]) A  $\mathcal{B}$ -graph is a type of directed hypergraph with only  $\mathcal{B}$ -arcs. A  $\mathcal{B}$ -arch is a type of a hyperedge that is directed to a single head vertex, and away from all its other vertices.
Definition 3. (Activity Symbolic System) The activity symbolic system  $(\mathcal{S},\mathcal{R})$  is a  $\mathcal{B}$ -Graph, where  $\mathcal{S}$  is a set of vertices/symbols, and  $\mathcal{R}$  is a set of pairs of subsets of  $\mathcal{S}$ . Each of these pairs  $r = (M,C)\in \mathcal{R}$  is a hyperedge/rule; the vertex subset  $M = \{m_i\}_{i = 1}^{n_r}$  is its domain/premises, and  $C = \{c\}$  is its codomain/conclusion. Equivalently, a rule takes the form  $r:\bigwedge_{m\in M}m\models c$ . Since  $|C| = 1$ ,  $r$  is a  $\mathcal{B}$ -arch.
Definition 4. (Decomposition of Activity Symbolic System) One conclusion  $c^*$  corresponds to one symbolic sub-system  $(\mathcal{S}_{c^*}, \mathcal{R}_{c^*})$ , where  $\mathcal{S}_{c^*} \subseteq \mathcal{S}$ ,  $\mathcal{R}_{c^*} \subseteq \mathcal{R}$ .  $\forall r = (M, C) \in \mathcal{R}_{c^*}$ ,  $C = \{c^*\}$ .  $\forall s \in \mathcal{S}_{c^*} \backslash \{c^*\}$ ,  $\exists r = (M, C) \in \mathcal{R}_{c^*}$ ,  $s \in M$ .
Def. 3,4 is depicted in Fig. 2. Def. 3 is based on Def. 1,2 and analysis above. In applications, we typically judge a specific conclusion with other symbols/rules removed. It is achieved by decomposing the activity symbolic system (graph) into sub-systems (sub-graphs) in Def. 4.
Figure 2: Our activity symbolic system. a) Structure and decomposition of the symbolic system (Sec. 3.1.1). b) Semantic coverage (Sec. 3.1.2). It can be approximated based on LLMs' knowledge and achieved via the symbol-rule loop (Sec. 3.2.1). c) Logical entailment (Sec. 3.1.2). It can be approximated based on an entailment scoring function and achieved by entailment check (Sec. 3.2.2).
3.1.2 Two Ideal Properties
Existing methods [21, 20] can also be formulated under this structure. However, they fail to cover the complex patterns of activities and lack compositional generalization. To overcome the defects, we propose two important ideal properties: 1) symbols should be with broad semantic coverage to express different conditions in the activity image database; 2) rules should satisfy logical entailment [8] to add rationality and avoid ambiguity, i.e., the premises set should lead to the conclusion without exception. They are formulated in Def. 5,6 and illustrated in Fig. 2b,c.
Definition 5. (Semantic Coverage of Activity Symbolic System) Given a very large-scale activity images database  $\mathcal{D} = \{(I, A_I, S_I)\}$  ( $I$ : image,  $A_I$ : ground-truth activities happening in  $I$ ,  $S_I$ : ground-truth symbols happening in  $I$ ), then  $\forall (I, A_I, S_I) \in \mathcal{D}, \forall s \in S_I, s \in S$ .
Definition 6. (Logical Entailment of Activity Symbolic System)  $\forall r = (M, C) \in \mathcal{R}$ ,  $M = \{m_i\}_{i=1}^{n_r}$ , we have:
1.  $\forall (I,A_I,\mathcal{S}_I)\in \mathcal{D},$  if  $M\subset \mathcal{S}_I$  , then  $c\in A_I$
2.  $\forall 1\leq i\leq n_r,\exists (I,A_I,\mathcal{S}_I)\in \mathcal{D},M\backslash \{m_i\} \subset \mathcal{S}_I$  , but  $c\notin A_{I}$
3.2 Instantiating the Symbolic System
To instantiate the proposed symbolic system, it is expensive to collect massive human knowledge via manual annotation as in HAKE [21, 20]. Instead, we use LLMs which have profoundly reshaped the acquisition of human knowledge and shown impressive language reasoning capabilities, i.e., Symbol-LLM framework. In Sec. 3.2.1 and Sec. 3.2.2, we discuss the approximation of the two ideal properties with the help of LLMs and provide practical solutions, then a summarized pipeline is presented in Sec. 3.2.3.
3.2.1 Solving Semantic Coverage
Approximation. In Def. 5, the target domain of semantic coverage is a very large-scale activity image database, which is too costly to implement. For instance, HAKE [21] collect symbols from  $100\mathrm{K}+$  image via manual annotations, which is a heavy workload but still very limited. Here, the target domain can be replaced with semantic coverage of language models since natural language is another carrier of human commonsense knowledge. LLMs profit from large-scale text pre-training and imply abundant text-based human knowledge, thus can handle complex visual activity patterns without extra visual data. The approximation of Def. 5 is:
Definition 7. (Approximation of Semantic Coverage of Activity Symbolic System) Given an LLM  $\mathcal{L}$  and an activity set  $\mathcal{A},\forall A\in \mathcal{A},\mathcal{L}$  implies a symbol set  $S_{A}$  as premises of  $A$  , then  $\forall s\in S_A,s\in S.$
Symbol-Rule Loop. In the implementation, generating rules connecting the symbols is non-trivial and challenging. Intuitively, we can ask LLMs "What is the rule of the activity". However, the instruction is ambiguous for LLMs to generate satisfying answers. More information should be provided to generate rules. Also, it is costly to generate all symbols and exhaustively query their connections. Thus, we propose to adopt symbol-rule loops to generate them alternately. Given a known symbol, we can add another symbol and extend it into a rule via prompting:
(Rule Extension)
In a picture, IF [<known symbols>] AND [condition] THEN [<activity>]. [condition] is one concise phrase. The format is "<The person's hands/arms/hip/legs/feet><verb><object)". What is [condition]?
Then, the answer for “[condition]” is the extended symbol. Also, a candidate rule “<known symbols> ∧ <extended symbol> |= <activity>]” is generated. Thus, we get a new extended symbol from this known symbol, and they are connected with rules. The new symbol can be repeatedly used as a known symbol to generate new rules. As is shown in Fig. 2b, a known symbol (the brown circle) is first “popped out” from a queue, and new symbols (the green circle) and rules (the pink edge) are generated by extending the known symbol. Then, the new symbol is “pushed in” to start the next loop. Thus, LLMs' implicit knowledge is fully exploited to satisfy Def. 7. To get the initial symbols, we ask an LLM to describe hand-related states related to the activity. The prompts mentioned above are determined via a trial-and-error process by human experts on randomly sampled activities (detailed in supplementary).
3.2.2 Solving Logical Entailment
Approximation. In Def. 6, it is costly to verify the logical entailment upon a very large-scale activity image database. Instead, we develop an entailment scoring function from an LLM based on its knowledge and language reasoning capability. As illustrated in Fig. 2c, as the rule extends from  $m_{1} \models c$ ,  $m_{1} \wedge m_{2} \models c$  to  $m_{1} \wedge m_{2} \wedge m_{3} \models c$ , new symbols are successively added, and the measured entailment score increases. With the scoring function, the approximation of Def. 6 is:
Definition 8. (Approximation of Logical Entailment of Activity Symbolic System) Given a function  $\tau(\cdot)$  to measure the entailment score of a rule, and an entailment threshold  $e_h$ , then  $\forall r = (M, C) \in \mathcal{R}$ ,  $M = \{m_i\}_{i=1}^{n_r}$ , we have:
1.  $\tau (\bigwedge_{m\in M}m\models c)\geq e_h$
2.  $\forall 1\leq i\leq n_r$  ，  $\tau (\bigwedge_{m\in M\backslash \{m_i\}}m\models c) <   e_h.$
Entailment Check. As is shown in Fig. 2c, to implement the scoring function based on an LLM, we rewrite the rule  $r$  as a sentence and design the prompt as:
(Entailment Check)
In a picture, <symbol 1>, <symbol 2> ... <symbol \( n_r \>). Estimate the probability that he is <activity> at the same time. Choose from: (a) 0.1, (b) 0.5, (c) 0.7, (d) 0.9, (e) 0.95, (f) unknown.
Then, the output answers can be used as the entailment score  $e = \tau(r)$ , whose credibility depends on the knowledge from the LLM. For one rule, the answering text is sampled five times and their average scores are taken as the final result to add stability. In practice, we set  $e_h = 0.9$ .
3.2.3 Summarized Pipeline
The pipeline of the symbolic system can be summarized as: given a target activity set  $\mathcal{A}$ , or equivalently, a conclusion set  $\mathcal{C}$ , we generate a sub-system  $(\mathcal{S}_c, \mathcal{R}_c)$  for each  $c \in \mathcal{C}$  (Def. 4). The symbolic system is merged as  $(\mathcal{S}, \mathcal{R}) = (\bigcup_{c \in \mathcal{C}} \mathcal{S}_c, \bigcup_{c \in \mathcal{C}} \mathcal{R}_c)$ . Alg. 1 shows how to get the sub-system  $(\mathcal{S}_c, \mathcal{R}_c)$  for a given conclusion  $c$ . Details are explained in Alg. 1 comment and the supplementary.
3.3 Reasoning with Visual Inputs
With the activity symbol system constructed, we can utilize it to reason with visual inputs. As is shown in Fig. 3, visual information is extracted and checked based on the defined symbols and
Algorithm 1 Instantiating the Symbolic System
Input: conclusion  $c$ , entailment threshold  $e_h$   
Output: Symbolic Sub-System  $(S_c, \mathcal{R}_c)$   
1:  $S_c^0 \gets$  Symbol_Initialization  $(c)$   
2:  $S_c^{cand}.push(S_c^0)$   
3:  $S_c \gets \{c\}, \mathcal{R}_c \gets \{\}$   
4: while not  $S_c^{cand}.$  is empty() do  
5:  $m_{kno} \gets S_c^{cand}.pop()$   
6:  $M = \{m_{kno}\}$   
7: while EntailmentChecking  $(M) < e_h$  do  
8:  $m_{new} \gets$  Rule_Extension  $(M, c)$   
9:  $M = M \cup \{m_{new}\}$   
10: end while  
11:  $S_c \gets S_c \cup M$   
12:  $\mathcal{R}_c \gets \mathcal{R}_c \cup \{r : \bigwedge_{m \in M} m \models c\}$   
13:  $S_c^{cand}.push(M \setminus S_c)$   
14: end while  
15: return  $(S_c, \mathcal{R}_c)$
Human ride a boat?
$m_1^{(1)}$ : Hands operate the steering wheel  
 $m_2^{(1)}$ ,  $m_1^{(2)}$ : Hip seated in a boat ....
Figure 3: Visual reasoning with the proposed activity symbol system.
Fuzzy-Logic Calculation
$\wedge (\cdot ,\cdot)\Rightarrow \min (\cdot ,\cdot)$
V  $(\cdot ,\cdot)\Rightarrow \max (\cdot ,\cdot)$
V( ）……
Activity Semantic
organized as a probability distribution on the hyper-graph. Then, the activity semantics can be reasoned out based on the rules and the probability of the symbols. We detail the key steps as follows.
Decomposition. Given the input image and the target activity/conclusion  $c$ , we first decompose the symbol system to exclude unrelated symbols and rules based on Def. 4. Thus, we get the rule set  $\mathcal{R}_c = \{r^{(j)}\}_{j=1}^{N_c}$ , where a rule takes the form  $r^{(j)} : m_1^{(j)} \wedge m_2^{(j)} \wedge \ldots \wedge m_{n_{r(j)}}^{(j)} \models c$ .
Extracting Visual Symbols. Given an image, to extract visual content and express it in a symbolic form, we measure the probability of each defined symbol (Fig. 3). In [20], the visual extractor is a customized model trained with annotations, which is costly, especially for our symbolic system with broad-range symbols. Instead, we utilize a simplified visual extractor based on existing System-1-like VLMs. To determine the probability of a symbol, we convert it into a text question and query the answer from a VLM. For example, given a premise symbol  $m_i^{(j)}$  "Hip seated in a boat", it is converted into a sentence "The person's hip is seated in a boat. Yes/No?". Used as a scoring function [1], the language model outputs the probability of answering "Yes", "No" as  $p_{y,i}^{(j)}, p_{n,i}^{(j)}$ . Then the probability  $p_i^{(j)}$  of the symbol can be obtained via normalizing the Yes/No answer as
$$
p _ {i} ^ {(j)} = \frac {e ^ {p _ {y , i} ^ {(j)}}}{e ^ {p _ {y , i} ^ {(j)}} + e ^ {p _ {n , i} ^ {(j)}}}. \tag {1}
$$
Checking Visual Symbols. To ensure the reliability of the symbols' probabilities, we developed a checker module to filter out visual symbols with uncertain predictions. Each symbol is paraphrased into several variants, and the predictions of variants should be similar due to semantic consistency. Thus, the standard deviation of the predictions can be used to filter symbols with uncertainty.
Reasoning. With the symbol extractor and checker, the probability  $p_i^{(j)}$  for each premise symbol  $m_i^{(j)}$  in the rule set  $\mathcal{R}_c$  is measured. Then, the probability  $p_c$  of the conclusion is reasoned based on premise symbols' probabilities and the rules. Premise symbols within a rule are connected with  $\wedge$ , while rules for a conclusion are connected with  $\lor$ . We utilize the fuzzy-logic calculation [35], where
Figure 4: Detailed process for instantiating the symbolic sub-system of the activity "Human board an airplane". As is shown in the left bottom box, there is some difference in graph presentation between this figure and Fig. 1-3: here the edge connects premises instead of connecting premise and conclusion. The numbers 1-17 correspond to the 17 symbols listed in Tab. 1. The “ $\rightarrow$ ” in “ $1 \rightarrow 6 \rightarrow 7 \rightarrow 8$ ” means that 6, 7, 8 is obtained via 1's rule extension one after another.
Symbols
Round1	(1) hold a boarding pass	(2) place luggage in overhead compartment
(3) adjust seatbelt	(4) wave goodbye to loved ones
(5) grip a luggage handle
Round2	(6) walk towards the boarding gate	(7) luggage visible beside him
(8) boarding pass is scanned by airport staff	(9) stand on the jet bridge
(10) luggage is loaded onto the plane
Round3	(11) reach for the airplane door handle	(12) stand in line with carry-on luggage
(13) hold the carry-on luggage
Round4	(14) open the airplane door	(15) move forward in the line
(16) move towards the airplane door
Round5	(17) airline staff checking the boarding pass
Table 1: Generated symbols & rules for "human board an airplane". Round  $i$  :  $i$  -th symbol-rule loop.
the operator  $x\wedge y,x\vee y$  are replaced with  $\min (x,y),\max (x,y)$  respectively. Thus, we have
$$
p _ {c} = \max  _ {1 \leq j \leq N _ {c}} \left(\min  _ {1 \leq i \leq n _ {r (j)}} p _ {i} ^ {(j)}\right). \tag {2}
$$
The fuzzy-logic calculation is more explainable and lightweight than the logical modules in [20], whose parameters need extra training samples. Benefiting from the sound definition of our symbolic system, it has the potential to achieve better performance even with simplified reasoning calculations.
4.1 Dataset and Metric