SlowGuess/NAACL_2021_OCR huggingface dataset:
Positive Query	Matched Effect	Matched Preconditions	Preconditions to Generate
Eq-Vq-Oq	S-Vq-Oq	S-Vp-Op	Eq-Vp-Op
Ep-Vp-Op	Ep-Vp-Op
Eq-Vq-Oq	S-Vp-Op	S-Vp-Op
Ep-Vp-Op	Ep-Vp-Op
S-Vq-Oq	S-Vq-Oq	S-Vp-Op	S-Vp-Op
Ep-Vp-Op	Ep-Vp-Op
Ep-Vq-Oq	S-Vp-Op	S-Vp-Op
Ep-Vp-Op	Ep-Vp-Op
No matched rule	directly return false
Negative Query	Matched Effect	Matched Preconditions	Preconditions to Generate
Eq-not-Vq-Oq	S-Vq-Oq	S-Vp-Op	Eq-Vp-Op
Ep-Vp-Op	Ep-Vp-Op
Eq-Vq-Oq	S-Vp-Op	S-Vp-Op
Ep-Vp-Op	Ep-Vp-Op
S-not-Vq-Oq	directly return true
No matched rule	directly return true
Table 10: Matching rules for pattern 10 data to generate preconditions that are matched. Note that for negative query, the rule with positive effect is matched. This is used for inverse proof of a negative query (i.e., prove a negative statement is false).
For example, given the query "bob is green" and "all rough things are green", the generated precondition should be "bob is rough". Using formal representations, the representation for "bob is green" is "bob-is-green", and the rule is "(something-is-rough)  $\rightharpoonup$  (something-is-green)". This situation is the row 1 in Table 10. The generated precondition should be  $E_{q} - V_{p} - O_{p}$ , i.e., the subject is the subject from the query, and the verb and the object are from the preconditions of the rule. So the representation of the generated precondition is "bob-is-rough"
A.5 An Actual Output of EVR
The following shows an actual output of our model on the first test problem of depth-4 and Table 11 shows the final result and the evaluation.
Evaluation Metric	Value
Question Label	True
Model's Prediction	True
Model's Proof	((((( triple8 NAF) -> rule8)) -> rule9)) -> rule4) NAF) -> rule8)
All Candidate Proofs Provided by Dataset	['( ((((( ( triple8 NAF) -> rule8)) -> rule9)) -> rule4) NAF) -> rule8)]
Table 11: Final result and the evaluation of our model on the first question of depth-4 test set.
proceeding instance 0
facts:
fact 1: the cat chases the rabbit.
fact 2: the cat is red.
fact 3: the cat sees the rabbit.
fact 4: the cat visits the mouse.
fact 5: the lion is green.
fact 6: the lion visits the rabbit.
fact 7: the mouse does not see the cat.
fact 8: the mouse sees the lion.
fact 9: the mouse visits the cat.
fact 10: the mouse does not visit the lion.
fact 11: the rabbit sees the cat.
rules:
rule 1: if something is red and kind then it does not visit the cat.
rule 2: if something sees the cat then it is not kind.
rule 3: if something does not visit the rabbit then it is big.
rule 4: if something chases the cat then the cat sees the lion.
rule 5: if the rabbit visits the mouse and the rabbit is big then the mouse visits the lion.
rule 6: if something is green then it sees the rabbit.
rule 7: if something chases the rabbit and it sees the mouse then the mouse sees the lion.
rule 8: if something sees the lion and it is not blue then it is kind.
rule 9: if something is kind then it chases the cat.
question and answer:
the cat is kind. True
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to prove "the cat is kind".']
generated operation: GENERATE_SUBGOALS
generated subgoal:i want to judge whether the facts can prove "the cat is kind". OR i want to judge whether the rules can prove "the cat is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the facts can prove "the cat is kind".]
generated subgoal:i want to judge whether fact buffer 1 can prove "the cat is kind". OR i want to judge whether fact buffer 2 can prove "the cat is kind". OR i want to judge whether fact buffer 3 can prove "the cat is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 1 can prove "the cat is kind".]
generated operation: GET( FACT_buffer_1 ) THEN RUN ( EPISODIC_buffer, FACT_buffer_1 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 1 can prove "the cat is kind". fact 1: the cat chases the rabbit. fact 2: the cat is red. fact 3: the cat sees the rabbit. fact 4: the cat visits the mouse. fact 5: the lion is green. operator: RUN  $
$  generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 2 can prove "the cat is kind".]
generated operation: GET( FACT_buffer_2 ) THEN RUN ( EPISODIC_buffer, FACT_buffer_2 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 2 can prove "the cat is kind". fact 6: the lion visits the rabbit. fact 7: the mouse does not see the cat. fact 8: the mouse sees the lion. fact 9: the mouse visits the cat. fact 10: the mouse does not visit the lion. operator: RUN  $
$
generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 3 can prove "the cat is kind".]
generated operation: GET( FACT_buffer_3 ) THEN RUN ( EPISODIC_buffer, FACT_buffer_3 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 3 can prove "the cat is kind". fact 11: the rabbit sees the cat. operator: RUN 
 generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the rules can prove "the cat is kind".]
generated subgoal:i want to judge whether rule buffer 1 can prove "the cat is kind". OR i want to judge whether rule buffer 2 can prove "the cat is kind". OR i want to judge whether rule buffer 3 can prove "the cat is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 1 can prove "the cat is kind".]
generated operation: GET( RULE_BUFFERER_1 ) THEN RUN ( EPISODIC_BUFFERER, RULE_BUFFERER_1 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 1 can prove "the cat is kind". rule 1: if something is red and kind then it does not visit the cat. rule 2: if something sees the cat then it is not kind. rule 3: if something does not visit the rabbit then it is big. operator: RUN
generated answer: false
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 2 can prove "the cat is kind".]
generated operation: GET( RULE_BUFFERER_2 ) THEN RUN ( EPISODIC_BUFFERER, RULE_BUFFERER_2 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 2 can prove "the cat is kind". rule 4: if something chases the cat then the cat sees the
lion. rule 5: if the rabbit visits the mouse and the rabbit is big then the mouse visits the lion. rule 6: if something is green then it sees the rabbit. operator: RUN 
  
generated answer:true
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 3 can prove "the cat is kind".]
generated operation:GET( RULE_BUFFERER_3)THEN RUN(EPISODIC_buffer,RULE_bufferER_3
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 3 can prove "the cat is kind". rule 7: if something chases the rabbit and it sees the mouse then the mouse sees the lion. rule 8: if something sees the lion and it is not blue then it is kind. rule 9: if something is kind then it chases the cat. operator: RUN
generated answer: according to rule 8, i need to prove the cat sees the lion and the cat is not blue.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: according to rule 8, i need to prove the cat sees the lion and the cat is not blue.'] generated operation: GENERATE_SUBGOALS
generated subgoal:i want to prove "the cat sees the lion". AND i want to prove "the cat is not blue".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to prove "the cat sees the lion".']
generated subgoal:i want to judge whether the facts can prove "the cat sees the lion". OR i want to judge whether the rules can prove "the cat sees the lion".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the facts can prove "the cat sees the lion".]
generated subgoal:i want to judge whether fact buffer 1 can prove "the cat sees the lion". OR i want to judge whether fact buffer 2 can prove "the cat sees the lion". OR i want to judge whether fact buffer 3 can prove "the cat sees the lion".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.'] 'episodic buffer: i want to judge whether fact buffer 1 can prove "the cat sees the lion".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 1 can prove "the cat sees the lion". fact 1: the cat chases the rabbit. fact 2: the cat is red. fact 3: the cat sees the rabbit. fact 4: the cat visits the mouse. fact 5: the lion is green. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 2 can prove "the cat sees the lion".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 2 can prove "the cat sees the lion". fact 6: the lion visits the rabbit. fact 7: the mouse does not see the cat. fact 8: the mouse sees the lion. fact 9: the mouse visits the cat. fact 10: the mouse does not visit the lion. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 3 can prove "the cat sees the lion".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 3 can prove "the cat sees the lion". fact 11: the rabbit sees the cat. operator: RUN  $
$  generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the rules can prove "the cat sees the lion".]
generated subgoal:i want to judge whether rule buffer 1 can prove "the cat sees the lion". OR i want to judge whether rule buffer 2 can prove "the cat sees the lion". OR i want to judge whether rule buffer 3 can prove "the cat sees the lion".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 1 can prove "the cat sees the lion".']
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 1 can prove "the cat sees the lion". rule 1: if something is red and kind then it does not visit the cat. rule 2: if something sees the cat then it is not kind. rule 3: if something does not visit the rabbit then it is big. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 2 can prove "the cat sees the lion".']
generated operation:GET( RULE_BUFFERER_2)THEN RUN(EPISODIC_buffer,RULE_buffer_2)
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 2 can prove "the cat sees the lion". rule 4: if something chases the cat then the cat sees the lion. rule 5: if the rabbit visits the mouse and the rabbit is big then the mouse visits the lion. rule 6: if something is green then it sees the rabbit. operator: RUN
generated answer: according to rule 4, i need to prove something chases the cat.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.'] 'episodic buffer: according to rule 4, i need to prove something chases the cat.']
generated subgoal:i want to prove "something chases the cat".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to prove "something chases the cat"'].
generated subgoal:i want to judge whether the facts can prove "something chases the cat". OR i want to judge whether the rules can prove "something chases the cat".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the facts can prove "something chases the cat".]
generated subgoal:i want to judge whether fact buffer 1 can prove "something chases the cat". OR i want
to judge whether fact buffer 2 can prove "something chases the cat". OR i want to judge whether fact buffer 3 can prove "something chases the cat".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 1 can prove "something chases the cat".]
generated operation: GET( FACT_BUFFER_1 ) THEN RUN ( EPISODIC_BUFFER, FACT_BUFFER_1 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 1 can prove "something chases the cat". fact 1: the cat chases the rabbit. fact 2: the cat is red. fact 3: the cat sees the rabbit. fact 4: the cat visits the mouse. fact 5: the lion is green. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 2 can prove "something chases the cat".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 2 can prove "something chases the cat". fact 6: the lion visits the rabbit. fact 7: the mouse does not see the cat. fact 8: the mouse sees the lion. fact 9: the mouse visits the cat. fact 10: the mouse does not visit the lion. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 3 can prove "something chases the cat".']
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 3 can prove "something chases the cat". fact 11: the rabbit sees the cat. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the rules can prove "something chases the cat".]
generated subgoal:i want to judge whether rule buffer 1 can prove "something chases the cat". OR i want to judge whether rule buffer 2 can prove "something chases the cat". OR i want to judge whether rule buffer 3 can prove "something chases the cat".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 1 can prove "something chases the cat".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 1 can prove "something chases the cat". rule 1: if something is red and kind then it does not visit the cat. rule 2: if something sees the cat then it is not kind. rule 3: if something does not visit the rabbit then it is big. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 2 can prove "something chases the cat".']
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 2 can prove "something chases the cat". rule 4: if something chases the cat then the cat sees the lion. rule 5: if the rabbit visits the mouse and the rabbit is big then the mouse visits the lion. rule 6: if something is green then it sees the rabbit. operator: RUN 
 generated answer: false
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.'] 'episodic buffer: i want to judge whether rule buffer 3 can prove "something chases the cat".]
generated operation: GET( RULE_BUFFERER_3 ) THEN RUN ( EPISODIC_BUFFERER, RULE_BUFFERER_3 )
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 3 can prove "something chases the cat". rule 7: if something chases the rabbit and it sees the mouse then the mouse sees the lion. rule 8: if something sees the lion and it is not blue then it is kind. rule 9: if something is kind then it chases the cat. operator: RUN
generated answer: according to rule 9, i need to prove something is kind.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: according to rule 9, i need to prove something is kind.']
generated subgoal:i want to prove "something is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to prove "something is kind".']
generated subgoal:i want to judge whether the facts can prove "something is kind". OR i want to judge whether the rules can prove "something is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the facts can prove "something is kind".']
generated subgoal:i want to judge whether fact buffer 1 can prove "something is kind". OR i want to judge whether fact buffer 2 can prove "something is kind". OR i want to judge whether fact buffer 3 can prove "something is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 1 can prove "something is kind".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 1 can prove "something is kind". fact 1: the cat chases the rabbit. fact 2: the cat is red. fact 3: the cat sees the rabbit. fact 4: the cat visits the mouse. fact 5: the lion is green. operator: RUN  $
$  generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.'] 'episodic buffer: i want to judge whether fact buffer 2 can prove "something is kind".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 2 can prove "something is kind". fact 6: the lion visits the rabbit. fact 7: the mouse
does not see the cat. fact 8: the mouse sees the lion. fact 9: the mouse visits the cat. fact 10: the mouse does not visit the lion. operator: RUN 
  
generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 3 can prove "something is kind".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 3 can prove "something is kind". fact 11: the rabbit sees the cat. operator: RUN 
 generated answer: false, CWA.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the rules can prove "something is kind".']
generated subgoal:i want to judge whether rule buffer 1 can prove "something is kind". OR i want to judge whether rule buffer 2 can prove "something is kind". OR i want to judge whether rule buffer 3 can prove "something is kind".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 1 can prove "something is kind".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 1 can prove "something is kind". rule 1: if something is red and kind then it does not visit the cat. rule 2: if something sees the cat then it is not kind. rule 3: if something does not visit the rabbit then it is big. operator: RUN 
  
generated answer: false
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.'] 'episodic buffer: i want to judge whether rule buffer 2 can prove "something is kind".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 2 can prove "something is kind". rule 4: if something chases the cat then the cat sees the lion. rule 5: if the rabbit visits the mouse and the rabbit is big then the mouse visits the lion. rule 6: if something is green then it sees the rabbit. operator: RUN 
  
generated answer: false
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether rule buffer 3 can prove "something is kind".]
generated operation:GET( RULE_BUFFERER_3)THEN RUN(EPISODIC_buffer,RULE_buffer_3)
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether rule buffer 3 can prove "something is kind". rule 7: if something chases the rabbit and it sees the mouse then the mouse sees the lion. rule 8: if something sees the lion and it is not blue then it is kind. rule 9: if something is kind then it chases the cat. operator: RUN  $
$
generated answer: according to rule 8, i need to prove something sees the lion and something is not blue.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: according to rule 8, i need to prove something sees the lion and something is not blue.']
generated subgoal:i want to prove "something sees the lion". AND i want to prove "something is not blue".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to prove "something sees the lion".']
generated subgoal:i want to judge whether the facts can prove "something sees the lion". OR i want to judge whether the rules can prove "something sees the lion".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the facts can prove "something sees the lion".']
generated subgoal:i want to judge whether fact buffer 1 can prove "something sees the lion". OR i want to judge whether fact buffer 2 can prove "something sees the lion". OR i want to judge whether fact buffer 3 can prove "something sees the lion".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 1 can prove "something sees the lion".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 1 can prove "something sees the lion". fact 1: the cat chases the rabbit. fact 2: the cat is red. fact 3: the cat sees the rabbit. fact 4: the cat visits the mouse. fact 5: the lion is green. operator: RUN
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether fact buffer 2 can prove "something sees the lion".]
buffer input: episodic buffer: there are 3 fact buffers and 3 rule buffers. episodic buffer: i want to judge whether fact buffer 2 can prove "something sees the lion". fact 6: the lion visits the rabbit. fact 7: the mouse does not see the cat. fact 8: the mouse sees the lion. fact 9: the mouse visits the cat. fact 10: the mouse does not visit the lion. operator: RUN
generated answer:true, this is confirmed by fact 8.
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to prove "something is not blue".]
generated subgoal:i want to judge whether the facts do not contradict "something is not blue". AND i want to judge whether the rules do not contradict "something is not blue".
episodic buffer ['episodic buffer: there are 3 fact buffers and 3 rule buffers.', 'episodic buffer: i want to judge whether the facts do not contradict "something is not blue".']