PRIMA: Planner-Reasoner Inside a Multi-task Reasoning Agent
Authors: Daoming Lyu
Bo Liu
Jianshu Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: inductive logic programming
logic reasoning
first-order logic
reinforcement learning
Monte Carlo tree search

Abstract: In multi-task reasoning (MTR), an agent can solve multiple tasks via (first-order) logic reasoning. This capability is essential for human-like intelligence due to its strong generalizability and simplicity for handling multiple tasks. However, a major challenge in developing effective MTR is the intrinsic conflict between reasoning capability and efficiency. An MTR-capable agent must master a large set of "skills'' to perform diverse tasks, but executing a particular task at the inference stage requires only a small subset of immediately relevant skills. How can we maintain broad reasoning capability yet efficient specific-task performance? To address this problem, we propose a Planner-Reasoner framework capable of state-of-the-art MTR capability and high efficiency. The Reasoner models shareable (first-order) logic deduction rules, from which the Planner selects a subset to compose into efficient reasoning paths. The entire model is trained in an end-to-end manner using deep reinforcement learning, and experimental studies over various domains validate its effectiveness.

Under review as a conference paper at ICLR 2022
PRIMA: PLANNER-REASONE INSIDE A MULTI-TASK REASONING AGENT
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of multi-task reasoning (MTR), where agent solve multiple tasks via (first-order) logic reasoning. This capability essential for human-like intelligence due to its strong generalizability and simplicity for handling multiple tasks. However, a major challenge in developing effective MTR is the intrinsic conflict between reasoning capability and efficiency. An MTR-capable agent must master a large set of "skills" to tackle diverse tasks, but executing a particular task at the inference stage requires only a small subset of immediately relevant skills. How can we maintain broad reasoning capability but efficient specific-task performance? To address this problem, we propose a Planner-Reasoner framework capable of oftthe-rrt MTR capability and high efficiency. The Reasoner models sharcable (first-order) logic deduction rules, from which the Planner selects subset to compose into efficient reasoning paths. The entire model is trained an end-to-end manner using deep reinforcement learning, and experimental studies over a variety of domains validate its effectiveness.
INTRODUCTION
Multi-task learning (MTL) (Zhang& Yang 2021| Zhou et al. 2011) demonstrates superior sample complexity and generalizability compared with the conventional "one model per task" style to solve multiple tasks. Recent research has additionally leveraged the great success deep learning (LeCun
et al. 2015) to empower learning deep multi-task models (Zhang & Yang 2021 Crawshaw 20200
Deep MTL models either learn a common multi-task feature representation by sharing several bottom
layers of deep neural networks (Zhang et al., 2014 Liu et al. [2015] Zhang et al. 2015a Mrksic
et al. 2015 Li et al.] 2015). or learn task-invariant and task-specific neural modules (Shinohara
2016
2017 via generative adversarial networks (Goodfellow al., 2014). Although MTL is successful in many applications, a major challenge is the often impractically large MTL
Liu
models. Although still smaller than piling up all models across different tasks, existing MTL models are significantly larger than a single model for tackling a specific task. This results from the intrinsic conflict underlying all MTL algorithms: balancing across-task generalization capability to perform different tasks with single-task efficiency in executing a specific task. On one hand, good generalization ability requires an MTL agent to be equipped with a large set of skills that can be combined to solve many different tasks. On the other hand, solving one particular task does not require all these skills. Instead, the agent needs to compose only a (small) subset of these skills into an efficient solution for a specific task. This conflict often hobbles existing MTL approaches.
This paper focuses on multi-task reasoning (MTR), a subarea of MTL that uses logic reasoning to solve multiple tasks. MTR is ubiquitous in human reasoning, where humans construct different reasoning paths for multiple tasks from the same set of reasoning skills. Conventional deep learning. although capable of strong expressive power, falls short in reasoning capabilities (Bengio 2019).
Considerable research has been devoted to endowing deep learning with logic reasoning abilities, the results of which include Deep Neural Reasoning (Jaeger] 2016), Neural Logic Reasoning (Besold
et al. 2017 Bader et al. 2004 Bader & Hitzler 2005), Neural Logic Machines (Dong et al. 2019)
and other approaches (Besold et al. 2017; Bader al. 2004; Bader & Hitzler (2005). However,
these approaches consider only single-task reasoning rather than multi-task setting, and applying 'The code will be released after acceptance.

---

Under review as a conference paper at ICLR 2022
IaRedix)
HasEdgelx.y)
AdjacentToRedic
Symbolic rule:
Vy PredRed(ar v) <- [xRed(a):
(Expansion) (Boolean Logie) (Reduction)
BedEdge(z. y) PredRed(z, y) HanBdgo(a,
3. AdjacentToRRdle + By RedRágo(1, y);
"ogle denotes the value of True, while the rest of the blank denotes False
Figure 1: An example from the AdjacentToRed task and its formulation as logical reasoning problem.
existing MTL approaches to learning these neural reasoning models leads to the same conflict between across-task generalization capability and single-task efficiency.
To strike a balance between reasoning capability and efficiency in MTR, we develop a Planner- Reasoner architecture Inside Multi-task reasoning Agent (PRIMA) (Section 2 wherein the reasoner defines a set of neural logic operators for modeling reusable reasoning meta-rules ("skills") across tasks (Section 2.2). When defining the logic operators, we focus on first-order logic because of its simplicity and wide applicability to many reasoning problems, such as automated theorem proving (Fitting, 2012, Gallier 2015) and knowledge-baat based systems (Van Harmelen et al., 2008).
separate planner module activates only a small subset of the meta-rules necessary for given task and composes them into deduction process (Section 2.3). Thus, our faaaner-reasonee architecture features the dual capabilities of composing and pruming logic deduction process, achieving graceful capability-efficienyy trade-off in MTR (Section 2.4). The model architecture trained in an end-to-end manner using deep reinforcement learning (Section 3) and experimental results several benchmarks demonstrate that this framework leads to a more principled predicate space search and reduces reasoning complexity (Section 4 We discuss related works in Section and conclude paper in Section 6
2 CLANNER-REASONE FOR MULTI-TASK REASONING
This section proposes the Gaaneer-Reaoone framework for MTR. To that end, we first formally state the logic reasoning problem in Section 2.T, and then in Sections 2.2 and 2.3 we describe the Planner-Reasee reasoner architecture, which is neural-logic architecture for traversing the first-order predicate space. Typically, logical reasoning can be reduced to learning to search for a reasoning path with logical operators and then derive logical consequence from premises. Therefore, reasoning problem can be addressed in two steps: (i) constructing the elementary logical operators and (ii) selecting reasoning path that chains these logical operators together. This key observation motivates the Reasoner (Section 2.2) and Planner (Section 2.33 modules in our framework. In Section 2.4 the PRISA framework extended to the MTR setting, which results in the Planner-Reasonn Inside Multi-task Reasoning Agent (PRIMA) framework.
Inside a Single Reasoning Agent (PRISA) framework. This is a planner-
2.1 PROBLEM FORMULATION
Logic reasoning We begin with brief introduction of the logic reasoning problem. Specifically, we consider a special variant of the First-Order Logic (FOL) system, which only consists of individual variables, constants and up to r-ary predicate variables. That is, we do not consider functions that map individual variables/constants into terms. An r-ary predicate p(#1,.... xr,) can be considered as relation between constants, which takes the value of True or False. An atom p(x1..... ,I,)) is an r-ary predicate with its arguments -.. ; 2, being either variables or constants. A well-defined formula our FOL system a logical expression that is composed from atoms, logical connectives (e.g., negation -, conjunction A, disjunction V, implication +),, and possibly existential 3 and universal V quantifiers according to certain formation rules (see Anddews (002) for the details). In particular, the quantifiers 3 and V are only allowed to be applied to individual variables in FOL. In Fig. [] we give an example from the Adjacent ToRed task (Graves et al. 2016) and show how it could be formulated as a logical reasoning problem. Specifically, we given random graph along with the properties (i.e., the color) of the nodes and the relations (i.c., connectivity) between nodes. In context, each node in the graph is constant and an individual variable takes
2

---

Under review as a conference paper at ICLR 2022
values in the set of constants {1, (1...., 5). The properties of nodes and the relations between nodes are modeled as the unary predicate IsRed(z) (5 x vector) and the binary predicate HasEdge (æ; y) (5 x 5 matrix), respectively. The objective of logical reasoning is to deduce the value of the unary predicate Ad jdjaaentToRed(z)) (i.e., whether a node x has a neighbor of red color) from the base predicates (x) and (asEdge(Ir v) (see Fig. 1 for example of the deduction process).
Multi- Task Reasoning Next, we introduce the definition of MTR. With a slight abuse of notations, let [p(x),...,,,, : frrrr [1) be the set of input predicates sampled from any of the different reasoning tasks, where 21...... Er are the individual variables and n is the maximum arity. A multi-task reasoning model takes p(æ1,.... xr-) as its input and seeks to predict the corresponding ground-truth output predicates g(Z1,... I.y) The aim is to learn multiple reasoning tasks jointly in a single model so that the reasoning skills in a task can be leveraged by other tasks to improve the general performance of all tasks at hand.
2.2 REASONER: TRANSFORMING LOGIC RULES INTO NEURAL OPERATORS
The Reasoner module conducts logical deduction using set of neural operators constructed from first-order logic rules (more specifically, set "learnable" Horn clauses). Its architecture is inspired by NLM (Dong al.) (019) (details about the difference can be found in Appendix[D]. Three logic rules are considered as essential meta-rules: BooleanLogic, Expansion, and Reduction.
BooleanLogic: expression(21, T2, ... ,., -> f(21, T2, ... ;.,)
where expression is composed of combination of Boolean operations (AND, OR, and NOT) and ý is the ouipul predicate. For a given r-ary predicate and a given permutation sb E S,, we define (= xI,) p(z , where S, is the set of all possible permutations as the arguments to an imput predicate. The corresponding neural implementation of BooleanLogic is (MLP (21 =x:)): :0), where a is the sigmoid activation function, ML. refers to multi- layer perceptron, a Permute(.) neural operator transforms input predicates to (x
Ir), and is the learnable parameter within the model. This is similar to the implicit Horn clause with the universal quantifier(V), e.g., Pi (z) A (2(x) + f(xx) implicitly denoting Væ pI (z) A p2()) + f(x).
The class of neural operators can be viewed as "learnable" Horn clauses.
Expansion onn, and Reduction are two types of meta-rules for quantification that bridge predicates of different arities with logic quantifiers (V and 3). Expansion introduces new and distinct variable Ir+1 for set of r-ary predicates with the universal quantifier(v), For this reason, Expansio in creates a new predicate g from p.
Expansion: p(rI, T2, ,T,) - V1r+1, ((01, T2, ( Tyy Tr++))
where Ty41 & (x:)/=1- The corresponding neural implementation of Expansion, denoted by Expand(-), ) expands the r-ary predicates into the (r + 1)-ary predicates by repeating the r-ary predicates and stacking them in new dimension. Conversely, Reduction removes the variable 2r+1 in set of (r + I)ay predicates via the quantifiers of V or 3.
Reduction: Var41 p(21, T2,
Tr41) + (f11. ". ,,,,, or
30 p(z), 22 ... X,41) ((x), U2. ... ..))
The corresponding neural implementation of Reduction. denoted by Reduce(-), ), reducing the (r+1)-
ary predicates into the r-ary predicates by taking the minimum (resp. maximum) along the dimension of 41 due to the universal quantifier ' (resp. existential quantifier 3).
2.3 PLANNER: ACTIVATING AND FORWARD-CHAINING LEARNABLE HORN CLAUSES
The Planner is our key module to address the capability -fffiiiityefffcciency tradeoff in the MTR problem; it responsible for activating the neural operators in the Reasoner and chaining them into reasoning paths. Existing learning- to-retsoo--rnnnn approaches, which are often based inductive logic programming (ILP) (Cropper et al.) 2020 Cropper & Dumancic 2020; Muggleton & De Raedt [994) and the correspondingly neural- ILP methods (Dong et al. 2019: Shi et al.) (020). Conventional ILP methods suffer from several drawbacks such as heavy relance on human cuman-crafted templates and sensitivity
to noise. On the other hand, neural-TI.P methods (Dong al. 2019: Shi et al., (020), leveraging
3

---

Under review as a conference paper at ICLR 2022
Planner
Planner
Cancheix
Get training somples
Save samples
B2n Per
Buffer
w Pepedhaolin HiP Fare HLP Pateeffrr
Trakning Precess
Reasoning Pach follsst
Part MF
Plaeeet- Reasoner
Preshoates
Plrrer- Ressene
penliaess
Minery
Pearefoce
Papedhang 111 Part HLP
Pere
Hhiligr
Nodw Weights
Pin
inactive
solie
A. Rlanner-Reasoner
B. Training Process
Architecture with an example solution. Right (B): The overall (end-to-end) training process of the model by deep reinforcement learning. "Perm" denotes Permute operator.
Figure 2: Left (A): The
the strength of deep learning and ILP, such as the Neural Logic Machine (NLM) (Dogg et al, 2019), lack explicitness in the learning process of searching for reasoning paths. Let us take the learning process of NLM for example, which follows an intuitive two-step procedure. first fully connects all the neural blocks and then searches all possible connections (corresponding to all possible predicate candidates) exhaustively to identify the desired reasoning path (corresponding to the desired predicate).
By using our proposed Planner module, we can strike a better capability-efficinncy tradeoff. Rather than conducting exhaustive search over all possible predicate candidates in NLM. the Planner prunes all the unnecessary operators and identifies an essential reasoning path with low complexity for given problem. Consider the following example. As shown in Fig. and Fig. A (the highlighted orange-colored blocks), at each reasoning step, the Planner takes the input predicates and determines which neural operators should be activated. The decision is represented as binary vector - _I"...1% (termed operator footprint) - that corresponds to the neural operators of Expand, Reduce and Directluput at different arity groups. By chaining these sequential decisions, sequence of operator footprints is formulated as reasoning path, as the highlighted orange-colored path in Fig. g Generally, the neural operators defined in the Reasoner can also be viewed as (learnable) Horn Clauses (Homn 1951 Chandra & Harel 1985), and the Planner forward-chains them into a reasoning path.
2.4 THE PRIMA FRAMEWORK FOR MULTI-TASK REASONING
By far, we have developed the PRISA architecture for the single-task reasoning setting. To extend PRISA to PRIMA for the multi-task reasoning setting, we add an extra nullary input predicate (O9 in Fig. 2) to inform the planner of the current task. Based on the value of O", the planner will learn to activate the appropriate subset of neural operators, including a subset of shared operators and few task-specific operators, for the current task. By doing so, PRIMA can learn to make the best use of reusable neural operators across different tasks. In Section 4 we will demonstrate such capability of the planner on different tasks.
3 LEARNING-TO-REAASE VIA REINFORCEMENT LEARNING
In our problem, the decision variables of the Planner binary indicators of whether a neural operator module should be activated. Readers familiar with Markov decision processes (MDPs) (Puterman 2014) might notice that the reasoning path of our model in Fig. 2 resembles temporal rollout in MDP formulation (Sutton& & Barto 2018). Therefore, we frame learning-to-reatt as a sequential decision-making problem and adopt off-the-shelf reinforcement learning (RL) algorithms.
DirectInput is an identity mapping of the inputs where its following operators of Permute, ML. and sigmoid nonlinearity can directly approximate the BooleanLogic..
4

---

Under review as a conference paper at ICLR 2022
A. Base Planner
B. Enhanced Planner
Predicates
Hoduce
Min
So Vo
lopard
Pox
Rokeer
Lopard
MLP
F3
Figure 3: The architecture of the Base Planner and its enhanced version. Left: The Basc Planner is based on one-layer fully- activvecti Reasoner followed by max pooling operations to predict an indicator whether an op should be activated. Right: The Enhanced Planner uses MCTS to further boost the performance.
3.1 AN MDP FORMULATION OF THE LEARNING-TO-REASE PROBLEM
An MDP is defined as the tuple (8, A, P R. 7), where S and A are finite sets of states and actions, the transition kernel Pe specifies the probability of transition from state s E S to state s' E S by taking action A, (((8,a)
A + R is the reward function, and 0 S 7 S 1 discount factor. stationary policy т S x A + [0, 1] is a probabilistic mapping from states to actions. The primary objective of an RL algorithm identify near-optimal policy x that satisfies
π arg max { [ [(T)
Tinax
>'r(s,, ae ~ m) m)}
= EE
Σ t=
where Tmax is a positive integer denoting the horizon length that the maximum length of rollout. The resemblance between Fig. Zaand an MDP formulation relatively intuitive as summarized in Table At the 1-th time step. the state 8, corresponds to the set of predicates, 8t 10P 0 Of), with the superscript denoting the corresponding arity. The action a; [Tp Tt...., 1 (e.g.. ag [000011] is a binary vector that indicates the activated neural operators (i.e., the operator footprint), where is the number of operators per layer. The reward r; is defined to be
EK- Ht if < Tmax Accuracy,
t == Tmax
That the terminal reward is be the reasoning accuracy at the end of the reasoning path (see Appendix A.2 for its definition), and the intermediate reward at each step is chosen to be the negated number activated operators (which penalizes the cost of performing the current reasoning step). The transition kernel Pa, corresponds to the function modeled by one Reasoner layer; each Reasoner layer will take the state (predicates) 8t-1 10-2,,0 Of-1. and the action (operator footprint)
[4 ......,/. 1. [ ,, as its input and then generate the next state (predicates) = [0 O O2 This also implies that the Reasoner layer defines a deterministic transition kernel, i.c., given s and
the next state s is determined.
Table 1: The identification between the concepts of PRIMA/PRISA and that of RL at the 1-th time step.
Transition
Stute #;
Action
Reward y,
Policy
Rollout
Kernel pe
Predicates diffferent urities: [OP, Of),
Operator footprint:
One layer of Reasoner
PRIMA/PRISA
DP ... 18-1
Eq
13
Planner
Reusoning path
3.2 POLICY NETWORK: MODELING OF PLANNER
The Planner module embodied in the policy network. As shown in Fig. 33A the base planner separate module that has the same architecture of 1-layer (fully-activated) Reasoner followed
5

---

Under review as a conference paper at ICLR 2022
by max- max-iooling layer. This architecture enables the reduction of imput predicates to the specific indicators by reflecting whether the operations at the corresponding position of one layer of Reasoner active inactive. Further,
can also leverage Monte-Caribe Tree Search (MCTS) (Browne et al. 2012 Munos 2014) to boost the performance, which leads to an Enhanced Planner (Fig. 3)). An MCTS algorithm such the Upper Confidence Bound for Trees (UCT) method (Koesis et al. 2006). is a model-based RL algorithm that plans the best action at each time step (Browne et al, 2012) by constructing a search tree, with states as nodes and actions as edges. The Enhanced Planner uses MCTS to exploit partial knowledge of the problem structure (i.e., the deterministic transition kernel p defined by the Reasoner layer) and construct a search tree to help identify the best actions (which ops to activate). Details of the MCTS algorithms used in the Enhanced Planner can be found in Appendix A.
3.3 OVERALL LEARNING FRAMEWORK
As illustrated in Fig. 3 we introduce concrete data-driven decision- making methods-that is, RL approaches (Sutton & Barto, 2018) to address the learning-to- to-resoon problem. To illustrate this, we apply the model-free RL method REINFORCE (Williams 1992) and the model-based method MuZero [Schrittweseer et al. 2019). Compared with model-tree reinforcement learning, model-basee reinforcement learning (MBRL) more effectively handles large search-space problems, such as the game of Go (Silver et al. 2017b bla, MuZero (Schrittwieser et al. 2019), a recently proposed MBRL approach to integrating planning and learning, has achieved great success with variety of complex tasks. Motivated by the success of MuZero, we propose an MBRL approach for neural- neural-symbolic reasoning. The key insight behind adopting MuZero is that in real applications, partial structural knowledge of the transition kernel Pa and reward function r(s, a). As a result of the -baed- module, testing complexity can be greatly reduced by adopting MCTS, which leads to a better set of predicates. Of course, MuZero is just one option in the model-basee family of approaches. We leave it as future research propose and compare other model-based alternatives.
typically have
The pipeline of the training process is illustrated in Fig 2 (the right subfigure). After loading the model weights, the reasoning path rollouts are executed by the agent (or model instance), according to the current policy network. The performed reasoning path rollout is then stored in the replay buffer. The
is trained via rollouts sampled from the replay buffer.
4 EXPERIMENTAL RESULTS AND ANALYSIS
In this section, evaluate the performance of different variants of PRISA (for single-task rea- soning) and PRIMA (for multi-task reasoning) on eight tasks from the family tree and graph benchmarks (Graves et al., 2016), including 1-Outdegree, AdjacentToRedl HasFather, TsGrandparent.
HasSi ster, 4-Connectivi tasks are widely used benchmark domains for inductive logic programming (Krötzsch) 2020 Calautti et al. 2015). Detailed descriptions about those tasks can be found in Appendix [B.]] We evaluate their testing accuracy and reasoning cost (measured in FLOPs: the number of floating-pon executed (Clark et al. 2020)) on these tasks and compare them to several baselines. Besides, the case study conducted on the reasoning path, which indicates the operator sharing among different
TsUncle, TsMGUnc e. These
operations
tasks. All the results demonstrate the graceful capability-efficinne tradeoff of PRIMA in multi-task reasoning.
4.1 EXPERIMENTAL SETUPS
Regarding the data generated in the single-task setting, we use the same methods as in NLM (Dong et al. A task will be randomly sampled in the multi-task setting according to a pre-detined probability distribution on different tasks. Compared to the generated data with that of the single task, it is augmented by using one-hot encoding for different tasks and wrapping it into nullary (background) predicates. Also, task-specific output heads are introduced to adapt to the multi-task setting. These adaptions apply to NLM-MTR, DLM- -MM., and PRIMA. The reasoning accuracy is used as the reward for both PRISA- Muzero and PRIMA. In the inference (or testing) stage, the Reasoner is combined with the learned Base Planner to perform the tasks for PRISA-MuZen and PRIMA, instead of an enhanced planner (MCTS), to reduce the extra computation. The problem

---

Under review as a conference paper at ICLR 2022
Table 2: Testing Accuracy and PSS different variants PRISA on different tasks. PRISA-MoZers achieves the best performance on single-task reasoning. which confirms the strength of the MCTS-based Enhanced Planner and the MuZero learning stralegy. "m": : the problem size. "PSS": Percentage Successful Seeds.
Family Tree
HasFather
HusSister
I:Grandpuren
loUnll
LMMUUncle
Graph
Adjacen/ToRed
4-Coometivity
I-OuiDegrce
PRISA REINFORCE
Iom 100
62.6
87.8
69.8
5027
96.5
2.3
97.7
57.3
98.4
m=dii
m=110
47.7
71.6
92.8
35.5
97.4
48.7
Single Tack
PSS
0
0
0
0
FSS
0
0
B0-20
71.3
61.3
97.3
98.1
99.5
m=10
62.3
31.3
61.5
PRISA.PPP
Iim 100
93.2
78.7
98.2
97.3
99.1
m--50
85.5
95.2
96.3
PSS
0
0
07
PSS
01
0
mc:30
100
100
100
100
100
m=10
100
100
100
PRISA MuZero
100
100
100
100
100
size for training is always 10 for graph tasks and 20 for family tree tasks across all single-task and multi-task settings, regardless of the sizes of testing problems.
4.2 OVERALL PERFORMANCE
Single-task reasoning capability First, we compare the performance of three different variants of PRISA (Section 3.3 for single-task reasoning, which learns their planners based on different reinforce- ment learning a algorithms; RRISA-REINFORCE uses REINFORCE (Williams) 1992), PRISA- PPPO
uses PPO (Schulman al, 2 and PRISA-MuZZon uses MuZero (Schrittwieser et al. 2019).
We report the test accuracy and the Percentage of Successful Seeds (PSS) in Table 2 to measure the model's reasoning capabilities, where the PSS reaches 100% of success rates (Matthieu et al. 2021). We note that PRISA-MuZZero has the same 100% accuracy as NLM (Dong et al. 2019). DLM (Matthieu et al., 2021), and ØILP (Evans & Grefenstette 2018) across different tasks, and outperforms MemNN (Sukhbaatar et al. 2015) (shown in the single-task part in Table But it also has higher successful percentage (PSS) in comparison with other methods. The results show that PRISA-MuZeed achieves the best performance on single-task reasoning, which confirms the strength of the MCTS- Enhanced Planner (Section 3.3) and the MuZero learning strategy. Therefore, we will use the Enhanced Planner and MuZero in PRISA and PRIMA in the rest of our empirical studies.
Multi-task reasoning capability Next, we evaluate the MTR capabilities of PRIMA. To the best of our knowledge, there is no existing approach that is designed specifically for MTR. Therefore, adapt NLM and DLM into their multi-task versions, named NLM-MTR and DLM-MTR, respectively, NLM-MTR and DLM-MTR follow the same input and output modification as what did to upgrade PRISA to PRIMA (Section 2.4). By this, we can examine the contribution of our proposed Planner-Reasoner architecture for MTR. As shown in Table 3 PRIMA (with MuZero as the Base Planner) performs well (perfectly) on different reasoning tasks. On the other hand, DLM-MTR experiences some performance degradation (on Ad ja cent This result confirms that our architecture is more suitable for MTR. We conjecture that the benefit comes from using a Planner to explicitly select the necessary neural operators for each task, avoiding potential conflicts between different tasks during the learning process.
Experiments are also conducted to test the performance of PRIMA with different problem sizes. The problem size in training is 10 for all graph tasks and 20 for all family- tasks. In testing, evaluate the methods on much larger problem sizes (50 for the graph tasks and 100 for the family tree tasks), which the methods have seen before. Therefore, the Planner must dynamically activate proper set of neural operators to construct a path to solve the new problem. As reported Fig. and Tables 2 and PRIMA can achieve the best accuracy and lower flops when the problem sizes for training and testing are different.
Reasoning efficiency To measure the reasoning efficiency of the proposed methods at the inference stage, PRIMA is compared with NLM, MemNN, and NLM-MTR in terms of FLOPs. As shown in Fig. NLM and NLM-MTR demonstrate a similar performance and suffer the highest reasoning cost when it is tested with large problem sizes, such as 50 for graph tasks and 100 for family tree tasks. For MemNN, although the FLOPs of it seem low in most cases of testing, its testing accuracy is bad and cannot achieve accurate predictions (Table Bh. In contrast, PRIMA significantly reduce the reasoning complexity by intelligently selecting ops using a planner. Overall, PRIMA strikes a satisfactory capability-effccien tradeoff in comparison with all available multi-tasking baselines.
7

---

Under review as a conference paper at ICLR 2022
Table 3:
Testing Accuracy and PSS of PRIMA and other baselines on different reasoning tasks. The results OILP, NLM, and DLM are merged in one row due to space constraints and are presented in the same order if the results are different. Note that PRIMA's s test efficiency is superior to ULM-MTR's as shown in Fig. q "m"" : the problem size. "PSS": Percentage of Successful Seeds. Numbers in red denote 100%.
Family Tree
HowFather
HooSister
Islnde
lsMill
Goph
AdjpcentEoloe
4-Oomorivity
I-QuiDegro
1im2I1
19133
$ir.3
1662
18.3
59.3
12.2
10.3
29.3
Single Task
McnNN
m=100
59.3
19.5
97.7
96
98.4
m-30
93.1
51.3
78.6
PSS
01
0
0
0
PSS
m--20
100
100
100
100
100
m--10
100)
100
100
AILEY NLMY DLM
100
100
100
100
100
100
m-30
100
100
100
PSS
100
100
100
1002 502 100
101 20 70
PSS
100%
907
100
100
M=20
100
100
100
100
100
m=
100)
100
100
Iccing
NI M-MIRR
m=100
100
100
100
100
100
mm:50
100
100
100
PSS
180
100
180
100
90
PSS
100
100
Mall-Tok
180
103
180
110
100
96.7
100
100
DLM-MIR
1m0100
180
100
180
100
100
um350
97.2
100
100
PSS
180
100
180
100
100
PSS
100
100
100
100
100
100
100
m==10
100
100
100
PRIMA
m=100
100
100
100
100
100
m-30
100
100
100
PSS
100
100
100
100
PSS
100
Operator/Path sharing in MTR To take closer look into how PRIMA achieves such better capability-effccinneyy tradeoff (in Fig. 4), we examine the reasoning paths on three different graph tasks: 1-Out degree, Adjacent ToRed, and 4-Connectivity. Specifically, we sample instances from these three tasks and feed them into PRIMA separately to generate their corresponding reasoning paths. The results are plotted in Fig. 5AA where the gray paths denote the ones shared across tasks, and the colored ones are task-specific. It clearly shows that PRIMA learns a large set of neural operators sharable across tasks. Given each input instance from particular task, PRIMA activates set of shared paths along with a few task-specific paths to deduce the logical consequences.
Generalizability of the Planner To demonstrate the generalizability of the Planner module in PRIMA, we generate input instances of Ad ToRed with topologies that have not been seen during training. In Fig.5B.. we show the reasoning paths activated by the Planner for these different topologies, which demonstrates that different input instances share large portion of reasoning paths. This fact is not surprising as solving the same task of Ad ToRet should rely on common set of skills. More interestingly, we notice that even for solving this same task, the Planner will have to call upon few instance-depender sub-paths to handle the subtle inherent differences (e.g., the graph topology) that exist between different input instances. For this reason, PRIMA maintains an instance- instance-dependent dependent dynamic architecture, which is in sharp contrast to the Neural Architecture Search (NAS) approaches. Although NAS may also use RL algorithms to seek for smaller architecture (Zoph & Le 2017). it only searches for a static architecture that will be applicd to all the input instances.
Testing cobject stoe-10)
Testing conpect stcee50)
Testing jobject 20)
Testing (otject sizee 1001
FLOPsNN
FLOPUN
FLOPS
NLM NLM-MTR PRIMA
MemNN
The reasoning costs (in FLOPs) of different models at the inference stage (1 M=1 G=Ix Compared to NLM and NLM-MTR, our PRIMA significantly reduces the reasoning complexity by intelligently selecting ops (short for neural operators) using a planner. Although the FLOPs of MemNN seem low in most cases of testing, its testing accuracy is bad and cannot achieve accurate predictions (see Table "OD" denotes 1 --uutdegree, likewise, jaceentt "HS": ster, "Granaarran,pareen, "UN":IsUncle IMGUncc
Figure 4:
inieetiiiity.. "HF":Hasrather
RELATED WORK
Multi-task learning Multi-task learning (Zhang & Yang. 2021 Zhou et al.) 2001 Pan & Yang 2009) has focused primarily on the supervised learning paradigm, which itself can be divided into several approaches. The first is feature-hased MTI, also called multi-task feature learning, (Argyriou
8

---

Under review as a conference paper at ICLR 2022
Gradi
1
P
11
1
Skill sharing among different tasks.
8. mostance-ddpendent reasoniin path withinn the same task.
Figure 5: The reasoning paths of PRIMA. The gray arrows denote the "shared path". The colored arrows in sub-figures (A) and (B) denote task-specific paths and instance-specific paths, respectively. In (A), besides the Adjacent Toeed task we have introduced carlier, we also consider the 1 d-gut task, which reasons about whether the out-degree of a node is exactly equal to 1, and the 4 CoConeect ivity task, which is to decide whether there are two nodes connected within 4 hops.
et al. 2008), which explores the sharing features across different tasks using regularization tech- miques (Shinohara 2016 Liu et al., (017)) The second approach assumes that tasks intrinsically
related, such as low low-rank learning (Ando & Zhang, 2005 Zhang et al. 2005), learning with task
clustering (Gu et al. 2011 Zhang 2016), and task-relation learning (such via task similarity,
task correlation, or task covariance) (Goncalves et al. (016)) Zhang 2013 Ciliberto et al. 2013).
MTL has also been explored under other paradigms such unsupervised learning, for example,
multi-task clustering in (Zhang & Zhang 2013 Zhang et al. 2015b Gu et al. 2011 Zhang 2015).
Neural-symbolic reasoning Neural-symbolic AI for reasoning and inference has a long his- tory (Besold et al., 2017, Bader et al., (004 Garcez et al. 2208)), and neural-ILP has developed primarily in two major directions (Cropper et al. 2020). The first direction applies neural operators such tensor calculus to simulate logic reasoning (Yng et al. 2017 | et al. 2019 Shi et al., 2020) and (Manhaeve et al. 2018). This approach uses binary tensors over constant domains to represent the predicates and tensor chain products to simulate logic clauses. The second direction involves relaxed subset selection (Evans & Grefenstette 2018 Si et al. 2019) with a predefined set of task-specific logic clauses. This approach reduces the task subset-selection problem by selecting a subset of clauses from the predefined set and using neural networks to search for relaxed solution.
Our work is different from the most recent work, such (Dong al. 2019) Shi al. 2020), in several ways. Compared with (Dong et al.) 2019), the most notable difference is the improvement in efficiency by introducing Marning-to-reason via reinforcement learning. A second major difference is that PRIMA offers more generalizability than NLM by decomposing the logic operators into more finely amely-grained units. We refer readers for more detailed related work Appendix D
CONCLUSION
standitanding challenge in multi-task learning the intrinsic conflict between capability and efficiency. In this paper, we propose a Planner-Reasoner framework for multi-task reasoning. To maintain broad capability but efficient specific performance, the Reasoner extracts shareable meta- rules via first-order logic, from which the Planner efficiently selects subset of meta-rules to formulate the reasoning path. The model's training follows a complete data-driven end-to-end approach via deep reinforcement learning, and the performance is validated across variety of benchmark tasks. Future work could include extending the framework to high-order logic and investigating scenarios when meta reta- have a hierarchical structure.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We commit to ensuring that other researchers with reasonable background knowledge in our area can reproduce our theoretical and empirical results. The algorithm details can be seen in Appendix AA Specifically, benchmark details are provided in Appendix B.1] in Appendix [.2] and computing infrastructure are detailed Appendix B.3 The experiment details can be seen in Appendix B
settings are provided
ETHICS STATEMENT
This work is about the methodology of achieving a capability-effciinney trade-off in multi-task first-order logic reasoning. The potential impact of this work is likely to further extend the framework high-order logic and investigate scenarios when meta-rules have hierarchical structure, which should be generally beneficial to the multi-task learning research community. We have not considered specific applications or practical scenarios as the goal of this work. Hence, does have any direct ethical consequences.
REFERENCES
Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 2005.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In JCML, 2017.
Peter B Andrews. An introduction to mathematical logic and type theory, volume 27, Springer Science & Business Media, 2002.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. MLJ, 2008
Peter Auer. Using confidence bounds for exploitationnexxllratiio trade-offs. Journal of Machine Learning Research, 3(Nov):397-422, 2002.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235 256, 2002.
Sebastian Bader and Pascal Hitzler. Dimensions of neural-symbolic integration-a structured survey, arXiv preprint cx/0511043 2005.
Sebastian Bader, Pascal Hitzler, and Steffen Hölldobler. The integration of connectionism and first-order knowledge representation and reasoning as challenge for artificial intelligence. arXiv preprint cs/0408069, 2004.
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interac- tion networks for learning about objects, relations and physics. arXiv preprint arXiv: 1612.0022, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1rXiv:1806.01 1806.01261, 2018.
Yoshua Bengio. From system 1 deep learning to system deep learning. In Thirty-third Conference Neural Information Processing Systems, 2019.
Tarek R Besold, Artur Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pas- cal Hitzler. Kai-Un uwe Kühnberger, Luis C Lamb, Daniel Lowd, Priscila Machado Vieira Lima, et al. Neural- symbolic learning and reasoning: survey and interpretation. arXiv preprint 1rXiv: 2017.
10

---

Under review as a conference paper at ICLR 2022
Timo Bräm, Gino Brunner, Oliver Richter, and Roger Wattenhofer. Attentive multi-tass deep reinforcement learning. In ECML/PKDD, 2019.
Cameron B Browne, Edward Powley. Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions Computational Intelligence and Al in games, 4(1):1-43 2012.
Salam El Bsat, Haitham Bou-Ammar, and Matthew E. Taylor. Scalable multitask policy gradient reinforcement learning. AAAI, 2017.
Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli. Sparse multi-task reinforcement learning. NIPS, 2014.
Marco Calautti, Georg Gottlob, and Andreas Pieris. Chase termination for guarded existential rules. In Proceedings of the 34th ACM Systems, pp. 91-103, 2015.
Symposiun on Principles of Database
L. Cao. Non-iid recommender systems: A review and framework of recommendation paradigm shifting. Engineering, 2(2):212-224, 2016.
Deepayan Chakrabarti and Christos Faloutsos. Graph mining: Laws, generators, and algorithms. ACM computing surveys (CSUR), 38(1):2-es, 2006.
Ashok K Chandra and David Harel. Horn clause queries and generalizations. The Journal of Logic Programming, 2(1):1-15, 1985.
Jianzhong Chen, Stephen Muggleton, and José Santos. Learning probabilistic logic models from probabilistic examples. Machine learning, 73(1):55-85, 2008.
K. Chen, F. Yang, and X. Chen. Planning with task-oriented knowledge acquisition for service robot. IJCAI, pp. 812-818, 2016.
Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinav Gupta. Iterative visual reasoning beyond convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7239- 7248, 2018.
Carlo Ciliberto, Youssef Mroueh, Tomaso Poggio, and Lorenzo Rosasco. Convex learning of multiple tasks and their structure. In ICML, 2015.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 105555, 2020.
Diane Cook and Lawrence B Holder. Mining graph data. John Wiley & Sons, 2006.
Michael Crawshaw. Multi-task learning with deep neural networks: survey, arXiv preprint 00XX::2009.09796, 2020.
Andrew Cropper and Sebastijan Dumaneic. Inductive logic programming at 30: a new introduction. arXiv preprint arXi::2008.07912 2020.
Andrew Cropper, Sebastijan Dumancic, and Stephen Muggleton. Turning 30: New ideas in inductive logic programming. arXiv preprint arXiv:2002. /002, 2020.
Aniket Anand Deshmukh, Ürtnn Dogan, and Clayton Scott. Multi-task learning for contextual bandits. In NIPS, 2017.
Carlos Diuk, Andre Cohen, and Michael L Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 240-247, 2008.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. arXiv preprint arXiv: /904 16944, 2019.

---

Under review as a conference paper at ICLR 2022
Sašo Džeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Machine learning, 43(1):7-52,, 2001.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In ICML, 2018.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
Melvin Fitting. First-order logic and automated theorem proving. Springer Science & Business Media, 2012.
Jean H Gallier. Logic for computer science: foundations of automatic theorem proving. Courier Dover Publications, 2015.
M. C. Ganiz, C. George, and W. M Pottenger. Higher order naive bayes: A novel non-iid approach to text classification. IEEE Transactions on Knowledge and Data Engineering, 23(7):1022- 1034, 2010.
Artur SD' Avila Garcez, Luis Lamb, and Dov M Gabbay. Neural-symbolic cognitive reasoning. Springer Science & Business Media, 2008.
M. Gebser, B. Kaufmann, and T. Schaub. Conflict-driven answer set solving: From theory to practice. Artificial Intelligence, 187-188:52-888 2012.
Michael Gelfond and Vladimir Lifschitz. Action languages. 1998.
André R. Goncalves, Fernando J. Von Zuben, and Arindam Banerjee. Multi-task sparse structure learning with Gaussian copula models. JMLR, 2016.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv: 140614466,,6, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626): 471-476, 2016.
Quanquan Gu, Zhenhui Li, and Jiawei Han. Learning a kernel for multi-task clustering. In AAAI, 2011.
Carlos Guestrin, Daphne Koller, Chris Gearhart, and Neal Kanodia. Generalizing plans new environments in relational mdps. In Proceedings of the 18th international joint conference on Artificial intelligence, pp. 1010, 2003.
Mare Hanheide, Moritz Göbelbecker, Graham Horn, Andrzej Pronobis, Kristoffer Sjöö, Alper Aydemir, Patric Jensfelt, Charles Gretton, Richard Dearden, Miroslav Janicek, et al. Robot task planning and explanation in open and uncertain worlds. Artificial Intelligence, 2015.
M. Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 191-246, 2006.
Alfred Horn. On sentences which are true of direct unions of algebrasl. The Journal of Symbolic Logic, 16(1):14-2)) 21, 1951
Maximilian Igl, Andrew Gambardella, Nantas Nardelli, N. Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. In UAI, 2020.
Herbert Jaeger. Deep neural reasoning. Nature, 53877626):467-468, 2016.
Ramtin Keramati, Jay Whang, Patrick Cho, and Emma Brunskill. Strategic object oriented rein- forcement learning. In Exploration in Reinforcement Learning Workshop at the 35th International Conference Machine Learning, 2018
12

---

Under review as a conference paper at ICLR 2022
P. Khandelwal. S. Zhang, J. Sinapov, M. Leonetti, J. Thomason, F. Yang, Gori, M. Svetlik, P. Khante, V. Lifschitz, and Stone. Bwibots: A platform for bridging the gap between ai and human human-robot interaction research. The International Journal of Robotics Research, 36(5-7) -0903355556666 2017.
Levente Kocsis, Csaba Szepesvári, and Jan Willemson. Improved monte-carlo search. Univ. Tartu, Estonia, Tech. Rep, 2006.
Daphne Koller. Probabilistic relational models. In International Conference on Inductive Logic Programmingg pp. 3-13. Springer, 1999.
Daphne Koller, Nir Friedman, Sašo Džeroski, Charles Sutton, Andrew McCallum, Avi Pfeffer, Pieter Abbeel, Ming-Fai Wong, David Heckerman, Chris Meek, al. Introduction to statistical relational learning. MIT press, 2007.
Markus Krötzsch. Computing cores for existential rules with the standard chase and asp. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, volume 17, pp. 603-613, 2020.
Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian multi-task reinforcement learning. In ICML, 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436444, 2015.
Hui Li, Xuejun Liao, and Lawrence Carin. Multi-task reinforcement learning in partially observable stochastic environments. JMLR, 2009.
Sijin Li. Zhi-Qian Zhi-Qiang Liu, and Antoni B. Chan. Heterogencous multi-task Iearning for human pose estimation with deep convolutional neural network. IJCV, 2015.
V. Lifschitz. What is answer programming? In Proceedings of the AAAI Conference Artificial Intelligence, pp. 1594-1597. MIT Press, 2008
Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In NeurlPS, 2019.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classification. In ACL, 2017.
Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo. Multi-task deep visual-semantic embedding for video thumbnail selection. In CVPR, 2015.
Robin Manhaeve, Sebastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Processing Systems, pp. 3749-3759, 2018
Zimmer Matthieu, Feng Xuening, Glanois Claire, Jiang Zhaohui, Zhang Jianyi, Weng Paul, Jianye Hao, Dong Li, and Wulong Liu. Differentiable logic machines. arXiv preprint arXiv:2102.115299 2021.
Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl-the planning domain definition language. 1998.
Nikola Mrksic, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gasic, Pei-hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Multi-domain dialog state tracking using recurrent neural networks. In ACL, 2015.
Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming. 19:629 679, 1994.
Remi Munos. From bandits to monte-carlo tree search: The optimistic principle applied to optimiza- tion and planning. Foundations and Trends in Machine Learning, 2014.
13

---

Under review as a conference paper at ICLR 2022
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In ICML, 2017.
Rasmus Berg Palm, Ulrich Paquet, and Ole Winther. Recurrent relational networks. arXiv preprint arXiv:1711.08023 2017.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1455-135, 2009.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer
reinforcement learning. In ICLR, 2016.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathemutics and Artificial Intelligence, 61(3):203-25 2000000,2202200000 2011.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2016.
Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. arXiv preprint
01427, 2017.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? arXiv preprint IrXiv:1005. 1604, 2018.
Andrew M. Saxe, Adam Christopher Earle, and Benjamin Rosman. Hierarchy through composition with multitask LMDPs. In ICML, 2017.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint 1rXiv:191..08265, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Sahil Sharma and Balaraman Ravindran. Online multi-task learning using active sampling. In ICLR Workshop, 2017.
Sahil Sharma, Ashutosh Kumar Jha, Parikshit Hegde, and Balaraman Ravindran. Learning to multi-task by active sampling. In ICLR, 2018
Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang. Neural logic reasoning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1375-1374, 2020.
Yusuke Shinohara. Adversarial multi-task learning of deep neural networks for robust speech recognition. In Interspeech, 2016.
Xujie Si, Mukund Raghothaman, Kihong Heo, and Mayur Naik. Synthesizing datalog programs using numerical relaxation. arXiv preprint arXiv: /06.006906 2019.
David Silver, Thomas Hubert, Julian Schrittiesser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 171.0/ 12.011 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-35) 2017b.
14

---

Under review as a conference paper at ICLR 2022
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 4440-2448, 2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Prasad Tadepalli, Robert Givan, and Kurt Driessens. Relational reinforcement learning: An overview. In Proceedings of the ICML-2004 workshop on relational reinforcement learning, pp. 1-9, 2004.
Yee Whye Teh, Victor Bapst, Wojeiech M. Czamnekki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In NIPS, 2017.
Rasul Tutunov, Dongho Kim, and Haitham Bom-am Distributed multitask reinforcement learning with quadratic convergence. In NeurlPS, 2018.
Frank Van Harmelen, Vladimir Lifschitz, and Bruce Porter. Handbook of knowledge representation. Elsevier, 2008
Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics, 9()):1363, 2020.
Tung-Long Vuong, Do Van Nguyen, Tai-Long Nguyen, Cong-Minh Bui, Hai-Dang Kieu, Viet-Cuong Ta. Quoc-Long Tran, and Thanh Ha Le. Sharing experience in multitask reinforcement learning. In IJCAI, 2019.
Ronald J Williams.
Simple statistical fradient- algorithms for connectionist reinforcement learning. Machine learning, 8(3-4)2229-25) 256, 1992.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: A hierarchical Bayesian approach. In ICML, 2007.
Victoria Xia, Zi Wang, and Leslie Pack Kaclbling. Learning sparse relational transition models. arXiv preprint ar 1810. 177, 2018
Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. In Advances in Neural Information Processing Systems, pp. 2319-2328, 2017.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In NIPS, pp. 1031-1042, 2018.
N. Yoshida, T. Nishio, M. Morikura, K. Yamamoto, and R. Yonetani. Hybrid-fl for wireless networks: Cooperative learning mechanism using non-iid data. In ICC 2020-2020 IEEE International Conference Communications (ICC), pp. 1-7. IEEE, 2020.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint 1806v:
2018.
Jian Zhang, Zoubin Ghahramani, and Yiming Yang. Learning multiple related tasks using latent independent component analysis. In NIPS, 2005.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: comprehensive review. Computational Social Newworks, 6(1):1-23, 2019.
Wenlu Zhang, Rongjian Li, Tao Zeng. Qian Sun, Sudhir Kumar, Jieping Ye, and Shuiwang Ji. Deep model based transfer and multi-task learning for biological image analysis. In KDD, 2015a.
Xianchao Zhang and Xiaotong Zhang. Smart multi-task Bregman clustering and multi-task kernel clustering. AAAI, 2013.
Xianchao Zhang, Xiaotong Zhang, and Han Liu. Smart multitask Bregman clustering and multitask kernel clustering. ACM TKDD, 2015b.
Xianchao Zhang, Xiaotong Zhang, and Han Liu. Self-adapted multi-task clustering. In IJCAI, 2016.
15

---

Under review as a conference paper at ICLR 2022
Xiao-Lei Zhang. Convex discriminative multitask clustering. IEEE TPAMI, 2015.
Yu Zhang. Heeerogeeeus--eiihhhoooooooooo based multi-task local learning algorithms. In NIPS, 2013.
Yu Zhang and Qiang Yang. survey on multi-task learning, IEEE Transactions on Knowledge and Data Engineering, 2021.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. ECCV, 2014.
Jiayu Zhou, Jianhui Chen, and Jieping Ye. Malsar: Multi-task learning via structural regularization. Arizona State University, 21, 2011.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning deep reinforcement learning: survey, arXiv preprint arXiv:2009.0788 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. 2017.
2017.
16

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
A ALGORITHM DETAILS
A.1 DETAILS OF MCTS: FOUR KEY STEPS
During each epoch, MCTS repeatedly performs four sequential steps: selection, expansion, simulation,, and backpropagation. The selection step traverses the existing search tree until the leaf node (or termination condition) by choosing actions (edges) a, at each node according tree policy, One widely used tree policy is the UCT (Koesis et al. (006) policy, which conducts the node-selection via
V + BN log N No¹
02 arg max s'EC(s)
(2)
where C(s) denotes the set of all child nodes for 8, the first term Var is an estimate of the long- cumulative reward that can be received when starting from the state represented by node s', cooffdencc-intervee size) of that estimate. The
and the second term represents the uncertainty confidence interval is calculated based on the upper confidence bound (UCB) (Auer
2002] Auer 2002) using N, and No., which denote the number of times that nodes and s' have been visited (respectively). The key idea UCT policy (2) is to select the best action according to an optimistic estimation (the upper confidence bound) of the expected return, which balances the exploitation (first term) and exploration (second term) with controlling the trade-off. The second step (node expansion) is conducted according to a prior policy by adding new child node if the selection process reaches a leaf node the search tree. Next, the simulation step estimates the value function (cumulative reward) V. by running the environment simulator with a default (simulation) policy. Finally, the buckpropagation step updates the statistics V, and N, from the leaf node ST the root node so of the selected path by recursively performing the following update (i.c., from = T 1 to t 0):
No. N +1, +1 Vs, + r(s, + + VN, + ((N,, 1)V V/.)/Ns..
where Vey is the simulation return of sT, and ag denotes the action selected following (2) at state St- Generally, the expansion step and the simulation step are more time-consuming than the other two steps, as they involve large number of interactions with the environment.
In our Enhanced Planner, we adopt a recently proposed variant of MCTS that is based on probabilistic upper confidence tree (PUCT) (Rosin) 2011). It conducts node-selection according to so-called PUCT score with two components PUCT (s, a) = Q(s, a) + (s, a).. Q(s, a) is the mean state-action value calculated from the averaged game result that takes action during current simulations, and Uss, a) is the exploration bonus calculated as
a( (alss) E, (s. 1+N(s,a)
01 + log E. (o,
C2
where a, b are possible actions, क (a]s)) the output of the Base Planner, N(s, a) is the visit count of the (8, a) pair during current simulations, and constants G, C2 are the exploration-comtrlliig hyper-parameters.
An MCTS algorithm is a model-based RL algorithm that plans the best action at each time step (Browne et al., 2012) by constructing search tree, with states nodes and actions as edges. It uses the MDP model to identify the best action at each time step until the leaf node (or until other termination conditions are satisfied) by choosing actions (edges) a, each node s according to a tree policy. For implementations of Monte-Catd Tree Search, we refer readers to https //gi thub. www/wwrmeeeeeeeeoo duwaud/ wwweauuddmueeeeeeeeeeeee -eeeegeeerraae for details.
A.2 DETAILS OF CALCULATING REASONING ACCURACY
Since the "target" predicates are available for all objects pairs of objects, the reasoning accuracy refers to the accuracy evaluated on all objects (for properties such as jajaneet ToRed (x) or 1 1-OutDegree (x) ) pairs of objects (for relations such 4 4-Connect vitty (x, y) ).
17

---

Under review as a conference paper at ICLR 2022
B EXPERIMENTAL DETAILS
B.1 REASONING TASKS INTRODUCTION
Graph For graph tasks, they have the same background knowledge (or background predicate): HasEdge(z, y). i.e., HasEdge(2 (z; y) is True if there is an undirected edge between node and node y. However, AdjacentToRed has an extra background predicate, Red(x). Red(x) is True if the color of node is red. Specifically, these graph tasks seeks to predict the target concepts (or target predicates) shown as below.
1 1-OutDegree: 1-OutDegreeez)) is True if the out-degree of node is exactly 1.
AdjacentoRRed adjacent is True if the node x has an edge with red node.
4 4-Connective ivity: 4 4-Connectivity(la y) is True if there are within 4 hops between node z and node y.
For family tree tasks, they have the same background knowledge (or background pred- icates): IsFather(z, y). IsMother y). IsSon (æ, y) and IsDaughter y). For instance, herather y) is True when y is x's father. Specifically, these family tree tasks seek to predict the target concepts (or target predicates) shown below.
Family Tree
HasFather: HasFather(z) is True if has father.
HasSisters HasSister(r) is True if x has at least one sister.
TsGrandparent: IsCrandparentt(r,
True if y r's grandparent.
IsUncle: IsUncle(I,g) y) is True if is z's uncle.
TsMGUncle: TsMGUncle(1s y) is True y is w's maternal great uncle.
B.2 HYPER-PARAMETHR SETTINGS
For simplicity, set discount factor y in all experiments. The Tmax depends the depth of the Reasoner. In single-task setting, we use the same hyper-parameter settings as in the original papers for MemNN (Sukhbaatar al., 2015), ØILP (Evans & Grefenstette (018). NLM (Dong
et al. 2019), and DLM (Matthieu et al.
2021 In multi-task setting, the probability distribution of sampling task for NLM-MTR is [0.09. 0.11, 0.11, 0.09, 0.09, 0.135, 0.135, 0.24] that is correspond-
ing to the task list of [1-Outdegree, AdjacentIoRed. 4 4-Connect ivitt ivity, HasSister, IsGrandparent, IsUncle, IsMGUncle], That distribution for DLM-MTR [0.1, 0.12, 0.12. 0.1, 0.13. 0.13, 0.2]. The details of hyper-parameters in NLM-MTR and DML- MTR can be found in Table 4 For all MLPs inside NLM-MTR, we keep the same settings as NLM. Similarly, the "Internal Logic", initial temperature and scale of the Gumbel distribution, and the dropout probability in DLM-MTR are also kept the same settings as those in DLM. Besides, the probability distribution of sampling task for PRIMA is [0.1, 0.12, 0.12, 0.1, 0.1, 0.13, 0.13, 0.2]. Regarding the hyper-pante parameters PRIMA, details can be found in Table 5 We also show the hyper-parameters of PRISA-REINFORGE PRISA-PPO, PRISA-MuZero Table
at
We note that the problem size for training is always 10 for graph tasks and 20 for family tree tasks in single-task and multi-task settings.
B.3 COMPUTING INFRASTRUCTURE
We conducted our experiments on CPU server where the CPU is "Intel(R) Xeon(R) Silver 4114 CPU" with 40 cores and 64 GB memory in total.
18

---

Under review as a conference paper at ICLR 2022
Table 4: Hyper- papper-arramterr settings for NLM-MTR and DML-MTR.
NLM-MTR
DML-MTR
learning rate
0.005
0.005
epochs
50
200
epoch size batch size (train)
8000
2000
breadth depth
3
Table 5: Hyper-parameter settings for PRISA and PRIMA. "Ir-reasoner" refers to the learning rate for the reasoner, "Ir-policy" denotes the learning rate for the policy network, "Ir-value" denotes the learning rate for the value network, "residual" refers to the residual connection in the reasoner, "Num Warmups" refers to the number of warm-ups before starting the training, "NumRollouts" refers to the number of roll-outs in MCTS, "RwdDecay" refers to the constant exponential decay applied on the reward, "c1 and 02" refers to the same constants in PUCT formula Section [..] "RBsize" refers to the replay buffer size that is used to count the number of stored trajectories, "BatchSize" refers to the batch size for training, "TrainingSteps" refers to the number of training steps. For REIN--OEOEO PRISA-PPO, PRISA-MuZZro, the "TrainingSteps" denotes the training steps for each task, while that of PRIMA represents the training steps for all tasks.
PRISA-
PRISA-
PRISA- MuZero
PRIMA
REINFORCE
PPO
Ir-reasoner Ir Ir-policy
0.005
0.005
0.005
0.004
0.085
0.085
0.075
0.075
Ir-value
0.15
0.075
0.075
breadth
3
3
depth
residual
False
False
False
False
Num Wurmus
200
200
NunRollouts
1200
1200
RwdDecay
5
5
C1
30
30
19652
19652
RBsize
16
400
400
BatchSize
16
16
16
16
TrainingSteps
x 10
7 x 104
x 10
39 x 104
Table 6: Additional results: The performance of PRIMA w.r.t different numbers of intermediate predicates. "# pred": the number of intermediate predicates.
Family
Tiee
HamFaheer
HasSester
IMOUsslle
Grogh
Adgacnnt ToRao
m=20
61.78
$1.97
96.76
97.19
99.67
m=10
12.62
B9.28
$1.35
pred
m=100
87.5
31.2
97.7
96.5
95.4
m=:50
11.1
95.6
95.5
PSSS
ppod
96.6
m=50
PSS
100
100
0
PSS
testing
m=20
100
1001
100
100
100
m=
99.7
100
100
pred 6
m=100
100
100
100
100
100
m=55
100
100
100
PSS
100
100
100
100
90
PSS
100
1000
prod
19

---

Under review as a conference paper at ICLR 2022
Testing (object size 10)
Testing (object Size=50)
Testing (object sizem20)
Testing Jobject sizem 1001
1LOPS (N)
FLOPs (M)
(4)
42
00
AR
CO
HF
H5
GP
UN
MG
HF
15
04
M6
PRIMA
PRISA-REINFORGE
PRISA-PPO
PRISA- MuZero
Figure 6: The reasoning costs (in FLOPs) of PRIMA and three variants of PRISA at the inference stage (1 M=1x10", G=1x10").
C ADDITIONAL EXPERIMENTS
C.I REASONING COSTS OF PRISA
To demonstrate the improved reasoning efficiency in PRISA at the inference stage, three variants of PRISA (e.g., PRISA REISFORE, PRISA-PPO, and PRISA-MuZero) compared with PRIMA in terms FLOPs. As shown in Fig. 6 three variants PRISA generally have lower reasoning complexity than PRIMA. The difference FLOPs between PRIMA and PRISA-MuZero is primarily from the difference between multi multi-tasking setting and single-task setting.
C.2 PREDICATE DIMENSION ANALYSIS
Since the Reasoner only realizes partial of Horn clauses of FOL, the number of intermediate predicates will greatly determine the expressive power of the model. The performance of PRIMA degrades gracefully when the number of intermediate predicates decreases. This shown in the additional experiments of examining the limiting performance of our PRIMA, where we examine the limiting performance of our PRIMA when the number of intermediate predicates at each layer decreases. The results in Table show that the performance of PRIMA degrades gracefully when the number intermediate predicates decreases; for example, still performs reasonably well on most tasks even when the number of predicates becomes 4.
C.3 ANALYSIS OF MULTI-TASKING CAPABILITIES
88088582 E5u88552 55ua8852 38v88651
85089588 88088883 85088551 33U85687
09688851 83u88585
Figure The quantitative evaluation of ops reuse between different Tasks in PRIMA.
To quantitatively measure how different tasks reuses the "skills", we show in Fig. 7 which is a heat-map showing the probability of overlapping about Ops reuse across different tasks, where red color denotes high-level of Ops overlapping. Ops reuse factor defined as min(Pa(T)
Why/how do different tasks share blocks (of ops) as in Fig. 7
P,(P)) where (I refers to the probability of ops Pj {O..... 91) being active for task and (I) denotes the probability of 1 being active for task From Fig. 7 it can be inferred that some tasks share skills more than others.
For example, in most of the actions, 8 is shared across different tasks.
20

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL RELATED WORK
This section provides complementary contents to Sec. 5
Model-based symbolic planning Another large body of related work is symbolic planning (SP) (Van Harmelen al., 2008: [Hanheide et al. 2015 Chen al., [0T6 Khandelwal et al., (017))
In SP approaches, a planning agent carries prior symbolic knowledge of objects, properties, and how they are changed by executing actions in the dynamic system, represented in formal, logic-based language such as PDDL (McDermott et al. 1998) or an action language (Gellond & Litschitz 1998) that relates logic programming under answer set semantics (answer set programming) (Lifschitz) 2008). The agent utilizes a symbolic planner, such as a PDDL planner FASTDOWNWARD (Helmert 2006) or an answer set solver CLINGO (Gebser et al. 2012) to generate a sequence of actions based on its symbolic knowledge, executes the actions to achieve its goal. Compared with RL approaches, an SP agent does not require a large number of trial-and-error to behave reasonably well, yet requires predefined symbolic knowledge the model prior. It should be noted that it remains questionable if PDDL-based methods can be applied directly to our problem, as there is no explicitly predefined prior knowledge under our problem setting, which is required by PDDL-bbsed approaches. In contrast, address the learning-to- to--resoonn problem via a complete data-driven deep reinforcement approach.
Multi-task RL Multi-task learning can help boost the performance of reinforcement learning, lead- ing to multi-task reinforcement learning /Vithayathil Varghese & MMammoud 2020, Zhu et al. 20200)
Some research (Wilson et al. 2007: Li et al. 2009; Lazaric & Ghavamzadehi 2010 Calandricllo
et al. 2014
2019 Vuong 2019 [gl et al. 2020) has adapted the ideas behind multi-task learning to RL. For example, in (Bräm et al. 2019). multi-task deep RL model based on attention can group tasks into sub-networks with seate-level granularity. The idea of compression and distillation has been incorporated into multi-task RI. as in (Parisotto et al., 2016; |usu al. 2016,] Omiidshffiii et al., 2017, [Teh al.] 2017). For example, in (Parisotto et 2016), the proposed actor-mimic method combines deep reinforcement learning with model-compression techniques to train a policy network that can learn to act for multiple tasks. Other research in multi-task RL focuses on online and distributed settings Ravindran| 2017. Sharma et al. 2018 Espeholt al., 2018 Tutunoy
Andreas et al. 2017 Deshmukh et al.
et al. 2017 Bräm et al.
et al.
(Bsat et al.
et al. 2018
Lin et al. 2019).
RL for logic reasoning Relational RL integrates RL with statistical relational learning and connects RL with classical AI for knowledge representation and reasoning. Most prominently, Džeroski et al.
(001I) originally proposed relational RL. Tadepalii et al. (2004) surveyed relational RL, Guestrin al. (2003) introduced relational MDPs, and Diuk al. (2008) introduced dnjected-orienteee MDPs (DDsMDPs). More recently, Battaglia et al. (2018) proposed to incorporate relational inductive bias, Zambaldi et al. (2018) proposed deep relational RI., Keramati et al. (2018) proposed strategic object-oriented RI., and some researchers have also adopted deep learning approaches for dealing
with relations and/or reasoning, for example (Battaglia ct al. 2016: Chen et al. 2018 Santoro et al.
2017: Santurkar et al. 2018 et al. 2017 Xia et al. 2018 Yi et al.
Statistical relational reasoning Statistical relational reasoning (SRL) (Koller et al. 2007) often
provides a better understanding of domains and predictive accuracy, but with more complex learning and inference processes. Unlike IIP., SRI. takes statistical and probabilistic learning perspective and extends the probabilistic formalism with relational aspects. Examples of SRL include text classification (Ganiz et al. 2010), recommendation systems (Cao 2016). and wireless networks (Yoshida et al., 2020). SRL can be divided into two research branches: probabilistic relational models (PRMs) (Koller 1999) and probabilistic logic models (PLMs) (Chen et al. 2008). PRMs start from probabilistic graphical models and then extend to relational aspects, while PLMs start from ILP and extend to probabilistic semantics. A related research area is graph relational reasoning, such using graph neural networks (Zhou et al., 2020) to conduct reasoning. A discussion of this approach is beyond the scope of this paper, but we refer interested readers to (Chakrabarti & Faloutsos 2006; Cook & Holder, 2006; Zhang et al. 2019) for a comprehensive review.
Comparisons with NLM in details The Reasoner shares the same functionality with NLM, in terms of approximating the meta-rules (BooleanLogic, Expansion, and Reduction). However,
21

---

Under review as a conference paper at ICLR 2022
Reasoner has major differences from NLM. First, the targets are different. Our Reasoner targets the trade-off between capability and efficiency. Compared with NLM, our Reasoner can greatly improve the efficiency by using Planner module. Second, our Reasoner has more explicitness in the reasoning path. Let [ 0 = denote the output predicates (in tensor representation) from the previous layer, and Concate() () denotes the concatenation operation. NLM performs the intra-group computation
0 (MLP (Permute (Z));0')),
where Of the output predicate, Z Concat (Expand (O . Reduce (0z+)) the sigmoid nonlinearity and or denotes learnable parameters. On the contrary, in the Reasoner module of our PRISA/PRIMA, the implementation
O = (MPPPemmcc(xxppn(((((1)))))))) )0M
-MIPPPemmt((ccccc((111111)
which allows for the neural implementations of BooleanLogic, Expansion, and Reduction (at the same arity group) to be executed independently to some extent and reduce the unnecessary computations.
22

---

Under review as a conference paper at ICLR 2022
PRIMA: PLANNER-REASONE INSIDE A MULTI-TASK REASONING AGENT
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of multi-task reasoning (MTR), where an agent can solve multiple tasks via (first-order) logic reasoning. This capability is essential for human-like intelligence due to its strong generalizability and simplicity for handling multiple tasks. However, major challenge in developing effective MTR is the intrinsic conflict between reasoning capability and efficiency. An MTR-capable agent must master large set of "skills" to tackle diverse tasks, but executing a particular task at the inference stage requires only a small subset of immediately relevant skills. How can we maintain broad reasoning capability but efficient specific-task performance? To address this problem, we propose a Planner-Reasoner
Reasoner framework capable of of-te-oo-thhaarr MTR capability and high efficiency. The Reasoner models shareable (first-order) logic deduction rules, from which the Planner selects subset to compose into efficient reasoning paths. The entire model is trained in an end-to-end manner using deep reinforcement learning, and experimental studies over a variety of domains validate its effectiveness.
INTRODUCTION
Multi-task learning (MTL) Zhang & Yang 2021| Zhou et al. 2011) demonstrates superior sample
complexity and generalizability compared with the conventional "one model per task" style solve
multiple tasks. Recent research has additionally leveraged the great success of deep learning (LeCun)
al. 2015) to empower learning deep multi-task models (Zhang & Yang 2021 Crawshaw 2020p.
Deep MTL models either learn a common multi-task feature representation sharing several bottom
layers of deep neural networks (Zhang et al., 2014] Liu et al. 2015 Zhang et al. 2015a Mrksic
et al. 2015 Li et al. 2015). learn task-invariant and task-specific neural modules (Shinohara
2016
Liu et al. 2017) via generative adversarial networks (Goodfellow al. 2014). Although
MTL successful in many applications, a major challenge is the often impractically large MTL models. Although still smaller than piling up all models across different tasks, existing MTL models significantly larger than single model for tackling specific task. This results from the intrinsic conffict underlying all MTL algorithms: balancing across-tass generalization capability to perform different tasks with single-task efficiency in executing specific task. On one hand, good generalization ability requires an MTL agent to be equipped with a large set of skills that can be combined solve many different tasks. On the other hand, solving one particular task does not require all these skills. Instead, the agent needs to compose only a (small) subset of these skills into an efficient solution for a specific task. This conflict often hobbles existing MTL approaches.
This paper focuses on multi-task reasoning (MTR), a subarea of MTL that uses logic reasoning to solve multiple tasks. MTR ubiquitous in human reasoning, where humans construct different reasoning paths for multiple tasks from the same set of reasoning skills. Conventional deep learning. although capable of strong expressive power, falls short in reasoning capabilities (Bengio 2019).
Considerable research has been devoted to endowing deep learning with logic reasoning abilities, the results of which include Deep Neural Reasoning (Jaeger 2016), Neural Logic Reasoning (Besold
et al. 2017 Bader et al. 2004 Bader & Hitzler 2005). Neural Logic Machines (Dong et al. 2019).
and other approaches (Besold et al. 2017: Bader et al. 2004;; Bader & Hitzler (2005). However, these approaches consider only single-task reasoning rather than multi-task setting, and applying existing MTL approaches to learning these neural reasoning models leads to the same conflict between across-task generalization capability and single-task efficiency.
23

---

Under review as a conference paper at ICLR 2022
IaRedix)
HasEdgelx.y)
AdjacentToRedic
Symbolic rule:
Vy PredRed(ar v) <- [xRed(r):
(Expansion) (Booleun Logie) (Reduction)
(edEdge(z. y) PredRed(z, y) HanBdgo( 3. AdjacentToRRdle + By RedRágo(1, y);
"ngle denotes the value of True, while the rest of the blank denotes False
Figure 1: An example from the AdjacentToRed task and its formulation as a logical reasoning problem.
To strike a balance between reasoning capability and efficiency in MTR, we develop a Planner- Reasoner architecture Inside Multi-task reasoning Agent (PRIMA) (Section 3). wherein the reasoner defines set of neural logic operators for modeling reusable reasoning meta-rules ("skills"" across tasks (Section 3.2). When defining the logic operators, we focus on first-order logic because of its simplicity and wide applicabilitt to many reasoning problems. such as automated theorem proving (Fitting 2012, Gallier 2015) and knowledge-basee
systems (Van Harmelen et al. 2008). A separate planner module activates only a small subset of the meta-rules necessary for given task and composes them into a deduction process (Section (33). Thus, our planner-reasoner architecture features the dual capabilities of composing and gpruning a logic deduction process, achieving trade-off in MTR. The model architecture is trained in an end-to-
graceful capability-efficieei end manner using deep reinforcement learning (Section 4) and experimental results several benchmarks demonstrate that this framework leads to a state-of-the-ar balance between capability and efficency (Section [5)
We discuss related works in Section and conclude our paper in Section
2 PROBLEM FORMULATION
Logic reasoning We begin with a brief introduction of the logic reasoning problem. Specifically, we consider a special variant of the First-Order Logic (FOL.) system, which only consists of individual variables, constants and up to r-ary predicate variables. That is, we do not consider functions that map individual variables/constants into terms. An r-ary predicate p(z1..... z.,) can be considered as relation between constants, which takes the value of True False. An atom p(r1,... ,I,,) is an r-ary predicate with its arguments 21. .... 3y being either variables or constants. A well-defined formula in our FOL system is a logical expression that is composed from atoms, logical connectives (e.g.. negation -, conjunction A, disjunction V, implication +-), and possibly existential and universal V quantifiers according to certain formation rules (see Andrews (2002) for the details). In particular, the quantifiers 3 and V are only allowed to be applied to individual variables in FOL. In Fig. we give an example from the Adjacent T ToRed task (Graves et al. 2016) and show how it could be formulated as a logical reasoning problem. Specifically, we are given random graph along with the properties (i.e., the color) of the nodes and the relations (i.e., connectivity) between nodes. In our context, each node i in the graph is constant and an individual variable takes values in the set of constants (l...., 5J. The properties of nodes and the relations between nodes .....,,).
are modeled as the unary predicate (s)) (5 x vector) and the binary predicate HasEdge (xr v) (5 x 5 matrix), respectively. The objective of logical reasoning is to deduce the value of the unary predicate Adj AdjacentToRed(n (i.e., whether a node x has a neighbor of red color) from the base predicates IsRed(r) and HasEdge(ır, y) (see Fig. for an example of the deduction process).
Multi-task reasoning Next, we introduce the definition of MTR. With slight abuse of notations, let (p(x1,...,.;, ,,, E n]) be the set of input predicates sampled from any of the different reasoning tasks, where 21,- Fy the individual variables and n is the maximum arity. A multi-task reasoning model takes p(æ1,.... σr) as its input and seeks to predict the corresponding ground-truth output predicates g( ). The aim is learn multiple reasoning tasks jointly in a single model so that the reasoning skills in a task can be leveraged by other tasks to improve the general performance of all tasks at hand.
PRIMA: PLANNER-REASONER FOR MULTI-TASK REASONING
In this section, develop PRIMA framework, which is a novel neural-logic model architecture for multi-task reasoning. We begin with an introduction to the overall architecture (Section 3.T along with its key design insights, and then proceed to diseuss its modules in Sections 3.2
24

---

Under review as a conference paper at ICLR 2022
Planner
Planner
Cancheix
Get training somples
Save samples
B2n Per
May 20 Paneler
Buffer
w Pepedhaolin HiP Fare HLP Pateoffrr
Trakning Precess
Reasoning Pach follsst
Part MF
Plaeeet- Reasoner
Preshoates
Plrrer- Ressene
penliaess
Minery
Pearefoce
Papedhang 111 Part PLP
Pere
Hhiligr
Nodw Weights
Pin
inactive
solie
A. Pianner-Reasoner
B. Training Process
Figure 2: Left (A): The Reaneer-eessoerr Architecture with an example solution. Right (B): The overall (end-to-end) training process of the model by deep reinforcement learning. "Perm" denotes Permute operator.
3.1 THE OVERALL MODEL ARCHITECTURE OF PRIMA
Logic reasoning typically seeks to chain an appropriate sequence of logic operators to reach desirable logical consequences from premises. There are two types of chaining to conduct logic reasoning: (i) forward-chaining and (ii) bockward-channing The forward-chaining strategy recursively deduces all the possible conclusions based on all the available facts and deduction rules until it reaches the answer. In contrast, backward-chhaiingg starts from the desired conclusion (i.e., the goal) and then works backward recursively to validate conditions through available facts. This paper adopts the barward-chaiiig strategy since the goal is unavailable here beforehand. Note that, in the MTR setting, it generally requires a large set of "skills" (logic operators) to tackle diverse tasks. Therefore, it is unrealistic to recursively generate all possible conclusions by chaining all the available deduction rules as in the vanilla forward-chaid -chaining strategy. Therefore, pruning the forward-chainig process is essential improve the reasoning efficiency in the MTR setting.
We note that reasoning via pruned forward-chaining could be implemented by: (i) constructing the elementary logic operators and (ii) selecting and chaining subset of these logic operators together. Accordingly, we develop a novel neural model architecture, named Maaner-Reasonee (see Figure 2 to jointly accomplish these two missions with two interacting neural modules. Specifically, "Reasoner" defines a set of neural-logic operators (representing learnable Horn clauses (Horn 195T)). Meanwhile, a "Planner" learns to select and chain a small subset of these logie operators together towards the correct solution (c.g., the orange-colored path in Figure The inputs to the ReasonrR model are the base predicates (p. p(z), p(z, .)....} that describe the premises. In particular, in the MTR setting, the nullary predicate p characterizes the current task to be solved. This design allows the Planner to condition its decisions on the input task type. We feed the outputs (i.e., the conclusion predicates) into multiple output heads, one for each task (just in standard multitask learning), to generate predictions. The entire model is trained in a complete data-driven end-to-end approach via deep reinforcement learning (Figure 2 and Section 4): trains the Planner to dynamically prune the -hwainig process while jointly learning the logic operators in the Reasoner, leading to a state-of oftee-of-the-ar balance between MTR capability and efficiency (Section (5).
3.2 REASONER: TRANSFORMING LOGIC RULES INTO NEURAL OPERATORS
The Reasoner module conducts logical deduction using set of neural operators constructed from first-order logic rules (more specifically, a set of "learable" Horn clauses). Its architecture is inspired by NLM (Dong al. 2019) (details about the difference can be found in Appendix(] Three logic rules are considered as essential meta-rules: BooleanLogic, Expansion, and Reduction.
BooleanLogic: expression( (xniiooo((111,1) T2, ... ,r,) > p(#1, 22, ,rr),
where expression is composed of a combination of Boolean operations (AND, OR, and NOT) and P is the output predicate. For given r-ary predicate and given permutation 1i S,, define (x), ,I,) = p(z ... (I(r)) where S, is the set of all possible permutations as the arguments an input predicate. The corresponding neural implementation of BooleanLogic is
25

---

Under review as a conference paper at ICLR 2022
(MLP (2o( ;zr)) where a is the sigmoid activation function, MLP refers to multi- layer perceptron, a Permute)() neural operator transforms input predicates to pa ... ...,, ,2y). and 0 is the learnable parameter within the model. This is similar to the implicit Horn clause with the universal quantifier(V), e.g., pi (x) A p2(2)) + f(x) implicitly denoting Va pı p1 A p2(r) + p(x))
The class of neural operators can be viewed as "learnable" Horn clauses.
Expansi on, and Reduction on are two types of meta-rules for quantification that bridge predicates of different arities with logic quantifiers (V and 3). Expansion introduces new and distinct variable 1r+1 for a set of r-ary predicates with the universal quantifier(v)). For this reason, Expansion creates a new predicate 9 from p.
Expansion: P(,1, ,.r) - (21, LT,, (rrrBr++))
where Iy+1 & {- The corresponding neural implementation of Expansion, denoted by Expand(-), expands the r-ary predicates into the (r + 1)-ary predicates by repeating the r-ary predicates and stacking them in a new dimension. Conversely, Reduction removes the variable 2 in set of (r 1)-ary predicates via the quantifiers of A or
Reduction: : Var+1 p(21, T2,
Cr. Tr+1) + q (11 T2. Fr), or
31, p(2 #2, Gr, q( (211 U2. (.r)
The corresponding neural implementation of Reduction, denoted by Reduce(-), ), reducing the (r+1)- ary predicates into the r-ary predicates by taking the minimum (resp. maximum) along the dimension of +r+1 due to the universal quantifier A (resp. existential quantifier 3).
3.3 PLANNER: ACTIVATING AND FORWARD-CHAINING LEARNABLE HORN CLAUSES
The Planner is our key module to address the capability tradeoff in the MTR problem: is responsible for activating the neural operators in the Reasoner and chaining them into reasoning paths. Existing learning- to-reason approaches, which are often based on inductive logic programming
(ILP) (Crppprr al. 2020 Cropper & Dumancic 2020; Muggleton & De Raedt 1994) and the
correspondingly neural-ILP ILP methods (Dong et al. 2019: Shi et al. 2020). Conventional ILP methods suffer from several drawbacks, such as heavy reliance human-crafted templates and sensitivity to noise. On the other hand, neural-ILP methods (Dong et al. 2019; Shi et al., 2020). leveraging the strength of deep learning and ILP, such as the Neural Logic Machine (NLM) (Dong al., 2019), lack explicitness in the learning process of searching for reasoning paths. Let us take the learning process of NLM for example, which follows an intuitive two-step procedure. first fully connects all the neural blocks and then searches all possible connections (corresponding to all possible predicate candidates) exhaustively to identify the desired reasoning path (corresponding to the desired predicate).
By using our proposed Planner module, we can strike a better capability-efficinney tradeoff. Rather than conducting exhaustive search over all possible predicate candidates in NLM. the Planner prunes all the unnecessary operators and identifies an essential reasoning path with low complexity for given problem. Consider the following example. As shown in Fig. and Fig. A (the highlighted orange-colored blocks), at each reasoning step. the Planner takes the input predicates binary
and determines which neural operators should be activated. The decision is represented vector - ./..../' [ (termed operator footprin() - that corresponds to the neural operators of Expand, Reduce and DirectInput at different arity groups. By chaining these sequential decisions, a sequence of operator footprints is formulated as reasoning path, as the highlighted orange-colores path in Fig. ZAA. Generally, the neural operators defined in the Reasoner can also be viewed as (learnable) Horn Clauses (Horn 1951 Chandra & Harel 1985), and the Planner forward-chains them into a reasoning path.
SEARNING-TO-REASE VIA REINFORCEMENT LEARNING
In our problem, the decision variables of the Planner are binary indicators of whether a neural operator module should be activated. Readers familiar with Markov decision processes (MDPs) [Puterman]
DDrectInput is an identity mapping of the inputs where its following operators of Permute, ML.P and sigmoid nonlinearity can directly approximate the BooleanLogic.
26

---

Under review as a conference paper at ICLR 2022
A. Base Planner
B. Enhanced Planner
Input Predicates:
Hoduce
Min
Aox Vo
lopare
Pox ing
Rokeer
Lxpand
MLP
F3
Figure 3: The architecture of the Base Planner and its enhanced version. Left: The Basc Planner is based on one-layer fully- autivvatii Reasoner followed by max nax-pooling operations to predict an indicator whether an op should be activated. Right: The Enhanced Planner uses MCTS to further boost the performance.
2014) might notice that the reasoning path of our model in Fig. 2 resembles temporal rollout in MDP formulation (Sutton & Barto 2018). Therefore, we frame learning- tarnn---t-rrronn as a sequential decision-making problem and adopt off-the-shelf reinforcement learning (RL) algorithms.
4.1 AN MDP FORMULATION OF THE LEARNING-TO-REASO PROBLEM
An MDP is defined as the tuple (S, P...... 7). where S and A are finite sets of states and actions, the transition kernel Pag specifies the probability transition from state a to state 8' E S by taking action E A. R(s, a): S A + R is the reward function, and 0 S 7 S 1 is a discount factor. stationary policy π S x A + [0. 1] is a probabilistic mapping from states to actions. The primary objective of an RL algorithm is to identify near-optimal policy x² that satisfies
Tmax 11
Ar((s, ar ~ m) m)]}
[(T) = E
Σ t=
arg max
3
where Taax is a positive integer denoting the horizon length that is, the maximum length of rollout. The resemblance between Fig. [aand an MDP formulation relatively intuitive as summarized in [ At the (-th time step, the state corresponds to the set of predicates, with the superscript denoting the corresponding arity. The action a [4, ..... 1 (c.g.. an [000011] is binary Vector that indicates the activated neural operators (i.e., the operator footprint), where is the number of operators per layer. The reward rt defined to be
10P O O
K-1 T/ ift < Tmax
Accuracy,
Tmax
That is, the terminal reward is be the reasoning accuracy at the end of the reasoning path (see Appendix A.2 for its definition), and the intermediate reward at each step is chosen to be the negated number of activated operators (which penalizes the cost of performing the current reasoning step). The transition kernel P corresponds to the function modeled by one Reasoner layer; each Reasoner layer will take the state (predicates) st-1 = [ 0! OZ-1; and the action (operator footprint)
a, = [70, ,,.....,.......... I, - as its input and then generate the next state (predicates) s, = [0, O O This also implies that the Reasoner layer defines a deterministic transition kernel, i.e., given 8 1 and
ax the next state 8, is determined.
Table 1: The identification between the concepts of PRIMA and that of RL the t-th time step.
Transition
RL
State
Action
Reward rz
Policy
Rollout
Kernel a
Predicates different arities: [0 O0,O% Of},
Operator footprint:
One layer of Reasoner
PRIMA
[K-1
Eq. D
Planner
Reasoning path
27

---

Under review as a conference paper at ICLR 2022
4.2 POLICY NETWORK: MODELING OF PLANNER
The Planner module is embodied in the policy network. As shown in Fig. BA, the base planner a separate module that has the same architecture of 1-layer (fully-activated) Reasoner followed by max-poolie max-pooling layer. This architecture enables the reduction of input predicates to the specific indicators by reflecting whether the operations at the corresponding position of one layer of Reasoner active inactive. Further, we can also leverage Monte-Cario Tree Search (MCTS) (Browne eet al. 2012 Munos 2014) to boost the performance, which leads to an Enhanced Planner (Fig. (3).. An MCTS algorithm such the Upper Confidence Bound for Trees (UCT) method (Kocsis et al. 2006), is a model-based RL algorithm that plans the best action at each time step (Browne et al. 2012 by constructing a search tree, with states as nodes and actions as edges. The Enhanced Planner uses MCTS to exploit partial knowledge of the problem structure (i.e., the deterministic transition kernel P defined by the Reasoner layer) and construct a search tree to help identify the best actions (which ops to activate). Details of the MCTS algorithms used in the Enhanced Planner can be found in Appendix A.
4.3 OVERALL LEARNING FRAMEWORK
As illustrated in Fig. 3 we introduce concrete data-driven ceriiiion-making methods-that is, RL approaches (Sutton & Barto, 2018 to address the learning Maaming-to-reaaee to-reason problem. To illustrate this, we apply the model-free RL method REINFORCE (Williams 1992) and the model-based method MuZero (Schrittwieser et al. 2019). Compared with model-free reinforcement learning, model-based reinforcement learning (MBRL) more effectively handles large search-space problems, such as the game of Go (Silver et al. b0j7b/bjai. MuZero (Schrittwieser et al. [2019), a recently proposed MBRL approach to integrating planning and learning, has achieved great success with variety of complex tasks. Motivated by the success of MuZero, we propose an MBRL approach for neural-symbot reasoning. The key insight behind adopting MuZero is that in real applications, partial structural knowledge of the transition kernel se' and reward function r(s, a). As a result of the model-based module, testing complexity can be greatly reduced by adopting MCTS, which leads to a better set of predicates. Of course, MuZero is just one option in the model-based family of approaches. We leave it as future research to propose and compare other model- based alternatives.
typically have
The pipeline of the training process is illustrated in Fig 2 (the right subfigure). After loading the model weights, the reasoning path rollouts are executed by the agent (or model instance), according to the current policy network. The performed reasoning path rollout is then stored in the replay buffer. The
is trained via rollouts sampled from the replay buffer.
5 EXPERIMENTAL RESULTS AND ANALYSIS
In this section, we evaluate the performance of different variants of PRIMA eight tasks from the family tree and graph benchmarks (Graavs et al. 2016), including 1 1-Ouudegree AdjacentTRRed, HasFather, HasSister, 4 C-Conectivvty, IsGrandparent, IsUncle, IsMGUncle. These tasks are widely used benchmarks for inductive logic program- ming Krötzsch 2020; Calautti et al. 2015). Detailed descriptions about those tasks can be found in Appendix | We evaluate their testing accuracy and reasoning cost (measured in FLOPs: the number floating-point operations executed (Clark et al., 2020)) on these tasks and compare them to several baselines. Furthermore, detailed case studies are conducted on the reasoning path, which indicates the operator sharing among different tasks. All the results demonstrate the graceful supabilit-efffciinne tradcoff of PRIMA in multi-task reasoning.
5.1 EXPERIMENTAL SETUPS
In the multi-task setting, we first randomly sample a task according to a pre-defined probability distribution (over different tasks), and then generate the data for the selected task using the same methods as in NLM (Dong et al. (201)). In addition, we also augment the generated data with one-hot encoding to indicate the selected task, which will be further converted into nullary (background) input predicates. Also, task-specific output heads are introduced to generate outputs for different tasks. These adaptions apply to NLM-MTR, DLMMMTR, and PRIMA. The reasoning accuracy is
28

---

Under review as a conference paper at ICLR 2022
Table 2: Testing Accuracy and PSS different variants PRISA on different tasks. PRISA-MoZers achieves the best performance on single-task reasoning, which confirms the strength of the MCTS-based Enhanced Planner and the MuZero learning stralegy. "m": the problem size. "PSS": Percentage Successful Seeds.
Family Tree
HasFather
HusSister
I:Grnndpuren
lsUnlle
LMMUUncle
Graph
Adjacen/ToRed
4-Coometivity
I-OuiDegree
PRISA REINFORCE
I0m 100
62.6
87.8
69.8
5027
96.5
2.3
97.7
57.3
98.4
m=d0i
m=110
47.7
71.6
92.8
35.5
97.4
48.7
Single Tack
PSS
0
0
0
0
FSS
0
0
B0-20
71.3
61.3
91.3
98.1
99.5
m=10
62.3
31.3
61.5
PRISA.PPP
Iim 100
93.2
78.7
98.2
97.3
99.1
m--50
85.5
95.2
96.3
PSS
0
0
07
PSS
01
0
mc:30
100
100
100
100
100
m=10
100
100
100
PRISA MuZerro
100
100
100
100
100
100
used as the reward. In the inference (or testing) stage, the Reasoner combined with the learned Base Planner to perform the tasks, instead of enhanced planner (MCTS), to reduce the extra computation. The problem size for training is always 10 for graph tasks and 20 for family tree tasks across all setrings, regardless of the sizes of testing problems.
5.2 OVERALL PERFORMANCE
Single-task reasoning capability Before we proceed evaluate the MTR capabilities of PRIMA, we first adapt it to the single-task setting by letting the nullary input predicates to be zero, and then train a separate model for each individual task. We name this single-task version of PRIMA as PRISA (i.e., -eateerr--esonerrr Inside a Single Reasoning Agen),, and compare it with existing (single-task) neural-logic reasoning approaches (e.g., NLM (Dong et al. 2019). DLM (Matthieu et al., 2021p, and MemNN (Sukhbaatar al.] 2015)). First, we compare the performance of three different variants of PRISA (Section 4.3) for single-tast task reasoning, which learns their planners based on differ- reinforcement learning algorithms: PRISA-REINFOR
uses REINFORCE (Williams) [992]. PRISA-PPO uses PPO (Schulman et al. 2017). and PRISA-MuZerro uses MuZero (Schrittwieser 2019). We report the test accuracy and the Percentage of Successful Seeds (PSS) in Table 2 to measure the model's 's reasoning capabilities, where the PSS reaches 100% of success rates (Matthieu
al.
al. 202T). We note that PRISA-MuZero has the same 100% accuracy as NLM (Dong et al. 2019).
DLM Matthieu et al., 2021), and ØILP (Evans & Grefenstette 2018) across different tasks, and outperforms MemNN (Sukhbaatar et al. 2015) (shown the single-task part in Table 3.. But also has a higher successful percentage (PSS) in comparison with other methods. The results show that PRISA A-MuZer achieves the best performance on single-task reasoning, which confirms the strength of the MCTS-based Enhanced Planner (Section 4.3) and the MuZero learning strategy. Therefore, will use the Enhanced Planner and MuZero in PRISA and PRIMA in the rest of our empirical studics.
Multi-task reasoning capability Next, we evaluate the MTR capabilities of PRIMA. To the best of our knowledge, there no existing approach that is designed speeifically for MTR. Therefore, we adapt NLM and DLM into their multi-task versions, named NLM-MTR and DLM-MTR, respectively. NLM-MTR and DLM-MTR follow the same input and output modification as what we did to upgrade PRISA to PRIMA (Section 3.T). By this, we can examine the contribution of our proposed Planner-Reasoner architecture for MTR. As shown in Table 3 PRIMA (with MuZero as the Base Planner) performs well (perfectly) different reasoning tasks. On the other hand, DLM-MTR experiences some performance degradation (on Ad adljaces jaajenen This result confirms that our Planner-Reasoner architecture is more suitable for MTR. We conjecture that the benefit comes from using a Planner to explicitly select the necessary neural operators for each task, avoiding potential conflicts between different tasks during the learning process.
Experiments are also conducted to test the performance of PRIMA with different problem sizes. The problem size in training is 10 for all graph tasks and 20 for all family-tree tasks. In testing, we evaluate the methods on much larger problem sizes (50 for the graph tasks and 100 for the family tree tasks), which the methods have not seen before. Therefore, the Planner must dynamically activate proper set of neural operators to construct a path to solve the new problem. As reported in Fig, 4 and Tables and 3 PRIMA can achieve the best accuracy and lower flops when the problem sizes for training and testing are different.
Reasoning efficiency To measure the reasoning efficiency of the proposed methods at the inference stage, PRIMA is compared with NLM, MemNN, and MLMM-TR in terms of FLOPSs As shown in
29

---

Under review as a conference paper at ICLR 2022
Table 3:
Testing Accuracy and PSS of PRIMA and other baselines on different reasoning tasks. The results of OILP, NLM, and DLM are merged in one row due to space constraints and are presented in the same order if the results are different. Note that PRIMA's s test efficiency is superior to ULM-MTR's as shown in Fig. q "m"" : the problem size. "PSS": Percentage of Successful Seeds. Numbers in red denote 100%.
Family Tre:
HowFather
HooSister
Islnce
IsMiil
Goph
AdjpcentEolle
4-Oomorivity
I-QuiDegro
1i=2I1
19133
$fr.3
1662
18.3
39.3
12.2
10.3
19.3
Single Task
McnNN
m=100
59.8
19.5
97.7
96
98.4
m-30
93.1
51.3
78.6
PSS
01
01
01
0
PSS
m--10
100
103
100
100
100
m--10
100)
100
100
AILEY NLMY DLM
100
100
100
100
100
100
m-50
100
100
100
PSS
100
100
100
1002 502 100
101 20V 70
PSS
100% 90V 90
100
100
M=20
100
100
100
100
100
m=
100)
100
100
NI M-M--
m=100
100
100
100
100
100
mm:50
100
100
100
PSS
180
100
180
100
90
PSS
100
100
Mall-Tak
180
103
180
1000
100
96.7
100
100
DLM-MIR
1m1100
180
100
180
100
100
um350
97.2
100
100
PSS
180
100
180
100
100
PSS
100
100
100
100
100
100
100
m==10
100
100
100
PRIMA
m-100
100
100
100
100
100
m-30
100
100
100
PSS
100
100
100
100
90
PSS
100
100
Fig. 4. NLM and NLM-MTR demonstrate similar performance and suffer the highest reasoning cost when is tested with large problem sizes, such as 50 for graph tasks and 100 for family tree tasks. For MemNN, although the FLOPs of it seem low in most cases of testing, its testing accuracy bad and cannot achieve accurate predictions (Table 3.. In contrast, PRIMA can significantly reduce the reasoning complexity by intelligently selecting ops using planner. Overall, PRIMA strikes satisfactory aapabilit--efficenn tradeoff in comparison with all available multi-tasking baselines.
Operator/Path sharing in MTR To take closer look into how PRIMA achieves such a better capability-efficinnyy tradeoff (in Fig. 4). we examine the reasoning paths on three different graph tasks: 1-Out degree, instances from these three tasks and feed them into PRIMA separately to generate their corresponding reasoning paths. The results are plotted in Fig. [5A, where the gray paths denote the ones shared across tasks, and the colored ones are task task-specific. It clearly shows that PRIMA learns a large set of neural operators sharable across tasks. Given each input instance from particular task, PRIMA activates a set of shared paths along with a few task-specific paths to deduce the logical consequences.
aarjaaeenoReee and 4-Connect i ivity. Specifically, we sample
Generalizability of the Planner To demonstrate the generalizability of the Planner module in PRIMA, we generate input instances of Ad jacentt ToRed with topologies that have not been seen during training. In Fig. [B. we show the reasoning paths activated by the Planner for these different topologies, which demonstrates that different input instances share a large portion of reasoning should rely on a common set of skills. More interestingly, we notice that even for solving this same task, the Planner will have to call upon few instance- dependent sub-paths to handle the subtle inherent differences (e.g., the graph topology) that exist between different input instances. For this reason, PRIMA maintains an instance- dependeed dynamic architecture, which is in sharp contrast to the Neural Architecture Search (NAS) approaches. Although NAS may also use RL algorithms to seek for a smaller architecture (Zoph & Le (2077), it only searches for static architecture that will be applied to all the input instances.
paths. This fact is not surprising solving the same task of Ad AdjacentToBee
Testing conject stoe=10)
Testiig conpect stce=30)
Testing lobject 202)=20)
Testing Sonject sizee 1001
FLOPUNN
FLOPLON
[LOP5 (6)
OP
NLM NLM-MTR
PRIMA
-MmmNN
The reasoning costs (in FLOPs) of different models at the inference stage (1 XM=lx100 1 G=1x100)) Compared to NLM and NLM-MTR, our PRIMA significantly reduces the reasoning complexity by intelligently selecting ops (short for neural operators) using a planner. Although the FLOPs of MemNN seem low in most cases of testing, its testing accuracy is bad and cannot achieve accurate predictions (see Table "OD" denotes 1 degree,, likewise, "AR"tadjacentToRee "HS": ster, "GP":IaGrandparee
Figure 4:
"HF" "UN":IsUnele "MG": IMGUncle.
30

---

Under review as a conference paper at ICLR 2022
Gradi
10
P
Fockors
1
A. Skili sharing among different tasks.
B. mostance-ddpendent reasoniin path withiin the same task.
Figure 5: The reasoning paths of PRIMA. The gray arrows denote the "shared path". The colored arrows in sub-figures (A) and (B) denote task-specific paths and instanco-specific paths, respectively. In (A), besides the Adjacent Toee task we have introduced carlier, we also consider the 1 d-gut task, which reasons about whether the out-degree of a node is exactly equal to 1, and the 4 Conneect ivity task, which is to decide whether there are two nodes connected within 4 hops.
6 RELATED WORK
Multi-task learning Multi-task Learning (Zhang & Yang 2021; Zhou et al. [011]] Pan & Yang 2009) has focused primarily on the supervised learning paradigm, which itself can be divided into several approaches. The first is feature-based MTI., also called multi-task feature learning, (Argyriou al. 2008). which explores the sharing features across different tasks using regularization tech- niques (Shinhhara 2016 Liu et al., 2017). The second approach assumes that tasks intrinsically related, such as low- low-rank learning (Ando & Zhang, 2005 Zhang et al. 2005), learning with task clustering (Gu et al. 201 Zhang al.] 2016), and task-relation learning (such via task similarity. task correlation, or task covariance) (Goncalves et al. 2016 Zhang 2013 Ciliberto et al. 2015).
MTL has also been explored under other paradigms such unsupervised learning, for example,
multi-task clustering in (Zhang & Zhang 2013 Zhang et al.
2015b Gu et al. 2011 Zhang 2015).
Neural-symbolic reasoning Neural-symbolic AI for reasoning and inference has a long his- tory (Besold et al.) 2017: Bader et al., [2044] Garcez et al. 2008), and neural-ILP has developed primarily in two major directions (Cropper et al. 2020). The first direction applies neural operators such tensor calculus simulate logic reasoning (Yang et al., 2017 Dong al. 2019 Shi et al. 2020) and (Manhaeve et al. 2018). The second direction involves relaxed subset selection (Evans & Grefenstette 2018 Si al. 2019) with a predefined set of task-specific logic clauses. This approach reduces the task to subset-selecte selection problem by selecting a subset of clauses from the predefined set and using neural networks search for relaxed solution.
Our work is different from the most recent work, such as (Dong et al. 2019; Shi et al. 2020), in several ways. Compared with [Dong et al.] 2019), the most notable difference the improvement in efficiency by introducing learning-to- re-soo via reinforcement learning. A second major difference is that PRIMA offers more generalizability than NLM by decomposing the logic operators into more finely-grained units. We refer readers for more detailed related work to Appendix D
7 CONCLUSION
A long-standing challenge in multi-task reasoning (MTR) is the intrinsic conflict between capability and efficiency. Our main contribution is the development a novel neural-logic architecture termed PRIMA, which learns to perform efficient multitask reasoning (in first-order logic) by dynamically chaining learnable Horn clauses (represented by the neural logic operators). PRIMA improves inference efficiency by dynamically pruning unnecessary neural logic operators, and achieves state- of-the-art balance between MTR capability and inference efficiency. The model's 's training follows complete data-driven end-to-end approach via deep reinforcement learning, and the performance validated across a variety of benchmarks. Future work could include extending the framework to high-order logic and investigating scenarios when meta-rules have hierarchical structure.
31

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We commit to ensuring that other researchers with reasonable background knowledge in our area can reproduce our theoretical and empirical results. The algorithm details can be seen in Appendix AA Specifically, benchmark details are provided in Appendix B.1] in Appendix [32] and computing infrastructure are detailed Appendix B.3 The experiment details can be seen in Appendix B
settings are provided
ETHICS STATEMENT
This work is about the methodology of achieving a capability-effciinncy trade-off in multi-task first-order logic reasoning. The potential impact of this work is likely to further extend the framework high-order logic and investigate scenarios when meta-rules have hierarchical structure, which should be generally beneficial to the multi-task learning research community. We have not considered specific applications or practical scenarios as the goal of this work. Hence, does have any direct ethical consequences.
REFERENCES
Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 2005.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In JCML, 2017.
Peter B Andrews. An introduction to mathematical logic and type theory, volume 27, Springer Science & Business Media, 2002.
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. MLJ, 2008
Peter Auer. Using confidence bounds for exploitationnexxllratiio trade-offs. Journal of Machine Learning Research, 3(Nov):3 2002.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235 256, 2002.
Sebastian Bader and Pascal Hitzler. Dimensions of neural-symbolic integration-a structured survey, arXiv preprint cx/0511043 2005.
Sebastian Bader, Pascal Hitzler, and Steffen Hölldobler. The integration of connectionism and first-order knowledge representation and reasoning as challenge for artificial intelligence. arXiv preprint cs/0408069, 2004.
Peter W Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interac- tion networks for learning about objects, relations and physics. arXiv preprint arXiv: 1612.0022, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1rXiv:1806.01 1806.01261, 2018.
Yoshua Bengio. From system 1 deep learning to system deep learning. In Thirty-third Conference Neural Information Processing Systems, 2019.
Tarek R Besold, Artur Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pas- cal Hitzler. Kai-U uwe Kühnberger, Luis C Lamb, Daniel Lowd, Priscila Machado Vieira Lima, et al. Neural- symbolic learning and reasoning: survey and interpretation. arXiv preprint 1rXiv: 2017.
32

---

Under review as a conference paper at ICLR 2022
Timo Bräm, Gino Brunner, Oliver Richter, and Roger Wattenhofer. Attentive multi-tass deep reinforcement learning. In ECML/PKDD, 2019.
Cameron B Browne, Edward Powley. Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions Computational Intelligence and Al in games, 4(1):1-43 2012.
Salam El Bsat, Haitham Bou-Ammar, and Matthew E. Taylor. Scalable multitask policy gradient reinforcement learning. AAAI, 2017.
Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli. Sparse multi-task reinforcement learning. NIPS, 2014.
Marco Calautti, Georg Gottlob, and Andreas Pieris. Chase termination for guarded existential rules. In Proceedings of the 34th ACM Systems, pp. 91-103, 2015.
Symposiun on Principles of Database
L. Cao. Non-iid recommender systems: A review and framework of recommendation paradigm shifting. Engineering, 2(2):212-224, 2016.
Deepayan Chakrabarti and Christos Faloutsos. Graph mining: Laws, generators, and algorithms. ACM computing surveys (CSUR), 38(1):2-es, 2006.
Ashok K Chandra and David Harel. Horn clause queries and generalizations. The Journal of Logic Programming, 2(1):1-15, 1985.
Jianzhong Chen, Stephen Muggleton, and José Santos. Learning probabilistic logic models from probabilistic examples. Machine learning, 73(1):55-85, 2008.
K. Chen, F. Yang, and X. Chen. Planning with task-oriented knowledge acquisition for service robot. IJCAI, pp. 812-818, 2016.
Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinav Gupta. Iterative visual reasoning beyond convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7239- 7248, 2018.
Carlo Ciliberto, Youssef Mroueh, Tomaso Poggio, and Lorenzo Rosasco. Convex learning of multiple tasks and their structure. In ICML, 2015.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 105555, 2020.
Diane Cook and Lawrence B Holder. Mining graph data. John Wiley & Sons, 2006.
Michael Crawshaw. Multi-task learning with deep neural networks: survey, arXiv preprint 0rXX::2009.09796, 2020.
Andrew Cropper and Sebastijan Dumaneic. Inductive logic programming at 30: a new introduction. arXiv preprint arXi::2008.07912 2020.
Andrew Cropper, Sebastijan Dumancic, and Stephen Muggleton. Turning 30: New ideas in inductive logic programming. arXiv preprint arXiv:2002. /002, 2020.
Aniket Anand Deshmukh, Ürtnn Dogan, and Clayton Scott. Multi-task learning for contextual bandits. In NIPS, 2017.
Carlos Diuk, Andre Cohen, and Michael L Littman. An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 240-247, 2008.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. arXiv preprint arXiv: /904 16944, 2019.
33

---

Under review as a conference paper at ICLR 2022
Sašo Džeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Machine learning, 43(1):7-52,, 2001.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukeuoglu. IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In ICML, 2018.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research, 61:1-64, 2018.
Melvin Fitting. First-order logic and automated theorem proving. Springer Science & Business Media, 2012.
Jean H Gallier. Logic for computer science: foundations of automatic theorem proving. Courier Dover Publications, 2015.
M. C. Ganiz, C. George, and W. M Pottenger. Higher order naive bayes: A novel non-iid approach to text classification. IEEE Transactions on Knowledge and Data Engineering, 23(7):1022- 1034, 2010.
Artur SD' Avila Garcez, Luis Lamb, and Dov M Gabbay. Neural-symbolic cognitive reasoning. Springer Science & Business Media, 2008.
M. Gebser, B. Kaufmann, and T. Schaub. Conflict-driven answer set solving: From theory to practice. Artificial Intelligence, 187-188:52-888 2012.
Michael Gelfond and Vladimir Lifschitz. Action languages. 1998.
André R. Goncalves, Fernando J. Von Zuben, and Arindam Banerjee. Multi-task sparse structure learning with Gaussian copula models. JMLR, 2016.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv: 140614466,,6 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626): 471-476, 2016.
Quanquan Gu, Zhenhui Li, and Jiawei Han. Learning a kernel for multi-task clustering. In AAAI, 2011.
Carlos Guestrin, Daphne Koller, Chris Gearhart, and Neal Kanodia. Generalizing plans new environments in relational mdps. In Proceedings of the 18th international joint conference on Artificial intelligence, pp. 1010, 2003.
Mare Hanheide, Moritz Göbelbecker, Graham Horn, Andrzej Pronobis, Kristoffer Sjöö, Alper Aydemir, Patric Jensfelt, Charles Gretton, Richard Dearden, Miroslav Janicek, et al. Robot task planning and explanation in open and uncertain worlds. Artificial Intelligence, 2015.
M. Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 191-246, 2006.
Alfred Horn. On sentences which are true of direct unions of algebrasl. The Journal of Symbolic Logic, 16(1):14-2)) 21, 1951
Maximilian Igl, Andrew Gambardella, Nantas Nardelli, N. Siddharth, Wendelin Böhmer, and Shimon Whiteson. Multitask soft option learning. In UAI, 2020.
Herbert Jaeger. Deep neural reasoning. Nature, 53877626):467-468, 2016.
Ramtin Keramati, Jay Whang, Patrick Cho, and Emma Brunskill. Strategic object oriented rein- forcement learning. In Exploration in Reinforcement Learning Workshop at the 35th International Conference Machine Learning, 2018
34

---

Under review as a conference paper at ICLR 2022
P. Khandelwal. S. Zhang, J. Sinapov, M. Leonetti, J. Thomason, F. Yang, Gori, M. Svetlik, P. Khante, V. Lifschitz, and Stone. Bwibots: A platform for bridging the gap between ai and human human-robot interaction research. The International Journal of Robotics Research, 36(5-7) -0903355556666 2017.
Levente Kocsis, Csaba Szepesvári, and Jan Willemson. Improved monte-carlo search. Univ. Tartu, Estonia, Tech. Rep, 2006.
Daphne Koller. Probabilistic relational models. In International Conference on Inductive Logic Programmingg pp. 3-13. Springer, 1999.
Daphne Koller, Nir Friedman, Sašo Džeroski, Charles Sutton, Andrew McCallum, Avi Pfeffer, Pieter Abbeel, Ming-Fai Wong, David Heckerman, Chris Meek, al. Introduction to statistical relational learning. MIT press, 2007.
Markus Krötzsch. Computing cores for existential rules with the standard chase and asp. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, volume 17, pp. 603-613, 2020.
Alessandro Lazaric and Mohammad Ghavamzadeh. Bayesian multi-task reinforcement learning. In ICML, 2010.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436444, 2015.
Hui Li, Xuejun Liao, and Lawrence Carin. Multi-task reinforcement learning in partially observable stochastic environments. JMLR, 2009.
Sijin Li. Zhi-Qian Zhi-Qiang Liu, and Antoni B. Chan. Heterogencous multi-task Iearning for human pose estimation with deep convolutional neural network. IJCV, 2015.
V. Lifschitz. What is answer programming? In Proceedings of the AAAI Conference Artificial Intelligence, pp. 1594-1597. MIT Press, 2008
Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In NeurlPS, 2019.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classification. In ACL, 2017.
Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo. Multi-task deep visual-semantic embedding for video thumbnail selection. In CVPR, 2015.
Robin Manhaeve, Sebastijan Dumancie, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Processing Systems, pp. 3749-3759, 2018
Zimmer Matthieu, Feng Xuening, Glanois Claire, Jiang Zhaohui, Zhang Jianyi, Weng Paul, Jianye Hao, Dong Li, and Wulong Liu. Differentiable logic machines. arXiv preprint arXiv:2102.115299 2021.
Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Pddl-the planning domain definition language. 1998.
Nikola Mrksic, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gasic, Pei-hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Multi-domain dialog state tracking using recurrent neural networks. In ACL, 2015.
Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming. 19:629 679, 1994.
Remi Munos. From bandits to monte-carlo tree search: The optimistic principle applied to optimiza- tion and planning. Foundations and Trends in Machine Learning, 2014.
35

---

Under review as a conference paper at ICLR 2022
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In ICML, 2017.
Rasmus Berg Palm, Ulrich Paquet, and Ole Winther. Recurrent relational networks. arXiv preprint arXiv:1711.08023 2017.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1455-135, 2009.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer
reinforcement learning. In ICLR, 2016.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathemutics and Artificial Intelligence, 61(3):203-25 2000000,2202200000 2011.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2016.
Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. arXiv preprint
01427, 2017.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? arXiv preprint IrXiv:1005. 1604, 2018.
Andrew M. Saxe, Adam Christopher Earle, and Benjamin Rosman. Hierarchy through composition with multitask LMDPs. In ICML, 2017.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint 1rXiv:191..08265, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Sahil Sharma and Balaraman Ravindran. Online multi-task learning using active sampling. In ICLR Workshop, 2017.
Sahil Sharma, Ashutosh Kumar Jha, Parikshit Hegde, and Balaraman Ravindran. Learning to multi-task by active sampling. In ICLR, 2018
Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang. Neural logic reasoning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1375-1374, 2020.
Yusuke Shinohara. Adversarial multi-task learning of deep neural networks for robust speech recognition. In Interspeech, 2016.
Xujie Si, Mukund Raghothaman, Kihong Heo, and Mayur Naik. Synthesizing datalog programs using numerical relaxation. arXiv preprint arXiv: /96.00630 2019.
David Silver, Thomas Hubert, Julian Schrittiesser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 171.0/ 12.011 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 2017b.
36

---

Under review as a conference paper at ICLR 2022
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 4440-2448, 2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Prasad Tadepalli, Robert Givan, and Kurt Driessens. Relational reinforcement learning: An overview. In Proceedings of the ICML-2004 workshop on relational reinforcement learning, pp. 1-9, 2004.
Yee Whye Teh, Victor Bapst, Wojeiech M. Czamnekki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In NIPS, 2017.
Rasul Tutunov, Dongho Kim, and Haitham Bom-ar Distributed multitask reinforcement learning with quadratic convergence. In NeurlPS, 2018.
Frank Van Harmelen, Vladimir Lifschitz, and Bruce Porter. Handbook of knowledge representation. Elsevier, 2008
Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics, 9()):1363, 2020.
Tung-Long Vuong, Do Van Nguyen, Tai-Long Nguyen, Cong-Minh Bui, Hai-Dang Kieu, Viet-Cuong Ta. Quoc-Long Tran, and Thanh Ha Le. Sharing experience in multitask reinforcement learning. In IJCAI, 2019.
Ronald J Williams.
Simple statistical fradient- algorithms for connectionist reinforcement learning. Machine learning, 8(3-4)2229-25) 256, 1992.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: A hierarchical Bayesian approach. In ICML, 2007.
Victoria Xia, Zi Wang, and Leslie Pack Kaclbling. Learning sparse relational transition models. arXiv preprint ar 1810. 177, 2018
Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. In Advances in Neural Information Processing Systems, pp. 2319-2328, 2017.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In NIPS, pp. 1031-1042, 2018.
N. Yoshida, T. Nishio, M. Morikura, K. Yamamoto, and R. Yonetani. Hybrid-fl for wireless networks: Cooperative learning mechanism using non-iid data. In ICC 2020-2020 IEEE International Conference Communications (ICC), pp. 1-7. IEEE, 2020.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint 1806v:
2018.
Jian Zhang, Zoubin Ghahramani, and Yiming Yang. Learning multiple related tasks using latent independent component analysis. In NIPS, 2005.
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: comprehensive review. Computational Social Newworks, 6(1):1-23, 2019.
Wenlu Zhang, Rongjian Li, Tao Zeng. Qian Sun, Sudhir Kumar, Jieping Ye, and Shuiwang Ji. Deep model based transfer and multi-task learning for biological image analysis. In KDD, 2015a.
Xianchao Zhang and Xiaotong Zhang. Smart multi-task Bregman clustering and multi-task kernel clustering. AAAI, 2013.
Xianchao Zhang, Xiaotong Zhang, and Han Liu. Smart multitask Bregman clustering and multitask kernel clustering. ACM TKDD, 2015b.
Xianchao Zhang, Xiaotong Zhang, and Han Liu. Self-adapted multi-task clustering. In IJCAI, 2016.
37

---

Under review as a conference paper at ICLR 2022
Xiao-Lei Zhang. Convex discriminative multitask clustering. IEEE TPAMI, 2015.
Yu Zhang. Heeerogeeeus--eiihhhoooooooooo based multi-task local learning algorithms. In NIPS, 2013.
Yu Zhang and Qiang Yang. survey on multi-task learning, IEEE Transactions on Knowledge and Data Engineering, 2021.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. ECCV, 2014.
Jiayu Zhou, Jianhui Chen, and Jieping Ye. Malsar: Multi-task learning via structural regularization. Arizona State University, 21, 2011.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning deep reinforcement learning: survey, arXiv preprint arXiv:2009.0788 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. 2017.
2017.
38


Reviews:
[{"review_id": "t9KaSRojRj", "reviewer": "Reviewer_z1ve", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the paper is a motivating description of the problem, detailed presentation of approach, evaluation and supplementary material. \n\nThe problems with the paper are that:\na)  it pays cursory attention to important details about planning, the main idea, of the approach. Specifically, the field of planning is diverse with model-based (PDDL-based) and model-free RL approaches. The paper is only referring to the latter but makes no mention of it. \n\nb) glosses that planning is not just chaining. The complexity of planning increases when actions interact with each other and there is increase in the \"type of actions\". The authors do not talk about diversity in type of actions and their interaction. If there is no interaction among actions, any chain will give us the solution. The trade-off between different search paths is just cost, not feasibility of finding a plan -- a simpler problem.\n\nc) does not talk how the different tasks relate to each other in evaluation. The ones of family relationship (father, sister, grandparent, uncle and mguncle) seem quite similar. \n\nd) does not discuss the implication of results. The approach is a rather heavy-weight machinery to use on such simple set of reasoning tasks. \n", "summary_of_the_paper": "The paper claims to present a new approach for multi-task reasoning where reasoning operators are defined as neural operators and a planner is used to chain them based on tasks. The approach is end-to-end trained and it is claimed to show good improvement across the demonstrated tasks.", "summary_of_the_review": "The paper is motivated on an important problem - MTR. The approach is also on a promising intuition that a planner should be able to chain the logical operators based on the needs to a task. However, the approach is not well explained with respect to related work and planning capability is not well utilized. \n\nBeyond improving explaining, I will suggest the authors to attempt more mainstream reasoning tasks and not on toy settings.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "amgzGgmXl98", "reviewer": "Reviewer_Phsj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, PRIMA seems like a strong approach for multi-task logical reasoning. While the paper motivates PRIMA using the general framing of the generalization/single-task performance trade-off in multi-task learning more generally, I find this a bit tenuous (not a reason to reject, but I recommend the authors maybe temper their claims a bit). I think multi-task logical reasoning is an important problem in itself, without the broader connection (as it’s hard to even think how we’d get an approach like this to generalize to say multi-task problems in NLP/Vision/Robotics). \n\nStrengths:\n\nIn terms of strengths, I think the approach is well-formulated, and easy to understand. Leveraging th structure of the problem to do MCTS and train a MuZero style policy (planner) is a great idea, and one that clearly paid dividends. I think the components of the proposed architecture were well motivated and well executed.\n\nWeaknesses:\n\nThat being said, I have identified a few weaknesses, and I am hoping the authors can either point me to relevant information (I totally could have misunderstood certain results), or engage with me to discuss the importance of these few points:\n\nAs it stands, we evaluate PRIMA in both multi-task and single-task settings, on a set of logical reasoning tasks on family trees/graphs. These tasks, while motivated in prior work, feel very simple to me, further evidenced by the fact that PRIMA, as well as most prior work get 100% accuracy across the board, except for a handful of tasks. I’m wondering if we can trust this evaluation to truly get at the broad capabilities of this model, relative to those that already exist.\n\nThe key “win” for PRIMA over related work seems to be the reduction in FLOPs relative to prior methods like MT-NLM, MT-DLM (variants of prior logical reasoning models augmented with the “simple” task conditioning information). Performance is the same. However, it’s hard for me to understand (1) if this reduction of FLOPs actually matters in practice, (2) if this is a fair comparison (I don’t know these prior models too well, but they could be using more FLOPs for expressivity, because they considered compute “cheap” for these settings), and (3) if the corresponding reduction is meaningfully significant (e.g., it’s a 2X reduction, but is it a 2X reduction that just translates to an extra second on a laptop, or an extra few hours on a huge cluster).\n\nThe second “win” is a single probing experiment done towards the end of the paper, where PRIMA (end of Section 4) is fed new input instances with novel topologies for a graph reasoning problem. The argument here, mostly graphical, is hard to follow, but seems to indicate that PRIMA has a bias for reusing reasoning components when possible, but uses instance=dependent sub-paths when dealing with novel features. I’m not really sure how to interpret this argument, nor am I sure whether this sort of functionality is present in the baselines.\n\nTypos/Style/Questions:\n\nPage 1: “Existing MTL models.” (fragment)\n\n---\nEDIT (Post Rebuttal): The authors have addressed some (but not all) of my weaknesses in a satisfactory way. I am raising my score up to a 6.", "summary_of_the_paper": "This paper presents PRIMA (“Planner-Reasoner Inside a Multi-Task Reasoning Agent”), a multi-task reasoning model that can be applied to several tasks that require first order logical reasoning. Notably, this paper seeks to address a problem that plagues multi-task learning more generally — the tradeoff between cross-task generalizability, and single-task effectiveness. Prior work in multi-task learning shows that it’s hard to strike this balance, and most work in building neural systems for logical reasoning focus on the single task setting.\n\nThe main contribution of this paper is first a single task planning-based agent that learns to reason for an individual logical task (e.g., a graph reasoning task like `4-Connectivity` or `1-Outdegree`) by routing input problem representations (predicates describing the problem) through a planning module; this planner acts as a high-level policy that (over multiple steps) routes the input through individual “reasoning” components that are “neural” versions of first-order logic meta-operators. This is a compact, two-party approach for solving complex reasoning problems; the reasoner defines the atomic operations, while the planner figures out what operations to apply, in what order.\n\nThe full system is learned via reinforcement learning, first via model-free methods such as REINFORCE and PPO, but with significantly better results using the structure of the problem (we can always roll-out through the planner) to use Model-based RL, specifically the MuZero formulation. Using MuZero/MCTS to learn the Planner/Reasoner, PRIMA shows competitive performance on single-tasks, and to generalize to multiple tasks, we simply add a “nullary predicate” that specifies the task ID. PRIMA performs well, comparably to other approaches for logical reasoning, even when prior approaches (such as the Neural Logic Machine) are extended to multi-task settings in the same way (adding this extra task conditioning information). Where PRIMA shows benefits over prior approaches are in aggregate FLOPs (showing that PRIMA may be learning to reason more efficiently), and in a single probing experiment showing how PRIMA can generalize to new input classes unseen at training.", "summary_of_the_review": "In general, PRIMA is a neat approach for multi-task logical reasoning. However, while the method and optimization are sound, I’m slightly concerned with the results. The two big wins for PRIMA over prior methods are seemingly a reduction in FLOPs (which is hard to contextualize), and possibly evidence that indicates that PRIMA uses shared structure (which has no comparison to baselines). I think this could be a strong paper, and I welcome responses from the authors during rebuttal; I’d be happy to change my score if the above questions are meaningfully addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8fky1MMOK1z", "reviewer": "Reviewer_Z6sN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The article is interesting, but in my opinion, badly structured. It is unclear what new features have been introduced. In fact, the title and introduction only refer to PRIMA, while the article spends a lot of space presenting PRISA (the single-task version) as well. The experimental section is also divided equally between the two systems, making the purpose of the article unclear.\nI would recommend reformulating the title and the introductory parts to make them fit better with the content of the article, which is not only about PRIMA but also about PRISA. The differences between the two systems are not many, since PRIMA is based on PRISA, but in my opinion, the organisation of the content and the title leave the reader a bit lost.\n\nIn addition, the article should be re-read to correct some grammatical errors. For example, on page 1 \"to solving multiple tasks\". Also on this page is a sentence disconnected from the rest of the text: \"Existing MTL models.\" A pagina 6, \"to integrating\". On page 8 \"we examine its the reasoning paths\".", "summary_of_the_paper": "The paper presents PRIMA, a framework designed to solve the problem of multi-task reasoning. PRIMA consists of two parts: a reasoner, which produces logical deductions using a set of neural operators built by translating FOL rules; and a planner, which activates the reasoner operators and creates reasoning paths to solve the task.\nThe paper presents several tests that achieve performance usually stable on 100% succesful seeds.", "summary_of_the_review": "The system is able to achieve really good results in the tests conducted. On the other hand, the article is not structured very well and not entirely consistent with the title.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Idu0ACGmGa", "reviewer": "Reviewer_eedL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning to improve reasoning is an interesting and overlooked problem, particularly in purely learning-based systems, though they do have a long history in more traditional reasoning circles (for example the authors may want to look up utility problems in search-based reasoning frameworks [1]). I think the use of RL methods to direct reasoning is an interesting solution. In terms of concerns regarding the work, I mostly feel the need to improve reasoning efficiency was not as well-motivated and there is missing information on additional training overhead introduced by including an RL component to the overall learning problem.\n\nReasoning Cost for Single Task Problem: The paper motivates the problem of improving reasoning efficiency by appealing to MTR problems. While I agree that achieving efficient reasoning may be harder in MTR settings due to the need to generalize across more diverse tasks, I don’t see how the same is not a problem in single task settings. I would imagine PRISA presents the same gains in improvements over vanilla NLM that is presented by PRIMA over NLM-MTR. For me, the problem of introducing a planner-reasoner into an NLM style framework seems independent of the problem of extending NLM into a multi-task learning setting. So I would encourage the authors to include computational costs for PRISA in the results too. If it does turn out that PRIMA somehow is more efficient than PRISA, that might be worth pointing out too.\n\nTraining Overhead: In terms of computational cost, the paper exclusively focuses on the cost during inference. Unfortunately, this doesn’t present the complete picture. By introducing a planner-reasoner into the mix, the authors are currently looking at a more complex learning problem. Any gains at the time of reasoning are achieved at the cost of performing trials at the training time that tries to identify policies that promote smaller operator footprints. I would imagine, this introduces additional computational costs (not to mention those introduced by the use of MCTS). Unless the authors are explicitly focusing on specific application settings where the computational cost during training can be ignored, the authors should provide a discussion on the overhead introduced during training (and may be provide a comparison of training computational costs). After all, most concerns regarding energy usage of modern deep learning systems are related to the training time. If the authors do have some application setting in mind then there should be a clearer discussion of the setting in the paper.\n\nSmaller Comments:\n\nThe description of neural implementation of BooleanLogic in section 2.2 seems to be incomplete. The corresponding text refers to the Permute operator which is missing from the description.\n\nSome of the claims in the empirical evaluation need to be more carefully worded. For example,\n\n“We note that PRISA-MuZero has the same 100% accuracy as NLM  and \\deltaILP across different tasks” -- doesn’t  \\deltaILP outperform PRISA-MuZero in AdjacentToRed task?\n\n“On the other hand, DLM-MTR experiences some performance degradation (on AdjacentToRed). This result confirms that our Planner-Reasoner architecture is more suitable for MTR.” -- But this statement overlooks the fact that NLM-MTR does as well as PRIMA for all the tasks. The second statement (and the following statement) makes it sound like planner-reasoner provides an advantage over all the baselines that don’t use a Planner-Reasoner architecture.\n\nReward Function for RL: Currently there is no hyper-parameter introduced to balance the penalty related to the operator footprint size against the reward-related to the final accuracy. I expect for larger arities there could be cases in the current setting, the RL algorithm could choose lower accuracy to avoid larger operator footprints.\n\n[1] Minton, Steven. \"Quantitative results concerning the utility of explanation-based learning.\" Artificial Intelligence 42.2-3 (1990): 363-391.", "summary_of_the_paper": "The paper introduces a neuro-symbolic framework for multi-task reasoning that builds on NLM. Specifically, the method tries to achieve higher efficiency at the inference time, by using an RL component that controls the operations performed as part of the reasoning process. They adapt the system for multi-task settings by introducing nullary predicates that capture the specific task details. The method is compared to multiple baselines, including an adaptation of NLM to the multi-task setting. While the methods perform similar to the baselines in terms of the overall accuracy, the approach does result in a lower number of floating-point operations at the inference time.", "summary_of_the_review": "I think the paper looks at an often overlooked problem and introduces an interesting mechanism to drive the reasoning process. Though I do have concerns about some experimental results that are missing and could provide more information about trade-offs introduced by including a planner-reasoner in the architecture.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews for this paper were mixed (6,6,6,5) with one review (Z6sN) being somewhat uninformative. During the rebuttal, some reviewers raised their scores to 6 but overall there was not strong excitement among the reviewers, AC, and SAC. From fresh readings (by SAC and researchers with relevant expertise), this paper’s technical approach looks reasonable but feels quite incremental (novelty is not high) and the experimental results are conducted on a small scale problem where up to 100% success rate is achievable by baseline methods. Therefore, the practical significance of this approach for real-world problems with complex and noisy environments is quite unclear. Overall, the paper looks below the ICLR acceptance threshold. For improvement, we suggest providing evidence/demonstration that this method can successfully tackle more challenging real-world problems.
Decision: Reject