Optimization of Information-Seeking Dialogue Strategy
for Argumentation-Based Dialogue System
Hisao Katsumi1 , Takuya Hiraoka2 , Koichiro Yoshino1,3 ,Kazeto Yamamoto2 , Shota Motoura2 , Kunihiko Sadamasa2 , Satoshi Nakamura1
1 NARA Institute of Science and Technology
2 NEC Central Research Laboratories
3 Japan Science and Technology Agency
Abstract
Argumentation-based dialogue systems, which can handle
and exchange arguments through dialogue, have been widely
researched. It is required that these systems have sufficient
supporting information to argue their claims rationally; however, the systems often do not have enough of such information in realistic situations. One way to fill in the gap
is acquiring such missing information from dialogue partners (information-seeking dialogue). Existing informationseeking dialogue systems are based on handcrafted dialogue strategies that exhaustively examine missing information. However, the proposed strategies are not specialized incollecting information for constructing rational arguments.
Moreover, the number of system’s inquiry candidates grows
in accordance with the size of the argument set that the system deal with. In this paper, we formalize the process of
information-seeking dialogue as Markov decision processes(MDPs) and apply deep reinforcement learning (DRL) for automatically optimizing a dialogue strategy. By utilizing DRL,
our dialogue strategy can successfully minimize objectivefunctions, the number of turns it takes for our system to collect necessary information in a dialogue. We conducted dialogue experiments using two datasets from different domainsof argumentative dialogue. Experimental results show that the
proposed formalization based on MDPs works well, and thepolicy optimized by DRL outperformed existing heuristic dialogue strategies.

Introduction
Argumentation-based dialogue systems are systems that canhandle and exchange arguments with their dialogue partner. These systems have been widely researched because
their ability to argue is required in various argumentative
situations, such as persuasion and negotiation (Amgoud,
Maudet, and Parsons 2000; McBurney and Parsons 2001;
Parsons, Wooldridge, and Amgoud 2002; 2003). In general,an argument consists of a claim and its facts that support it(Besnard et al. 2014; Besnard and Hunter 2014). Systems
are required to provide rational arguments in such situations
with sufficient information that supports their claims. For example, if a system works as a prosecutor and tries to argue
the claim “The accused is guilty of larceny,” it is required to
Copyright c 2019, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

provide supporting facts such as “a witness saw the accusedsteal the goods.”
The ability to collect facts in order to construct rational
arguments is important for argumentation-based dialoguesystems because it is not often that the systems have sufficient facts to support their claims beforehand. One way
for the systems to cover for this lack of supporting facts
is collecting the required facts from a dialogue partner or
third parties through interactions, as human being do. For
examples, if a system tries to argue that “The accused is
guilty of larceny” but does not have facts that support thisclaim, it can try to collect them by asking the witnessquestions such as “Did you see the accused steal something?” The process of collecting facts can be modeled
as information-seeking dialogue (Walton and Krabbe 1995;
Fan and Toni 2015).
In some existing research, the use of handcrafted dialogue strategies for information-seeking dialogue systems
has been proposed (Parsons, Wooldridge, and Amgoud
2002; Fan and Toni 2012; 2015). These strategies are designed to examine all facts exhaustively; however, facts that
support the main claim must be collected quickly because
the available time for the systems to collect facts is limited.
Dialogue modeling based on Markov decision processes(MDPs) is widely used and these researches have tried to
optimize dialogue strategies by using reinforcement learning (RL) (Levin, Pieraccini, and Eckert 2000; Williams
and Young 2007; Misu et al. 2012; Yoshino and Kawahara2015). These strategies are defined as mapping functions(policies) from states to actions and are trained to maximize
expected future rewards, e.g., successful dialogue tasks andnumber of steps (turns) in each dialogue.
In this paper, we formalized information-seeking dialogue
as MDPs and applied deep reinforcement learning (DRL) to
find the optimal policy in dialogues. We used a double deep
Q-network (DDQN) to find a good policy that can increase
the rationality of a system’s claim quickly. We compared
our optimized dialogue strategy with existing strategies on
the basis of two heuristics, depth-first search and breadthfirst search. Experimental results show that the proposed
method outperformed existing dialogue strategies in several
information-seeking dialogue tasks for argumentation-based
dialogue systems.

Related Work
Information-Seeking Process in Dialogue
Information-seeking, which is a kind of argumentationbased dialogue, in particular frequently appears in the realworld (Walton and Krabbe 1995). It is a process of collecting information in which participants in a dialogue exchange questions and answers to collect various facts. In
previous research, heuristic dialogue strategies which exhaustively examine all facts, have been proposed (Parsons, Wooldridge, and Amgoud 2002; Fan and Toni 2012;2015). These strategies do not consider the communication
overhead cost, although, in real-world dialogue, it is important to reduce the number of inquiries because, in general,
the available time for information-seeking dialogue is limited. In addition, it is demanding to construct a strategy forsuch exhaustive examination based on heuristic rules because the rules usually depends on relations of collectedfacts or domain-specific knowledge. In contrast, in this paper, we present an automatic optimization method for building a dialogue strategy by considering the communication
overhead cost (time-pressure) and the success rate throughstatistical learning based on RL.

Dialogue Strategy Optimization forArgumentation-based Dialogue Systems
Some previous researchers applied RL to optimize dialogue
strategies for argumentation-based dialogue systems: persuasive dialogue systems (Hadoux et al. 2015; Rosenfeld
and Kraus 2016; Alahmari, Yuan, and Kudenko 2017) andnegotiation dialogue systems (Georgila and Traum 2011;
Papangelis and Georgila 2015). They focused on optimizing the dialogue strategy for presenting arguments given inadvance; nonetheless, ordinal argumentation-based dialoguesystems do not always have sufficient facts to construct rational arguments. In comparison, our research focuses on
optimizing the dialogue strategy for constructing new arguments. It contributes to developing the ability of the systems
to construct such rational arguments adaptively.

Information-Seeking Dialogue forConstructing Rational Arguments
We formulate a formal framework for information-seeking
dialogue for constructing rational arguments. We introduce
a formal definition of arguments, a method for constructing
arguments, a method for evaluating the rationality of methodof the arguments, and a dialogue protocol.

Formal Framework for Information-Seeking
Dialogue for Rational Argument Construction
Our proposed dialogue is a kind of an information-seeking
dialogue. An outline of the information-seeking process isshown in Figure 1. Questioner Que tries to provide a rational argument including a certain claim but it does not haveenough supporting facts. In the dialogue, Que asks answerer
Ans questions to collect supporting facts. Ans provides facts
if it knows, i.e., a fact is contained in its own knowledge
KA . Que updates its knowledge KQ with the facts that Ans

Figure 1: Framework of information-seeking dialogue forconstructing rational arguments.
provides. The dialogue continues until Que constructs an argument with enough rationality.
An argument hΦ, αi consists of a claim α and its support Φ. Here, Φ is a set of facts and inference rules. Formally, facts and inference rules are defined by logical expressions. Given logical expressions p1 ...pn , q satisfying
p1 ∧...∧ pn → q, a fact is represented by a logical expression
with n = 0 (i.e., q), and the inference rule is a logical expression with n 6= 0 (i.e., p1 → q). Fact q can be substituted withan assumption when the fact is necessary for an argumentation even if it is missing (Dung, Kowalski, and Toni 2009).
Figure 2 shows an example of an argument. In this example, α is q1 (yellow rounded box), and Φ is a set of two
facts (q3 and q4 ), two assumptions [asm(q2 ) and asm(q5 )]and two inference rules (l1 and l2 ). Here, asm indicates thata fact is an assumption. In addition, two inference rules, l1
and l2 , are also contained in the argument. l1 means that fact
q1 is built if we have observed three facts, q2 , q4 , and q5 ,and l2 means that fact q3 is built if we have observed fact q2 .
Note that two of the facts, q2 and q5 , are not contained in
KQ , and, thus, assumptions are used instead of them.
Given a claim α and Que’s knowledge KQ , the argumenthΦ, αi is constructed by defining Φ as Φ = KQ ∪ H, where,
H is a set of assumptions. H is interpolated as:arg minH∈H E({α} ∪ KQ ∪ H),
s.t., KQ ∪ H |= α, {α} ∪ KQ ∪ H 6|= ⊥.
Here, H is a set of all possible hypothesis candidates1 . E
is a cost function of an abduction model, and E({x, y, . . .})explains the plausibility of observing x, y, . . . at the sametime (Charniak and Shimony 1990). E is also used in calculating the rationality of the argument R(Φ, α). We will givedetails on the cost function and rationality in the next section.
1 |= means that the right term can be derivable from the left term,and ∪ means the union of sets. In addition, unnecessary inference
rules in KQ for deriving α are excluded from Φ.

Optimization of Information-Seeking Dialogue
Strategy based on Reinforcement Learning
In this section, we first describe the formulation of theinformation-seeking dialogue for constructing rational arguments, on the basis of MDPs; then, we describe how to apply
deep reinforcement learning to the formalized informationseeking dialogue to find the optimal strategy efficiently.

Formulation based on MDPs

Figure 2: Example of questioner’s argument.
In our proposed dialogue, given α, a dialogue proceeds
by repeating the following. Que asks Ans about a fact, Ans
returns a corresponding fact q if the fact is in KA , Que adds
q to KQ (KQ ← KQ ∪ q), and Que constructs the argument
and evaluates its rationality. If we process the dialogue withthe example of Figure 2, α is given as q1 . When Que asks
Ans “Did not the old man, who lived near the scene of themurder, hear any loud noises?” to collect q5 , which is necessary to support α for inference rule l1 , Ans returns q5 as
the answer. Then, Que updates KQ according to the answer(KQ ← KQ ∪ q5 ). Note that only if the corresponding fact
to the question, such as q5 , is included in KA , Ans returnthe fact. This loop is repeated until the system successfully
constructs a rational argument or the number of questions
reaches the upper limit.

Evaluation Function of Rationality
We define the rationality of arguments following an existingpiece of work (Ovchinnikova et al. 2014):R(Φ, α) = (minE({α} ∪ H 0 ) + min
E(KQ ∪ H 00 ))
0
00
H

H

− E({α} ∪ Φ). (1)We use a cost function that is used in weighted abduction
for E (see (Inoue and Inui 2011) for more details), and it
represents the plausibility of arguments of E occurring at
the same time (Charniak and Shimony 1990). The smaller
E(x) gets, the more plausible x occurring is; thus, equation(1) describes how much KQ explains the situation, where afact α and those included in KQ occur at the same time. To
calculate E, we use Phillip2 , an open source fast abductive
reasoner.
2 https://github.com/kazeto/phillip

In this work, st ∈ S, at ∈ A(st ) and rt ∈ R denote the internal
state of Que, the action taken by Que, and the reward for Que
at time-step t, respectively. We also define Φt as a support of
the argument at step t.
at corresponds to the question of Que to Ans. The action
space A(st ) at step t is a list of candidates of facts, and at
is expressed as an index for accessing an element in the list.
The size of the action space is equal to the number of possible supporting facts, e.g., four in the case of Figure 2. In thiswork, once Que selects at , Que can not take the same action
after step t (A(st+1 ) = (A(st )\{at })).
st is composed of an action history record, collected facts,
and the rationality of the argument at step t, and s0 represents the initial state of the system. “Action history record”
is the history of actions that Que has taken by step t, which isrepresented as a binary vector vh,t ∈ {0, 1}|A(s0 )| . It is necessary for avoiding selecting a question that has been alreadyasked. “Collected facts” {q1 , q2 , ..., qt 0 }(t 0 ≤ t) are facts that
Que has collected from Ans. We use bag-of-facts (BoF) torepresent collected facts. BoF is a representation similar tobag-of-words (BoW) that expresses collected facts with binary vector vqt . The size of the vector is equal to the numberof all possible facts. As, in the case of Figure 2, there are
four possible facts, q2 , q3 , q4 , and q5 , the size of the BoF
vector is four. The initial element of each dimension of thevector is set to zero, and once a fact is collected, the element corresponding to it is updated to one. “Rationality” atstep t is calculated by our rationality function R(Φt ). The
scalar value of the rationality is simply converted to one dimensional vector, vR,t = [R(Φt )]. Finally, we concatenate theproposed three features as one vector. Thus, the state st attime t is defined as:
st = [vh,t ⊕ vq,t ⊕ vR,t ].

(2)Note that ⊕ represents a concatenation of vectors.
A reward, rt , is given to encourage Que to collect facts inorder to construct more rational arguments as fast as possible. Formally, the reward is defined as:rtime + rgoal (ΘR ≤ R(Φt , α)
rt =
,
rtime
(otherwise)
where rtime and rgoal represent a time pressure penalty andgoal reward, respectively. The time pressure penalty is constantly fed as a time-lapse to value dialogues with shortsteps. The rewards are designed to encourage the effective
collection of facts for a rational argument. A goal reward is
fed if Que can construct an argument with higher rationality than a given threshold ΘR by the maximum number of
dialogue steps (Tlimit ).

Optimization with Double Deep Q-Network
We consider strategy π(a|s) that gives the probability of selecting an action a ∈ A(s) given a state s ∈ S. The objectiveof reinforcement learning is to find the optimal strategy π ∗that maximizes expected future rewards.
To find π ∗ , we use a double deep Q-network (DDQN).
DDQN (Van Hasselt, Guez, and Silver 2016) is one of the
deep RL methods based on Q-learning (Watkins and Dayan
1992). The algorithm introduces double Q-learning (Hasselt2010) into a deep Q-network (Mnih et al. 2015). DDQN updates the action-value function Q to find π ∗ . Q is a functionthat returns the expected future reward for a pair of st and at .
DDQN uses a neural network (Q-network) as Q. Given transitions hst , at , rt+1 , st+1 i sampled from the environment, theQ-network is updated to minimize the difference between atarget signal, YtDDQN , and its prediction. YtDDQN is given as:
YtDDQN = rt+1
+ γQ(st+1 , arg max Q(st+1 , a0 ; θt ); θt− ).

(3)

a0 ∈A

Here, both θt and θt− are parameters of neural networks att. θt is used for both selecting actions and calculating target
values, while θt− is used only for calculating target values. In
general, during the learning, hst , at , rt+1 , st+1 i are collectedby agents in accordance with ε-greedy exploration, in which
agents choose an action at from A(st ), randomly selectingwith probability ε or by following arg maxat ∈A(st ) Q(st , at )with probability 1 − ε. Once the update of Q converges tothe optimal Q∗ , π ∗ is given by arg maxat Q(st , at ).
When we apply DDQN, it is necessary to build a simulator that outputs the next state st+1 given the current state
st and the action at . We build an answerer simulator, which
returns the required fact if it is included in the knowledge
KA ; if otherwise, it answers with “I don’t know” and returns
nothing. KA is given at the beginning of the dialogue.

Figure 3: Fact graph (FG) made from 122 facts and 72 inference rules extracted from “Twelve Angry Men dataset”
This dataset records deliberative arguments (utterances)
between the 12 jurors of a homicide trial, and all of thearguments are paired into 80 pairs, and each pair is annotated with their relationship: support/attack. We formally extracted facts and inference rules from the dataset in accordance with the following steps.
1. Consider two utterances as two facts, utt1 and utt2
respectively.
2. Consider the relationship between utt1 and utt2 as aninference rule, utt1 → utt2.
As a result, we extracted 160 facts and 80 inference rules
by following the procedure above. In addition, to complicate the information-seeking task, we combined these extracted rules into more complicated ones. For example, we
combined three rules, utt1 → utt4, utt2 → utt4, andutt3 → utt4, into one complicated rule,utt1 ∧ utt2 ∧ utt3 → utt4.

Experiments
We conducted experimental evaluations to investigate the effectiveness of dialogue strategies optimized by RL in two
different dialogue domains: legal discussion and complianceviolation detection.
We prepared a large number of facts and inference rules
for KA and KQ during training and evaluation 3 . In this section, we first describe the prepared facts for KA s and inference rules for KQ s and then perform experiments with themand discuss the results.

Legal Discussion Domain
We prepared 550 KA s and 72 inference rules for KQ generated from an original argumentation dataset, “Twelve Angry
Men dataset (Cabrio and Villata 2014)4 .”3 Prepared facts and inference rules for K
A and KQ are
available at: https://github.com/HiKat/Data_for_
AAAI-DEEPDIAL19
4 http://www-sop.inria.fr/NoDE/NoDE-xml.
html#12AngryMen

Further, removing duplicate facts and inference rules, 122
facts (including a man claim ”The boy is not guilty”) and 72
inference rules remained.
All of the 72 inference rules were given to KQ at the beginning of each dialogue, and 121 action candidates corresponding to all facts except the main claim were defined for Que. For the training and evaluation, each of 550
KA s was constructed with 20 randomly chosen facts fromthe 121 facts except the main claim. 500 of the KA s wereused for training to find an optimal strategy, and the others were used for evaluation. At the beginning of eachdialogue, the answerer Ans was initialized with one randomly chosen KA , and the Que aimed to construct an argument that claims the fixed fact α “The boy is not guilty(The-boy-is-innocent)” by drawing facts repeatedly
from Ans. The rationality R(Φ) of hΦ, αi will increase
through the information-seeking dialogue if the system hasa good dialogue strategy.
To optimize and evaluate Que’s strategy, we prepared asimulator for Ans. The simulator returned one fact corresponding to Que’s inquiry if the inquired fact existed in KA .

Table 1: Upper three lines show results for three baseline
heuristic strategies, random, depth-first search (DFS), andbreadth-first search (BFS), strategies, and proposed strategy
based on DDQN in legal discussion domain. Lower threelines show results in compliance violation detection domain.
Each cell shows average result of five models generated inparallel and their one standard error.
Average Completed Average
Score
EpisodesSteps
Legal Discussion Domain-2.684
3.6
9.884
Random(1.88)(0.92)(0.04)
7.176
8.4
9.624
Depth-first search(8.46)(4.14)(0.18)
0.66
5.2
9.74
Breadth-first search(3.30)(1.61)(0.08)
45.456
26.6
7.744
DDQN
(5.74)(2.75)(0.26)
Compliance Violation Detection Domain
-10
0
-10
Random
(0.00)(0.00)(0.00)-100-10Depth-first search(0.00)(0.00)(0.00)-10
0-10
Breadth-first search(0.00)(0.00)(0.00)
65.304
35
4.696
DDQN
(2.37)(1.13)(0.12)

Compliance Violation Detection Domain
We also prepared 250 KA s, and 106 inference rules as KQ
from a compliance violation detection dataset. We generated this dataset by gathering thousands of Japanese mailthreads from role-played experiments between pairs of subjects. Each subject played a role as a worker for a certain
company or government, and he or she made plans together
with a partner in compliance violation matters, such as bidrigging or making a cartel. After the role-plays, we extracted
250 mail threads, each of which was composed of about
10 mails. We generated each KA from each mail thread byconverting each mail thread into 100 ∼ 600 facts by using Japanese semantic analysis tools (ChaPAS5 , KNP6 andzunda7 ) removing several facts that has nothing to do withcompliance violation. Finally, each mail thread was converted to 20∼30 facts, and 3782 kinds of facts remained in
all of the 250 threads in total.
The 106 inference rules were manually created to detect compliance violations from each mail thread. For
example, they included the inference rule “When x is
a competitor of y and x provides the price information to y, x and y form a cartel./competitor(x, y) ∧provide price information(x, y) → cartel(x, y).” All of therules were given to KQ at the beginning of each dialogue.
We used 100 to 250 KA s for training and the others for theevaluation.
5 https://sites.google.com/site/yotarow/

chapas6 http://nlp.ist.i.kyoto-u.ac.jp/?KNP
7 https://jmizuno.github.io/zunda/

Experimental Settings
We compared the optimized policy based on RL with threebaselines strategies based on heuristics in each of the twodomains. In this section, we give details on the compared
strategies and their settings as follows.
Random Strategy: A random selection strategy. The system randomly asks Ans about the existence of facts.
Depth-first Search (DFS) Strategy: A heuristic strategy
based on depth-first search (DFS). Que, with this strategy,
takes an action in depth-first order across a fact graph (FG).
In a FG, nodes are facts, and undirected edges are inference
rules. For example, if we have an inference rule, p1 ∧ p2 → q,
three nodes, p1 , p2 , and q, are connected with two undirected edges, (p1 , q) and (p2 , q). A part of the FG we used in
the experiment is shown in Figure 3. We consider the nodecorresponding to Que’s claim as a claim node, which is thestarting node of DFS. During DFS, Que randomly selects
which adjacent node of the current node to ask next. This
strategy is basically the same as the existing work proposedby (Fan and Toni 2012).
Breadth-first Search (BFS) Strategy: A heuristic strategy
based on breadth-first search (BFS). This strategy also uses
the FG as a DFS strategy; however, Que, with this strategy,
takes an action in breadth-first order across the graph. Note
that, Que, following this strategy, randomly selects one next
fact from nodes at the same depth.
DDQN: A strategy optimized with DDQN. In our experiments, the DDQN uses the Q-network, which takes the proposed feature vector (Eqn 2) as the input. The network has
two hidden nonlinear layers, each of which has 50 units followed by a hyperbolic tangent. The network finally outputs
action-value function values with one linear layer. The exploration rate ε for the ε-greedy strategy was linearly annealed from 0.1 to 0.01 during the first 2000 actions. The
discount rate γ was fixed to 0.95, and it was trained on 1000
episodes.
In this work, the rewards were set at rgoal = 100 andrtime = −1.0. During training, Tlimit was set to 10. ΘR was
fixed to 0.7 for the legal discussion domain and 0.5 for thecompliance violation detection domain.
We trained strategies based on DDQN with simulated dialogues, where a simulated Ans played Que’s partner. At the
beginning of each simulated training dialogue, the KA for
Ans was randomly selected from 500 or 200 training KA s in
response to each domain. In the evaluation, we also used thesimulator; however, the KA s for Ans were selected from different 50 KA s in the test set. These strategies interacted withthe test set simulator for the evaluation. Note that there wasno big difference from conducting a dialogue with humans
because the operation of Ans is simple in this setting, that is
only answering regarding the existence of the inquiry.

Results
Table 1 shows the scores of the strategies that we compared.
We evaluated strategies in terms of “test score,” “completed
dialogues,” and “average steps.” “Test score” shows the averages of cumulative rewards at the end of each test dialogue. “Completed dialogues” shows the number of evalu-

Table 2: We show example dialogue with strategy optimized with DDQN from evaluation. Each line of table shows each stepof interaction between system and answerer. First column shows step number of interaction. Second column shows speaker of
each step; “Sys” stands for system as questioner, and “Ans” stands for answerer. Others show each utterance of speaker andrationality of system’s argument after collection, respectively. Answerer’s bold-faced utterance corresponds to fact that system
successfully collects.
Step
1
2
3
4
5
6

Speaker
Sys
Ans
Sys
Ans
Sys
Ans
Sys
Ans
Sys
Ans
Sys
Ans

Utterance
Did not the woman across the street see the murder committed?
The woman across the street did not see the murder committed.
Did not the old man hear the kid yell, “I’m gonna kill you”?
The old man did not hear the kid yell, “I’m gonna kill you”.
Is the old man a liar?
I do not know.
Did not the boy crouch down and stab a taller man in the chest?
I do not know.
Did not the woman across the street see the boy stab his father?
I do not know.
Did not the boy admit buying the knife?
The boy did not admit buying the knife.

ated dialogues, in which Que successfully constructed a rational argument whose rationality exceeded the given thresholds within the 10-step limit (i.e., Tlimit ). “Average steps” describes the average of the number of steps required to finish
dialogues. The upper part of the table shows the results forthe legal discussion domain, and the lower one shows the results for the compliance violation detection domain. The results show that the proposed DDQN based strategy achieved
the best score in both domains. For further analysis, in legal
discussion domain, we investigated the relationship between
the upper time limit of each evaluated dialogue and the number of completed dialogues, as shown in Figure 4. This figure
indicates that the strategy optimized with DDQN collectedfacts that support a claim much more quickly than the otherhandcrafted strategies.

Rationality
0.1
0.4
0.4
0.4
0.4
0.7

DDQN strategy. Each question written in English are prepared by hand with one-to-one mapping from an answer(fact) to a question. As shown in the example, the questioner
uses the strategy based on DDQN to collect the fact “Theboy did not admit buying the knife” at the early stage ofthe dialogue. In the same setting as in Table 2, we observedthat both the BFS and DFS strategies collected these facts ina later stage, and, as a result, these strategies required three
extra steps to complete the dialogue. The aforementioned
facts were frequently contained as supports in a rational argument, i.e., an argument with higher rationality than a given
threshold, in these experiments. We found that the DDQN
strategy tried to collect such frequently contained facts at an
early stage in many dialogue examples.

Conclusion

Figure 4: Comparison of number of completed dialogues
where questioner constructed rational arguments with eachstrategy within time limit in legal discussion domain. Horizontal axis shows upper limit of questions for each questioner (Tlimit ), and vertical axis shows number of completed
dialogues. Note that the maximum number of dialogues was
50.
Table 2 shows a dialogue example generated by the

In this paper, we proposed formulating information-seeking
dialogue for constructing rational arguments with MDPs.
We also proposed dialogue strategy optimization based on
DDQN. Experimental results showed that strategies optimized with DDQN achieved the best scores in terms ofeffective information collection to construct rational arguments for the main claims of the system.
As future work, considering language understanding errors or the deceptive response of adversarial Ans’s are important. To deal with these problems, we can expand our
proposed MDPs formulation to a formulation based on partially observable Markov decision processes (POMDPs).
Furthermore, we will extend our formulation to enriched
information-seeking situations, where 1) Ans can provide arguments based on its own knowledge (inference rules andfacts) as well as facts, and 2) Que can challenge Ans to getfurther information about the provided arguments.

References
Alahmari, S.; Yuan, T.; and Kudenko, D. 2017. Reinforcement learning for abstract argumentation: A Q-learning ap-

proach. In Proceedings of AAMAS.
Amgoud, L.; Maudet, N.; and Parsons, S. 2000. Modelling
dialogues using argumentation. In Proceedings of the ICMAS, 31–38.
Besnard, P., and Hunter, A. 2014. Constructing argument
graphs with deductive arguments: a tutorial. Argument &Computation 5(1):5–30.
Besnard, P.; Garcia, A.; Hunter, A.; Modgil, S.; Prakken, H.;
Simari, G.; and Toni, F. 2014. Introduction to structured
argumentation. Argument & Computation 5(1):1–4.
Cabrio, E., and Villata, S. 2014. Node: A benchmark ofnatural language arguments. In Proceedings of the COMMA,
449–450.
Charniak, E., and Shimony, S. E. 1990. Probabilistic semantics for cost based abduction. Brown University, Departmentof Computer Science.
Dung, P. M.; Kowalski, R. A.; and Toni, F.
2009.
Assumption-Based Argumentation. Boston, MA: Springer
US. 199–218.
Fan, X., and Toni, F. 2012. Agent strategies for ABA-based
information-seeking and inquiry dialogues. In Proceedingof the ECAI, 324–329.
Fan, X., and Toni, F. 2015. Mechanism design forargumentation-based information-seeking and inquiry. In
Proceedings of PRIMA, 519–527. Springer.
Georgila, K., and Traum, D. 2011. Reinforcement learningof argumentation dialogue policies in negotiation. In Proceedings of the Interspeech.
Hadoux, E.; Beynier, A.; Maudet, N.; Weng, P.; and Hunter,
A. 2015. Optimization of probabilistic argumentation with
Markov decision models. In Proceedings of IJCAI, 2004–2010.
Hasselt, H. V. 2010. Double q-learning. In Lafferty, J. D.;
Williams, C. K. I.; Shawe-Taylor, J.; Zemel, R. S.; and Culotta, A., eds., Advances in Neural Information Processing
Systems 23. Curran Associates, Inc. 2613–2621.
Inoue, N., and Inui, K. 2011. ILP-based reasoning forweighted abduction. In Proceedings of PAIR, 25–32.
Levin, E.; Pieraccini, R.; and Eckert, W. 2000. A stochastic model of human-machine interaction for learning dialog
strategies. IEEE Transactions on speech and audio processing 8(1):11–23.
McBurney, P., and Parsons, S. 2001. Representing epistemic
uncertainty by means of dialectical argumentation. Annalsof Mathematics and Artificial Intelligence 32(1-4):125–169.
Misu, T.; Georgila, K.; Leuski, A.; and Traum, D. 2012. Reinforcement learning of question-answering dialogue policies for virtual museum guides. In Proceedings of the
SIGDIAL-12, 84–93. ACL.
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.;
Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Humanlevel control through deep reinforcement learning. Nature
518(7540):529.

Ovchinnikova, E.; Montazeri, N.; Alexandrov, T.; Hobbs,
J. R.; McCord, M. C.; and Mulkar-Mehta, R. 2014. Abductive reasoning with a large knowledge base for discourse
processing. In Proceeding of the IWCS. Springer. 107–127.
Papangelis, A., and Georgila, K. 2015. Reinforcement learning of multi-issue negotiation dialogue policies. In Proceedings of the SIGDIAL, 154–158.
Parsons, S.; Wooldridge, M.; and Amgoud, L. 2002. An
analysis of formal inter-agent dialogues. In Proceedings of
the AAMAS, 394–401. ACM.
Parsons, S.; Wooldridge, M.; and Amgoud, L. 2003. On theoutcomes of formal inter-agent dialogues. In Proceedings of
the AAMAS, 616–623. ACM.
Rosenfeld, A., and Kraus, S. 2016. Strategical argumentative agent for human persuasion. In Proceeding of the ECAI,
320–328.
Van Hasselt, H.; Guez, A.; and Silver, D. 2016. Deep reinforcement learning with double Q-learning. In Proceedingsof the AAAI, 2094–2100.
Walton, D., and Krabbe, E. C. 1995. Commitment in dialogue: Basic concepts of interpersonal reasoning. SUNY
press.
Watkins, C. J., and Dayan, P. 1992. Q-learning. Machinelearning 8(3-4):279–292.
Williams, J. D., and Young, S. 2007. Partially observable
Markov decision processes for spoken dialog systems. Computer Speech & Language 21(2):393–422.
Yoshino, K., and Kawahara, T. 2015. Conversational system
for information navigation based on pomdp with user focus
tracking. Computer Speech & Language 34(1):275–291.