Inventory Management with Attention-Based Meta
Actions
Keisuke Izumiya∗ and Edgar Simo-Serra∗∗ Department of Computer Science and Communications Engineering

Waseda University, Tokyo, Japan
Email: k-izumiya@ruri.waseda.jp, ess@waseda.jp

Abstract—Roguelike games are a challenging environmentfor Reinforcement Learning (RL) algorithms due to having torestart the game from the beginning when losing, randomized
procedural generation, and proper use of in-game items being
essential to success. While recent research has proposed roguelike
environments for RL algorithms and proposed models to handle
this challenging task, to the best of our knowledge, none havedealt with the elephant in the room, i.e., handling of items. Items
play a fundamental role in roguelikes and are acquired during
gameplay. However, being an unordered set with a non-ﬁxed
amount of elements which form part of the action space, it is notstraightforward to incorporate them into an RL framework. In
this work, we tackle the issue of having unordered sets be part of
the action space and propose an attention-based mechanism thatcan select and deal with item-based actions. We also propose amodel that can handle complex actions and items through a metaaction framework and evaluate them on the challenging game of
NetHack. Experimental results show that our approach is able
to signiﬁcantly outperform existing approaches.
Index Terms—reinforcement learning, attention, NetHack

I. I NTRODUCTION
Designing autonomous agents to play video games can play
an important role in game balancing, testing, and design.
Furthermore, video games play an important role in developing robust Reinforcement Learning (RL) algorithms that
can then be applied to other real-world situations. In thiswork, we tackle the challenging game of NetHack, a roguelike
game based on procedurally generated content, and develop
a general attention-based approach to handle unordered setsof actions, such as inventory management, with autonomous
agents.
RL, a branch of machine learning, is a method that has thegreat advantage of being able to learn without directly using
the label data. In particular, Deep Reinforcement Learning(DRL), which incorporates deep learning techniques, has been
actively researched in recent years and is often applied togames such as Atari 2600 [1], Go [2], and StarCraft II [3].
Among games, roguelikes are suitable as a challenging target
problem for DRL because they contain elements that current
DRL methods have not entirely overcome, such as a hugestate space, sparseness and delay of rewards, and the need forstrategies. According to [4], roguelikes are turn-based games,
and their grid-based environments are randomly procedurally
generated each time the game begins. The variety of enemies
and items makes the game complex enough that there are
multiple ways to complete the game. The required tasks

include resource management, combat with large numbers ofenemies, and exploration.
NetHack, which is the target problem of this research, is
one of the most popular open-source roguelikes, and it is still
being updated even though it is one of the earliest roguelikes.
The game’s objective is to search through over 50 levelsof procedurally generated dungeons using various items toﬁnd the Amulet of Yendor and bring it back. It is incrediblychallenging even for humans because there is a wide variety ofenemies, items, and actions the player can take. Furthermore,
the dungeons are not straight paths but rather branching paths
that must be traversed back and forth. Examples of NetHack
screens are shown in Figure 1. The player can use fourmain types of information: messages, dungeon, status, andinventory. Messages show events and conﬁrmation messages.
The dungeon shows the ﬂoor where the player @ is. Most
ﬂoors consist of square rooms and passages # (top ﬁgure), butsome ﬂoors do not (middle ﬁgure). An empty area indicatesthat nothing exists or the area has not been explored. The statusindicates the player’s attributes such as strength, experience,
intelligence, and hunger. The inventory shows the items theplayer possesses (bottom ﬁgure).
As described in Section II, there are several studies onusing DRL for roguelikes. However, most of these studies
focus on constructing the environment itself or on learningin non-general situations. It is important that, to the best of
our knowledge, there is no study that deals with items in
the general situation that exist in most role-playing games,
including roguelikes, and whose use is essential. Therefore,
we propose a mechanism that handles items appropriately andshow its effectiveness in experimental results1 .
Our main contributions are as follows:
We propose an attention-based mechanism for handlingan unordered set of items in a deep reinforcement learning
framework.
• We develop a new agent model that signiﬁcantly outperforms existing approaches in terms of in-game score on
the challenging game of NetHack.
• We perform an in-depth evaluation of the proposed approach and compare with existing approaches.
•

1 The source code is available at https://github.com/izumiya-keisuke/
inventory-management

B. Reinforcement Learning for Games
Passage

Most of the modern RL algorithms are evaluated on Atari
2600 games provided by ALE [9] or the environments provided by OpenAI Gym [10] as a benchmark. For more populargames, some studies tackled perfect information games such
as Go and Chess [2], [11]. There have also been studies on
multiplayer imperfect information games, such as StarCraft
II [3] and Dota 2 [12]. These methods combine the DRL
methods with MCTS [13]. There is also the research [14] thatused an environment based on Minecraft [15], [16], a gamethat uses items similar to NetHack. In this research we focus
on roguelike game of Nethack.

Item
Room
PlayerStairs

Enemy

Message

DungeonStatusFig. 1. Examples of NetHack gameplay screens. The top two images show
examples of the main in-game screen, while the bottom images shows anexample of the inventory screen. Different parts of the screens are annotatedfor ease of understanding.

C. Reinforcement Learning for Roguelikes
Some research has focused on tackling roguelikes with
RL. To facilitate RL’s application, highly customizable environments based on Rogue, the game that originated theroguelike genre and a simpler game than NetHack, havebeen created [17], [18]. With these environments, DRL techniques aimed at the exploration of the dungeon [17]–[21](the exploration of the current ﬂoor and descending to thedeeper ﬂoor) have been proposed. For NetHack, the subjectof this study, some recent research has created environments
based on NetHack and tackled various tasks. Campbell andVerbrugge [22] aimed at learning enemy combat with an
abstracted state and action space. They also focused on ﬁnding
hidden doors and passages, which is essential for dungeon
exploration, and proposed a method using occupancy maps toachieve the goal efﬁciently. Küttler et al. [23] tackled several
tasks such as scoring a game and collecting gold in a generalsituation by using random network distillation, which is one of
the exploration facilitation techniques. Most of these methods
do not take into account items, which are almost essential forcompleting the game, and this is one of the issues to be solved.
In this research, we propose a technique which is able perform
inventory management based on attention allowing for significant performance improvements over existing approaches.

II. R ELATED W ORK
D. Attention
A. Reinforcement Learning Algorithms
Mnih et al. [5] proposed a method that does not use experience replay but instead uses data collected by running multiple
agents in parallel, represented by asynchronous advantage
actor-critic (A3C). In addition to A3C, there are several othermethods belonging to the actor-critic method, such as ActorCritic with Experience Replay (ACER) [6], UNsupervised
REinforcement and Auxiliary Learning (UNREAL) [7], which
introduces auxiliary tasks in addition to experience replay, andimportance weighted actor-learner architecture (IMPALA) [8],which can be extended to thousands of machines by elaborating on the data communication method and off-policy correction. Our approach is based on IMPALA with modiﬁcations te
be able to do inventory management. The details of IMPALA
are shown in Section III-A.

Attention is a mechanism widely used in deep learningwhere the model can learn which parts of the data andfeatures to pay attention to has been actively studied in natural
language processing. In the past, attention had often used in
combination with an RNN, as in [24], but many methods
using attention alone have been proposed and showed better
performance than existing methods since the appearance ofTransformer models [25], which used self-attention without
RNN. Now models that use self-attention are commonly
employed in the ﬁeld of image recognition, where Convolution
Neural Network (CNN) have been commonly used [26]. In theﬁeld of RL, Berner et al. [12] used an attention mechanism tochoose the target unit, and Zambaldi et al. [27] used selfattention to describe the relationship between entities. We
focus on the position-independent property of attention andpropose a mechanism to handle inventory using this property.

III. M ETHOD
A. Reinforcement Learning Background
RL is often modeled by Markov Decision Processes (MDP).
The state space and the action space are denoted as S and A,
respectively. The state at the discrete time t = 0, 1, 2, . . . isdenoted as St ∈ S. The action At ∈ A the agent takes attime t brings a reward Rt and a new state St+1 . The goal of
RL is to learn the policy π(a | s) = Pr{AP tt = a | St = s}that maximizes the expected return Eπt γ Rt , where γ isa discount factor.
We base our work on IMPALA [8], which is an asynchronous off-policy actor-critic method. As in IMPALA, we
prepare multiple independent agents, actors, and environments
and update the parameters of the learning agent, learner, using
the data collected by these agents. We use an off-policy
correction method called V-trace because each actor has its
own local policy µ, which lags behind the learner’s policy π.
At the state s, the agent outputs the state value V (s), which
estimates the return from the state s, and a policy π(a | s).
The gradient of the loss function for the policy parameter isρt ∇ log π(At | St ) Rt + γvt+1 − V (St ){z}|policy gradient term

+β∇
|

where

X

a∈A

−π(a | St ) log π(a | St ),{z}

entropy term

t+n−1
X

s−t

s−1
Y

(1)

!

δs ,

(2)


δt = ρt Rt + γV (St+1 ) − V (St ) ,π(At | St )ρt = min ρ,,µ(At | St )π(At | St )ct = min c,.
µ(At | St )(3)

vt = V (St ) +

s=t

γ

i=t

ci

(4)
(5)

β, n, ρ, and c are hyperparameters. The ﬁrst term in (1) is apolicy gradient, and the second one is entropy. The gradient
of the loss function for the state value parameter is vt −V (St ) ∇V (St ).
B. Baseline Model
As a baseline, we use a slightly modiﬁed version of themodel used in [28], which is an unsurpassed model to ourbest knowledge. The diagram of the model is shown above
the dashed line in Figure 2. The model can handle three ofthe four types of main information as described in Section I,
all excluding the inventory. Since a message is a string of
256 characters in length, each character is embedded with its
ASCII code, and then the features of the message are extracted
by a 1D CNN. Although this is an outdated method in the ﬁeldof natural language processing, we use it because it is one of
the simplest models and message processing is not the focus
of our experiments. Status is a vector consisting of numericalvalues such as HP and strength, and the feature is extracted
by a multilayer perceptron (MLP). The dungeon’s shape is
21 × 79, and each grid is a vector embedded with attributes
such as its color and its kind. We extract the feature of thedungeon with a 2D CNN. In addition, the feature of 9 × 9grids centered on the agent is extracted in the same way as
the entire dungeon. This has been validated by [29] and [30]to help train the agent. These features are then concatenatedand input to the MLP and Gated Recurrent Unit (GRU) in thatorder. h and h′ represent the hidden state at the previous timeand at the current time, respectively. Finally, the GRU output,
which can be regarded as the feature of the current state, isinput to two MLPs to obtain the state value and the policy.
C. Action Recursion
We add the embedding layer to the model to incorporate
knowledge of the previous action.In this layer, the agent’s
previous action is embedded and concatenated with the main
information’s features. It is intuitively that a player may
plan a sequence of actions. In particular, in NetHack, someactions are often repeated multiple times, and others consistof multiple actions. An example of the former is the searchfor hidden passages or doors. If there are hidden passages ordoors in grids adjacent to the agent, it can ﬁnd them witha certain probability by using the search action. Therefore,
a common strategy is to repeat the search action a certain
number of times to ﬁnd them. An example of the latter iskicking and actions which use items. Kicking is composedof two actions: the kick action itself and a kicking direction,
while the item-using action is composed of two actions: thecontext of the action such as quaff and read and the item tobe used. Therefore, we need to input the actions that the agent
has taken before into the model. Although it is theoreticallypossible for the model to learn to propagate this information
through the hidden state of the GRU layer, we found that inpractice, this effect did not occur and that it was beneﬁcial todirectly use the previous action as an input.
It is possible to treat all pairs of actions and their targets as
separate actions to handle actions that require multiple inputs,
but we do not do so because this increases the size of the action
space and makes learning difﬁcult. In NetHack, for example,
there are more than ten actions using items and more thanﬁfty types of items that can be held, so treating each potentialcommand pair as an action makes the size of the action spacebeyond 500. Also, there is room for the number of actions
input to the model, but we decide to use only the previous
action because it should possible to consider any number ofactions thanks to GRU. Note that we embed the action withthe action space A′ , which is deﬁned in Section III-F.
D. Meta Actions
In NetHack, specifying an item is limited to a few situations,such as selecting a potion to quaff immediately after selecting
a quaff action and selecting an armor to wear immediately
after selecting a wear action. In these situations, it is ofprimary importance to specify an item. Therefore, we add theDungeonDungeon(crop)
StatusMessage

Action

Emb
Emb

CNN

h

MLP

CNN
Concat

MLP

GRU

MLP
Emb

Emb

Policyπ(a | s)

MLP

h'

CNN

State value
V(s)

‧
‧
‧

Item N

Emb

MLP

MLP

MLP

CNN

CNN

GRU

GRU

Sum

Summation layer

Attention Attention layer

y1

MLP

‧
‧
‧

Attention
Emb

Embedding layer

Concat Concatenating layer

Sum
Item 1

Emb

yN

MLP

Fig. 2. Overview of our agent model architecture. The baseline model is shown above the dashed line. The whole model represents our proposed model
where we use attention-based inventory management framework. We note that there are still some other differences in low-level between our full model andthe baseline, such as the usage of meta actions.

meta action use item to the action space. When the agent
chooses this action, P
it speciﬁes the item according to theprobability exp(yi )/ i exp(yi ) where yi is the ith item’sscore. The method of calculating the score is described in
Section III-E.
The introduction of the meta action also has the advantagethat items can be handled independently from normal actions.
It is not easy for policy to incorporate items directly because
the number of items is not constant. Using meta actions
makes the size of the action space ﬁxed and makes theimplementation easier. Furthermore, we can handle the normal
action space and the action space of inventory separately. We
propose the computation of the entropy term in the loss function described in Section III-F as one example of separating.
Besides, for the same reason as described in Section III-C, it
can prevent the action space from becoming too large.
E. Attention-Based Inventory Feature Extractor
It is not easy to handle the inventory appropriately withneural networks because it is an unordered set of items. To
express the unordered nature, we propose the attention-based
mechanism to extract the feature of the inventory. First, each
item in the inventory is transformed into a vector xi using theembedding and the MLP. This embedding is done in a similar
way to the embedding of each gridPin the dungeon. Then, thesum of vectors of all items, x = i xi , is input to another
MLP, and the output is deﬁned as the feature of the inventory.
Note that the summation does not depend on the order ofthe items. The inventory feature is concatenated with othermain information features and the embedded previous action,and the feature representation is processed in the same way

as the baseline model. Then, to specify the items to be used,each item’s score is calculated using the attention operation.
Speciﬁcally, we denote the feature of the current state as f
and prepare the matrices WQ and WK and the vector wV ,and calculate each item’s score yi by the following equation:q ⊤ kiyi = √ v i ,
dk

(6)

⊤
where q = WQ f , ki = WK xi , vi = wV
xi , and dk is thedimension of q and ki .
The feature vector is then used with the item meta actions
to determine what item to use while being invariant to theorder of the items.

F. Loss Function
The use of the meta action and the attention-based item
selection policy necessitates modifying the loss function. First,
we deﬁne modiﬁed action spaces and policies. We denote theuse item action as b0 and the action using the ith item as bi .
The virtual action space is deﬁned as Av = A ∪ {b0 }, theaction space for the inventory is deﬁned as Ai = {b1 , b2 , . . .},and the actual action space is deﬁned as A′ = A ∪ Ai .
Similarly, we deﬁne the virtual policy as πv : S × Av → [0, 1],
the policy for the inventory as πi : S × Ai → [0, 1], and theactual policy as π ′ : S × A′ → [0, 1]. Note that these policies
satisfy the following relations for all s ∈ S:(
πv (a | s)for a ∈ A,
π ′ (a | s) =(7)πv (b0 | s)πi (a | s) for a ∈ Ai ,XXX
π ′ (a | s) = 1. (8)πi (a | s) =πv (a | s) =
a∈Av

a∈Ai

a∈A′

The RL algorithm used in this study is IMPALA [8]. Amongthe gradient of the loss function for the policy parameter, as
can be seen in (1), two modiﬁcations are applied: all π are
replaced with π ′ in the policy gradient term, and the entropy
term is replaced with
∇

X

a∈Av

TABLE IAVERAGE SCORE OVER TEN EPISODES OF TESTING . W E SHOW RESULTSFOR DIFFERENT APPROACHES AND THREE DIFFERENT CHARACTERSCENARIOS . B EST RESULTS ARE SHOWN IN BOLD .

[28]
Baseline
Ours w/o Inventory
Ours w/o Action Recursion
Ours−πv (a | St ) log πv (a | St )+ λπv (b0 | St )∇

X

b∈Ai

−πi (b | St ) log πi (b | St ),(9)where λ is a hyperparameter. When λ = 1, it means that
all actions belonging to A′ are treatedequally in the entropyPcalculation, and (9) is equal to ∇ a∈A′ −π ′ (a | St ) log π ′ (a |
St ).

2000

Valkyrie

Tourist807.1
1431.2
1348.41500.12345.0

645.4
686.2
840.9
890.5
906.7

42.2
56.8
191.1
153.0
283.7

Monk

[28]
Baseline
w/o Inventory

1000

w/o Recursion
00

IV. E XPERIMENTS AND R ESULTS

0.5

Ours

1 (x109)

Valkyrie

A. Experiment Settings
In this study, we used the in-game score as a reward,
which shown in the status bar at the bottom as seen in
Figure 1. In NetHack, the in-game score can be earned byvarious events, such as descending to a new level, defeating
enemies, or getting gold. We used this setting because it
contains many elements necessary to complete the game andconsiders combat with enemies, which is often done with theuse of items, for which our model is explicitly designed tohandle. Furthermore, previous research described in Section II
restrict the actions that agents can take to movement only or
movement and a few other actions. In this study, however,
we used an action space covering most of the actions directly
related to the game.
We compared against the approach of Küttler et al. [28],which is based on IMPALA [8] and TorchBeast [31] and uses
a limited amount of actions. In particular, only movementin 8 directions, climbing up/down, reading messages, eating,searching, and kicking are allowed for a total 14 different
actions. Our baseline extended the amount of actions byconsidering an additional 11 actions (apply, drop, pickup,
puton, quaff, read, takeoff, throw, wear, wield, and zap) in
addition to meta actions. Our proposed approach extended
the baseline with action recursion, meta action, and attentionbased inventory feature extraction. We trained agents for all
approaches for one billion steps. In addition, we conducted
the tests over ten episodes with the same seed after training.
Most evaluation on NetHack was done using the characteras mon-hum-neu-mal (indicating that the role is monk, therace is human, the alignment is neutral, and the gender ismale). The game content in NetHack varies greatly depending on the characters, especially on the roles. Therefore, in
addition to mon-hum-neu-mal, two other characters withdifferent characteristics were used in the experiment. The
additional characters were val-dwa-law-fem (valkyrie,dwarf, lawful, and female) and tou-hum-neu-fem (tourist,
human, neutral, and female), and the characteristics of the usedcharacters are as follows:

Monk

Tourist

1000

200

500

100

0
0

0.5

1 (x109)

00

0.5

1 (x109)

Fig. 3. Average return during training. The horizontal axis shows thetraining step, and the vertical one shows the average return of all agents.

monk: strong at the beginning of the game, with various
items and high combat performance, but more challenging to conquer from the middle of the game.
• valkyrie: has very few items at the beginning of thegame, but the character has high combat performance.
• tourist: has various items but is very weak at the beginning of the game, so tactics speciﬁc to this character isrequired.
We evaluated all approaches independently in each of the threecharacter settings.
•

B. Comparison against Existing Approaches
We compared against existing approaches and testing results
are summarized in Table I and the training evolution is shownin Figure 3. We can see that the proposed approach is able tosigniﬁcantly outperform existing approaches. In particular, we
can see the largest relative increase in the most challenging
class Tourist, which does not start out with any combat skills
and must rely on inventory usage to stay alive.
C. Ablation Study
We compared four models’ performance: the baselinemodel, the model without action recursion, the model without
the attention-based mechanism for the inventory, and the full
proposed model. Figure 3 shows the average return during
training, and Table I shows the average return in the test. We
can see that all the different components of our approach play
an important role in obtaining good results.

TABLE II
AVERAGE NUMBER OF TIMES THE AGENT TOOK THE ACTION OF USINGITEMS WITHIN AN EPISODE . R ESULTS ARE SHOWN FOR THE POLICYLEARNED WHEN PLAYING WITH THE CLASS OF Monk. ACTIONS THATHAVE NOT BEEN TAKEN EVEN ONCE ARE OMITTED .

Baseline
Ours w/o Inventory
Ours w/o Action Recursion
Ours

drop

eat

quaff

read

0.0
0.0
0.0
0.6

4.0
6.8
4.0
6.2

1.0
2.6
2.7
2.7

0.2
0.0
0.0
1.7

use item
0.956

0.000
0.000
0.000
0.020
0.000
0.000
0.000
0.378
0.179
0.040
0.005
0.000
0.378
0.000
0.000
0.000
0.000
0.000
0.000

West
0.028
NorthEast
0.003

D. Effect of Difﬁculty
We can see that the difﬁculty of the game plays a fundamental role in the total score obtained. As Monk does not rely on
items and obtains strong intrinsic abilities when leveling up,
it shows the highest performance overall. Valkyrie and Tourist
rely heavily on items to stay alive and show lower performanceoverall.
E. Analysis of the Policy
We show characteristic examples of the learned policy of
our model in Figure 4. The ﬁgures with black backgroundshow the state of the game, and the probabilities on the rightside show the actions the agent will take in this state and theirprobabilities. In the top three ﬁgures, the three actions with thehighest probability in the action space Av are shown. In thebottom ﬁgure, the transitions of probabilities of the two actions
with the highest probability, search and North, are shown until
the agent takes search action ﬁve times in a row and ﬁnds thehidden door. In addition, the inventory screens are shown inthe blue boxes and the probability of using each item, or πi , isshown to the left of each item in the top two examples because
the action with the highest probability is use item.
The ﬁrst example shows the screen immediately after theagent takes the eat action, and the message at the top of thescreen asks the agent which item to eat. In the second example,
the agent is in the state immediately after taking the quaff
action and is asked the item to quaff. The third example shows
the state immediately after taking the kick action, and the agent
is asked in which direction to kick. The fourth example shows
the screen immediately after the agent moves to the left by theWest action.
We can see that in many cases, the agent is able to learn touse items depending on the game state to stay alive. We cansee that the attention is able to understand the different usageof the types of items and is invariant to the order of the items.

use item
0.986

Player
Enemy

Enemy

eat
0.007

0.000
0.000
0.000
0.000
0.000
0.981
0.009
0.000
0.000
0.000
0.009
0.000

quaff
0.002

HP

South
0.983

Player

kick
0.011
East
0.004

Door

search North
0.957 0.030
0.717 0.214
0.717 0.219
0.679 0.250

Player

0.616 0.304

V. D ISCUSSION AND C ONCLUSION
We have presented an attention-based approach that isable to handle unordered sets, such as item inventories, in
reinforcement learning models. We evaluate on the challenging
benchmark that is the roguelike game NetHack and haveshown that our proposed inventory management framework,in combination with our handling of meta actions and action
recursion, our approach is able to signiﬁcantly outperform
existing approaches.

Fig. 4. Examples of the policy output of the proposed model. The rightside shows the probabilities of taking actions that belong to Av with highprobabilities. In the top two ﬁgures, the blue boxes represent the inventory
screens, and the probabilities to the left of the box represent πi . In the bottom
ﬁgure, the transitions of the probabilities are shown.

TABLE III
T HE ITEM SPECIFIED BY THE ACTION THAT USES ITEMS . U P TO THREEITEMS ARE SHOWN IN ORDER OF THE NUMBER OF TIMES THEY HAVE BEEN
USED . T HE NUMBER IN PARENTHESES INDICATES THE AVERAGE NUMBER
OF TIMES THE ITEM WERE USED . R ESULTS ARE FOR THE CLASS OF Monk.

ActionUsed Items

drop
eat
quaff
read

unlabeled scroll (0.5), unknown potion (0.1)
food ration (3.0), apple (1.5), orange (1.0)
potion of healing (2.7)
spellbook (1.3), unknown scroll (0.2), unlabeled scroll (0.2)

As shown in Figure 3 and Table I, the proposed model
outperforms the existing model. The ﬁrst example in Figure 4
shows that the agent was able to use items appropriately
because it selected edible items as targets to eat with a very
high probability at the right time. The bottom two examples
show the effect of action recursion. In the second example, theagent was ﬁghting two enemies and had very little HP left.
Therefore, the possible effective tactics are to ﬂee or quaff
a potion of healing to recover HP, and this example showsthat the agent chose the latter. In the third example, the agent
was able to take kick action appropriately that requires twoconsecutive actions: kick and the direction to kick. The door
+ on the south of the agent @ can be opened only by kicking
at the stage because the door is locked and the agent does
not have a key, so kicking is one of the appropriate actions
to take here. In the last example, some rooms likely exist on
the dungeon’s left side because it is unexplored there, but it is
difﬁcult to imagine a path to the left side based on the currently
observable dungeon. Therefore, it is natural to assume that
there is a hidden door somewhere in the leftmost part of the
explored area where the agent @ is now, and it is appropriate
to search for it. In fact, the agent found the hidden door on theleft side after ﬁve search actions. Also, since the search action
searches around the agent, the agent cannot ﬁnd the door if
it exists in the room’s upper left area. Therefore, one of thenatural strategies is to search upward again if the door is notfound after a certain amount of search actions. The probability
transitions of the search and North actions indicate that thisstrategy is being taken.
The ablation study shown in Table II indicates that it isnecessary to incorporate both of mechanisms to use items
appropriately. The most frequently performed actions were eat
and quaff ; both are directly related to increasing the agent’s
survival time. The character we used, Monk, always has somefood and three potions of healing at the beginning of thegame. In NetHack, a hunger level progresses with actions, andhigh hunger level makes the agent’s combat power weak andeventually causes death by starvation. Therefore, eating foodis essential for the agent to survive for a long time and to beable to get a higher reward. A potion of healing also helps
the agent’s survival since quafﬁng it restores HP. Besides, theproposed model also took read actions. Monk starts the gamewith a random scroll and is trained to read it if it is a scroll of
enchant armor. This is because reading it enhance the agent’s

AC, i.e. defense and evasion, which is advantageous in combat,
where most agent deaths occur. The list of items used is given
in Table III.
On the other hand, other actions of using items were notlearned. Many of them have a temporary disadvantage whentaken or use items that the Monk does not have at the beginningof the game, i.e., items that need to be picked up to use. In
addition, the identity of items that the agent does not have atthe beginning of the game is unknown. It can only be revealed
by limited actions such as using the item or using a speciﬁcitem. However, some items have disadvantageous effects, andthe Monk does not have identifying items initially, so learningthese actions is difﬁcult.
The model without the action recursion or inventory handling mechanism successfully took actions of using items acertain number of times. As mentioned earlier, an action touse an item usually consists of multiple inputs, so it seems
impossible for a model that does not use action recursion tolearn this kind of action. However, a message often indicateswhat input is required for the second action, as shown in
Figure 4. Therefore, although it is not easy to learn messages,
which are natural language, it is possible to learn actions that
require multiple inputs without action recursion. However, weobserved that the model often failed to take appropriate theseactions in the test. Also, the use of items is learned even though
the model does not use the inventory feature. In NetHack,
items are speciﬁed by the alphabetic characters shown on theleft of the inventory screen in the bottom ﬁgure in Figure 1.
Therefore, it is possible to learn using items that the agent has
at the beginning of the game without the inventory feature if
their alphabets are used to specify standard actions.
Figure 3 and Table I show that the proposed method’s
effectiveness is large for the Monk and the Tourist and small
for Valkyrie. The Monk and Tourist have many items at the
beginning of the game, but the Valkyrie has few items. As
mentioned earlier, it is not easy to learn to use items thatit does not possess at the beginning of the game, which is
the reason why the proposed method is not very effective for
Valkyrie.
Although our model allows for inventory usage, the agent
still has issues with long-term planning and is unable to learn
to equip new equipment. This is also partially due to the factthat in NetHack, equipment is not always a strict upgrade,it can be cursed and give large penalties in many situations.
Detecting if equipment is cursed or uncursing it is a hardendeavour which discourages the agent from learning suchcomplex behaviours. We hope that with further developments
in reinforcement learning that it may be possible to overcome
such issues.
In conclusion, we have shown that our proposed model
is able to properly use items, which are essential elements
in role-playing games, including roguelikes. As a result, thein-game score increased signiﬁcantly. On the other hand, theproposed model could not sufﬁciently learn the use of items
that were not possessed at the beginning of the game because
their identities were unknown at ﬁrst. Also, our proposedmodel did not consider spells, which is a common element
in many of role-playing games. Spells often have the samefeature: players can cast multiple spells, and casting themconsumes a shared resource. The proper handling of theseremains as future work.
R EFERENCES[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” in NIPS Deep Learning Workshop, 2013.
[2] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van
Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam,
M. Lanctot et al., “Mastering the game of go with deep neural networks
and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016.
[3] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik,
J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev et al., “Grandmaster level in starcraft ii using multi-agent reinforcement learning,”
Nature, vol. 575, no. 7782, pp. 350–354, 2019.
[4] International Roguelike Development Conference, “Berlin interpretation,” 2008, accessed: March 14th, 2021. [Online]. Available:
http://www.roguebasin.com/index.php?title=Berlin_Interpretation[5] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley,
D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep
reinforcement learning,” in Proceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning
Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New
York, New York, USA: PMLR, Jun 2016, pp. 1928–1937.
[6] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, andN. de Freitas, “Sample efﬁcient actor-critic with experience replay,” inInternational Conference on Learning Representations, 2017.
[7] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo,
D. Silver, and K. Kavukcuoglu, “Reinforcement learning with unsupervised auxiliary tasks,” in International Conference on Learning
Representations, 2017.
[8] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward,
B. Yotam, F. Vlad, H. Tim, I. Dunning, S. Legg, and K. Kavukcuoglu,“IMPALA: Scalable Distributed Deep-RL with Importance Weighted
Actor-Learner Architectures,” 35th International Conference on Machine Learning, ICML 2018, vol. 4, pp. 2263–2284, 2018.
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The Arcade
Learning Environment: An evaluation platform for general agents,”
Journal of Artiﬁcial Intelligence Research, 2013.
[10] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint
arXiv:1606.01540, 2016.
[11] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre,
S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel et al.,“Mastering atari, go, chess and shogi by planning with a learned model,”Nature, vol. 588, no. 7839, pp. 604–609, 2020.
[12] C. Berner, G. Brockman, B. Chan, V. Cheung, P. D˛ebiak, C. Dennison,
D. Farhi, Q. Fischer, S. Hashme, C. Hesse et al., “Dota 2 with large
scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680,
2019.
[13] R. Coulom, “Efﬁcient selectivity and backup operators in monte-carlo
tree search,” in International conference on computers and games.
Springer, 2006, pp. 72–83.
[14] S. Frazier and M. Riedl, “Improving deep reinforcement learning in
minecraft with action advice,” in Proceedings of the AAAI Conferenceon Artiﬁcial Intelligence and Interactive Digital Entertainment, vol. 15,
no. 1, 2019, pp. 146–152.
[15] W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani,
S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin et al., “TheMineRL competition on sample efﬁcient reinforcement learning using
human priors,” NeurIPS Competition Track, 2019.
[16] W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno,
C. Loomis, S. Milani, S. Mohanty, K. Nakata, R. Salakhutdinov,
J. Schulman, S. Shiroshita, N. Topin, A. Ummadisingu, and O. Vinyals,“Neurips 2020 competition: The MineRL competition on sample efﬁcient reinforcement learning using human priors,” NeurIPS Competition
Track, 2020.
[17] A. Asperti, C. De Pieri, and G. Pedrini, “Rogueinabox: an environment
for roguelike learning,” International Journal of Computers, vol. 2, 2017.

[18] Y. Kanagawa and T. Kaneko, “Rogue-gym: A new challenge for generalization in reinforcement learning,” in 2019 IEEE Conference on Games(CoG), 2019, pp. 1–8.
[19] A. Asperti, C. De Pieri, M. Maldini, G. Pedrini, and F. Sovrano, “A
modular deep-learning environment for rogue,” WSEAS Trans. Syst.
Control, vol. 12, pp. 362–373, 2017.
[20] A. Asperti, D. Cortesi, and F. Sovrano, “Crawling in rogue’s dungeons
with (partitioned) a3c,” in Machine Learning, Optimization, and Data
Science, G. Nicosia, P. Pardalos, G. Giuffrida, R. Umeton, and V. Sciacca, Eds. Cham: Springer International Publishing, 2019, pp. 264–275.
[21] A. Asperti, D. Cortesi, C. De Pieri, G. Pedrini, and F. Sovrano,“Crawling in rogue’s dungeons with deep reinforcement techniques,”
IEEE Transactions on Games, vol. 12, no. 2, pp. 177–186, 2020.
[22] J. Campbell and C. Verbrugge, “Learning combat in NetHack,” in Thirteenth Annual AAAI Conference on Artiﬁcial Intelligence and Interactive
Digital Entertainment (AIIDE 2017), October 2017, pp. 16–22.
[23] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration byrandom network distillation,” in International Conference on Learning
Representations, 2019.
[24] Y. Kim, C. Denton, L. Hoang, and A. M. Rush, “Structured attention
networks,” in International Conference on Learning Representations,
2017.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advancesin Neural Information Processing Systems, I. Guyon, U. V. Luxburg,
S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett,
Eds., vol. 30. Curran Associates, Inc., 2017.
[26] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image
recognition,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), June 2020.
[27] V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin,
K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, M. Shanahan,
V. Langston, R. Pascanu, M. Botvinick, O. Vinyals, and P. Battaglia,“Deep reinforcement learning with relational inductive biases,” in International Conference on Learning Representations, 2019.
[28] H. Küttler, N. Nardelli, A. H. Miller, R. Raileanu, M. Selvatici,
E. Grefenstette, and T. Rocktäschel, “The NetHack Learning Environment,” in Proceedings of the Conference on Neural Information
Processing Systems (NeurIPS), 2020.
[29] F. Hill, A. Lampinen, R. Schneider, S. Clark, M. Botvinick, J. L.
McClelland, and A. Santoro, “Environmental drivers of systematicity
and generalization in a situated agent,” in International Conference on
Learning Representations, 2020.
[30] C. Ye, A. Khalifa, P. Bontrager, and J. Togelius, “Rotation, translation,and cropping for zero-shot generalization,” in 2020 IEEE Conference onGames (CoG). IEEE, 2020, pp. 57–64.
[31] H. Küttler, N. Nardelli, T. Lavril, M. Selvatici, V. Sivakumar, T. Rocktäschel, and E. Grefenstette, “TorchBeast: A PyTorch Platform forDistributed RL,” arXiv preprint arXiv:1910.03552, 2019.