Mobile Robot Navigation Using Learning-Based Method Based on
Predictive State Representation in a Dynamic Environment
Kohei Matsumoto1 , Akihiro Kawamura2 , Qi An2 , and Ryo Kurazume2
Abstract— Mobile robot navigation in a dynamic environment
with pedestrians is essential for service robots operating in aliving environment. Accordingly, the robot needs to understandand predict the behavior of pedestrians. However, predicting
pedestrian behavior in advance is difficult because humanbehavior may be affected by factors that cannot be directly
observed or modeled in advance, such as intentions andenvironmental influences. In addition, pedestrian behavior may
be affected by the behavior of the robot.
In this study, we apply a deep reinforcement learning method
based on a novel predictive state representation (PSR) model
to mobile robot navigation for realizing a navigation method
considering the changes in pedestrian behavior caused by robot
actions and other pedestrians. In addition, we propose two
methods for integrating the states of the PSRs corresponding toeach pedestrian and evaluate these methods in situations where
the number of pedestrians differs between learning and testing.

I. I NTRODUCTION
Mobile robot navigation in dynamic environments is an
important element for service robots that provide services
in spaces inhabited by humans. Handling this task requires
the robots to understand the behavior of pedestrians. However, pedestrian behavior may change depending on theirintentions and other factors that depend on particular environments. Thus, it cannot be easily modeled in advancewith sufficient accuracy, particularly if the road shape andattractions in the environment (e.g., stores or scenery) areconsidered. Furthermore, the behavior of the robot may affectthe pedestrian behavior.
To address these issues, we propose a learning-based
method based on predictive state representation (PSR) thatconsiders the changes in pedestrian behavior caused by robot
actions. This method can learn the behavior of pedestrians
by observing their interactions with the environment andthe changes in their behavior due to the robot’s actions.
Mobile robot navigation methods based on deep reinforcement learning (RL) have been actively studied in recent
years [1]–[5]. However, the methods proposed so far mainly
use predesigned models or do not specifically consider thechanges in pedestrian behavior caused by robot actions.
In this study, we apply a deep RL method based on the
new PSR model to mobile robot navigation for considering
the changes in pedestrian behavior caused by robot actions
1 Kohei Matsumoto is with the Graduate School of Information Scienceand Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japanmatsumoto@irvs.ait.kyushu-u.ac.jp
2 Akihiro Kawamura, Qi An, and Ryo Kurazume are with the Faculty of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan {kawamura, anqi, kurazume}@ait.kyushu-u.ac.jp

Current step

Next stepbservation
obs

Fig. 1: PSR modeling for mobile robot navigation in a dynamic environment
with pedestrians

and other pedestrians. In addition, we consider the situation
where the number of pedestrians changes between training
and testing. The main contributions of this study are as
follows:• the application of a deep RL method using PSR for
mobile robot navigation and the verification of theeffects of the structure based on PSR for mobile robot
navigation tasks;• proposing a new PSR architecture for mobile robot
navigation in a dynamic environment with pedestrians
considering the interactions among pedestrians and theverification of the effectiveness of the architecture;• proposing methods for integrating the states of the
PSRs corresponding to each pedestrian for handling thechange in the number of pedestrians between training
and testing, and comparing these integrating methods.
II. R ELATED WORKS
Various methods based on deep RL techniques have been
proposed. Everett et al. [1] proposed a method based onpolicy-based RL and long short-term memory. Chen et al. [2]proposed a pooling module for learning the relationships between the attention mechanism, changes in human behavior,and surrounding pedestrians; they also proposed a method
to learn robot behavior, including human–robot and human–
human interactions. Chen et al. [3] proposed an excellent
navigation method that uses a graph convolutional network
and attention mechanism based on human gaze. Chen etal. [4] used model-based deep RL methods and a graphconvolutional network to achieve efficient navigation byencoding the interactions among agents. Liu [5] proposed a
deep RL method that focuses on robot navigation in crowded
real-world environments.
PSR [6] is used for modeling dynamics by considering
partial observability. Various methods for utilizing PSR havebeen proposed. Boots et al. [7] proposed an extended PSR
model based on the Hilbert space embeddings of distributions to manage infinite sets of continuous observations andactions. Hefny et al. [8] proposed a supervised learning
method based on the kernel version of instrumental variable
regression for PSR models. They also proposed an efficient PSR model for learning controlled dynamical systems
using random Fourier features with a kernel function [9].
In addition, methods utilizing PSR for deep learning havebeen proposed recently. Venkatraman et al. [10] studied theuse of PSR for representing the latent states of recurrent
neural networks (RNNs). They observed that the proposed
method could improve the performance and compensate forthe disadvantages of RNN in probabilistic filtering, imitation
learning, and RL. Hefny et al. [11] proposed a fundamental
deep RL method, called the recurrent predictive state policy
network, to utilize PSR.
In robotics, PSR has been applied in human–robot communication by utilizing its ability to model the results after
performing actions [12]. It has also been applied in inhand manipulation, which includes interactions with the environment [13]. This method extends PSR to address partial
observability using a new kernel-based feature that integrates
actions and observations. However, PSR has not yet been
applied to mobile robot navigation in dynamic environments.
III. P RELIMINARIES
A. Predictive state representation
In PSR [6], the dynamics are modeled considering theinteractions between an agent and the environment. The
characteristic of PSR is that the state is expressed in a
way that facilitates the prediction of the future. The state isrepresented by observations and actions. Therefore, values
other than the fully observable elements do not need to be
defined. In this section, we describe PSR in detail.
1) Original PSR: The basic concept of PSR is that, if the
expected results of all possible tests are known, the dynamical system becomes completely understood. By expressing
the state using observable information, partially observable
dynamic systems can be modeled without prior knowledge.
Consider a discrete system with a finite set of observations,O = {o1 , o2 , . . . , ok }, and actions, A = {a1 , a2 , . . . , ak }.
The state representation of the system at time t is a vector
composed of the probability of occurrence of a test basedon the latest history. Each test is a sequence of actions
and observations starting at time t + 1; the history at time
t is a sequence of actions and observations up to andincluding time t. The probability of success of a test, τ ,of length m for history h, i.e., the probability of obtaining
an observation sequence in τ when taking the sequence ofactionsin τ , is represented by p(τ | h) = p(h, τ )/p(h) =QmPr(oi | h, ai ).
i=1
Knowing the success probabilities of some tests may help
us understand those of other tests. Given a test set T ={τ1 , τ2 , . . . , τk }, if there exists a function ft whose prediction
vector p(T | h) = [p (τ1 | h) p (τ2 | h) . . . p (τk | h)] satisfiesp(τl | h) = ft (p(T | h)) for any test τl , then T is called thecore test and the prediction vector, p(T | h), represents thestate of PSR.

2) Recurrent PSR: The original PSR model can only beapplied to systems consisting of discrete observations andactions. Methods to make PSR compatible with continuous
systems have been proposed [7], [9]. Herein, these methods
are collectively referred to as recurrent PSR (RPSR). In
RPSR, the state qt is represented as a conditional distribution of future observations, ot:t+k−1 , conditioned by future
actions, at:t+k−1 . The state update of RPSR is performed in
two steps.
• Extension: The linear map Wext is applied to the state qt
to obtain the extended state, et . The extended state, et , isa conditional distribution of the extended observations,
ot:t+k , conditioned by the extended action, at:t+k :et = Wext qt .
•(1)

Conditioning: For the action at and the observation ot
at time t, the known conditioning function, fcnd , updates
the state as follows:
qt+1 = fcnd (et , at , ot ) .

(2)

In discrete systems, qt and et are represented by conditional
establishment tables, and a Bayesian rule is applied by fcond .
To apply these to a continuous system, the Hilbert spaceembeddings of the distributions [7] and the kernel Bayes’rule [14] are used.
IV. A PPROACH
A. Application of PSR to mobile robot navigation in a
dynamic environment with pedestrians
In this study, we applied a PSR-based deep RL method
to mobile robot navigation in a dynamic environment withmultiple pedestrians. The elements of mobile robot navigation and PSR can be associated as follows:• Observations: The position and velocity data of N
pedestrians and the robot are treated as observations.
The observation data for each pedestrian and robot include vector (pix , piy , vxi , vyi ), where (pix , piy ) represents
the position and (vxi , vyi ) represents the velocity of theith pedestrian or robot.
• Actions: In this study, we assumed a holonomic omnidirectional mobile robot and used a two-dimensional
vector (vx , vy ) consisting of the input velocity vx in thex-axis direction and the input velocity vy in the y-axis
direction of the robot in a two-dimensional space.
The PSR used in this study becomes a model that can predictthe position of the pedestrian and the velocity in the next
step by inputting the command velocity of the robot. The
conceptual diagram of the PSR model is shown in Fig. 1.
B. Architecture of proposed method
The architecture of the proposed method is shown in Fig.
2. The proposed method consists of a PSR consisting of afeature extractor, state updater, and observation predictor, as
well as state integrator and value estimator.
• Feature extractor: It extracts features from each input
datum using a nonstationary spectral kernel function[15], [16].

Feature extractor
PSR
State updater
Observation predictor
Graph convolution
State integrator
Value estimator

Fig. 2: Architecture of the proposed method

State updater: It updates states via the state updater
using the features extracted from the observation andaction.
• Observation predictor: It predicts the next observations
using the states features extracted from action.
• State integrator: It integrates the states of the PSR corresponding to each pedestrian and provides the integrated
state to the value estimator.
• Value estimator: It estimates the value from the integrated state and robot feature.
•

State B
State CState A

State CState AGraph
convolution

Robot feature

Robot feature

Current step

Next step

Fig. 3: Procedure of GC-RPSR

The basic RPSR model can consider the effect of theaction of the agent on the environment. Through this feature,
the effects of robot actions on each pedestrian can be considered. However, the RPSR cannot consider the effects of theinteractions among pedestrians. For handling this problem,
we propose graph convolutional RPSR (GC-RPSR).
This architecture has a graph convolution part in the state
update phase in the RPSR for describing the relation effects
among pedestrians. The state update process is shown in Fig.
3. The kernel function for the adjacency matrix At is basedon the inverse of the L2 norm of the difference of eachposition [17]. This function is expressed in Eq. (3).
(1/ pit − pjt
if pit − pjt 6= 0
ij
at =
(3)
2
2
0
Otherwise
Finally, we use the normalized adjacency matrix. The normalization process is expressed in Eq. (4).
−1

et = Wext q̃t(6)

qt+1 = fcnd (et , at , ot ) ,(7)

where fgc represents the function for graph convolution.
D. State integrator

C. Graph convolutional RPSR
State B

where Ât = At + I and Λt is the diagonal node degree
matrix of At .
By using the graph convol05tion and Eqs. (1) and (2), thestate update process of GC-RPSR is expressed as follows:(5)q̃t = fgc qt , Ãt−1

Ãt = Λt 2 Ât Λt 2 ,We need an integration process to construct input data forthe value estimator from the state data corresponding to eachpedestrian. We propose two methods for the integration. The
first method uses graph convolution and the second method
is based on an occupancy map.
1) Integration using graph convolution: This method usesgraph convolution for integrating the states of the PSRs
corresponding to each pedestrian. The value estimator only
receives the robot feature to which graph convolution isapplied. The kernel function to be used for the adjacency
matrix is the same as the state update of the GC-RPSR
expressed in Eq. (3). The conceptual diagram of this method
is shown in Fig. 4.
2) Integration using occupancy map: This method uses an
occupancy map for integrating the states of the PSRs corresponding to each pedestrian. The states of the pedestrians are
stored in the corresponding cell depending on the position ofeach pedestrian. If multiple states are required to be stored
in the same cell, an average value is stored. The conceptual
diagram of this method is shown in Fig. 5.
E. Action generation
The action is generated using the value estimator fv , state
updater fpq , and observation predictor fpo from the PSR model
fp by selecting an action obtaining the maximum reward
State A

State BGraph
convolution

State CState D

Robot feature(4)

State A

State B

State CState D

Robot feature

Fig. 4: State integration using graph convolution

Algorithm 1: Optimization of the proposed method
Initialize the PSR fp and the value estimator fv with
two-stage regression and imitation learning
Initialize the target value estimator fˆv
for i = 1 to E do
Select at following the exploration policy and obtain
the reward rt , observation ot , and robot position pt
After finishing an episode, store the trajectory of(ot , at , rt , pt ) to buffer B
Fig. 5: State integration using an occupancy map

Sample from the buffer B and obtain M set of
trajectories
for j = 1 to M do
Obtain the trajectoriesbelow,
observation oj = oj1, oj2 , . . . , ojT ,jjjjrobot position
 pj =j p1 , p2j , . . . , pT ,jaction a = a1 , a2 , . . . , aT
Calculate the trajectories
below,
state qj = q1j , q2j ,. . . , qTj ,target value yj = y1j, y2j , . . . , yTj ,integrated state sj = sj1 , sj2 , . . . , sjT
endfrom the action space A. The formulation is shown in Eq.
(8), where γ is the discount factor, st is the integrated state attime t, pt is the robot position at time t, and ôt = fpo (qt , at ).
at ← argmaxat ∈A R (ôt ) + γ ∆t fv (st , pt )(8)

In addition, R(ot ) is a reward function at time t. Thisfunction is expressed in Eq. (9), where dt is the minimum
separation distance between the robot and the pedestrians
and pg is the goal position of the navigation task.
−0.25
if dt < 0

−0.1 + dt /2 else if dt < 0.2
R(ot ) =(9)1else if pt = pg0otherwise
Accordingly, we use d-step planning [18] that considers ad-step future. By rollout using the PSR and value estimatord-step, the action is selected with the maximum return along
the d-step prediction using Eq. (10).
fvd (st , pt ) = fv (st , pt )1 1
f (s , p ) + d−1
d maxat ( d v t t
R (ôt+1 ) + γfvd−1 (ŝt+1 , p̂t+1 )if d = 1otherwise(10)

where ôt+1 = fpo (qt+1 , at+1 ), and q̂t+1 = fpq (qt , at , ot ).
F. Training process
Training is performed in two steps. The first step is an
initialization of the PSR model and the imitation learningof the value estimator. Subsequently, we train the value
estimator using RL and the PSR model using supervised
learning.
1) Preliminary learning step: In the first step, the initialvalues of the PSR parameters are determined. This initialization is performed through two-stage regression [8] after collecting data using an exploration policy that follows ORCA
[19] for initialization. Subsequently, the value estimator istrained in imitation learning using the collected data.
2) RL and supervised learning step: In the second step,
we train the entire model with Algorithm 1, where E is
the number of episodes for training and d is the target
update frequency. The value estimator is trained with the
RL procedure based on the bootstrapped DQN [20].

Update fp by minimizing Lpred = MSE(fpo (q, a), o)Update fv by minimizing Lvalue = MSE(fv (s, p), y)
if i mod d thenUpdate fˆv by fˆv ← fv
end
end

V. E XPERIMENT
A. Implementation details
The structure of the value estimator is a multilayer perceptron with a hidden size of {500, 300}. In the RPSR structure,
the feature extractor uses 500 random Fourier features andsets the PCA dimensionality reduction to 40 dimensions.
The parameters are trained using AdaBelief [21] in theimitation, supervised, and reinforcement learning phases, andthe learning rate is 10−4 . We trained our models in 10k
scenarios by setting the batch size of the trajectory to 5 andthe target update frequency to 1k. The discount factor γ isset to be 0.9. The ε-greedy policy is used for the exploration
with decays from 0.5 to 0.1 linearly in the first 4k episodes.
The action space A consists of 80 discrete actions, consistingof 5 velocities exponentially spaced between (0, 1] and 16
headings evenly spaced between [0, 2π). Regarding action
generation, we set the depth and width of d-step planning to
2. For the state integration of the second model, we use an
occupancy map with a radius of 3 m, a distance resolutionof 0.3 m, and an angular resolution of 30◦ .
B. Experimental environment
We use the circle crossing scenarios in the CrowdNav
environment used in some related works [2], [4]. In thisenvironment, pedestrians are controlled by ORCA [19]. The
pedestrians are randomly placed on a circle of radius 4m, andtheir position (x, y) is subjected to a random perturbation.
The robot and pedestrians are influenced by each other,and their behavior changes with position and velocity. The
evaluation is conducted with 500 random test cases.
In the training phases of all the experiments and the testing
phases of the first and second experiments, we set the numberof pedestrians N to 5.
C. Comparison of GC-RPSR and RPSR
Here, we confirm the effectiveness of GC-RPSR by comparing it with the basic RPSR. We compare the success rate,
collision rate, execution time, and average return. We use
a simple concatenate for the state integrator. The result isshown in Table I.
TABLE I: Numerical comparison of RPSR and GC-RPSR
MethodRPSR
GC-RPSR

Success [%]
57.8
64.0

Collision [%]
33.8
11.0

Exec. time [s]
8.84
8.73

Avg. return
0.258
0.356

Thus, GC-RPSR outperforms RPSR in terms of all theevaluation items. Accordingly, the proposed PSR model
enhances the performance of the mobile robot navigation
task compared with basic RPSR.

Fig. 7: Sample trajectories of the results of the OGC-RPSR. The black trajectories describe the robot, and the colored trajectories describe pedestrians.

D. Comparison models with baseline

TABLE II: Numerical comparison of the proposed methods and the baseline
methodHere, we compare the proposed models with a method
proposed in a related study [4], which is considered as thebaseline. The evaluation items are the same as earlier. Fig. 6
shows samples of the result trajectory of the proposed modelwith state integration using graph convolution (GGC-RPSR)
and Fig. 7 shows samples of the result trajectory of the proposed model with state integration using an occupancy map(OGC-RPSR). In each figure, the black trajectory represents
the robot, the colored trajectories represent pedestrians, andthe numbers indicate time steps. Both methods can navigate
the robot to the goal while avoiding pedestrians. However,
GGC-RPSR has a higher efficiency because it can generate
shorter paths. The numerical comparison of the two proposed
methods and the baseline method is shown in Table II.

MethodRGL [4]GGC-RPSR
OGC-RPSR

Success [%]
92.0
94.6
91.4

Collision [%]
7.0
5.4
6.8

Exec. time [s]
9.09
7.788
11.536

Avg. return
0.560
0.587
0.512

The GGC-RPSR outperforms the other methods in termsof all the evaluation items. This result demonstrates theeffectiveness of the proposed method using GC-RPSR andstate integration using graph convolution. The GGC-RPSR
significantly outperforms the OGC-RPSR in terms of theexecution time.
E. Comparison of the models in the situation where thenumber of pedestrians differs between training and testing
Here, we compared the two proposed methods in thesituation where the number of pedestrians differs between
training and testing. Both the models were trained using
an environment in which the number of pedestrians was 5
and were tested in environments in which the number ofpedestrians was 1 to 10.
The results of the two models are shown in Table III,
where N indicates the number of pedestrians. Thus, theperformance difference between the two methods can beobserved from the success rate by comparing with the result
for N = 5. The maximum decline of the success rate of the
GGC-RPSR is 43.4%; in contrast, the corresponding result
for the OGC-RPSR is 7.4%. This result shows that the OGCRPSR is more stable when the number of pedestrians isvaried between training and testing.
VI. C ONCLUSION AND F UTURE W ORK

Fig. 6: Sample trajectories of the results of the GGC-RPSR. The black trajectories describe the robot, and the colored trajectories describe pedestrians.

This study applied a deep RL method based on PSR tomobile robot navigation tasks in a dynamic environment withpedestrians. We proposed a new PSR architecture for mobile
robot navigation in a dynamic environment with pedestrians
considering the interactions among pedestrians. In addition,

TABLE III: Results of the GGC-RPSR and OGC-RPSR while changing the number of pedestrians. The values in the brackets in the success rate column
show the difference from the result in the situation with N = 5.
N
1
2
3
4
5
6
7
8
9
10

Success [%]63.6 (− 31.0)
68.2 (− 26.4)
65.8 (− 28.8)
91.0 (− 3.6)
94.6 (± 0.0)
76.2 (− 18.4)
93.6 (− 1.0)
89.0 (− 5.6)
51.2 (− 43.4)
89.0 (− 5.6)

GGC-RPSR
Collision [%]Exec. time [s]
2.4
8.171
16.4
8.109
22.0
8.122
5.8
7.988
5.4
7.788
20.8
7.985
5.8
7.889
10.2
7.885
39.6
8.135
10.8
7.878

Avg. return
0.419
0.393
0.361
0.556
0.587
0.431
0.578
0.541
0.220
0.540

we proposed methods for integrating the state of the PSRs
corresponding to each pedestrian for handling the change in
the number of pedestrians between training and testing.
The experiments conducted showed the effectiveness ofthe proposed model. In addition, we confirmed that thesecond version of the proposed model, which uses an
occupancy-map-based state integration, is more stable in thesituation where the number of pedestrians differs between
training and testing.
In future work, the two types of state integration methods
proposed in this paper have not yet been fully discussed;
further experiments are needed to clarify in detail the causesof the differences between the two methods. For the occupancy map-based method, further experiments are needed
to clarify the relationship between parameters such as thenumber of grids and the performance change. In addition,
we will consider the more effective model and the learning
process for a real-world scenario. The learning costs andexploration processes are critical problems in real-world deep
RL tasks. If a robot can learn in the real world directly,
the trained system can perform better in actual environments
than in simulated environments. However, the robot would be
required to repeat its interactions with the environment andlearn from numerous success and failure patterns, which isdifficult to achieve practically. Thus, a more efficient learning
process is required to overcome these problems.
ACKNOWLEDGMENT
This work was partially supported by JSPS KAKENHI
Grant Number JP20H00230.
R EFERENCES[1] M. Everett, Y. F. Chen, and J. P. How, “Motion Planning among
Dynamic, Decision-Making Agents with Deep Reinforcement Learning,” in IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS), pp. 3052–3059, 2018.
[2] C. Chen, Y. Liu, S. Kreiss, and A. Alahi, “Crowd-robot interaction:
Crowd-aware robot navigation with attention-based deep reinforcement learning,” in Proceedings - IEEE International Conference on
Robotics and Automation, vol. 2019-May, pp. 6015–6022, 2019.
[3] Y. Chen, C. Liu, B. E. Shi, and M. Liu, “Robot Navigation in
Crowds by Graph Convolutional Networks with Attention Learned
from Human Gaze,” IEEE Robotics and Automation Letters, vol. 5,
pp. 2754–2761, apr 2020.
[4] C. Chen, S. Hu, P. Nikdel, G. Mori, and M. Savva, “Relational graphlearning for crowd navigation,” in IEEE/RSJ International Conferenceon Intelligent Robots and Systems, 2020.
[5] L. Lucia, D. Daniel, C. Gianluca, S. Roland, and D. Renaud, “Robot
Navigation in Crowded Environments Using Deep Reinforcement
Learning,” in 2020 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2020.

Success [%]98.2 (+ 6.8)
95.0 (+ 3.6)
94.4 (+ 3.0)
92.0 (+ 0.6)
91.4 (± 0.0)
88.0 (− 3.4)
91.2 (− 0.2)
91.2 (− 0.2)
84.0 (− 7.4)
85.8 (− 5.6)

OGC-RPSR
Collision [%]Exec. time [s]0.6
10.983
3.4
11.017
4.8
11.126
6.8
11.048
6.8
11.536
10.0
11.251
7.6
11.327
7.6
11.27013.0
11.505
11.8
11.714

Avg. return
0.587
0.560
0.548
0.527
0.512
0.485
0.502
0.500
0.422
0.447

[6] S. Singh, M. James, and M. Rudary, “Predictive State Representations:
A New Theory for Modeling Dynamical Systems,” in 20th The
Conference on Uncertainty in Artificial Intelligence (UAI), 2004.
[7] B. Boots, A. Gretton, and G. J. Gordon, “Hilbert space embeddingsof predictive state representations,” in 29th Conference on Uncertainty
in Artificial Intelligence (UAI), pp. 92–101, 2013.
[8] A. Hefny, C. Downey, and G. J. Gordon, “Supervised learning for
dynamical system learning,” in Advances in Neural Information Processing Systems, vol. 2015-Janua, pp. 1963–1971, 2015.
[9] A. Hefny, C. Downey, and G. Gordon, “An efficient, expressive andlocal minima-free method for learning controlled dynamical systems,”
in 32nd AAAI Conference on Artificial Intelligence (AAAI), pp. 3191–
3198, feb 2018.
[10] A. Venkatraman, N. Rhinehart, W. Sun, L. Pinto, M. Hebert, B. Boots,
K. M. Kitani, and J. A. Bagnell, “Predictive-state decoders: Encoding
the future into recurrent networks,” in Advances in Neural Information
Processing Systems, vol. 2017-Decem, pp. 1173–1184, Neural information processing systems foundation, 2017.
[11] A. Hefny, Z. Marinho, W. Sun, S. S. Srinivasa, and G. Gordon,“Recurrent predictive state policy networks,” in 35th International
Conference on Machine Learning, ICML 2018, vol. 5, pp. 3104–3119,
2018.
[12] E. Meisner, S. Das, V. Isler, J. Trinkle, S. Śabanović, and L. R.
Caporael, “Predictive state representations for grounding human-robot
communication,” in Proceedings - IEEE International Conference on
Robotics and Automation, pp. 178–185, 2010.
[13] J. A. Stork, C. H. Ek, Y. Bekiroglu, and D. Kragic, “Learning
Predictive State Representation for in-hand manipulation,” in Proceedings - IEEE International Conference on Robotics and Automation,
vol. 2015-June, pp. 3207–3214, Institute of Electrical and Electronics
Engineers Inc., jun 2015.
[14] K. Fukumizu, L. Song, and A. Gretton, “Kernel Bayes’ rule: Bayesianinference with positive definite kernels,” Journal of Machine Learning
Research, vol. 14, pp. 3753–3783, 2013.
[15] S. Remes, M. Heinonen, and S. Kaski, “Non-stationary spectral
kernels,” in Advances in Neural Information Processing Systems,
vol. 2017-Decem, pp. 4643–4652, 2017.
[16] J. F. Ton, S. Flaxman, D. Sejdinovic, and S. Bhatt, “Spatial mappingwith Gaussian processes and nonstationary Fourier features,” Spatial
Statistics, vol. 28, pp. 59–78, dec 2018.
[17] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel, “SocialSTGCNN: A Social Spatio-Temporal Graph Convolutional Neural
Network for Human Trajectory Prediction,” in Proceedings of theIEEE Computer Society Conference on Computer Vision and Pattern
Recognition, pp. 14412–14420, 2020.
[18] J. Oh, S. Singh, and H. Lee, “Value prediction network,” in Advances in Neural Information Processing Systems, vol. 2017-Decem,
pp. 6119–6129, 2017.
[19] J. Van Den Berg, S. J. Guy, M. Lin, and D. Manocha, “Reciprocal
n-body collision avoidance,” in Springer Tracts in Advanced Robotics,
vol. 70, pp. 3–19, 2011.
[20] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration
via bootstrapped DQN,” in Advances in Neural Information Processing
Systems, pp. 4033–4041, 2016.
[21] J. Zhuang, T. Tang, Y. Ding, S. Tatikonda, N. Dvornek, X. Papademetris, and J. S. Duncan, “AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients,” in Advances in Neural
Information Processing Systems, vol. 2020-Decem, 2020.