Adaptive Online Packing-guided Search for POMDPs

1 Introduction . POMDPs generalize the MDPs by considering the state uncertainty [ 1 ] . In MDPs , the agent knows its state exactly . In POMDPs , though an MDP determines the underlying dynamics , the agent can not access its actual state at each timestep and , instead , receives an observation serving as a clue . In order to make decisions in POMDPs , the state uncertainty must be handled . A popular method for tackling the uncertainty is to maintain a belief , a distribution on the state space , where the probability of a state indicates how possible the agent believes it is . However , an exact belief updating requires O ( |S|2 ) computations , which is unaffordable in immense state space . A practical way is to approximate the belief with a collection of weighted particles ( or samples ) and update it with particle filtering [ 2 ] . Finding an optimal solution for finite-horizon POMDPs or a near-optimal solution for discounted infinite-horizon POMDPs is proved to be PSPACE-complete [ 3 , 4 ] . Therefore , online planning , which computes a real-time solution , plays a crucial role in solving POMDPs , following for two reasons . The first is that online planning searches for a solution for not all possible beliefs but the current belief alone , relieving the computational load significantly . Second , online planning can take advantage of the approximate offline solution and other heuristics to expedite the tree search process . Online planning algorithms have made substantial progress in solving large-scale POMDP problems [ 5–9 ] . These methods approximate the belief with a collection of sampled states instead of updating ∗Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . it exactly and consider a set of sampled observations instead of all possible observations . However , the estimation derived by simple belief approximation methods that have been applied , such as direct sampling and sequential importance sampling [ 10 ] , have a variance increasing exponentially with searching depth . Besides , naive sampling for observations , either the direct sampling as in [ 7 ] or importance sampling as in [ 9 ] , yields a large variance when the sample size is small . Inspired by this , we take one step further and propose an online planning algorithm , Adaptive Online Packing-guided Search ( AdaOPS ) , to better approximate beliefs and better balance estimation bias and variance . It features two innovations , adaptive particle filtering , and belief packing . The adaptive particle filter is designed for online planning . During the belief updating , it resamples adaptively and adjusts the sample size on the fly according to the dispersion of the belief distribution , which allows achieving high approximation accuracy with only a handful of particles . AdaOPS forms a belief packing by merging similar beliefs . Only beliefs in the packing are explored , hence the name “ packing-guided search ” . Belief packing is meant to balance estimation bias and variance . With the use of beliefs in their exact form , it is guaranteed that similar beliefs yield a slight difference in their optimal values [ 11 ] . We further extend this conclusion to cases where beliefs are approximated by a collection of weighted particles and prove that by merging similar beliefs , our algorithm can converge to the -optimal policy with high probability under some mild assumptions . 2 Background . 2.1 POMDPs . An MDP is mathematically defined as a tuple ( S , A , T , R , γ ) , where S and A are the state and action spaces , respectively , and T and R are the transition and reward functions , respectively . γ ∈ [ 0 , 1 ] is the discount factor . A POMDP problem can be seen as an extension to an MDP problem , which introduces two additional components , the observation space O and the observation function Z . At each timestep , instead of observing the actual state as an MDP agent does , a POMDP agent can only get an observation according to the observation function Z ( o | s′ , a ) = Pr ( o | s′ , a ) , which determines the probability of observing o ∈ O after executing action a ∈ A and reaching state s′ ∈ S . The transition function T and observation function Z are sometimes hard to define explicitly . An alternative is the generative model s′ , o , r ← G ( s , a ) , which implicitly defines the transition and observation distributions by randomly generating a possible combination of next state s′ , observation o , and reward r given state s and action a . The POMDP agent maintains a belief b in states , a distribution on S , and seeks to find a policy π mapping beliefs to actions , which maximizes the expected sum of discounted rewards , E [ ∑∞ t=0 γ tR ( st , π ( bt ) ] . 2.2 Particle Filters . Bayes filter recursively updates the posterior of the hidden states from a sequence of noisy and incomplete observations . In POMDPs , this technique is usually used to maintain a belief approximation . Particle filter [ 10 ] is an essential class of Bayes filter , which approximates the belief with a collection of weighted particles , { ( wi , si ) } i∈ [ N ] , where si is the i-th particle , and wi is the corresponding normalized weight . Sequential importance sampling ( SIS ) is a widely used particle filtering method . It updates the belief by propagating particles with action a , s′i ∼ T ( · | si , a ) , and reweighting particles with observation o , i.e. , w′i = wiZ ( o | s′i , a ) /ρ , where ρ is a normalizing factor . However , continuously updating weights will lead to sample degeneracy where most particles become impossible and have negligible weights . This problem causes the estimation variance of SIS to increase exponentially with depth [ 10 ] . In another particle filtering method , sequential importance resampling ( SIR ) , the weight updating is followed by a resampling step , which samples N particles from { ( w′i , s′i ) } i∈ [ N ] with probability proportional to the weights , and the weights of the resampled particles are set to be 1/N . Resampling alleviates the sample degeneracy by replicating high-weight particles and dropping low-weight ones . Theoretically , resampling is critical , without which we can not derive the uniform convergence in the time horizon for particle filters [ 12 ] . However , frequent resampling induces sample impoverishment , which means that particles ’ diversity is diminished , and the whole collection of particles contains few distinct ones . In practice , resampling is only performed when necessary . A rule of thumb is to resample when the ratio of particle number N to effective sample size ( ESS ) is greater than a threshold , say two . Here , ESS is the sample size required from the true distribution to achieve the same sampling error as sampling N samples from the proposal distribution and can be approximately calculated [ 13 , 14 ] : ESS ≈ 1∑N n=1 w 2 n . ( 1 ) Intuitively , this strategy ensures that we only resample when the weight disparity is significant . 2.3 KLD-Sampling . Instead of using a fixed sample size , [ 15 ] proposes KLD-Sampling for adapting the sample size in particle filters according to the dispersion of the posterior distributions . Assuming the true posterior can be represented by a discrete piece-wise constant distribution consisting of k different multidimensional bins , it calculates the number of samples needed to ensure that the Kullback-Leibler divergence ( KLD ) between the approximated distribution and the true distribution is less than an error ζ with a confidence level 1− η . The number of samples N is given by N ≈ k − 1 2ζ ( 1− 2 9 ( k − 1 ) + √ 2 9 ( k − 1 ) z1−η ) 3 , ( 2 ) where z1−η is the 1− η quantile of the standard normal distribution . This dynamic strategy ensures approximation quality while uses less particles on average . During the sampling , one should keep track of the number of bins that the approximated distribution occupies and estimate the number of samples needed accordingly . 2.4 Belief Packing . Definition 1 ( δ-packing ) . Let ( X , ‖ · ‖ ) be a normed space and Y ⊆ X . { y1 , y2 , . . . , yn } ⊆ Y is a δ-packing of Y if mini 6=j ‖yi − yj‖ > δ . The δ-packing number of Y , denoted P δ ( Y ) , is the maximum cardinality of any δ-packing of Y . Remark . If X is totally bounded and Y ⊆ X , then we have P δ ( Y ) < ∞ for all δ > 0 . A belief packing is a special case of δ-packing , where the X is the belief space , a probability simplex of |S| − 1 dimensions , and the norm ‖ · ‖ is the L1 norm . Since a simplex is totally bounded , it follows that the packing number of a belief packing is finite . 2.5 Related Work . The covering number is the minimum number of balls needed to cover the entire space . It is closely related to the packing number and is extensively used to identify the complexity of POMDP problems [ 11 , 16–18 ] . SARSOP [ 19 ] is an offline solver which builds an approximate cover of the beliefs space reachable by the optimal policy and performs point-based backup at beliefs in the cover . PGVI [ 17 ] improves SARSOP by maintaining belief packings and only expanding beliefs in the packing . FMP [ 20 ] applies the idea of PGVI to online planning and builds a δ-cover of beliefs . Notwithstanding , all these methods only work in discrete state spaces and require enumerating all observations . Online POMDP solvers mainly include POMCP [ 5 ] , DESPOT [ 6 ] , and their variants [ 7–9 ] . We will introduce them in terms of approximating beliefs and addressing large observation spaces . On the belief approximation , DESPOT and POMCP sample from the joint distribution of states and observations and approximate a belief with samples having consistent observations . These methods are doomed to failure in large or continuous observation space because the same observation is unlikely to be produced twice . POMCPOW [ 7 ] weights particles according to the likelihood , which makes its belief updating close to SIR . Nonetheless , most of its beliefs are approximated by merely several particles causing the approximation to be inaccurate . PTF-DPW [ 7 ] updates beliefs with SIR particle filters . However , standard particle filters are excessively costly . The best performance of PTF-DPW is achieved when the number of particles used is only 20 . DESPOT-α [ 9 ] and LABECOP [ 21 ] approximate beliefs with SIS , which is , as explained , ineffective compared to SIR . On addressing observations , DESPOT and POMCP expand all sampled observations , which causes the observation branches to explode quickly . POMCPOW and PTF-DPW introduce the progressive widening technique [ 22 ] , which restricts the number of observation branches and gradually relaxes the restriction when a node is revisited multiple times . DESPOT-α instead fixes the maximum number of observation branches to be C. However , restricting the number of observation branches inevitably produces extra estimation variance , damaging the performance . LABECOP adopts the lazy belief extraction technique , which always extracts the belief from scratch with the newly sampled observation sequence . This method naturally comes with a higher computational cost . 3 Method Algorithm 1 General Procedure Input : initial belief bel 1 : b← bel 2 : while True do 3 : a∗ ← PLANNING ( b ) 4 : Execute action a∗ 5 : Receive observation o 6 : Update the belief b with a∗ and o Online planning ( Alg . 1 ) consists of two alternating stages , planning and execution . A lookahead search is conducted during the planning stage to find the best action for the current belief . At the execution stage , the algorithm performs in the real environment the action found , gets a new observation , and updates the belief with a Bayes filter . Then a planning stage follows . AdaOPS follows the general procedure and makes changes in the planning part . 3.1 Planning At each timestep , AdaOPS gradually builds up a belief tree ( as shown in Figure 1 ) consisting of alternating action and belief nodes , of which the root belief b̄0 is sampled from the current belief b0 using KLDSampling . Each belief b̄ on the tree is represented by a collection of weighted particles and is an approximation to a corresponding true belief b . Note that an approximated belief also corresponds to a belief node , and we will not distinguish between them in the following . Sibling beliefs on the tree share the particles and differ in the corresponding weights . Initially , the belief tree is comprised of the root node alone . A series of explorations will then be made , each traversing a path on the tree and expanding new nodes in the meantime . The complete pseudocode of AdaOPS is shown in Appendix A. AdaOPS maintains an upper bound u ( b̄ , a ) and a lower bound l ( b̄ , a ) of the optimal value for every action node a of belief b̄ . During the exploration , the action a∗ with the highest upper bound will be chosen ( Line 10 in Alg . 2 ) . This strategy ensures that an action branch gets explored only when its upper bound is higher than or equal to the optimal value . AdaOPS also maintains lower and upper bounds , l ( b̄ ) and u ( b̄ ) , for each belief node b̄ . After choosing the action , it will choose ( Line 11 in Alg . 2 ) the child belief with the highest probability-weighted excess uncertainty ( EU ) [ 6 , 23 ] , p̂ ( o | b̄ , a∗ ) EU ( τ ( b̄ , a∗ , o ) ) , where p̂ ( o | b̄ , a∗ ) is an estimation of the probability Pr ( o | b̄ , a∗ ) , and τ ( b̄ , a∗ , o ) is the belief reached after executing a∗ and receiving o . The way to estimate Pr ( o | b̄ , a∗ ) is stated in Section 3.2.1 . The excess uncertainty is defined as EU ( b̄ ) = u ( b̄ ) − l ( b̄ ) − ξ · ( u ( b̄0 ) − l ( b̄0 ) ) /γdepth ( b̄ ) , ( 3 ) where ξ is a parameter for adjusting the desired gap between upper and lower bounds at the root and defaults to 0.95 , and depth ( b̄ ) is the depth of belief b̄ on the belief tree . The excess uncertainty measures how much uncertainty we need to eliminate at the belief node b̄ if we want the gap at the root , u ( b̄0 ) −l ( b̄0 ) , to be reduced to ξ · ( u ( b̄0 ) −l ( b̄0 ) ) . The excess uncertainty has a desirable property : EU ( b̄ ) ≤ ∑ o∈Ob̄ , a∗ p̂ ( o | b̄ , a∗ ) EU ( τ ( b̄ , a∗ , o ) ) , where Ob̄ , a∗ is the set of expanded observations Algorithm 2 PLANNING ( b0 ) 1 : b̄0 ← KLD-SAMPLING ( b0 ) 2 : while time permitting and l ( b̄0 ) < u ( b̄0 ) do 3 : b̄← b̄0 4 : while depth ( b̄ ) < D do 5 : if b̄ is a leaf node then 6 : EXPAND ( b̄ ) 7 : BACKUP ( b̄ ) 8 : if best action branch changes for some ancestors of b̄ or EU ( b̄ ) ≤ 0 then 9 : break 10 : a∗ ← arg maxa∈A u ( b̄ , a ) 11 : o∗ ← arg maxo∈Ob̄ , a p̂ ( o | b̄ , a ) EU ( τ ( b̄ , a ∗ , o ) )

Summary: This paper presents an online planning algorithm, called Adaptive Online Packing-guided Search (AdaOPS), for Partially Observable Markov Decision Processes. The article addresses the problem of improving the approximations of a belief using a particle filter. This improvement comes from two elements: adaptive particle filtering and belief packing.  Specifically, During the planning step, a particle resampling procedure is performed adaptively on new leaf nodes. This happens when the number of particles in the belief of the node is greater than twice the Effective Sample Size (ESS), as explained in (Kish [11], Liu [12]). AdaOPS use a KLD-sampling for this adaptive resampling (Fox [13], Li, Sun, Satter [15]), this guarantees a good approximation using a limited number of particles.  The article also presents a technique called Belief Packing, inspired by (Zhang, Hsu, Lee [17]). It is used to fuse similar beliefs, thus the planning procedure explores only beliefs that are significantly different. This improves the scalability of the online algorithm.  The paper shows that with enough planning time, this packing produces results that are $\epsilon$-optimal (i.e., the difference between the original belief and the belief that employs the packing is lower than a constant $\epsilon$ with a probability of at least $1 - \eta$). The article also presents an empirical evaluation of AdaOPS in four different domains (Laser Tag, Rocksampe, Roomba, and Light-Dark). The results show that AdaOPS can outperform state of the art planning algorithms (POMCPOW, ARDESPOT).