Adaptive Online Packing-guided Search for POMDPs

1 Introduction . POMDPs generalize the MDPs by considering the state uncertainty [ 1 ] . In MDPs , the agent knows its state exactly . In POMDPs , though an MDP determines the underlying dynamics , the agent can not access its actual state at each timestep and , instead , receives an observation serving as a clue . In order to make decisions in POMDPs , the state uncertainty must be handled . A popular method for tackling the uncertainty is to maintain a belief , a distribution on the state space , where the probability of a state indicates how possible the agent believes it is . However , an exact belief updating requires O ( |S|2 ) computations , which is unaffordable in immense state space . A practical way is to approximate the belief with a collection of weighted particles ( or samples ) and update it with particle filtering [ 2 ] . Finding an optimal solution for finite-horizon POMDPs or a near-optimal solution for discounted infinite-horizon POMDPs is proved to be PSPACE-complete [ 3 , 4 ] . Therefore , online planning , which computes a real-time solution , plays a crucial role in solving POMDPs , following for two reasons . The first is that online planning searches for a solution for not all possible beliefs but the current belief alone , relieving the computational load significantly . Second , online planning can take advantage of the approximate offline solution and other heuristics to expedite the tree search process . Online planning algorithms have made substantial progress in solving large-scale POMDP problems [ 5–9 ] . These methods approximate the belief with a collection of sampled states instead of updating ∗Corresponding Author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . it exactly and consider a set of sampled observations instead of all possible observations . However , the estimation derived by simple belief approximation methods that have been applied , such as direct sampling and sequential importance sampling [ 10 ] , have a variance increasing exponentially with searching depth . Besides , naive sampling for observations , either the direct sampling as in [ 7 ] or importance sampling as in [ 9 ] , yields a large variance when the sample size is small . Inspired by this , we take one step further and propose an online planning algorithm , Adaptive Online Packing-guided Search ( AdaOPS ) , to better approximate beliefs and better balance estimation bias and variance . It features two innovations , adaptive particle filtering , and belief packing . The adaptive particle filter is designed for online planning . During the belief updating , it resamples adaptively and adjusts the sample size on the fly according to the dispersion of the belief distribution , which allows achieving high approximation accuracy with only a handful of particles . AdaOPS forms a belief packing by merging similar beliefs . Only beliefs in the packing are explored , hence the name “ packing-guided search ” . Belief packing is meant to balance estimation bias and variance . With the use of beliefs in their exact form , it is guaranteed that similar beliefs yield a slight difference in their optimal values [ 11 ] . We further extend this conclusion to cases where beliefs are approximated by a collection of weighted particles and prove that by merging similar beliefs , our algorithm can converge to the -optimal policy with high probability under some mild assumptions . 2 Background . 2.1 POMDPs . An MDP is mathematically defined as a tuple ( S , A , T , R , γ ) , where S and A are the state and action spaces , respectively , and T and R are the transition and reward functions , respectively . γ ∈ [ 0 , 1 ] is the discount factor . A POMDP problem can be seen as an extension to an MDP problem , which introduces two additional components , the observation space O and the observation function Z . At each timestep , instead of observing the actual state as an MDP agent does , a POMDP agent can only get an observation according to the observation function Z ( o | s′ , a ) = Pr ( o | s′ , a ) , which determines the probability of observing o ∈ O after executing action a ∈ A and reaching state s′ ∈ S . The transition function T and observation function Z are sometimes hard to define explicitly . An alternative is the generative model s′ , o , r ← G ( s , a ) , which implicitly defines the transition and observation distributions by randomly generating a possible combination of next state s′ , observation o , and reward r given state s and action a . The POMDP agent maintains a belief b in states , a distribution on S , and seeks to find a policy π mapping beliefs to actions , which maximizes the expected sum of discounted rewards , E [ ∑∞ t=0 γ tR ( st , π ( bt ) ] . 2.2 Particle Filters . Bayes filter recursively updates the posterior of the hidden states from a sequence of noisy and incomplete observations . In POMDPs , this technique is usually used to maintain a belief approximation . Particle filter [ 10 ] is an essential class of Bayes filter , which approximates the belief with a collection of weighted particles , { ( wi , si ) } i∈ [ N ] , where si is the i-th particle , and wi is the corresponding normalized weight . Sequential importance sampling ( SIS ) is a widely used particle filtering method . It updates the belief by propagating particles with action a , s′i ∼ T ( · | si , a ) , and reweighting particles with observation o , i.e. , w′i = wiZ ( o | s′i , a ) /ρ , where ρ is a normalizing factor . However , continuously updating weights will lead to sample degeneracy where most particles become impossible and have negligible weights . This problem causes the estimation variance of SIS to increase exponentially with depth [ 10 ] . In another particle filtering method , sequential importance resampling ( SIR ) , the weight updating is followed by a resampling step , which samples N particles from { ( w′i , s′i ) } i∈ [ N ] with probability proportional to the weights , and the weights of the resampled particles are set to be 1/N . Resampling alleviates the sample degeneracy by replicating high-weight particles and dropping low-weight ones . Theoretically , resampling is critical , without which we can not derive the uniform convergence in the time horizon for particle filters [ 12 ] . However , frequent resampling induces sample impoverishment , which means that particles ’ diversity is diminished , and the whole collection of particles contains few distinct ones . In practice , resampling is only performed when necessary . A rule of thumb is to resample when the ratio of particle number N to effective sample size ( ESS ) is greater than a threshold , say two . Here , ESS is the sample size required from the true distribution to achieve the same sampling error as sampling N samples from the proposal distribution and can be approximately calculated [ 13 , 14 ] : ESS ≈ 1∑N n=1 w 2 n . ( 1 ) Intuitively , this strategy ensures that we only resample when the weight disparity is significant . 2.3 KLD-Sampling . Instead of using a fixed sample size , [ 15 ] proposes KLD-Sampling for adapting the sample size in particle filters according to the dispersion of the posterior distributions . Assuming the true posterior can be represented by a discrete piece-wise constant distribution consisting of k different multidimensional bins , it calculates the number of samples needed to ensure that the Kullback-Leibler divergence ( KLD ) between the approximated distribution and the true distribution is less than an error ζ with a confidence level 1− η . The number of samples N is given by N ≈ k − 1 2ζ ( 1− 2 9 ( k − 1 ) + √ 2 9 ( k − 1 ) z1−η ) 3 , ( 2 ) where z1−η is the 1− η quantile of the standard normal distribution . This dynamic strategy ensures approximation quality while uses less particles on average . During the sampling , one should keep track of the number of bins that the approximated distribution occupies and estimate the number of samples needed accordingly . 2.4 Belief Packing . Definition 1 ( δ-packing ) . Let ( X , ‖ · ‖ ) be a normed space and Y ⊆ X . { y1 , y2 , . . . , yn } ⊆ Y is a δ-packing of Y if mini 6=j ‖yi − yj‖ > δ . The δ-packing number of Y , denoted P δ ( Y ) , is the maximum cardinality of any δ-packing of Y . Remark . If X is totally bounded and Y ⊆ X , then we have P δ ( Y ) < ∞ for all δ > 0 . A belief packing is a special case of δ-packing , where the X is the belief space , a probability simplex of |S| − 1 dimensions , and the norm ‖ · ‖ is the L1 norm . Since a simplex is totally bounded , it follows that the packing number of a belief packing is finite . 2.5 Related Work . The covering number is the minimum number of balls needed to cover the entire space . It is closely related to the packing number and is extensively used to identify the complexity of POMDP problems [ 11 , 16–18 ] . SARSOP [ 19 ] is an offline solver which builds an approximate cover of the beliefs space reachable by the optimal policy and performs point-based backup at beliefs in the cover . PGVI [ 17 ] improves SARSOP by maintaining belief packings and only expanding beliefs in the packing . FMP [ 20 ] applies the idea of PGVI to online planning and builds a δ-cover of beliefs . Notwithstanding , all these methods only work in discrete state spaces and require enumerating all observations . Online POMDP solvers mainly include POMCP [ 5 ] , DESPOT [ 6 ] , and their variants [ 7–9 ] . We will introduce them in terms of approximating beliefs and addressing large observation spaces . On the belief approximation , DESPOT and POMCP sample from the joint distribution of states and observations and approximate a belief with samples having consistent observations . These methods are doomed to failure in large or continuous observation space because the same observation is unlikely to be produced twice . POMCPOW [ 7 ] weights particles according to the likelihood , which makes its belief updating close to SIR . Nonetheless , most of its beliefs are approximated by merely several particles causing the approximation to be inaccurate . PTF-DPW [ 7 ] updates beliefs with SIR particle filters . However , standard particle filters are excessively costly . The best performance of PTF-DPW is achieved when the number of particles used is only 20 . DESPOT-α [ 9 ] and LABECOP [ 21 ] approximate beliefs with SIS , which is , as explained , ineffective compared to SIR . On addressing observations , DESPOT and POMCP expand all sampled observations , which causes the observation branches to explode quickly . POMCPOW and PTF-DPW introduce the progressive widening technique [ 22 ] , which restricts the number of observation branches and gradually relaxes the restriction when a node is revisited multiple times . DESPOT-α instead fixes the maximum number of observation branches to be C. However , restricting the number of observation branches inevitably produces extra estimation variance , damaging the performance . LABECOP adopts the lazy belief extraction technique , which always extracts the belief from scratch with the newly sampled observation sequence . This method naturally comes with a higher computational cost . 3 Method Algorithm 1 General Procedure Input : initial belief bel 1 : b← bel 2 : while True do 3 : a∗ ← PLANNING ( b ) 4 : Execute action a∗ 5 : Receive observation o 6 : Update the belief b with a∗ and o Online planning ( Alg . 1 ) consists of two alternating stages , planning and execution . A lookahead search is conducted during the planning stage to find the best action for the current belief . At the execution stage , the algorithm performs in the real environment the action found , gets a new observation , and updates the belief with a Bayes filter . Then a planning stage follows . AdaOPS follows the general procedure and makes changes in the planning part . 3.1 Planning At each timestep , AdaOPS gradually builds up a belief tree ( as shown in Figure 1 ) consisting of alternating action and belief nodes , of which the root belief b̄0 is sampled from the current belief b0 using KLDSampling . Each belief b̄ on the tree is represented by a collection of weighted particles and is an approximation to a corresponding true belief b . Note that an approximated belief also corresponds to a belief node , and we will not distinguish between them in the following . Sibling beliefs on the tree share the particles and differ in the corresponding weights . Initially , the belief tree is comprised of the root node alone . A series of explorations will then be made , each traversing a path on the tree and expanding new nodes in the meantime . The complete pseudocode of AdaOPS is shown in Appendix A. AdaOPS maintains an upper bound u ( b̄ , a ) and a lower bound l ( b̄ , a ) of the optimal value for every action node a of belief b̄ . During the exploration , the action a∗ with the highest upper bound will be chosen ( Line 10 in Alg . 2 ) . This strategy ensures that an action branch gets explored only when its upper bound is higher than or equal to the optimal value . AdaOPS also maintains lower and upper bounds , l ( b̄ ) and u ( b̄ ) , for each belief node b̄ . After choosing the action , it will choose ( Line 11 in Alg . 2 ) the child belief with the highest probability-weighted excess uncertainty ( EU ) [ 6 , 23 ] , p̂ ( o | b̄ , a∗ ) EU ( τ ( b̄ , a∗ , o ) ) , where p̂ ( o | b̄ , a∗ ) is an estimation of the probability Pr ( o | b̄ , a∗ ) , and τ ( b̄ , a∗ , o ) is the belief reached after executing a∗ and receiving o . The way to estimate Pr ( o | b̄ , a∗ ) is stated in Section 3.2.1 . The excess uncertainty is defined as EU ( b̄ ) = u ( b̄ ) − l ( b̄ ) − ξ · ( u ( b̄0 ) − l ( b̄0 ) ) /γdepth ( b̄ ) , ( 3 ) where ξ is a parameter for adjusting the desired gap between upper and lower bounds at the root and defaults to 0.95 , and depth ( b̄ ) is the depth of belief b̄ on the belief tree . The excess uncertainty measures how much uncertainty we need to eliminate at the belief node b̄ if we want the gap at the root , u ( b̄0 ) −l ( b̄0 ) , to be reduced to ξ · ( u ( b̄0 ) −l ( b̄0 ) ) . The excess uncertainty has a desirable property : EU ( b̄ ) ≤ ∑ o∈Ob̄ , a∗ p̂ ( o | b̄ , a∗ ) EU ( τ ( b̄ , a∗ , o ) ) , where Ob̄ , a∗ is the set of expanded observations Algorithm 2 PLANNING ( b0 ) 1 : b̄0 ← KLD-SAMPLING ( b0 ) 2 : while time permitting and l ( b̄0 ) < u ( b̄0 ) do 3 : b̄← b̄0 4 : while depth ( b̄ ) < D do 5 : if b̄ is a leaf node then 6 : EXPAND ( b̄ ) 7 : BACKUP ( b̄ ) 8 : if best action branch changes for some ancestors of b̄ or EU ( b̄ ) ≤ 0 then 9 : break 10 : a∗ ← arg maxa∈A u ( b̄ , a ) 11 : o∗ ← arg maxo∈Ob̄ , a p̂ ( o | b̄ , a ) EU ( τ ( b̄ , a ∗ , o ) )

Summary: This paper proposes an online tree-search algorithm for partially observable Markov decision processes (POMDPs). Taking sampling-based tree search style (e.g., POMCP) approach as a basis, the paper proposes to employ an adaptive particle filter that tunes the number of samples in each branch of the tree using the KLD-sampling method (Fox, 2001). To prune the search tree, the paper proposes a new method of observation aggregation called belief packing: here, observation branches resulting in similar beliefs (and hence similar values) are aggregated together. The benefits of these ideas applied to online tree search planning are demonstrated empirically.