ARMCMC: Online Model Parameters full probability Estimation in Bayesian Paradigm

1 INTRODUCTION . Bayesian methods are powerful tools to not only obtain a numerical estimate of a parameter but also to give a measure of confidence ( Kuśmierczyk et al. , 2019 ; Bishop , 2006 ; Joho et al. , 2013 ) . In particular , Bayesian inferences calculate the probability distribution of parameters rather than a point estimate , which is prevalent in frequentist paradigms ( Tobar , 2018 ) . One of the main advantages of probabilistic frameworks is that they enable decision making under uncertainty ( NoormohammadiAsl & Taghirad , 2019 ) . In addition , knowledge fusion is significantly facilitated in probabilistic frameworks ; different sources of data or observations can be combined according to their level of certainty in a principled manner ( Agand & Shoorehdeli , 2019 ) . Nonetheless , Bayesian inferences require high computational effort for obtaining the whole probability distribution and prior general knowledge on noise distribution before estimation . One of the most effective methods for Bayesian inferences is the Markov Chain Monte Carlo ( MCMC ) methods . In the field of system identification , MCMC variants such as the one recently proposed by Green ( 2015 ) are mostly focused on offline system identification . This is partly due to computational challenges which prevent real-time use ( Kuindersma et al. , 2012 ) . The standard MCMC algorithm is not suitable for model variation since different candidates do not share the same parameter set . Green ( 1995 ) first introduced reversible jump Markov chain Monte Carlo ( RJMCMC ) as a method to address the model selection problem . In this method , an extra pseudo random variable is defined to handle dimension mismatch . There are further extensions of MCMC in the literature , however , an online implication of it has yet to be reported . Motion filtering and force prediction of robotic manipulators are important fields of study with interesting challenges suitable for Bayesian inferences to address ( Saar et al. , 2018 ) . Here , measurements are inherently noisy , which is not desirable for control purposes . Likewise , inaccuracy , inaccessibility , and costs are typical challenges that make force measurement not ideal for practical use ( Agand et al. , 2016 ) . Different environmental identification methods have been proposed in the literature for linear and Gaussian noise ( Wang et al. , 2018 ) ; however , in cases of nonlinear models like HuntCrossley that does not have Gaussian noise ( e.g . impulsive disorder ) , there is no optimal solution for the identification problem . Diolaiti et al . ( 2005 ) proposed a double-stage bootstrapped method for online identification of the Hunt-Crossley model , which is sensitive to parameter initial conditions . Carvalho & Martins ( 2019 ) proposed a method to determine the damping term in the Hunt-Crossley model . A neural network-based approach was introduced to control the contact/non-contact HuntCrossley model in ( Bhasin et al. , 2008 ) This paper proposes a new technique , Adaptive Recursive Markov Chain Monte Carlo ( ARMCMC ) , to address address certain weaknesses of traditional online identification methods , such as only being appllicable to systems Linear in Parameter ( LIP ) , having Persistent Excitation ( PE ) requirements , and assuming Gaussian noise . ARMCMC is an online method that takes advantage of the previous posterior distribution , given that there is no sudden change in the parameter distribution . To achieve this , we define a new variable jump distribution that accounts for the degree of model mismatch using a temporal forgetting factor . The temporal forgetting factor is computed from a model mismatch index and determines whether ARMCMC employs modification or reinforcement to either restart or refine parameter distribution . As this factor is a function of the observed data rather than a simple user-defined constant , it can effectively adapt to the underlying dynamics of the system . We demonstrate our method using two different examples : soft bending actuator and Hunt-Crossley model and show favorable performance compared to state-of-the-art baselines . The rest of this paper is organized as follows : In Sec . 2 , introductory context about the Bayesian approach and MCMC is presented . Sec 3 is devoted to presenting the proposed ARMCMC approach in a step-by-step algorithm . Simulation results on a soft bending actuator with empirical results on a reality-based model of a soft contact environment capturing a Hunt-Crossley dynamic are presented in Sec . 4 . Lastly , the final remarks and future directions are concluded in Sec 5 . 2 PRELIMINARIES . 2.1 PROBLEM STATEMENT . In the Bayesian paradigm , estimates of parameters are given in the form of the posterior probability density function ( pdf ) ; this pdf can be continuously updated as new data points are received . Consider the following general model : Y = F ( X , θ ) + ν , ( 1 ) where Y , X , θ , and ν are concurrent output , input , model parameters and noise vector , respectively . To calculate the posterior probability , the observed data along with a prior distribution are combined via Bayes ’ rule ( Khatibisepehr et al. , 2013 ) . The data includes input/output data pairs ( X , Y ) . We will be applying updates to the posterior pdf using batches of data points ; hence , it will be convenient to partition the data as follows : Dt = { ( X , Y ) tm , ( X , Y ) tm+2 , · · · , ( X , Y ) tm+Ns+1 } , ( 2 ) where Ns = Ts/T is the number of data points in each data pack with T , Ts being the data and algorithm sampling times , respectively . This partitioning is convenient for online applications , as Dt−1 should have been previously collected so that the algorithm can be executed from tm to tm + Ns+1 or algorithm time step t. Ultimately , inferences are completed at tm+Ns+2 . Fig . 1 illustrates the timeline for the data and the algorithm . It is worth mentioning that the computation can be done in parallel by rendering the task of the adjacent algorithm step ( e.g . phase A of algorithm t , phase B of algorithm t− 1 and phase C of algorithm t− 2 can all be done simultaneously ) According to Bayes ’ rule and assuming data points are independent and identically distributed ( i.i.d ) in equation 1 , we have P ( θt| [ Dt−1 , Dt ] ) = P ( Dt|θt , Dt−1 ) P ( θt|Dt−1 ) ∫ P ( D1|θt , Dt−1 ) P ( θt|Dt−1 ) dθt , ( 3 ) where θt denotes the parameters at current time step . P ( θt|Dt−1 ) is the prior distribution over parameters , which is also the posterior distribution at the previous algorithm sampling time . P ( Dt|θt , Dt−1 ) is the likelihood function which is obtained by the one-step-ahead prediction : Ŷ t|t−1 = F ( Dt−1 , θt ) , ( 4 ) where Ŷ t|t−1 is the prediction of the output in ( 1 ) . If the model in ( 4 ) is valid , then the difference between the real output and predicted should be measurement noise , ( i.e. , Y t|t−1 − Ŷ t|t−1 = ν ) . Therefore , the model parameter may be updated as follows : P ( Dt|θt , Dt−1 ) = tm+Ns+1∏ tm+1 Pν ( Y t|t−1 − Ŷ t|t−1 ) , ( 5 ) where Pν is the probability distribution of noise . Note that there is no restriction on the type of noise probability distribution . Remark 1 : As it was mentioned before , there is no need to know the exact probability distribution of noise . This probability distribution can be simply substituted with a Gaussian distribution , if one has minimal knowledge of the mean and variance of the data which can be easily obtained with preprocessing ( Bishop , 2006 ) . 2.2 MARKOV CHAIN MONTE CARLO . MCMC is often employed to compute the posterior pdf numerically . The multidimensional integral in ( 3 ) is approximated by samples drawn from the posterior pdf . The samples are first drawn from a different distribution called proposal distribution , denoted q ( . ) , which can be sampled easier compared to the posterior . Brooks et al . ( 2011 ) discuss different types of MCMC implementations which may employ various proposal distributions and corresponding acceptance criteria . The main steps of the Metropolis-Hastings algorithm are listed as follows ( Ninness & Henriksen , 2010 ) : 1 . Set initial guess θ0 while P ( θ0|Y ) > 0 for iteration k = 1 , 2 . Draw candidate parameter θcnd , at iteration k , from the proposal distribution , q ( θcnd|θk−1 ) 3 . Compute the acceptance probability , α ( θcnd|θk−1 ) = min { 1 , P ( θcnd|D ) q ( θk−1|θcnd ) P ( θk−1|D ) q ( θcnd|θk−1 ) } , ( 6 ) 4 . Generate a uniform random number γ in [ 0 , 1 ] , 5 . ‘ Accept ’ candidate if γ ≤ α and ‘ ignore ’ it if γ > α , 6 . Set iteration to k + 1 and go to step 2 . 2.3 PRECISION AND RELIABILITY . Two important notions in probabilistic framework to compare results are precision ( ) and reliability ( δ ) . The former represents the proximity of a sample to the ground truth , and the latter represents the probability that an accepted sample lies within of the ground truth . Lemma : Let Pk be k samples from MCMC , and E ( Pk ) denote its expected value . According to Chernoff bound ( Tempo et al. , 2012 ) , given , δ ∈ [ 0 , 1 ] , if the number of samples ( k ) satisfies k ≥ 1 2 2 log ( 2 1− δ ) , ( 7 ) then Pr { { Pk − E ( Pk ) } ≤ } ≥ δ. Algorithm 1 ARMCMC Assumptions : 1 ) roughly noise mean ( µν ) 2 ) roughly noise variance ( σν ) 3 ) desired precision and reliability ( 0 , δ0 ) 4 ) desired threshold for model mismatch ( ζth ) Goal : Online calculation of parameters posterior distribution given the consecutive t-th pack of data ( P ( θt|Dt ) ) Initialization : Prior knowledge for θ01 , n=0 Consider desire precision and reliability ( , δ ) repeat Put t0 = n ∗Ns + 1 from ( 2 ) , n++ Add new data pack to dataset Dt Model mismatch index : ζt from ( 10 ) if ζt < ζth then Reinforcement : set prior knowledge equal to the latest posterior of previous pack Temporal forgetting factor : λt from ( 9 ) else Modification : set prior knowledge θn1 Temporal forgetting factor : λt = 0 end if Set minimum iteration kmin from ( 12 ) for k = 1 to kmax do Proposal distribution : • draw λk ∼ U ( 0 , 1 ) • Variable jump distribution : qtk ( . ) from ( 8 ) Draw θt∗k ∼ qtk ( . ) Acceptance rate : α ( . ) from ( 6 ) Draw γ ∼ U ( 0 , 1 ) if γ ≤ α then ‘ Accept ’ the proposal end if end for Wait to build Dtm+Ns+1t0 ( algorithm sample time ) until No data is obtained 3 ARMCMC ALGORITHM . At each time interval , ARMCMC recursively estimates the posterior distribution by drawing samples . The number of samples drawn is constrained by the desired precision and reliability , and the real-time requirement . On the other hand , the maximum number of data points in each data pack , Ns , is limited by the frequency of model variation , and the minimum is confined by the shortest required time such that the algorithm is real-time . We propose a variable jump distribution that enables both enriching and exploring . This will necessitate the definition of the temporal forgetting factor as a measure to reflect current underlying dynamics of the data . In other words , this parameter will show the validity of the previous model for the current data . We also prove that ARMCMC achieves the same precision and reliability with fewer samples compared to the traditional MCMC . Algorithm 1 summarizes ARMCMC .

Summary: The paper introduces a new Markov chain Monte-Carlo (MCMC) algorithm to obtain and track the posterior distribution over unknown parameters in a non-linear system. Despite its simple elegance, i.e., the introduction of a data-driven _temporal forgetting factor_ into the usual Metropolis-Hastings algorithm, the approach is, to my knowledge, novel. Its discovery seems to be the result of the intersection between fields: system identification and Bayesian sampling techniques, leading to new bridges.