Title: Randomized Exploration for Reinforcement Learning with Multinomial Logistic Function Approximation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement learning

Function approximation

Multinomial logistic regression

Regret analysis
Abstract: We study reinforcement learning with _multinomial logistic_ (MNL) function approximation where the underlying transition probability kernel of the _Markov decision processes_ (MDPs) is parametrized by an unknown transition core with features of state and action. For the finite horizon episodic setting with inhomogeneous state transitions, we propose provably efficient algorithms with randomized exploration having frequentist regret guarantees. For our first algorithm, $\texttt{RRL-MNL}$, we adapt optimistic sampling to ensure the optimism of the estimated value function with sufficient frequency and establish that $\texttt{RRL-MNL}$ is both _statistically_ and _computationally_ efficient, achieving a $\tilde{\mathcal{O}}(\kappa^{-1} d^{\frac{3}{2}} H^{\frac{3}{2}} \sqrt{T})$ frequentist regret bound with constant-time computational cost per episode. Here, $d$ is the dimension of the transition core, $H$ is the horizon length, $T$ is the total number of steps, and $\kappa$ is a problem-dependent constant. Despite the simplicity and practicality of $\texttt{RRL-MNL}$, its regret bound scales with $\kappa^{-1}$, which is potentially large in the worst case. To improve the dependence on $\kappa^{-1}$, we propose $\texttt{ORRL-MNL}$, which estimates the value function using local gradient information of the MNL transition model. We show that its frequentist regret bound is $\tilde{\mathcal{O}}(d^{\frac{3}{2}} H^{\frac{3}{2}} \sqrt{T} + \kappa^{-1} d^2 H^2)$. To the best of our knowledge, these are the first randomized RL algorithms for the MNL transition model that achieve both computational and statistical efficiency. Numerical experiments demonstrate the superior performance of the proposed algorithms.

Randomized Exploration for Reinforcement Learning with Multinomial Logistic Function Approximation

Abstract

We study reinforcement learning with multinomial logistic (MNL) function approximation where the underlying transition probability kernel of the Markov decision processes (MDPs) is parametrized by an unknown transition core with features of state and action. For the finite horizon episodic setting with inhomogeneous state transitions, we propose provably efficient algorithms with randomized exploration having frequentist regret guarantees. For our first algorithm, RRL-MNL , we adapt optimistic sampling to ensure the optimism of the estimated value function with sufficient frequency. We establish that RRL-MNL achieves a ˜ O ( κ -1 d 3 2 H 3 2 √ T ) frequentist regret bound with constant-time computational cost per episode. Here, d is the dimension of the transition core, H is the horizon length, T is the total number of steps, and κ is a problem-dependent constant. Despite the simplicity and practicality of RRL-MNL , its regret bound scales with κ -1 , which is potentially large in the worst case. To improve the dependence on κ -1 , we propose ORRL-MNL , which estimates the value function using the local gradient information of the MNL transition model. We show that its frequentist regret bound is ˜ O ( d 3 2 H 3 2 √ T + κ -1 d 2 H 2 ) . To the best of our knowledge, these are the first randomized RL algorithms for the MNL transition model that achieve statistical guarantees with constant-time computational cost per episode. Numerical experiments demonstrate the superior performance of the proposed algorithms.

1 Introduction

Reinforcement learning (RL) is a sequential decision-making problem in which an agent tries to maximize its expected cumulative reward by interacting with an unknown environment over time. Despite significant empirical progress in RL algorithms for various applications [47, 52, 65, 66, 25], the theoretical understanding of RL algorithms had long been limited to tabular methods [40, 56, 10, 77, 79], which explicitly enumerate the entire state and action spaces and learn the value (or the policy) for each state and action. Recently, there has been an increasing body of research in RL with function approximation to extend beyond the tabular problem setting. In particular, linear function approximation has served as a foundational model [43, 73, 22, 9, 37]. On the other hand,

the linear transition model assumption poses significant constraints: 1) the output of the function must be within [0 , 1] , and 2) the sum of the probabilities for all possible next states must be exactly 1. These constraints make it challenging to apply RL with linear function approximation to real-world applications [35]. To overcome such challenges, there has been literature on RL with general function approximation [21, 28, 37, 44, 4, 18]. Despite the guarantee of sample efficiency achieved by their algorithms, this accomplishment might be impeded by computational intractability or the necessity to rely on stronger assumptions. As a result, the resulting methods may not be as general or practical.

On the other hand, Hwang and Oh [35] introduce specific non-linear parametric MDPs called MNLMDPs (Assumption 1) where the transition probability of MDPs is given by an MNL model. They consider an upper confidence bound (UCB) approach to balance exploration and exploitation. Since it is costly or even intractable to compute UCB explicitly, randomized exploration methods such as Thompson Sampling (TS) are widely studied in RL with linear function approximation as well as tabular MDPs. This is because, in various decision-making problems ranging from multi-armed bandits to RL, randomized exploration algorithms have been shown to perform better than UCB methods in empirical evaluations [16, 57, 64, 49]. Furthermore, randomized exploration can be easily integrated with linear function approximation. This is because the value function in linear MDPs can be linearly parameterized, allowing perturbations of the estimator to directly control the perturbations of the value function. However, although there has been some literature aiming to propose randomized algorithms for general function classes [37, 4, 5, 75], these methods do not discuss how to define the posterior distribution supported by the given function class and how to draw the optimistic sample from the posterior [4, 5, 75], or they require stronger assumptions on stochastic optimism [37], which is one of the most challenging elements in frequentist regret analysis. Thus, the design of a tractable randomized exploration RL algorithm and the feasibility of frequentist regret analysis for randomized exploration remain open challenges. Hence, the following question arises:

Can we design a provably efficient and tractable randomized algorithm for RL with MNL function approximation?

We answer the above question by proposing the first randomized algorithm, RRL-MNL , achieving ˜ O ( κ -1 d 3 2 H 3 2 √ T ) frequentist regret with constant-time computational cost per episode. RRL-MNL is not only the first algorithm with randomized exploration for MNL-MDPs, but also, to the best of our knowledge, it provides the first frequentist regret analysis for a non-linear model-based algorithm with randomized exploration without assuming stochastic optimism [37].

While RRL-MNL is statistically efficient, the current method used to analyze the regret of MNL function approximation introduces a problem-dependent constant κ (Assumption 4), which reflects the level of non-linearity of the MNL transition model. This constant κ originates from the use of generalized linear models (GLMs) for contextual bandit settings [26, 51, 45] and MNL bandit settings [54, 17, 55]. The magnitude of the constant κ can be exponentially small with respect to the size of the decision set, hence the regret bound scaling with κ -1 could be prohibitively large in the worst case [23]. Worse yet, the situation is even more challenging in RL, as in the worst case, κ -1 can be much larger than in the case of bandits. To overcome the prohibitive dependence on κ , algorithms based on new Bernstein-like inequalities and the self-concordant-like property of the log-loss have been proposed for logistic bandits [23, 3, 24] and for MNL bandits [61, 6, 50]. As an extension of these works, the following fundamental question remains open:

Is it possible for RL algorithms with MNL function approximation to have a sharper dependence on the problem-dependent constant κ ?

For the above question, we propose the second randomized algorithm referred to as ORRL-MNL , which establishes a regret bound of ˜ O ( d 3 2 H 3 2 √ T + κ -1 d 2 H 2 ) with constant-time computational cost per episode. We summarize our main contributions as follows:

We propose computationally tractable randomized algorithms for RL with MNL function approximation: RRL-MNL and ORRL-MNL . To the best of our knowledge, these are the first randomized model-based RL algorithms with MNL function approximation that achieve the frequentist regret bounds with constant-time computational cost per episode.

We establish that RRL-MNL enjoys ˜ O ( κ -1 d 3 2 H 3 2 √ T ) frequentist regret bound with constanttime computational cost per episode, where d is the dimension of the transition core, H is horizon length, T is the total number of rounds, and κ is a problem-dependent constant. We

derive the stochastic optimism of RRL-MNL , and to our knowledge, this is the first frequentist regret analysis for a non-linear model-based algorithm with randomized exploration without assuming stochastic optimism.

To achieve a regret bound with improved dependence on κ , we introduce ORRL-MNL , which constructs the optimistic randomized value functions by taking into account the effects of the local gradient information for the MNL transition model at each reachable state. We prove that ORRL-MNL enjoys an ˜ O ( d 3 2 H 3 2 √ T + κ -1 d 2 H 2 ) regret with constant-time computational cost per episode, significantly improving the regret of RRL-MNL without requiring prior knowledge of κ .

We evaluate our algorithms on tabular MDPs and demonstrate the superior performance of our proposed algorithms compared to the existing state-of-the-art MNL-MDP algorithm [35]. The experiments provide evidence that our proposed algorithms are both computationally and statistically efficient.

Related works on RL with function approximation and MNL contextual bandits are provided in Appendix A.

2 Problem Setting

We consider the episodic Markov decision processes (MDPs) denoted by M ( S , A , H, { P } H h =1 , r ) , where S is the state space, A is the action space, H is the horizon length of each episode, { P } H h =1 is the collection of probability distributions, and r is the reward function. Every episodes start from the initial state s 1 and for every step h ∈ [ H ] := { 1 , ..., H } in an episode, the learning agent interacts with the environment represented as M . The agent observes the state s h ∈ S , chooses an action a h ∈ A , receives a reward r ( s h , a h ) ∈ [0 , 1] and the next state s h +1 is given by the transition probability distribution P h ( ·| s h , a h ) . Then this process is repeated throughout the episode. A policy π : S × [ H ] → A is a function that determines the action of the agent at state s h , i.e., a h = π ( s h , h ) := π h ( s h ) .

We define the value function of the policy π , denoted by V π h ( s ) , as the expected sum of rewards under the policy π until the end of the episode starting from s h = s , i.e., V π h ( s ) =

r ( s, a ) + E s ′ ∼ P h ( ·| s,a ) [ V π h +1 ( s ′ ) ] . We define an optimal policy π ∗ to be a policy that achieves the highest possible value at every ( s, h ) ∈ S × [ H ] . We denote the optimal value function by V ∗ h ( s ) = V π ∗ h ( s ) and the optimal action-value function by Q ∗ h ( s, a ) = Q π ∗ h ( s, a ) . To simplify, we introduce the notation P h V h +1 ( s, a ) = E s ′ ∼ P h ( ·| s,a ) [ V h +1 ( s ′ )] . Recall that the Bellman equations are,

The goal of the agent is to maximize the sum of rewards for K episodes. In other words, the goal is to minimize the cumulative regret of the policy π over K episodes where π = { π k } K k =1 is a collection of policies π k at k-th episode. The regret is defined as

where s k 1 is the initial state at the k -th episode.

2.1 Multinomial Logistic Markov Decision Processes (MNL-MDPs)

Even though a lot of provable RL algorithms for linear MDPs are proposed, there is a simple but fundamental problem with the linear transition model assumption on the linear MDPs. In other words, the output of a linear function approximating the transition model must be in [0 , 1] and the probability of all possible following states must sum to 1 exactly. Such restrictive assumption can affect the regret performances of algorithm suggested under the linearity assumption. To resolve

these challenges, Hwang and Oh [35] propose a setting of a multinomial logistic Markov decision processes (MNL-MDPs), where the state transition model is given by a multinomial logistic model. We introduce the formal definition for MNL-MDP as follows:

̸

Assumption 1 (MNL-MDPs [35]) . An MDP M ( S , A , H, { P h } H h =1 , r ) is an MNL-MDP with a feature map φ : S × A × S → R d , if for each h ∈ [ H ] , there exists θ ∗ h ∈ R d , such that for any ( s, a ) ∈ S × A and s ′ ∈ S s,a := { s ′ ∈ S : P ( s ′ | s, a ) = 0 } , the state transition kernel of s ′ when an action a is taken at a state s is given by,

We call each unknown vector θ ∗ h transition core. Furthermore, we denote the maximum cardinality of the set of reachable states as U , i.e., U := max s,a |S s,a | .

Remark 1. While Hwang and Oh [35] assume a homogeneous transition kernel, we assume an inhomogeneous transition kernel, in which the probability varies depending on the current time step h even for the same state transition, which is a more general setting. Also, for notational simplicity, we denote the true transition kernel P h as P θ ∗ h , and the estimated transition kernel by θ as P θ .

2.2 Assumptions

We introduce some standard regularity assumptions.

Assumption 2 (Boundedness) . We assume ∥ φ ( s, a, s ′ ) ∥ 2 ≤ L φ for all ( s, a, s ′ ) ∈ S × A × S s,a , and ∥ θ ∗ h ∥ 2 ≤ L θ for all h ∈ [ H ] .

Assumption 3 (Known reward) . We assume that the reward function r is known to the agent.

̸

Assumption 4 (Problem-dependent constant) . Let B d ( L θ ) := { θ ∈ R d : ∥ θ ∥ 2 ≤ L θ } . There exists κ > 0 such that for any ( s, a ) ∈ S × A and s ′ , ˜ s ∈ S s,a with s ′ = ˜ s ,

Discussion of assumptions Assumption 2 is common in the literature on RL with function approximation [43, 72, 73, 37, 35] to make the regret bounds scale-free. Assumption 3 is used to focus on the main challenge of model-based RL that learning about P of the environment is more difficult than learning r . In the model-based RL literature [71, 9, 72, 81, 35], the known reward r assumption is widely used. Assumption 4 is typical in generalized linear contextual bandit [26, 51, 23, 3, 24] and MNL contextual bandit literature [54, 8, 55, 61, 6, 76, 50] to guarantee non-singular Fisher information matrix.

3 Randomized Algorithm for MNL-MDPs having constant-time computational cost

Previous work for MNL-MDPs [35] proposed a UCB-based exploration algorithm. Constructing a UCB-based optimistic value function is not only computationally intractable but also tends to overly optimistically estimate the true optimal value function. Additionally, their algorithm incurs increasing computation costs as episodes progress, as it requires all samples from the previous episode to estimate the transition core. In this section, we present a novel model-based RL algorithm that incorporates randomized exploration and online parameter estimation for MNL-MDPs.

3.1 Algorithm: RRL-MNL

Online transition core estimation While Hwang and Oh [35] estimate the transition core using maximum likelihood estimation over all samples from previous episodes, we employ an efficient online parameter estimation method by exploiting the particular structure of the MNL transition model. The key insight is that the negative log-likelihood function for the MNL model in each episode is strongly convex over a bounded domain. This property allows us to utilize a variation of the online Newton step [30, 31], which inspired online algorithms for logistic bandits [74] and MNL contextual bandits [55]. Specifically, for ( k, h ) ∈ [ K ] × [ H ] , we define the response variable y k h =

Algorithm 1 RRL-MNL (Randomized RL for MNL-MDPs)

1: Inputs: Episodic MDP M , Feature map φ : S × A × S → R d , Number of episodes K , Regularization parameter λ , Exploration variance { σ k } K k =1 , Sample size M , Problem-dependent constant κ

2: Initialize: θ 1 h = 0 d , A 1 ,h = λ I d for h ∈ [ H ]

3: for episode k = 1 , 2 , · · · , K do

4: Observe s k 1 and sample i.i.d. noise vector ξ ( m ) k,h ∼ N ( 0 d , σ 2 k A -1 k,h ) for m ∈ [ M ] and h ∈ [ H ]

5: Set { Q k h ( · , · ) } h ∈ [ H ] as described in (4)

6: for horizon h = 1 , 2 , · · · , H do

7: Select a k h = argmax a ∈A Q k h ( s k h , a ) and observe s k h +1

8: Update A k +1 ,h = A k,h + κ 2 ∑ s ′ ∈S k,h φ ( s k h , a k h , s ′ ) φ ( s k h , a k h , s ′ ) ⊤ and θ k +1 h as in (2)

9: end for

10: end for

[ y k h ( s ′ ) ] s ′ ∈S k,h such that y k h ( s ′ ) = 1 ( s k h +1 = s ′ ) for s ′ ∈ S k,h := S s k h ,a k h . Then, y k h is sampled from the following multinomial distribution: y k h ∼ multinomial(1 , [ P θ ∗ h ( s ′ | s k h , a k h ) ] s ′ ∈S k,h ) , where 1 represents that y k h is a single-trial sample. We define the per-episode loss ℓ k,h ( θ ) as follows:

Then, the estimated transition core for θ ∗ h is given by

where θ 1 h can be initialized as any point in B d ( L θ ) and A k,h is the Gram matrix defined by

Stochastically optimistic value function First of all, we introduce the key challenges of regret analysis for randomized algorithms, explain how previous works have overcome these challenges, and then describe why the techniques from previous works cannot be applied to MNL-MDPs. Ensuring that the estimated value function is optimistic with sufficient frequency is a crucial challenge in analyzing the frequentist regret of randomized algorithms. A common way to promote sufficient exploration in randomized algorithms is by perturbing the estimated value function or by performing posterior sampling in the transition model class. Frequentist regret analysis of randomized exploration in an RL setting has been conducted for tabular [59, 7, 62, 60, 67], linear MDPs [73, 37], and general function classes [37, 4, 5, 75]. In the case of linear MDPs [73, 37], since the property that the action-value function is linear in the feature map allows perturbing the estimated parameter directly to control the perturbation of the estimated value function. Also, even though Ishfaq et al. [37] presented a randomized algorithm for the general function class using eluder dimension, they assume stochastic optimism (anti-concentration), which is in fact one of the most challenging aspects of frequentist analysis. Other posterior sampling algorithms in RL for the general function class such as [4, 5, 75], except for very limited examples, do not discuss how to define the posterior distribution supported by the given function class and how to draw the optimistic sample from the posterior. That is why even after there exists a so-called general function class -based result, it is often the case that results in specific parametric models are still needed.

Note that in episodic RL, the perturbed estimated value functions are propagated back through horizontal steps, requiring careful adjustment of the perturbation scheme to maintain a sufficient probability of optimism without decaying too quickly with the horizon. For example, if the probability of the estimated value function being optimistic at horizon h is denoted as p , this would result in the probability that the estimated value function in the initial state is optimistic being on the order of p H , implying that the regret can increase exponentially with the length of the horizon H .

Additionally, the non-linearity and substitution effect of the next state transition in the MNL-MDPs make applying the existing TS techniques infeasible to guarantee optimism in MNL-MDPs with sufficient frequency. Instead, we design the stochastically optimistic value function by exploiting the structure of the MNL transition model. In other words, the prediction error of MNL transition model (Definition 1) can be bounded by the weighted norm of the dominant feature ˆ φ (Lemma 4). Based on such dominant feature, we perturb the estimated value function by injecting Gaussian noise whose variance is proportional to the inverse of the Gram matrix to encourage the perturbation with higher variance in less explored directions. To guarantee the optimism with fixed probability, we adapt optimistic sampling technique [7, 54, 37, 36]. For each m ∈ [ M ] , sample i.i.d. Gaussian noise vector ξ ( m ) k,h ∼ N ( 0 d , σ 2 k A -1 k,h ) where σ k is an exploration parameter, and add the most optimistic inner product value max m ∈ [ M ] ˆ φ k,h ( s, a ) ⊤ ξ ( m ) k,h to the estimated value function. To summarize for any ( s, a ) ∈ S × A , set Q k H +1 ( s, a ) = 0 and for h ∈ [ H ] ,

where V k h ( s ) = max a ′ Q k h ( s, a ′ ) and ˆ φ k,h ( s, a ) := φ ( s, a, ˆ s ) for ˆ s = argmax s ′ ∈S s,a ∥ φ ( s, a, s ′ ) ∥ A -1 k,h . Based on these stochastically optimistic value function, the agent plays a greedy action a k h = argmax a ′ Q k h ( s k h , a ′ ) . We layout the procedure in Algorithm 1.

Remark 2. Note that RRL-MNL only requires constant-time computational cost and storage cost per episode, as it does not require storing all samples from previous episodes, and the Gram matrix A k,h can be updated incrementally.

3.2 Regret bound of RRL-MNL

We present the regret upper bound of RRL-MNL . The complete proof is deferred to Appendix C.

Theorem 1 (Regret Bound of RRL-MNL ) . Suppose that Assumption 1- 4 hold. For any 0 < δ < Φ( -1) 2 , if we set the input parameters in Algorithm 1 as λ = L 2 φ , σ k = ˜ O ( H √ d ) and M = ⌈ 1 -log H log Φ(1) ⌉ where Φ is the normal CDF, then with probability at least 1 -δ , the cumulative regret of the RRL-MNL policy π is upper-bounded as follows:

where T = KH is the total number of steps.

Discussion of Theorem 1 To our best knowledge, this is the first result to provide a frequentist regret bound for the MNL-MDPs. Among the previous RL algorithms using function approximation, the most comparable techniques to our method are model-free algorithms with randomized exploration [73, 37]. To guarantee stochastic optimism, Zanette et al. [73] established a lower bound on the difference between the estimated value and the optimal value by the summation of linear terms with respect to the average feature (Lemma F.1 in [73]). This property is achievable due to the linear expression of the value function in linear MDPs. Instead, we established a lower bound on the difference between value functions by the summation of the Bellman errors (Definition 1) along the sample path obtained through the optimal policy (Lemma 7). Hence, our analysis significantly differs from that of Zanette et al. [73] since the value function in MNL-MDPs is no longer linearly parametrized, and there is no closed-form expression for it.

Compared to [37], they also used an optimistic sampling technique; however, our theoretical sampling size M = O (log H ) is much tighter than that of [37], i.e., O ( d ) for the linear function class, O (log( T |S||A| )) for the general function class. While Ishfaq et al. [37] extend the results of the linear function class to general function class under the assumption of stochastic optimism (Assumption C in [37]), we provide the frequentist regret analysis for a non-linear model-based algorithm with randomized exploration without assuming stochastic optimism .

Compared to the optimistic exploration algorithm for MNL-MDPs [35], our randomized exploration requires a more involved proof technique to ensure that the perturbation of the estimated value function has enough variance to maintain optimism with sufficient frequency (Lemma 6). As a result,

the established regret of RRL-MNL differs by a factor of √ d , which aligns with the difference in the existing bounds of linear bandits between a TS-based algorithm [2] and a UCB-based algorithm [1]. Additionally, we achieve statistical efficiency for the inhomogeneous transition model , which is a more general setting than that of Hwang and Oh [35]. Our computation cost per episode is O (1) while the computation cost per episode of Hwang and Oh [35] is O ( K ) .

Proof Sketch of Theorem 1 We provide the proof sketch of Theorem 1. By decomposing the regret into the estimation part and the pessimism part, we have

We bound these two parts separately. For the estimation part, for each k ∈ [ K ] , h ∈ [ H ] , we first show that the online estimated transition core θ k h (2) concentrates around the unknown transition core parameter θ ∗ h with high probability (Lemma 1). Then, we show that the prediction error induced by the estimated transition core can be bounded by the weighted norm of the dominant feature ˆ φ , multiplied by the confidence radius of the estimated transition core (Lemma 4). The bounded prediction error, together with the concentration of Gaussian noise, implies the desired bound on the estimation part (Lemma 10). For the pessimism part, we first show that the stochastically optimistic value function V k 1 is optimistic than the true optimal value function V ∗ 1 with sufficient frequency (Lemma 6). In the next step, we show that the pessimism part is upper bounded by a bound of the estimation part times the inverse probability of being optimistic (Lemma 11). Combining all the results, we can conclude the proof. Refer to Appendix C for detailed proofs.

4 Statistically Improved Algorithm for MNL-MDPs

Although RRL-MNL is provably efficient and achieves constant-time computational cost per episode, the current analysis makes its regret bound scale with κ -1 . Recall that the problem-dependent constant κ introduced in Assumption 4 indicates the curvature of the MNL function, i.e., how difficult it is to learn the true transition core parameter. It is required to ensure the non-singular Fisher information matrix, hence is typically used in GLM or MNL bandit algorithms that use the maximum likelihood estimator. As introduced in Faury et al. [23], κ -1 can be exponentially large in the worst case. The appearance of κ in existing bounds originates in the connection between the difference of estimators and the difference of gradients of negative log-likelihood, usually denoted as G in Filippi et al. [26]. Without considering local information at all, using a loose lower bound for G incurs κ -1 in regret bound (see Section 4.1 in Agrawal et al. [6]). Recently, improved dependence on κ has been achieved in bandit literature [23, 3, 61, 6, 76, 50] through the use of generalization of the Bernstein-like tail inequality [23] and the self-concordant-like property of the log loss [11]. However, a direct adaptation of the MNL bandit technique would result in sub-optimal dependence on the assortment size in MNL bandit, which corresponds to the size of the set of reachable states, such as U . In this section, we introduce a new randomized algorithm for MNL-MDPs, equipped with a tight online parameter estimation and feature centralization technique that achieves a regret bound with improved dependence on κ and U .

4.1 Algorithms: ORRL-MNL

Tight online transition core estimation Zhang and Sugiyama [76] presented a jointly efficient UCB-based MNL contextual bandit algorithm using online mirror descent algorithm. Adapting the update rule from [76], the estimated transition core run by the online mirror descent is given by

where ˜ θ 1 h can be initialized as any point in B d ( L θ ) , η is a step size, and ˜ B k,h is defined as

Algorithm 2 ORRL-MNL (Optimistic Randomized RL for MNL-MDPs)

1: Inputs: Episodic MDP M , Feature map φ : S × A × S → R d , Number of episodes K , Regularization parameter λ , Exploration variance { σ k } K k =1 , Confidence radius { β k } K k =1 , Sample size M , Step size η

2: Initialize: ˜ θ 1 h = 0 d , B 1 ,h = λ I d for all h ∈ [ H ]

3: for episode k = 1 , 2 , · · · , K do

4: Observe s k 1 and sample i.i.d. noise vector ξ ( m ) k,h ∼ N ( 0 d , σ 2 k B -1 k,h ) for m ∈ [ M ] and h ∈ [ H ]

5: Set { ˜ Q k h ( · , · ) } h ∈ [ H ] as described in (7)

6: for horizon h = 1 , 2 , · · · , H do

7: Select a k h = argmax a ∈A ˜ Q k h ( s k h , a ) and observe s k h +1

8: Update ˜ B k,h = B k,h + η ∇ 2 ℓ k,h ( ˜ θ k h ) and ˜ θ k +1 h as in (5)

9: Update B k +1 ,h = B k,h + ∇ 2 ℓ k,h ( ˜ θ k +1 h )

10: end for

11: end for

Note that the MNL model in Zhang and Sugiyama [76] operates in a multiple-parameter setting, where there are multiple unknown choice parameters and one given context feature. In contrast, our MNL model operates in a single-parameter setting, where there is one unknown transition core and features for up to U reachable states. This difference results in variations in applying the self-concordant-like property of the log-loss for the MNL model. For instance, Zhang and Sugiyama [76] utilized the fact that the log-loss for the multiple parameter MNL model is √ 6 -self-concordantlike (Lemma 2 in Zhang and Sugiyama [76]). On the other hand, Lee and Oh [50] revisit the self-concordant-like property and demonstrate that the log-loss of the single-parameter MNL model is 3 √ 2 -self-concordant-like (Proposition B.1 in Lee and Oh [50]). This results in a concentration bound that is independent of κ and U , introduced in Lemma 12.

Remark 3. Note that the online estimated parameters θ k h (2) and ˜ θ k h (5) do not aim to minimize the sum of negative log-likelihoods, ∑ k k ′ =1 ℓ k ′ ,h ( θ ) . Instead, we show that the online estimated parameter concentrates around the unknown transition core θ ∗ h with high probability (Lemma 1 & 12). This online update approach allows us to estimate the transition core with constant-time computational cost per episode, as the agent does not need to store all samples from previous episodes.

Optimistic randomized value function To achieve improved dependence on κ , a crucial point is to utilize the local gradient information of MNL transition probabilities for each reachable state when constructing the Gram matrix. In MNL bandit problems [61, 76], this can be accomplished by substituting the Hessian of the negative log-likelihood with the Gram matrix using global gradient information κ . However, there are fundamental differences between the settings in Perivier and Goyal [61], Zhang and Sugiyama [76] and ours. Perivier and Goyal [61] address the case where the reward for each product is uniform (i.e., all products have a reward of 1), and the reward for not selecting a product from the given assortment (also known as the outside option) is 0. On the other hand, Zhang and Sugiyama [76] deal with non-uniform rewards where the reward for each product may vary; however, the rewards for individual products are known a priori to the agent. In contrast, in MNL-MDPs, the value for each reachable state may vary (non-uniform) and is not known beforehand. Due to these differences, the analysis techniques in MNL bandits [61, 76] cannot be directly applied to our setting. Instead, we adapt the feature centralization technique [50]. Then, the Hessian of the per-round loss ℓ k,h ( θ ) is expressed in terms of the centralized feature as follows:

where ¯ φ ( s, a, s ′ ; θ ) := φ ( s, a, s ′ ) -E ˜ s ∼ P θ ( ·| s,a ) [ φ ( s, a, ˜ s )] is the centralized feature by θ . For more details, please refer to Appendix D.2.

Now we introduce the optimistic randomized value function ˜ Q k h ( · , · ) for ORRL-MNL . The key point is that when perturbing the estimated value function, we use the centralized feature by the estimated

transition parameter ˜ θ k h . For any ( s, a ) ∈ S × A , set ˜ Q k H +1 ( s, a ) = 0 and for each h ∈ [ H ] ,

where ˜ V k h ( s ) := max a ∈A ˜ Q k h ( s, a ) and ν rand k,h ( s, a ) is the randomized bonus term defined by

Here we sample i.i.d. Gaussian noise ξ ( m ) k,h ∼ N ( 0 d , σ 2 k B -1 k,h ) for each m ∈ [ M ] and set ξ s ′ k,h := ξ m ( s ′ ) k,h where m ( s ′ ) := argmax m ∈ [ M ] ¯ φ ( s, a, s ′ ; ˜ θ k h ) ⊤ ξ m k,h is the most optimistic sampling index for a reachable state s ′ . Based on these optimistic randomized value function, at each episode the agent plays a greedy action with respect to ˜ Q k h as summarized in Algorithm 2.

Remark 4. Note that the second term in the randomized bonus always has a positive value, but it rapidly decreases as episode proceeds. While due to the randomness of ξ , the randomized bonus ν rand k,h itself cannot be guaranteed to always have a positive value. Consequently, the constructed value function ˜ Q k h ( · , · ) can be optimistic or pessimistic. However, as shown in Lemma 18, optimistic sampling technique ensures that the optimistic randomized value function ˜ Q k h has at least a constant probability of being optimistic than the true optimal value function.

Remark 5. As with RRL-MNL , since the transition core is estimated in an online manner and the Gram matrices with local gradient information B k,h and ˜ B k,h are updated incrementally, ORRL-MNL also requires constant-time computational cost and storage cost per-episode. Although ORRL-MNL requires an additional O ( U ) computation cost for feature centralization, the computation complexity order is the same as that of RRL-MNL because it also needs to go over reachable states to calculate the dominant feature ˆ φ , which also incurs a O ( U ) computation cost. On the other hand, ORRL-MNL does not require prior knowledge of κ and achieves a regret with a better dependence on κ .

4.2 Regret Bound of ORRL-MNL

We present the regret upper bound of ORRL-MNL . The complete proof is deferred to Appendix D. Theorem 2 (Regret Bound of ORRL-MNL ) . Suppose that Assumption 1- 4 hold. For any 0 < δ < Φ( -1) 2 , if we set the input parameters in Algorithm 2 as λ = O ( L 2 φ d log U ) , β k = O ( √ d log U log( kH )) , σ k = Hβ k , M = ⌈ 1 -log( H U ) log Φ(1) ⌉ , and η = O (log U ) , then with probability at least 1 -δ , the cumulative regret of the ORRL-MNL policy π is upper-bounded as follows:

where T = KH is the total number of time steps.

Dicussion of Theorem 2 Theorem 2 establishes that the leading term in the regret bound does not suffer from the problem-dependent constant κ -1 and the second term of the regret bound is independent of the size of set of reachable states. To the extent of our knowledge, this is the first algorithm that provides a frequentist regret guarantee with improved dependence on κ -1 in MNLMDPs. Compared to RRL-MNL , the technical challenge lies in ensuring the stochastic optimism of the estimated value for ORRL-MNL . Note that the prediction error (Definition 1) for ORRL-MNL is characterized by two components: one related to the gradient information of the MNL transition model at each reachable state, and the other related to the dominant feature with respect to the Gram matrix B k,h (Lemma 16). Hence, the probability of the Bellman error at each horizon, when following the optimal policy, being negative can depend on the size of the reachable states. This implies that the probability of stochastic optimism can be exponentially small, not only in the horizon H but also in the size of the reachable states U . However, as shown in Lemma 18, this challenge has been overcome by using a sample size M that logarithmically increases with U , effectively addressing the issue.

0

2000

4000

6000

8000

10000

The Number of Episodes

1.0

1.5

2.0

2.5

3.0

RRL-MNL

UCRL-MNL

ORRL-MNL

Optimal Policy

UCRL-MNL+

(a)

S

= 4

, H

= 12

0

2000

4000

6000

8000

10000

The Number of Episodes

1.0

1.5

2.0

2.5

3.0

RRL-MNL

UCRL-MNL

ORRL-MNL

Optimal Policy

UCRL-MNL+

(b)

S

= 8

, H

= 24

RRL-MNL

UCRL-MNL

ORRL-MNL

UCRL-MNL+

Algorithms

0

1000

2000

3000

4000

Time (seconds)

20.3

4302.3

76.2

70.1

RRL-MNL

UCRL-MNL

ORRL-MNL

UCRL-MNL+

(c) Runtime for 1,000 episodes

Proof Sketch of Theorem 2 The overall proof pipeline for Theorem 2 is similar to that of Theorem 1. The main differences lie in the concentration of the estimated transition core (Lemma D.2), the bound on the prediction error (Lemma D.2), and the stochastic optimism (Lemma 18). Please refer to Appendix D for detailed proofs.

Optimistic exploration extension In general, since TS-based randomized exploration requires a more rigorous proof technique than UCB-based algorithms, our technical ingredients enable the use of optimistic exploration in a straightforward manner. We introduce UCRL-MNL+ (Algorithm 3) in the Appendix E, an optimism-based algorithm for MNL-MDPs. It is both computationally and statistically efficient compared to UCRL-MNL [35], achieving the tightest regret bound for MNLMDPs.

Corollary 1. UCRL-MNL+ (Algorithm 3) has ˜ O ( dH 3 / 2 √ T + κ -1 d 2 H 2 ) regret with high probability.

5 Numerical Experiments

We perform a numerical evaluation on a variant of RiverSwim [58] to demonstrate practicality of our proposed algorithms. We compare our algorithms ( RRL-MNL , ORRL-MNL , UCRL-MNL+ ) with the state-of-the-art UCRL-MNL [35] for MNL-MDPs. For each configuration, we report the averaged results over 10 independent runs. Figure 1a and 1b show the episodic return of each algorithm, which is the sum of all the rewards obtained in one episode. First, our proposed algorithms ( RRL-MNL , ORRL-MNL , UCRL-MNL+ ) outperform UCRL-MNL [35] for both cases of |S| = 4 , 8 . Second, ORRL-MNL and UCRL-MNL+ reach the optimal values quickly compared to the other algorithms, demonstrating improved statistical efficiency. Figure 1c illustrates the comparison in running time of the algorithms for the first 1,000 episodes. Our proposed algorithms are at least 50 times faster than UCRL-MNL . These differences become more pronounced as the episodes progress because our algorithms have a constant computation cost, whereas the computation cost of UCRL-MNL increases over time.

6 Conclusions

We propose randomized algorithms with provable efficiency and constant-time computational cost for MNL-MDPs. For the first algorithm, RRL-MNL , we use an optimistic sampling technique to ensure the stochastic optimism of the estimated value functions and provide the frequentist regret analysis. This is the first frequentist regret analysis for a non-linear model-based algorithm with randomized exploration without assuming stochastic optimism. To achieve a statistically improved regret bound, we propose ORRL-MNL by constructing the optimistic randomized value function using the effects of the local gradient of the MNL transition model equipped with the centralized feature. As a result, we achieve a frequentist regret guarantee with improved dependence on κ in RL with the MNL transition model, which is a significant contribution. The effectiveness and practicality of our methods are supported by numerical experiments.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24:2312-2320, 2011.

Marc Abeille and Alessandro Lazaric. Linear Thompson Sampling Revisited. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54 of Proceedings of Machine Learning Research , pages 176-184. PMLR, PMLR, 20-22 Apr 2017.

Marc Abeille, Louis Faury, and Clément Calauzènes. Instance-wise minimax-optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 3691-3699. PMLR, 2021.

Alekh Agarwal and Tong Zhang. Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. Advances in Neural Information Processing Systems , 35: 35284-35297, 2022.

Alekh Agarwal and Tong Zhang. Non-linear reinforcement learning in large action spaces: Structural conditions and sample-efficiency of posterior sampling. In Conference on Learning Theory , pages 2776-2814. PMLR, 2022.

Priyank Agrawal, Theja Tulabandhula, and Vashist Avadhanula. A tractable online learning algorithm for the multinomial logit contextual bandit. European Journal of Operational Research , 2023.

Shipra Agrawal and Randy Jia. Posterior sampling for reinforcement learning: worst-case regret bounds. In Advances in Neural Information Processing Systems , pages 1184-1194, 2017.

Sanae Amani and Christos Thrampoulidis. Ucb-based algorithms for multinomial logistic regression bandits. Advances in Neural Information Processing Systems , 34:2913-2924, 2021.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics , 4(2):384 - 414, 2010.

Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local rademacher complexities. The Annals of Statistics , 2005.

Steven J Bradtke and Andrew G Barto. Linear least-squares algorithms for temporal difference learning. Machine learning , 22(1):33-57, 1996.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Nicolo Campolongo and Francesco Orabona. Temporal variability in implicit online learning. Advances in neural information processing systems , 33:12377-12387, 2020.

Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. Advances in neural information processing systems , 24, 2011.

Xi Chen, Yining Wang, and Yuan Zhou. Dynamic assortment optimization with changing contextual information. Journal of machine learning research , 2020.

Zixiang Chen, Chris Junchi Li, Huizhuo Yuan, Quanquan Gu, and Michael Jordan. A general framework for sample-efficient function approximation in reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023.

Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. On oracle-efficient pac rl with rich observations. In Advances in Neural Information Processing Systems , volume 31, 2018.

Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR, 2021.

Simon S. Du, Sham M. Kakade, Ruosong Wang, and Lin F. Yang. Is a good representation sufficient for sample efficient reinforcement learning? In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020.

Louis Faury, Marc Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning , pages 3052-3060. PMLR, 2020.

Louis Faury, Marc Abeille, Kwang-Sung Jun, and Clément Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 546-580. PMLR, 2022.

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, 2022.

Sarah Filippi, Olivier Cappé, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 1 , NIPS'10, page 586-594, Red Hook, NY, USA, 2010. Curran Associates Inc.

Dylan J Foster, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Logistic regression: The importance of being improper. In Conference On Learning Theory , pages 167-208. PMLR, 2018.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2):169-192, 2007.

Elad Hazan, Tomer Koren, and Kfir Y Levy. Logistic regression: Tight bounds for stochastic and online optimization. In Conference on Learning Theory , pages 197-209. PMLR, 2014.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Logarithmic regret for reinforcement learning with linear function approximation. In International Conference on Machine Learning , pages 4171-4180. PMLR, 2021.

Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear markov decision processes. In International Conference on Machine Learning , pages 12790-12822. PMLR, 2023.

Taehyun Hwang and Min-hwan Oh. Model-based reinforcement learning with multinomial logistic function approximation. In Proceedings of the AAAI conference on artificial intelligence , pages 7971-7979, 2023.

Taehyun Hwang, Kyuwook Chai, and Min-Hwan Oh. Combinatorial neural bandits. In Proceedings of the 40th International Conference on Machine Learning . PMLR, 2023.

Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang, Zhaoran Wang, Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning , volume 139, pages 4607-4616. PMLR, PMLR, 2021.

Haque Ishfaq, Qingfeng Lan, Pan Xu, A. Rupam Mahmood, Doina Precup, Anima Anandkumar, and Kamyar Azizzadenesheli. Provable and practical: Efficient exploration in reinforcement learning via langevin monte carlo. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=nfIAEJFiBZ .

Haque Ishfaq, Yixin Tan, Yu Yang, Qingfeng Lan, Jianfeng Lu, A Rupam Mahmood, Doina Precup, and Pan Xu. More efficient randomized exploration for reinforcement learning via approximate sampling. Reinforcement Learning Journal , 3(1), 2024.

Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement

learning. Journal of Machine Learning Research , 11(4), 2010.

Zeyu Jia, Lin Yang, Csaba Szepesvari, and Mengdi Wang. Model-based reinforcement learning with value-targeted regression. In Learning for Dynamics and Control , pages 666-686. PMLR, 2020.

Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR, 2017.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021.

Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett. Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems , 30, 2017.

Yeoneung Kim, Insoon Yang, and Kwang-Sung Jun. Improved regret analysis for varianceadaptive linear bandits and horizon-free linear mixture mdps. Advances in Neural Information Processing Systems , 35:1060-1072, 2022.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Pac reinforcement learning with rich observations. Advances in Neural Information Processing Systems , 29:1840-1848, 2016.

Branislav Kveton, Csaba Szepesvári, Mohammad Ghavamzadeh, and Craig Boutilier. Perturbedhistory exploration in stochastic linear bandits. In Uncertainty in Artificial Intelligence , pages 530-540. PMLR, 2020.

Joongkyu Lee and Min-hwan Oh. Nearly minimax optimal regret for multinomial logistic bandit. arXiv preprint arXiv:2405.09831 , 2024.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Aditya Modi, Nan Jiang, Ambuj Tewari, and Satinder Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics , pages 2010-2020. PMLR, 2020.

Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. Advances in Neural Information Processing Systems , 32:3151-3161, 2019.

Min-hwan Oh and Garud Iyengar. Multinomial logit contextual bandits: Provable optimality and practicality. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 9205-9213, 2021.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. In Advances in Neural Information Processing Systems , pages 1466-1474, 2014.

Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learning , pages 2701-2710. PMLR, 2017.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning , pages 2377-2386. PMLR, 2016.

Aldo Pacchiano, Philip Ball, Jack Parker-Holder, Krzysztof Choromanski, and Stephen Roberts. Towards tractable optimism in model-based reinforcement learning. In Uncertainty in Artificial Intelligence , pages 1413-1423. PMLR, 2021.

Noemie Perivier and Vineet Goyal. Dynamic pricing and assortment under a contextual mnl demand. Advances in Neural Information Processing Systems , 35:3461-3474, 2022.

Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems , 32, 2019.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems , pages 2256-2264, 2013.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Mark Rowland, Michal Valko, and Pierre Ménard. Optimistic posterior sampling for reinforcement learning with few samples and tight guarantees. Advances in Neural Information Processing Systems , 35:10737-10751, 2022.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33, 2020.

Yining Wang, Ruosong Wang, Simon Shaolei Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2021.

Gellért Weisz, Philip Amortila, and Csaba Szepesvári. Exponential lower bounds for planning in mdps with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory , pages 1237-1264. PMLR, 2021.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning , pages 10746-10756. PMLR, 2020.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR, 2020.

Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-Hua Zhou. Online stochastic linear optimization under one-bit feedback. In International Conference on Machine Learning , pages 392-401. PMLR, 2016.

Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

Yu-Jie Zhang and Masashi Sugiyama. Online (multinomial) logistic bandit: Improved regret and constant computation cost. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learningvia reference-advantage decomposition. In Advances in Neural Information Processing Systems , volume 33, pages 15198-15207, 2020.

Zihan Zhang, Jiaqi Yang, Xiangyang Ji, and Simon S Du. Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems , 34:4342-4355, 2021.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Model-free reinforcement learning: from clipped pseudo-regret to sample complexity. In International Conference on Machine Learning , pages 12653-12662. PMLR, 2021.

Dongruo Zhou and Quanquan Gu. Computationally efficient horizon-free reinforcement learning for linear mixture mdps. Advances in neural information processing systems , 35:36337-36349, 2022.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning , pages 12793-12802. PMLR, 2021.

Contents of Appendix

A Related Work

RL with linear function approximation There has been a growing interest in studies that extend beyond tabular MDPs and focus on function approximation methods with provable guarantees [42, 71, 43, 73, 53, 22, 14, 9, 68, 70, 33, 81, 82, 37, 35, 38]. In particular, for minimizing regret in linear MDPs, Jin et al. [43] propose an optimistic variant of the Least-Squares Value Iteration (LSVI) algorithm [13, 59] under the assumption that the transition model and reward function of the MDPs are linear function of a d -dimensional feature mapping and they guarantee ˜ O ( d 3 2 H 3 2 √ T ) regret. Zanette et al. [73] propose a randomized LSVI algorithm that incorporates exploration by perturbing the least-square approximation of the action-value function, and this algorithm guarantees ˜ O ( d 2 H 2 √ T ) regret. Ishfaq et al. [37] propose a variant of the randomized LSVI algorithm that combines optimism and TS by perturbing the training data with i.i.d. scalar noise, achieving a regret bound of ˜ O ( d 3 2 H 3 2 √ T ) . Similarly, Ishfaq et al. [38] introduce a randomized RL algorithm that employs Langevin Monte Carlo (LMC) to approximate the posterior distribution of the action-value

√

function, also ensuring a regret bound of ˜ O ( d 3 2 H 3 2 T ) . Also, there have been studies on modelbased methods with function approximation in linear MDPs, such as Yang and Wang [72], which assume that the transition probability kernel is a bilinear model parametrized by a matrix and propose a UCB-based algorithm with an upper bound of ˜ O ( d 3 2 H 2 √ T ) for regret. He et al. [34] propose an algorithm achieving nearly minimax optimal regret ˜ O ( dH √ T ) . Jia et al. [41] consider a specific type of MDPs called linear mixture MDPs in which the transition probability kernel is a linear combination of different basis kernels. This model encompasses various types of MDPs studied previously in Modi et al. [53], Yang and Wang [72]. For this model, Jia et al. [41] propose a UCB-based RL algorithm with value-targeted model parameter estimation that guarantees an upper bound of ˜ O ( dH 3 2 √ T ) for regret. The same linear mixture MDPs have been used in other studies such as Ayoub et al. [9], Zhou et al. [81, 82]. Specifically, in Zhou et al. [81], a variant of the method proposed by Jia et al. [41] is suggested and proved that the algorithm guarantees an upper bound of ˜ O ( dH √ T ) regret with a matching lower bound of Ω( dH √ T ) for linear mixture MDPs. More recently, there are also works achieving horizon-free regret bounds for linear mixture MDPs [78, 46, 80].

RL with non-linear function approximation Studies have been conducted on extending function approximation beyond linear models. Ayoub et al. [9], Wang et al. [68], Ishfaq et al. [37] provide upper bound for regret based on eluder dimension [63]. Also, there has been an effort to develop sample-efficient methods with more 'general' function approximation [48, 42, 19-21, 28, 37, 44, 4, 5, 75, 18, 39] However, these attempts may have been hindered by the difficulty of solving computationally intractable problems [48, 42, 19, 21, 28, 44, 18], the necessity of relying on stronger assumptions [20, 37], or the lack of discussion on how to define the posterior distribution supported by a given function class and how to draw the optimistic sample from the posterior [4, 5, 75]. That is why even after there exists a so-called 'general function class'-based result, it is often the case that the results in specific parametric models are still needed. Despite the large number of studies on RL with linear function approximation, there is limited research on extending beyond linear models to other parametric models. Wang et al. [69] use generalized linear function approximation, where the Bellman backup of any value function is assumed to be a generalized linear function of feature mapping. Hwang and Oh [35] discuss the limitations of linear function approximation and propose a UCB-based algorithm for MNL transition model in feature space achieving ˜ O ( dH 3 2 √ T ) . Ishfaq et al. [39] present TS-based RL algorithms that utilize approximate samplers, such as LMC or Underdamped LMC, to enhance the implementation and computational tractability of TS for RL with general function classes.

Contextual bandits Faury et al. [23] first provide a UCB-based algorithm with κ -independent regret for binary logistic bandit and Abeille et al. [3] present UCB & TS based algorithms achieving nearly minimax optimal regret for the same setting. Faury et al. [24] propose a jointly efficient UCB-based algorithm that achieve κ -independent regret bound with O (log t ) computation cost. In the context of MNL model, Oh and Iyengar [54] employ TS approach, while Oh and Iyengar [55] incorporate a combination of UCB exploration and online parameter updates for MNL bandits. Both of the methods have O ( κ -1 √ T ) regret. Amani and Thrampoulidis [8] propose an optimistic algorithm with better dependence on κ . Agrawal et al. [6] design a UCB-based algorithm with

O ( √ T ) regret bound without κ in its leading term, and Perivier and Goyal [61] establish O ( √ T/κ ∗ ) regret for the uniform reward setting. Zhang and Sugiyama [76] develop jointly efficient UCB-based algorithm for non-uniform MNL bandit problem. Lee and Oh [50] propose nearly minimax optimal MNL bandit algorithm for both uniform and non-uniform reward structures.

B Notations & Definitions

In this section, we formally summarize some definitions and notations used to analyze the proposed algorithm.

Inhomogeneous MNL transition model

For h ∈ [ H ] , the probability of state transition to s ′ ∈ S s,a when an action a is taken at a state s is given by

The estimated transition probability parameterized by θ is denoted as

Feature vector

We abbreviate the feature vector as follows:

Response variable & per-episode loss

The response variable y k h is given by

The per-episode loss ℓ k,h ( θ ) is given by

Regularity constants

H :

Horizon length

K : Episode number

T = KH : Total number of interactions

L φ : ℓ 2 -norm upper bound of φ ( s, a, s ) , i.e., ∥ φ ( s, a, s ′ ) ∥ 2 ≤ L φ ,

L θ : ℓ 2 -norm upper bound of θ ∗ h , i.e., ∥ θ ∗ h ∥ 2 ≤ L θ ,

κ : Problem-dependent constant such that inf θ ∈B d ( L θ ) P θ ( s ′ | s, a ) P θ ( ˜ s | s, a ) ≥ κ,

U : Maximum cardinality of the set of reachable states, i.e., U := max s,a |S s,a | .

Estimated transition core

The estimated transition core for RRL-MNL is given by

and the estimated transition core for ORRL-MNL is given by

Gram matrices

The Gram matrix with global gradient information κ is given by

The Gram matrices with local gradient information are given by

Confidence radius

For some absolute constants C β , C ξ > 0 , α k := α k ( δ )

Filtration

For an arbitrary set X , we denote the Σ -algebra generated by X as Σ( X ) . Then we define the following filtrations

Pseudo-noise

For RRL-MNL , the pseudo-noise is sampled as

and for ORRL-MNL , the pseudo-noise is sampled as

for M times independently.

Estimated value functions

The stochastically optimistic value function for RRL-MNL is defined as follows:

The optimistic randomized value function for ORRL-MNL is defined as follows:

where

Prediction error & Bellman error

Definition 1 (Prediction error & Bellman error) . For any ( s, a ) ∈ S × A and ( k, h ) ∈ [ K ] × [ H ] , we define the prediction error about θ k h as

Also we define the Bellman error as follows:

Good events

For any δ ∈ (0 , 1) , we define the following good events:

For RRL-MNL ,

For ORRL-MNL ,

Derivative of MNL transition model

Proposition 1 (Derivative of MNL transition model) . The gradient and Hessian of P θ ( · | · , · ) can be calculated as follows:

and

Proof of Proposition 1. Let θ = ( θ 1 , . . . , θ d ) and [ φ s,a,s ′ ] i be the i -th component of φ s,a,s ′ . Then, we have

Then, the gradient of P ( s | s, a ) is given by

On the other hand, the second derivative ∂ ∂θ i ∂θ j P θ ( s ′ | s, a ) can be obtained as follows:

C Detailed Regret Analysis for RRL-MNL (Theorem 1)

In this section, we provide the complete proof of Theorem 1. First, we introduce all the technical lemmas needed to prove Theorem 1 along with their proofs. At the end of this section, we present the proof of Theorem 1.

C.1 Concentration of Estimated Transition Core θ k h

In this section, we provide the concentration inequality for the estimated transition core run by the approximate online Newton step. The proof is similar to that given by Oh and Iyengar [55]. For completeness, we provide the detailed proof.

Lemma 1 (Concentration of online estimated transition core) . For each h ∈ [ H ] , if λ ≥ L 2 φ , then we have where α k ( δ ) is given by

α k ( δ )

Proof of lemma 1. Recall that the per-round loss ℓ k,h ( θ ) and its gradient G k,h ( θ ) is defined as follows:

For the analysis, we define the conditional expectations of ℓ k,h ( θ ) & G k,h ( θ ) as follows:

By Taylor expansion with ¯ θ = ν θ k h +(1 -ν ) θ ∗ h for some ν ∈ (0 , 1) , we have

where H k,h ( θ ) is the Hessian of the per-round loss evaluated at θ , i.e.,

Note that for ¯ θ = ν θ k h +(1 -ν ) θ ∗ h with ν ∈ (0 , 1) , we have

where the inequality utilizes the fact that xx ⊤ + yy ⊤ ⪰ xy ⊤ + yx ⊤ for any x , y ∈ R d . Therefore, we have

̸

̸

where ˙ s k,h is the state satisfying φ ( s k h , a k h , ˙ s k,h ) = 0 d and the last inequality comes from the Assumption 4.

Using the lower bound of the Hessian of the per-round loss evaluated at ¯ θ , from (10) we have

By rearranging, we have

On the other hand, for any θ ∈ R d , since we have

where D ( P ∥ Q ) is the Kullback-Leibler divergence of P from Q , from (12) we have

̸

̸

̸

̸

̸

To get an upper bound of G k,h ( θ k h ) ⊤ ( θ k h -θ ∗ h ) , recall that the estimated transition core is given by

Since the objective function in (14) is convex, by the first-order optimality condition for any θ ∈ B d ( L θ ) , we have

which gives

Then, we have

where the last inequality follows by the fact that

Therefore, from (16) we have

By substituting (17) into (13), we have

Note that since we have

where the first inequality utilizes the inequality x ⊤ Ay + y ⊤ Ax ≤ x ⊤ Ax + y ⊤ Ay for any positivesemidefinite matrix A , and the last inequality holds since 0 ≤ P θ k h ( s ′ | s k h , a k h ) ≤ 1 and ∑ s ′ P θ k h ( s ′ | s k h , a k h ) = 1 .

Combining the results of (18) and (19), we have

where for the first equality we use A k +1 ,h = A k,h + κ 2 W k,h . By rearranging the terms, we have

Then summing over k gives

For the final step, note that ( ¯ G i,h ( θ i h ) -G i,h ( θ i h ) ) ⊤ ( θ i h -θ ∗ h ) is a martingale difference sequence. To bound this term, we invoke the following lemmas:

Lemma 2. For δ ∈ (0 , 1) and ( k, h ) ∈ [ K ] × [ H ] , with a probability at least 1 -δ we have

Lemma 3 (Generalized elliptical potential) . Let S t := { x t, 1 , . . . , x t,K } ⊂ R d . For any 1 ≤ t ≤ T and i ∈ [ K ] , suppose ∥ x t,i ∥ 2 ≤ L . Let V t := λ I d + ∑ t -1 τ =1 ∑ i ∈ S τ x τ,i x ⊤ τ,i for some λ > 0 . If λ ≥ L 2 , then we have

By Lemma 2, with probability at least 1 -δ , we have

where the second inequality comes from Lemma 3. Note that the Gram matrix A k,h in Algorithm 1 and the Gram matrix V in Lemma 3 are different by the factor of κ 2 , which results in additional 2 κ factor for the bound of ∑ k i =1 max s ′ ∈S i,h ∥ φ i,h,s ′ ∥ 2 A -1 i +1 ,h .

In the following, we provide all the proofs of the lemmas used to prove Lemma 1.

C.1.1 Proof of Lemma 2

Proof of Lemma 2. Note that ( ¯ G i,h ( θ i h ) -G i,h ( θ i h ) ) ⊤ ( θ i h -θ ∗ h ) is a martingale difference sequence, i.e.,

On the other hand, for any θ ∈ R d , since we have

then, it follows by

where the last inequality follows by ∥ θ i h -θ ∗ h ∥ 2 ≤ ∥ θ i h ∥ 2 + ∥ θ ∗ h ∥ 2 ≤ 2 L θ . Hence, if we denote M k,h := ∑ k i =1 ( ¯ G i,h ( θ i h ) -G i,h ( θ i h ) ) ⊤ ( θ i h -θ ∗ h ) , then M k,h is a martingale. Note that we also

have

where (21) holds by the Cauchy-Schwarz inequality, (22) holds because

However, if we denote B k,h := 2 ∑ k i =1 ∥ θ i h -θ ∗ h ∥ 2 W i,h , since B k,h is itself a random variable, to apply Freedman's inequality to M k,h , we consider two cases depending on the values of B k,h .

Case 1 : B k,h ≤ 4 k U

Suppose that B k,h = 2 ∑ k i =1 ∥ θ i h -θ ∗ h ∥ 2 W i,h ≤ 4 k U . Then we have

Case 2 : B k,h > 4 k U

Suppose that B k,h = 2 ∑ k i =1 ∥ θ i h -θ ∗ h ∥ 2 W i,h > 4 k U . Then, we have both a lower and upper bound for B k,h as follows:

Then by the peeling process from Bartlett et al. [12], for any η k > 0 , we have

where m = 1 + ⌈ 2 log 2 k U L φ L θ ⌉ . For I j , note that from (20) we have

By Freedman's inequality (Lemma 29), we have

By substituting Eq. (24) into Eq. (23), we have

Then, combining with the result of Case 1 & 2, letting η k = log m δ/k 2 = log (1+ ⌈ 2 log 2 k U L φ L θ ⌉ ) k 2 δ and taking union bound over k , with probability at least 1 -δ , we have

By applying 2 √ ab ≤ a + b to the first term on the right hand side, we have

Combining the results of Eq. (25) & Eq. (26), we have

C.1.2 Proof of Lemma 3

Proof of Lemma 3. By definition of V t , we have

Since λ ≥ L 2 , we have

Since for any z ∈ [0 , 1] , it follows that z ≤ 2 log(1 + z ) . Hence, we have

where the second inequality comes from Eq. (27) and the last inequality follows by the determinanttrace inequality (Lemma 28).

C.2 Bound on Prediction Error

In this section, we provide the bound on the prediction error induced by estimated transition core θ k h . Lemma 4 (Bound on Prediction Error) . For any δ ∈ (0 , 1) , suppose that Lemma 1 holds. Then for any ( s, a ) ∈ S × A , we have

Proof of Lemma 4. Recall that

Then by the mean value theorem, there exists ¯ θ = ρ θ k h +(1 -ρ ) θ ∗ h for some ρ ∈ [0 , 1] satisfying that

Since V k h ( s ′ ) ≤ H for all s ′ ∈ S , k ∈ [ K ] , and h ∈ [ H ] , we have

where the second inequality comes from the fact that P ¯ θ ( s ′ | s, a ) ≤ 1 is a multinomial probability, the third inequality holds due to the Cauchy-Schwarz inequality, and the last inequality follows from Lemma 1 and the definition of ˆ φ k,h , i.e., ˆ φ k,h ( s, a ) := φ ( s, a, ˆ s ) for ˆ s = argmax s ′ ∈S s,a ∥ φ ( s, a, s ′ ) ∥ A -1 k,h .

C.3 Good Events with High Probability

Lemma 5 (Good event probability) . For any K ∈ N and δ ∈ (0 , 1) , the good event G ( K,δ ′ ) holds with probability at least 1 -δ where δ ′ = δ/ (2 KH ) .

Proof of Lemma 5. For any δ ′ ∈ (0 , 1) , we have

On the other hand, for any ( k, h ) ∈ [ K ] × [ H ] , by Lemma 30, G ξ k,h ( δ ′ ) holds with probability at least 1 -δ ′ . Then, for δ ′ = δ/ (2 KH ) by taking union bound, we have the desired result as follows:

C.4 Stochastic Optimism

Lemma 6 (Stochastic optimism) . For any δ with 0 < δ < Φ( -1) / 2 , let σ k = Hα k ( δ ) = ˜ O ( H √ d ) . If we take multiple sample size M = ⌈ 1 -log H log Φ(1) ⌉ , then for any k ∈ [ K ] , we have

Proof of lemma 6. Before presenting the proof, we introduce the following lemmas.

Lemma 7. For any k ∈ [ K ] , it holds

where ι k h ( s, a ) := r ( s, a ) + P h V k h +1 ( s, a ) -Q k h ( s, a ) .

Lemma 8. Let δ ∈ (0 , 1) be given. For any ( k, h ) ∈ [ K ] × [ H ] , let σ k = Hα k ( δ ) . If we define the event G ∆ k,h ( δ ) as

then conditioned on G ∆ k,h ( δ ) , for any ( s, a ) ∈ S × A , we have

Lemma 9. Let δ ∈ (0 , 1) be given. For any ( h, k ) ∈ [ H ] × [ K ] , let σ k = Hα k ( δ ) . If we take multiple sample size M = ⌈ 1 -log H log Φ(1) ⌉ , then conditioned on the event G ∆ k ( δ ) := ⋂ h ∈ [ H ] G ∆ k,h ( δ ) , we have

Now, we define the event of the estimated value function being optimistic at the start of the k -th episode as

Then for the event G k ( δ ) =: G k , we have

where the last inequality comes from lemma 5.

On the other hand, by Lemma 7, we have

If we define an event

then, by Lemma 9, we have

Note that since X c k ∩ G k ⊂ Y c k ∩ G k , we can conclude that

where the last inequality comes from the choice of δ .

In the following, we provide all the proofs of the lemmas used to prove Lemma 6.

C.4.1 Proof of Lemma 7

Proof of lemma 7. In this proof, we use x k h as the states sampled under the π ∗ to distinguish with s k h . Since we have,

then by applying this argument recursively, we finally have

C.4.2 Proof of Lemma 8

Proof of Lemma 8. Since we have

it is enough to show that

at least with constant probability.

On the other hand, under the event G k,h ( δ ) , by Lemma 4 we have

Now, for ∀ m ∈ [ M ] , since ξ ( m ) k,h ∼ N ( 0 d , σ 2 k A -1 k,h ) , we have

which means,

C.4.3 Proof of Lemma 9

Proof of Lemma 9. For each h ∈ [ H ] and k ∈ [ K ] , define an event E k h := {-ι k h ( s h , a h ) ≥ 0 } Then it holds

where the first inequality uses the union bound, the second inequality comes from the Lemma 8 and the last inequality holds due to the choice of M = ⌈ 1 -log H log Φ(1) ⌉ .

C.5 Bound on Estimation Part

We decompose the regret into the estimation part and the pessimism part as follows:

and we bound these two parts in the following sections, respectively.

Lemma 10 (Bound on estimation part) . For any δ ∈ (0 , 1) , if λ ≥ L 2 φ , then with probability at least 1 -δ/ 2 , we have

Proof of lemma 10. For any given k ∈ [ K ] ,

where the second equality holds due to the variant of ι k h ( s k h , a k h ) as follows:

Then, by applying this argument recursively for whole horizon, we have

where ˙ ζ k h := P h ( V k h +1 -V π k h +1 )( s k h , a k h ) -( V k h +1 -V π k h +1 )( s k h +1 ) .

Let δ ′ = δ/ (8 KH ) . By Lemma 5, the good event G ( K,δ ′ ) holds with probability at least 1 -δ/ 4 . Then under the event G ( K,δ ′ ) , for any h ∈ [ H ] we have

where (30) comes from the Cauchy-Schwarz inequality and (31) holds due the the Lemma 4 & 30. Then, with probability at least 1 -δ/ 4 , we have

On the other hand, for ˙ ζ k h , we have | ˙ ζ k h | ≤ 2 H and E [ ˙ ζ k h | F k,h ] = 0 , which means { ˙ ζ k h | F k,h } k,h is a martingale difference sequence for any k ∈ [ K ] and h ∈ [ H ] . Hence, by applying the AzumaHoeffding inequality with probability at least 1 -δ/ 4 , we have

Combining the results of (32) and (33), with probability at least 1 -δ/ 2 , we have

where (34) follows from the fact that both α k ( δ ) and γ k ( δ ) are increasing in k , (35) comes from Cauchy-Schwarz inequality and (36) holds by the generalized elliptical potential lemma (Lemma 3).

C.6 Bound on Pessimism Part

Lemma 11 (Bound on pessimism) . For any δ with 0 < δ < Φ( -1) / 2 , let σ k = Hα k ( δ ) . If λ ≥ L 2 φ and we take multiple sample size M = ⌈ 1 -log H log Φ(1) ⌉ , then with probability at least 1 -δ/ 2 , we have

Proof of lemma 11. Similar to the techniques used in [73], we show that the difference between the optimal value function V ∗ 1 and the estimated value function V k 1 can be controlled by constructing an upper bound on V ∗ 1 and a lower bound on V k 1 . In this proof, we consider three kinds of pseudo-noises, ξ , ¯ ξ and ξ that we define later in the proof. Also, for δ ′ = δ/ 10 , we denote G ( K,δ ′ ) , ¯ G ( K,δ ′ ) and G ( K,δ ′ ) as the good events induced by ξ , ¯ ξ and ξ respectively. From now on, we denote G ( K,δ ′ ) by the event G ( K,δ ′ ) ∩ ¯ G ( K,δ ′ ) ∩G ( K,δ ′ ) . Then, by Lemma 5, the event G ( K,δ ′ ) holds with high probability at least 1 -3 δ/ 10 .

First, we construct the lower bound of V k 1 . For any given k ∈ [ K ] , let ˜ ξ := { ˜ ξ ( m ) k,h } m ∈ [ M ] ⊂ R d be a set of vectors for h ∈ [ H ] and V k h ( · ; ˜ ξ ) be the value function obtained by the Algorithm 1 with non-random ˜ ξ ( m ) k,h in place of ξ ( m ) k,h . Then consider the following minimization problem:

And we denote ξ := { ξ ( m ) k,h } h ∈ [ H ] ,m ∈ [ M ] by a minimizer and V k 1 ( s k 1 ) by the minimum of the above minimization problem, i.e., V k h ( · ) := V k h ( · ; ξ ) . Then, under the event G ( K,δ ′ ) , since { ξ ( m ) k,h } h ∈ [ H ] ,m ∈ [ M ] is also a feasible solution of the above optimization problem, and since V k h = V k h ( ; ξ ) , thus we have

Second, to find an upper bound for V ∗ , considering i.i.d copies { ¯ ξ ( m ) k,h } h ∈ [ H ] ,m ∈ [ M ] of { ξ ( m ) k,h } h ∈ [ H ] ,m ∈ [ M ] and run Algorithm 1 to get a corresponding value function ¯ V k h and ¯ Q k h for all h ∈ [ H ] . Define the event that ¯ V k 1 ( s k 1 ) is optimistic in the k -th episode as

Then by Lemma 6, for given δ , we have

Then by the definition of optimism, under the event G ( K,δ ′ ) , we have

where the expectations are over the ¯ ξ 's conditioned on the event ¯ X k and the second inequality comes from (37). On the other hand, under the event ¯ G ( K,δ ′ ) by the law of the total expectation, we have

where (39) comes from the fact that { ¯ ξ ( m ) k,h } h ∈ [ H ] ,m ∈ [ M ] is also a feasible solution of the above optimization problem under the event ¯ G ( K,δ ′ ) , i.e., ¯ V k 1 ( s k 1 ) ≥ V k 1 ( s k 1 ) . Then, by combining the results of (39) and (38), under the event G ( K,δ ′ ) , we have

where we denote

Note that since ¯ ξ is the i.i.d copy of ξ , therefore ¯ V k, 1 and V k, 1 are independent, which means { ¨ ζ k | F k -1 } K k =1 is a martingale difference sequence with | ¨ ζ k | ≤ 2 H Φ( -1) . Therefore by applying Azuma-Hoeffiding inequality under the event G ( K,δ ′ ) , with probability at least 1 -δ ′ , we have

On the other hand, by dividing the first term in (40) into two terms we have

For I 1 , note that since it is related to the estimation error, under the event G ( K,δ ′ ) we can bound the sum of I 1 for the total episode number using Lemma 10 as follows:

For I 2 , since we have

where (43) comes from a k 1 = argmax a Q k 1 ( s k 1 , a ) and (44) holds by the following definition of ι k h ( s k h , a k h ) :

Then by applying the same argument recursively for the whole horizon, we have

where we denote

Note that { ... ζ k h | F k,h } k,h is a martingale difference sequence with | ... ζ k h | ≤ 2 H . Then, under the event G ( K,δ ′ ) by applying the Azuma-Hoeffding inequality with probability at least 1 -δ ′ , we have

To bound ∑ H h =1 ι k h ( s k h , a k h ) , we divide the whole horizon index set into two groups as follows: H +

On the other hand, for j ∈ H -, under the event G ( K,δ ′ ) we have

where (47) holds by Lemma 4.

By combining the result of (46) and (48), we have

Then summing I 2 over the total number of episodes, under the event G ( K,δ ′ ) , we have

where the last inequality holds due to the Lemma 3 and (45).

Finally, by summing (40) over k and plugging the results of (42), (50) and (41) then, we have

To conclude the proof, by setting δ ′ = δ/ 10 and we take a union bound over the two applications of Azuma-Hoeffding ( ¨ ζ k , ... ζ k h ) and the event G ( K,δ ′ ) , we get the desired result with probability at least 1 -δ/ 2 .

C.7 Regret Bound of RRL-MNL

Proof of Theorem 1. Wecan decompose the regret with estimation part and pessimism part as follows:

Since both Lemma 10 and Lemma 11 holds with probability at least 1 -δ/ 2 respectively, by taking the union bound the following holds with probability at least 1 -δ :

D Detailed Regret Analysis for ORRL-MNL (Theorem 2)

D.1 Concentration of Estimated Transition Core ˜ θ k h

In this section, we provide the detailed proof of Lemma 12, which demonstrates the concentration result for ˜ θ k h independently of κ and U . Note that we adapt the proof provided by Zhang and Sugiyama [76] in the MNL contextual bandit setting to MNL-MDPs and improve the result, making it independent of U . We provide the lemmas for the concentration of the online transition core for completeness, noting that there are slight differences compared to their work, which stem from the different problem setting.

Lemma 12 (Concentration of online estimated transition core) . Let η = O (log U ) and λ = O ( d log U ) . Then, for any δ ∈ (0 , 1] and for any h ∈ [ H ] , we have

where β k ( δ ) = O ( √ d log U log( kH )) .

Proof of Lemma 12. Recall that the transition core updated by the online mirror descent is represented by

where ˜ ℓ k,h ( θ ) = ℓ k,h ( ˜ θ k h ) + ( θ -˜ θ k h ) ⊤ ∇ ℓ k,h ( ˜ θ k h ) + 1 2 ∥ ∥ ∥ θ -˜ θ k h ∥ ∥ ∥ ∇ 2 ℓ k,h ( ˜ θ k h ) . We introduce the following lemma providing that the estimation error of the online estimator ˜ θ k h can be bounded by the regret.

Lemma 13 (Lemma 12 in [76]) . Let α = log U +2(1 + L θ L φ ) and λ > 0 . If we set the step size η = α/ 2 , then we have

Now, we bound the first term of (51). To simplify the presentation, for all ( k, h ) ∈ [ K ] × [ H ] , we define the softmax function σ k,h : R |S k,h | → [0 , 1] |S k,h | as follows:

where [ · ] s ′ denote the element corresponding to s ′ ∈ S of the input vector. We also define the pseudo-inverse of the softmax function σ k,h via [ σ + k,h ( p )] s ′ = log([ p ] s ′ ) which has the property that for all p ∈ ∆ |S k,h | , we have σ k,h ( σ + k,h ( p )) = p and ∑ s ∈S k,h exp ( [ σ + k,h ( p )] s ) = 1 .

We denote Φ k,h = [ φ k,h,s ′ ] s ′ ∈S k,h ∈ R d ×|S k,h | for simplicity. Then, the transition model can also be written as P θ ( s ′ | s k h , a k h ) = [ σ k,h ( Φ ⊤ k,h θ ∗ h )] s ′ . We further define ˜ z i,h = σ + i,h ( E θ ∼N ( ˜ θ i h ,c B -1 i,h ) [ σ i,h ( Φ ⊤ i,h θ )] ) for our analysis. Then, we have

We can bound the first term of (52) by the following lemma.

Lemma 14. Let δ ∈ (0 , 1] . Then, for all ( k, h ) ∈ [ K ] × [ H ] , with probability at least 1 -δ , we have

where Γ A k ( δ ) = 5 4 (3 log( U k ) + L φ L θ ) λ +4(3log( U k ) + L φ L θ ) log ( H √ 1+2 k δ ) +2 .

Furthermore, we can bound the second term of (52) by the following lemma.

Lemma 15. Let λ ≥ 72 L 2 φ cd . Then, for any c > 0 and all ( k, h ) ∈ [ K ] × [ H ] , we have

where Γ B k ( δ ) = √ 6 cd log ( 1 + 2 kL 2 φ dλ ) .

Combining Lemma 13, Lemma 14, and Lemma 15, and by setting η = α/ 2 , c = 2 α/ 3 and λ ≥ max { 12 √ 2 L 3 φ α, 48 L 2 φ dα } , we derive that

where C > 0 is an absolute constant. In the above, we choose λ = O ( d log U ) , α = O (log U ) . The second inequality of (53) is derived from the fact that

The first inequality holds from B i,h ⪰ λ I d , and the second inequality is obvious from our setting of λ . Therefore, we can conclude that

In the following section, we provide the proofs of the lemmas used in Lemma 12.

D.1.1 Proof of Lemma 13

second-order Taylor expansion of ℓ i,h ( θ ) at ˜ θ i h . Since we have

by Lemma 31, if we define ψ ( θ ) = 1 2 ∥ θ ∥ 2 B i,h we obtain

By applying Lemma 33, we have

By setting η = α i,h / 2 and merging equations (54) and (55), we arrive at

Meanwhile, we obtain

by taking the gradient over both sides of the Taylor approximation of ℓ i,h ( θ ) . Using (57), we proceed to bound the first term of (56) as follows:

where ¯ θ i +1 h is a convex combination of ˜ θ i h and ˜ θ i +1 h . The second equality arises from the Taylor expansion, the first inequality is due to the self-concordant property, and the final inequality is justified by the following:

By summing over i and reorganizing the terms, we arrive at the final result as follows:

where the first inequality holds by Assumption 2 and the last inequality holds since α = log U + 2(1 + L φ L θ ) ≥ α i,h for all i ∈ [ k ] .

D.1.2 Proof of Lemma 14

Proof of Lemma 14. The norm of ˜ z i,h = σ + i,h ( E θ ∼N ( ˜ θ i h ,c B -1 i,h ) [ σ i,h ( Φ ⊤ i,h θ )] ) is generally unbounded [27]. In this proof, we utilize the smoothed version of ˜ z i,h , defined as follows:

where the smooth function smooth u i,h ( p ) = (1 -u ) p +( u/ U ) 1 with u ∈ [0 , 1 / 2] , and 1 ∈ R |S i,h | is an all-one vector.

Exploiting the property of σ + i,h such that σ i,h ( σ + i,h ( p )) = p for any p ∈ ∆ |S i,h | , it is straightforward to show that ˜ z u i,h = σ + i,h (smooth u i,h ( σ i,h ( ˜ z i,h ))) . Then, by Lemma 34, we have

Given the definition of ℓ i,h , we know that ℓ ( z ∗ i,h , y i h ) = ℓ i,h ( θ ∗ h ) , where z ∗ i,h = Φ ⊤ i,h θ ∗ h . We can bound the gap between the loss of θ ∗ h and ˜ z u i,h as follows:

where M i,h = log( |S i,h | ) + 2 log( U /u ) , and the second equality holds by a direct calculation of the first order and Hessian of the logistic loss.

Now, we first bound the first term of the right-hand side. Let d i,h = ( z ∗ i,h -˜ z u i,h ) / ( M + L φ L θ ) , where M = log U + 2log( U /u ) . Then, one can check that ∥ d i,h ∥ ∞ ≤ 1 since ∥ z ∗ i,h ∥ ∞ ≤ max s ′ ∈S i,h ∥ φ i,h,s ′ ∥ 2 ∥ θ ∗ h ∥ 2 ≤ L φ L θ and ∥ ˜ z u i,h ∥ ∞ ≤ log( U /u ) . Moreover, since z ∗ i,h and ˜ z u i,h are independent of y i h , d i,h is F i,h -measurable. Since E [( σ i,h ( z ∗ i,h ) -y i h )( σ i,h ( z ∗ i,h ) -y i h ) ⊤ | F i,h ] = ∇ σ i,h ( z ∗ i,h ) and ∥ σ i,h ( z ∗ i,h ) -y i h ∥ 1 ≤ 2 , we can apply Lemma 32. For any k and δ ∈ (0 , 1] ,

with probability at least 1 -δ/H , we have

where the second inequality holds since ∥ d i,h ∥ 2 ∇ σ i,h ( z ∗ i,h ) = d ⊤ i,h ∇ σ i,h ( z ∗ i,h ) d i,h ≤ 2 and λ ≥ 1 . Plugging (60) into (59) and rearranging the term, we get

Finally, combining (58) and (61), by setting u = 1 /k , we derive that

where the last inequality holds by the definition of M = log U +2log( U /u ) . Taking the union bound over h ∈ [ H ] , we conclude the proof.

D.1.3 Proof of Lemma 15

Proof of Lemma 15. We start the proof from the observation of Proposition 2 in Foster et al. [27], stating that ˜ z i,h represents the mixed prediction, which adheres to the following property:

where L i,h ( θ ) := ℓ i,h ( θ ) + 1 2 c ∥ ∥ ∥ θ -˜ θ i h ∥ ∥ ∥ 2 B i,h and Z i,h := √ (2 π ) d c | B -1 i,h | .

Consider the quadratic approximation

Using the property that ℓ i,h is 3 √ 2 L φ -self-concordant-like function as asserted by Proposition B.1 in [50], and applying Lemma 35, we obtain

Also, we have

where we define the function ˜ f i,h : B ( 0 d , 1) → R as

We denote ˜ Z i +1 ,h = ∫ R d ˜ f i +1 ,h ( θ ) d θ ≤ + ∞ and define ˜ Θ i +1 ,h as the distribution whose density function is ˜ f i +1 ,h ( θ ) / ˜ Z i +1 ,h . Then, we can rewrite (63) as follows:

where the second inequality is by Jensen's inequality and the last inequality holds because ˜ Θ i +1 ,h is symmetric around ˜ θ i +1 h and thus E θ ∼ ˜ Θ i +1 ,h [〈 ∇ L i,h ( ˜ θ i +1 h ) , θ -˜ θ i +1 h 〉] = 0 .

Combining (62) and (64), we get

Moreover, we have

where ̂ Θ i +1 ,h = N ( ˜ θ i +1 h , c B -1 i,h ) and ̂ Z i +1 ,h = ∫ R d exp ( -1 2 c ∥ ∥ ∥ θ -˜ θ i +1 h ∥ ∥ ∥ 2 B i,h ) d θ , and the last inequality holds because ̂ Z i +1 ,h and Z i,h are identical normalizing factors. Integrating (65) and (66) and summing over k , yields

Moreover, we can further bound the second term on the right-hand side of (66). By Cauchy-Schwarz inequality, we get

where λ j ( B -1 i,h ) denotes the j -th largest eigenvalue of B -1 i,h and { e 1 , . . . , e d } are orthogonal basis of R d . Furthermore, since we know that B -1 i,h ≤ λ -1 I d , we can bound the term (I) by

where χ 2 is the chi-square distribution and the last inequality holds due to Jensen's inequality. By choosing λ ≥ 72 L 2 φ cd , we arrive that

where the last inequality holds because the moment-generating function for χ 2 -distribution is bounded by E W ∼ χ 2 [exp( tW )] ≤ 1 / √ 1 -2 t for all t ≤ 1 / 2 . Now, we bound the term (II) .

where ¯ B i,h = ( ∇ 2 ℓ i,h ( ˜ θ i +1 h ) ) -1 / 2 B i,h ( ∇ 2 ℓ i,h ( ˜ θ i +1 h ) ) -1 / 2 . Let ¯ λ j := λ j ( c ¯ B -1 i,h ) be the j -th largest eigenvalue of the matrix. Then, a similar analysis as (67) gives that

where the last inequality holds due to E X j ,X j ′ ∼N (0 , 1) [ X 2 j X 2 j ′ ] ≤ 3 when considering the case where j = j ′ and the last equality is derived from the fact that ( ∑ d j =1 ¯ λ j ) 2 = tr ( c ¯ B -1 i,h ) . Here, we denote tr( A ) as the trace of the matrix A .

We define matrix R i +1 ,h := λ I d / 2 + ∑ i τ =1 ∇ 2 ℓ τ,h ( θ τ +1 ,h ) . Under the condition λ ≥ 2 L 2 φ , we have ∇ 2 ℓ i,h ( θ i +1 ,h ) ⪯ L 2 φ I d ≤ λ 2 I d . Then, we have B i,h ⪰ R i +1 ,h . Therefore, we can bound the trace by

where the last inequality holds due to Lemma 4.7 of Hazan et al. [32]. Therefore we can bound the term (II) as

Combining (68) and (69), we get

Plugging (66) and (70) into (65), and taking summation over k , we derive that

where the last inequality holds because ∑ k i =1 log det( R i +1 ,h ) det( R i,h ) = log(det( R k +1 ,h ) / det( λ/ 2 I d )) ≤ d log ( 1 + 2 kL 2 φ dλ ) . By rearranging the terms, we conclude the proof.

D.2 Bound on Prediction Error

In this section, we present the bound on the prediction error of parameters updated by ORRL-MNL . First, we compare the problem setting of MNL contextual bandits with ours and introduce the challenges of applying their analysis to our setting.

MNLdynamic assortment optimization (single-parameter & uniform reward) [61] Perivier and Goyal [61] consider an assortment selection problem where the user choice is given by a MNL choice model with the single-parameter. At each time t , the agent observes context features { x t,i } i M =1 ⊂ R d . Then the agent decides on the set S t ⊂ [ M ] to offer to a user, with | S t | ≤ N . Without loss of generality, we may assume | S t | = N . Then the user purchases one single product j ∈ S t ∪{ 0 } and the probability of each product j is purchased by a user follows the MNL model parametrized by a unknown fixed parameter θ ∗ ∈ R d ,

Then the difference between the revenue induced by θ ∗ and that by an estimator θ in Perivier and Goyal [61] is expressed as follows:

If we define Q : R N → R , such that for all u = ( u 1 , . . . , u N ) ∈ R N , Q ( u ) := ∑ N i =1 exp( u i ) 1+ ∑ N j =1 exp( u j ) and let v ∗ = ( x ⊤ t,i 1 θ ∗ , . . . , x ⊤ t,i N θ ∗ ) and v = ( x ⊤ t,i 1 θ , . . . , x ⊤ t,i N θ ) , then Eq. (71) can be expressed as follows:

where ¯ v is a convex combination of v ∗ and v . For the first term in Eq. (72), we have

where H t ( θ ) is the Gram matrix used in [61] defined by

Note that the term ∥ θ ∗ -θ ∥ H t ( θ ∗ ) can be bounded by the concentration result of the estimated parameter. On the other hand, to apply the elliptical potential lemma to the term

∑ j ∈ S t q t,j ( S t , θ ∗ ) q t, 0 ( S t , θ ∗ ) ∥ x t,j ∥ H -1 t ( θ ∗ ) , note that H t ( θ ∗ ) can be bounded as follows:

Now since the coefficient q t,j ( S t , θ ∗ ) q t, 0 ( S t , θ ∗ ) of ∥ x ∥ H -1 t ( θ ∗ ) in Eq. (73) aligns with the coefficients of the lower bound of H t ( θ ∗ ) in Eq. (74), the elliptical potential lemma can be applied. Note that such a lower bound in Eq. (74) holds since Perivier and Goyal [61] deals with the uniform reward, i.e., 1 -∑ i ∈ S t q t,i ( S t , θ ∗ ) = q t, 0 ( S t , θ ∗ ) .

Mulitinomial logistic bandit problem [76] Zhang and Sugiyama [76] address the multipleparameter MNL contextual bandit problem where at each time step t the agent selects an action x t ∈ R d and receives response feedback y t ∈ { 0 } ∪ [ N ] with N + 1 possible outcomes. Each outcome i ∈ [ N ] is associated with a ground-truth parameter θ ∗ i ∈ R d , and the probability of the outcome P ( y t = i | x t ) follows the MNL model,

In this model, there are N unknown choice parameter Θ ∗ := [ θ ∗ 1 , . . . , θ ∗ N ] ∈ R d × N and the agent chooses one context feature x t , that is why we call multiple-parameter MNL model. Then, the expected revenue of an action x t in [76] is given by

where we define the softmax function σ : R N → [0 , 1] N by

and ρ := [ ρ 1 , . . . , ρ N ] ∈ R N +1 + represents the reward for each outcome j ∈ [ N ] with ρ 0 = 0 . Then, the difference between the revenue induced by Θ ∗ and that by an estimator ˆ Θ in [76] is expressed by

where Ξ k = ∫ 1 0 (1 -ν ) ∇ 2 [ σ ( ˆ Θx t + ν ( Θ ∗ -ˆ Θ ) x t )] k dν . Then for the first term in Eq. (75), we have

where H t is the Gram matrix used in [76] defined by

Note that the term ∥ vec( Θ ∗ ) -vec( ˆ Θ ) ∥ H t in Eq. (76) can be bounded by the concentration result of the estimated parameter, and the term ∥ H -1 2 t ( I N ⊗ x ⊤ t ) ∇ σ ( ˆ Θx t ) ρ ∥ 2 also can be bounded as follows:

Here Zhang and Sugiyama [76] bound the term ∥ H -1 2 t ( I N ⊗ x ⊤ t ) ∇ σ ( ˆ Θx t ) ∥ 2 using a matrix version of elliptical lemma. However, they assume ∥ ρ ∥ 2 ≤ R (Assumption 2 in [76]).

Now, regarding the prediction error in our setting, the estimated values ( ˜ V k h +1 ( · ) ) for each reachable state are typically distinct, and we do not assume a constant upper bound on the ℓ 2 -norm of the estimated value vector for all reachable states. Instead, we can bound the ℓ 2 -norm of the estimated value vector for all reachable states as follows:

where ˜ V k h +1 ( s, a ) := [ ˜ V k h +1 ( s ′ ) ] s ′ ∈S s,a ∈ R |S s,a | . However, such a bound leads to a looser regret √

by a factor of U . To address, we adapt the feature centralization technique [50] to bound the prediction error independently of U , without making any additional assumptions. The key point is that the Hessian of per-round loss ℓ k,h ( θ ) is expressed in terms of the centralized feature as follows:

where ¯ φ ( s, a, s ′ ; θ ) := φ ( s, a, s ′ ) -E ˜ s ∼ P θ ( ·| s,a ) [ φ ( s, a, ˜ s )] is the centralized feature by θ . Now, we provide the bound on prediction error of the estimated parameter updated by ORRL-MNL .

Lemma 16 (Bound on the prediction error) . For any δ ∈ (0 , 1) , suppose that Lemma 12 holds. Let us denote the prediction error about ˜ θ k h by

Then, for any ( s, a ) ∈ S × A , we have

Proof of Lemma 16. Let us define F ( θ ) := ∑ s ′ ∈S s,a P θ ( s ′ | s, a ) ˜ V k h +1 ( s ′ ) . Then, by Taylor expansion we have

where ¯ θ = (1 -v ) θ ∗ h + v ˜ θ k h for some v ∈ (0 , 1) . By Proposition 1, we have

and

Then, the prediction error can be bounded as follows:

For the first term in Eq. (77),

where in the first inequality we use ˜ V k h +1 ( s ′ ) ≤ H and Cauchy-Scharwz inequality, and the second inequality follows by the concentration result of Lemma 12.

For the second term in Eq. (77), since 0 ≤ ˜ V k h +1 ( s ′ ) ≤ H ,

where for the second inequality we use Cauchy-Schwarz inequality, xx ⊤ + yy ⊤ ⪰ xy ⊤ + yx ⊤ for any x , y ∈ R d , and triangle inequality. Note that

By substituting Eq. (80) into Eq. (79) we have

where for the second inequality follows by Lemma 12 and ∑ s ′ ∈S s,a P ¯ θ ( s ′ | s, a ) = 1 . Combining the results of Eq. (78) and Eq. (81) and , we conclude the proof.

D.3 Good Events with High Probability

In this section, we introduce the good events used to prove Theorem 2 and show that the good events happen with high probability.

Lemma 17 (Good event probability) . For any K ∈ N and δ ∈ (0 , 1) , the good event G ( K,δ ′ ) holds with probability at least 1 -δ where δ ′ = δ/ (2 KH ) .

Proof of Lemma 17. For any δ ′ ∈ (0 , 1) , we have

On the other hand, for any ( k, h ) ∈ [ K ] × [ H ] , by Lemma 30 G ξ k,h ( δ ′ ) holds with probability at least 1 -δ ′ . Then, for δ ′ = δ/ (2 KH ) by taking union bound, we have the desired result as follows:

D.4 Stochastic Optimism

Lemma 18 (Stochastic optimism) . For any δ with 0 < δ < Φ( -1) / 2 , let σ k = Hβ k ( δ ) . If we take multiple sample size M = ⌈ 1 -log( H U ) log Φ(1) ⌉ , then for any k ∈ [ K ] , we have

Proof of Lemma 18. First, we introduce the following lemmas.

Lemma 19. Let δ ∈ (0 , 1) be given. For any ( k, h ) ∈ [ K ] × [ H ] , let σ k = Hβ k ( δ ) . If we define the event G ∆ k,h ( δ ) as

then conditioned on G ∆ k,h ( δ ) , for any ( s, a ) ∈ S × A , we have

Lemma 20. Let δ ∈ (0 , 1) be given. For any ( h, k ) ∈ [ H ] × [ K ] , let σ k = Hβ k ( δ ) . If we take multiple sample size M = ⌈ 1 -log( H U ) log Φ(1) ⌉ , then conditioned on the event G ∆ k ( δ ) := ∩ h ∈ [ H ] G ∆ k,h ( δ ) , we have

Based on the result of Lemma 20, using the same argument as in Lemma 6 we obtain the desired result.

In the following section, we provide the proofs of the lemmas used in Lemma 18.

D.4.1 Proof of Lemma 19

Proof of Lemma 19. Recall the definition of Bellman error (Definition 1), we have

Then, it is enough to show that

at least with constant probability. On the other hand, under the event G ∆ k,h ( δ ) , by Lemma 16 we have

Note that since ξ ( m ) k,h ∼ N ( 0 , σ 2 k B -1 k,h ) , it follows that

Therefore, by setting σ k = Hβ k ( δ ) , we have for m ∈ [ M ] and s ′ ∈ S s,a ,

Consequently, we arrive at the conclusion as follows:

where (84) comes from the fact that max s,a |S s,a | = U and the union bound, and (85) follows by (82).

D.4.2 Proof of Lemma 20

Proof of Lemma 20. It holds

where the first inequality uses the Bernoulli's inequality, the second inequality follows by Lemma 19, and the last inequality holds due to the choice of M = ⌈ 1 -log( U H ) log Φ(1) ⌉ .

D.5 Bound on Estimation Part

In this section, we provide the upper bound on the estimation part of the regret: ∑ K k =1 ( ˜ V k 1 -V ∗ 1 )( s k 1 ) . Lemma 21 (Bound on estimation) . For any δ ∈ (0 , 1) , if λ = O ( L 2 φ d log U ) , then with probability at least 1 -δ/ 2 , we have

Proof of Lemma 21. With the same argument in Lemma 10, we have

where ˙ ζ k h := P h ( ˜ V k h +1 -V π k h +1 )( s k h , a k h ) -( ˜ V k h +1 -V π k h +1 )( s k h +1 ) . Note that

where the last inequality follows by Lemma 16. Now we introduce the following lemma.

Lemma 22. For any ( k, h ) ∈ [ K ] × [ H ] and ( s, a ) ∈ S × A , it holds

By plugging the result of Lemma 22 into Eq. (87), we have

By letting us denote

and summing over all episodes, we have

Note that ∑ K k =1 ∑ H h =1 ∑ s ′ ∈S k,h is hereafter abbreviated as ∑ k,h,s ′ . For term (i) , we have

where the last inequality follows by the following lemma:

Lemma 23. For each h ∈ [ H ] , if λ ≥ L 2 φ , then we have

Then, term (i) can be bounded as follows:

For term (ii) , we introduce the following lemma:

Lemma 24. Let δ ∈ (0 , 1) be given. For any ( k, h ) ∈ [ K ] × [ H ] and ( s, a ) ∈ S ×A , with probability at least 1 -δ , it holds

where γ k ( δ ) := C ξ σ k √ d log( Md/δ ) for an absolute constant C ξ > 0 .

where the last inequality follows by Eq. (90). Note that

where the first inequality holds since B -1 k,h ⪯ A -1 k,h , the second inequality follows from ( x + y ) 2 ≤ 2 x 2 + 2 y 2 , and the third inequality uses the triangle inequality, and the fourth inequality uses ∑ ˜ s ∈S k,h P ˜ θ k +1 h ( ˜ s | s k h , a k h ) = 1 , and the last inequality follows by Lemma 3. By substituting

Eq. (93) into Eq. (92), we have

For term (iii) ,

where for the second inequality we use the same argument used to derive Eq. (93) and Lemma 3.

For term (iv) , since we have | ˙ ζ k h | ≤ 2 H and E [ ˙ ζ k h | F k,h ] = 0 , which means { ˙ ζ k h | F k,h } k,h is a martingale difference sequence for any k ∈ [ K ] and h ∈ [ H ] . Hence, by applying the AzumaHoeffding inequality with probability at least 1 -δ/ 4 , we have

Combining all results of Eq. (91), (94), (95), and (96), we have the desired result.

In the following, we provide the proof of the lemmas used in Lemma 21.

D.5.1 Proof of Lemma 22

Proof of Lemma 22. Note that

where the first inequality holds by triangle inequality.

For (i) , we have

where in the equality we apply the mean value theorem with ϑ k h = v ˜ θ k h +(1 -v ) ˜ θ k +1 h for some v ∈ [0 , 1] , and the inequality follows by Cauchy-Schwarz inequality. Meanwhile, since we have

by substituting (98) into (97) we have

Note that by Jensen's inequality, we have

Also, we introduce the following lemma:

Lemma 25. For any k ∈ [ K ] and h ∈ [ H ] , the following holds:

Then, substituting (100) into (99), we have

where the second inequality comes from Lemma 25, and the last inequality holds due to ∑ s ′ ∈S s,a P ϑ k h ( s ′ | s, a ) = 1 .

For (ii) , we have

where the last inequality is obtained through the same argument as used to bound (i) . Combining the results of Eq. (101) and Eq. (102), we have

D.5.2 Proof of Lemma 23

Proof of Lemma 23. Note that

where we define ˜ φ k,h,s ′ ( ˜ θ k +1 h ) := √ P ˜ θ k +1 h ( s ′ | s k h , a k h )¯ φ k,h,s ′ ( ˜ θ k +1 h ) . Then, we have

Taking the logarithm on both sides yields

On the other hand, since λ ≥ L 2 φ ,

where the last inequality uses ∑ s ′ ∈S k,h P ˜ θ k +1 h ( s ′ | s k h , a k h ) = 1 . From the fact that z ≤ 2 log(1 + z ) for any z ∈ [0 , 1] , it follows that

Finally, we obtain

where the last inequality follows by the determinant-trace inequality (Lemma 28).

D.5.3 Proof of Lemma 24

Proof of Lemma 24. Since ξ ( m ) k,h ∼ N ( 0 , σ 2 k B -1 k,h ) , by Lemma 30 for each m ∈ [ M ] , we have ∥ ξ ( m ) k,h ∥ B k,h ≤ C ξ σ k √ d log( Md/δ ) .

Following the result of Lemma 22, we have

Then, we obtain

D.5.4 Proof of Lemma 25

Proof of Lemma 25. We provide a proof for Lemma 25 since it is slight modification of Lemma 20 of [76]. From the definition, we know that

By rearranging the terms, the following holds:

Thus, we get

Since B k,h ⪯ ˜ B k,h and ˜ B -1 k,h ⪯ λ -1 I d , we obtain

For the last inequality of (103), we provide the upper bound of l 2 -norm of ∇ ℓ k,h ( θ ) . Since

the gradient of the loss function is given by

Therefore, we have

and this concludes the proof.

D.6 Bound on Pessimism Part

In this section, we provide the upper bound on the pessimism part of the regret: ∑ K k =1 ( V ∗ 1 -˜ V k 1 )( s k 1 ) . Lemma 26 (Bound on pessimism) . For any δ with 0 < δ < Φ( -1) / 2 , let σ k = Hβ k . If λ = O ( L 2 φ d log U ) and we take multiple sample size M = ⌈ 1 -log( H U ) log Φ(1) ⌉ , then with probability at least 1 -δ/ 2 , we have

Proof of Lemma 26. As seen in Lemma 18, by using multiple sampling technique we show that the optimistic randomized value function ˜ V of ORRL-MNL is optimistic than the true optimal value with constant probability Hence, with the same argument used in Lemma 11, we can show that the pessimism term of ORRL-MNL is upper bounded by a bound of the estimation term times the inverse probability of being optimistic, i.e.,

D.7 Regret Bound of ORRL-MNL

Proof of Theorem 2. Since both Lemma 21 and Lemma 26 holds with probability at least 1 -δ/ 2 respectively, by taking the union bound we conclude the proof.

E Optimistic Exploration Extension

In this section, we introduce UCRL-MNL+ (Algorithm 3), which is both computationally and statistically efficient for MNL-MDPs with UCB-based exploration. The main difference compared to ORRL-MNL is that UCRL-MNL+ constructs an optimistic value function that is greater than the optimal value function with high probability. At each episode k ∈ [ K ] , with the estimated transition core parameter ˜ θ k h (5), for ( s, a ) ∈ S × A , set ˆ Q k H +1 ( s, a ) = 0 . For each h ∈ [ H ] ,

where ˆ V k h ( s ) := min { max a ∈A ˆ Q k h ( s, a ) , H } and ν opt k,h ( s, a ) is the optimistic bonus term defined by

Based on these optimistic value function ˆ Q k h , at each episode the agent plays a greedy action with respect to ˆ Q k h as summarized in Algorithm 3.

Algorithm 3 UCRL-MNL+ (Upper Confidence RL for MNL-MDPs)

1: Inputs: Episodic MDP M , Feature map φ : S × A × S → R d , Number of episodes K , Regularization parameter λ , Confidence radius { β k } K k =1 , Step size η

2: Initialize: ˜ θ 1 h = 0 d , B 1 ,h = λ I d for all h ∈ [ H ]

3: for episode k = 1 , 2 , · · · , K do

4: Observe s k 1 and set { ˆ Q k h ( · , · ) } h ∈ [ H ] as described in (104)

5: for horizon h = 1 , 2 , · · · , H do

6: Select a k h = argmax a ∈A ˆ Q k h ( s k h , a ) and observe s k h +1

7: Update ˜ B k,h = B k,h + η ∇ 2 ℓ k,h ( ˜ θ k h ) and ˜ θ k +1 h as in (5)

8: Update B k +1 ,h = B k,h + ∇ 2 ℓ k,h ( ˜ θ k +1 h )

9: end for

10: end for

The main difference in regret analysis lies in ensuring the optimism of the estimated value function ˆ Q k h (Lemma 27). In the following statement (formal statement of Corollary 1), we provide a regret guarantee for UCRL-MNL+ , which enjoys the tightest regret bound for MNL-MDPs.

Theorem 3 (Regret Bound of UCRL-MNL+ ) . Suppose that Assumption 1- 4 hold. For any δ ∈ (0 , 1) , if we set the input parameters in Algorithm 3 as λ = O ( L 2 φ d log U ) , β k = O ( √ d log U log( kH ))

η = O (log U ) , then with probability at least 1 -δ , the cumulative regret of the UCRL-MNL+ policy π is upper-bounded by

where T = KH is the total number of time steps.

Proof of Theorem 3. By Lemma 17, suppose that the good event G ( K,δ ′ ) holds with probability at least 1 -δ . Then, we show that the optimistic value function ˆ Q k h is deterministically greater than the true optimal value function as follows:

Lemma 27 (Optimism) . Suppose that the event G ∆ k,h ( δ ) holds for all k ∈ [ K ] and h ∈ [ H ] . Then for any ( s, a ) ∈ S × A , we have

Conditioned on G ( K,δ ′ ) , by Lemma 27 we have

Note that

where we denote ζ k h := ( ˆ V k h +1 -V π k h +1 )( s k h +1 ) -E ˜ s | s k h ,a k h [ ( ˆ V k h +1 -V π k h +1 )( ˜ s ) ] . Then, with the same argument, we have

By summing over all episodes, we have

On the other hand, note that

where the last inequality follows by Lemma 22. Term (i) can be bounded as in Eq. (91):

For term (ii) , recall that as in Eq. (93) we have

Then, we have

For term (iii) , since we have

Combining the results of Eq. (106), (107), and (108), we have

Finally, by Azuma-Hoeffiding inequality as in Eq. (96) we have

This concludes the proof.

In the following, we provide the proof of Lemma 27.

E.1 Optimism

Proof of Lemma 27. We prove this by backwards induction on h . For the base case h = H , since V ∗ H +1 ( s ) = ˆ V k H +1 ( s ) = 0 for all s ∈ S , we have

Suppose that the statement holds for h +1 where h ∈ [ H -1] . Then, for h and for any ( s, a ) ∈ S ×A ,

where the first inequality follows from the induction hypothesis and the second inequality holds by Lemma 16.

F Experiment Details

s

1

s

2

...

s

n

-

1

s

n

0

.

4

0

.

6

0

.

05

0

.

6

0

.

35

0

.

05

0

.

6

0

.

35

0

.

05

0

.

35

0

.

4

(1

, r

=

5

1000

)

1

1

1

1

(0

.

6

, r

= 1)

The RiverSwim environment (Figure 2) consists of n states that are arranged in a chain. The agent starts in the leftmost state with a relatively small reward of 0 . 005 and aims to reach the rightmost state, which has a relatively large reward of 1 . Choosing to swim to the left moves the agent deterministically to the left, while swimming to the right has a probability of transitioning the agent toward the right state, but also a high chance of remaining in the current state or even moving left due to the strong current of river. Therefore, efficient exploration is crucial in order to learn the optimal policy for this environment.

We fine-tuned the hyperparameters for each algorithm within specific ranges. Figures 1a and 1b show the episodic returns in the RiverSwim environment over 10 independent runs with |S| = 4 , H = 12 , and K = 10 , 000 and |S| = 8 , H = 24 , and K = 10 , 000 , respectively. The shaded areas represent the standard deviations (1-sigma error). Figure 1c compares the running time of the algorithms over the first 1,000 episodes. All experiments were conducted on a Xeon(R) Gold 6226R CPU @ 2.90GHz (16 cores).

G Auxiliary Lemmas

Lemma 28 (Determinant-trace inequality [1]) . Suppose x 1 , . . . , x t ∈ R d and for any 1 ≤ τ ≤ t , ∥ x τ ∥ 2 ≤ L . Let V t = λ I d + ∑ t τ =1 x τ x ⊤ τ for some λ > 0 . Then,

Lemma 29 (Freedman's inequality [29]) . Consider a real-valued martingale { Y k : k = 0 , 1 , 2 , . . . } with difference sequence { X k : k = 0 , 1 , 2 , 3 , . . . } . Assume that the difference sequence is uniformly

bounded, X k ≤ R almost surely for k = 1 , 2 , 3 , . . . . Define the predictable quadratic variation process of the martingale:

Then, for all t ≥ 0 and σ 2 > 0 ,

Lemma 30 (Gaussian noise concentration (Lemma D.2 in [37])) . Let ξ (1) , ξ (2) , . . . , ξ ( M ) be M independent d -dimensional multivariate normal distributed vector with mean 0 d and covariance σ 2 A -1 for some σ > 0 and a positive definite matrix A -1 , i.e., ξ ( m ) ∼ N ( 0 d , σ 2 A -1 ) for m ∈ [ M ] . Then for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

where C ξ is an absolute constant.

Lemma 31 (Proposition 4.1 of 15) . Let the w t +1 be the solution of the update rule

where V ⊆ W ⊆ R d is a non-empty convex set and D ψ ( w 1 , w 2 ) = ψ ( w 1 ) -ψ ( w 2 ) -⟨∇ ψ ( w 2 ) , w 1 -w 2 ⟩ is the Bregman Divergence w.r.t. a strictly convex and continuously differentiable function ψ : W → R . Further supposing ψ ( w ) is 1 -strongly convex w.r.t. a certain norm ∥ · ∥ in W , then there exists a g t ∈ ∂ℓ t ( w t +1 ) such that

for any u ∈ W .

Lemma 32. Let {F t } ∞ t =1 be a filtration. Let { z t } ∞ t =1 be a stochastic process in B 2 ( U ) = { z ∈ R U | ∥ z ∥ ∞ ≤ 1 } such that z t is F t measurable. Let { ε t } ∞ t =1 be a martingale difference sequence such that ε t ∈ R U is F t +1 measurable. Furthermore, assume that conditional on F t , we have ∥ ε t ∥ 1 ≤ 2 almost surely, and denote by Σ t = E [ ε t ε ⊤ t |F t ] . Let λ > 0 and for any t ≥ 1 define

Then, for any δ ∈ (0 , 1] , we have

Lemma 33 (Lemma 1 of 76) . Let ℓ ( z , y ) = ∑ K k =0 1 { y = k } · log ( 1 [ σ ( z )] k ) , a ∈ [ -C, C ] K , y ∈ { 0 } ∪ [ K ] and b ∈ R K where C > 0 . Then, we have

Lemma 34 (Lemma 17 of 76) . Let ℓ ( z , y ) = ∑ K k =0 1 { y = k } · log ( 1 [ σ ( z )] k ) and z ∈ R K be a K -dimensional vector. Define z µ ≜ σ + (smooth µ ( σ ( z ))) , where smooth µ ( p ) = (1 -µ ) p + µ 1 / ( K +1) . Then, for µ ∈ [0 , 1 / 2] , we have

for any y ∈ { 0 } ∪ [ K ] . We also have ∥ z µ ∥ ∞ ≤ log( K/µ ) .

Lemma 35 (Lemma 18 of 76) . Let L i,h ( θ ) := ℓ i,h ( θ ) + 1 2 c ∥ θ -θ i h ∥ 2 B i,h . Assume that ℓ i,h is a √ N -self-concordant-like function. Then, for any θ , θ i h ∈ B ( 0 d , 1) , the quadratic approximation ˜ L i,h ( θ ) = L i,h ( ˜ θ i +1 h ) + ⟨∇ L i,h ( ˜ θ i +1 h ) , θ -˜ θ i +1 h ⟩ + 1 2 c ∥ ∥ ∥ θ -˜ θ i +1 h ∥ ∥ ∥ 2 B i,h satisfies

H Limitations

We make an assumption about the transition model of MDPs by using the MNL model, which is a specific parametric model. This assumption implies that we assume the realizability of the MNL model. It's worth noting that the realizability assumption has also been commonly made in previous literature on provable reinforcement learning with function approximation, including works such as [72, 43, 73, 53, 22, 14, 9, 68, 70, 33, 81, 82, 37, 35]. However, we hope that this condition can be relaxed in the future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract is to propose provably efficient randomized algorithms for MNL-MDPs. In Section 1 (Introduction), we provide the motivation and main contributions of this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of this work in Appendix H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions in Section 2.2 and a complete proof of main results in Appendix C and D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide numerical experiments that support our main claims in Section 5 and the detailed information of experiments in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have attached the data and code with sufficient instructions to reproduce the main experimental results in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the detailed explanation for the experimental setting in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars (standard deviation) in our numerical experiment results shown in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the detailed information on the computer resources used to conduct numerical experiments in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper adheres to the NeurIPS Code of Ethics in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no negative societal impacts of the work performed because this research focuses on theoretical aspects.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The research conducted in this paper does not pose any such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use any external assets such as code, data, or models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes the two randomized RL algorithms, RRL-MNL and ORRL-MNL, for the MNL transition model that for the first time achieve both computational and statistical efficiency without stochastic optimism. The complexity of ORRL-MNL has better dependence on the large problem-related constant $\\kappa^{-1}$ than RRL-MNL.\n\nSTRENGTHS:\nMNL parameterization is important and convenient since the parameterized transition kernel is always a probability distribution. This work is computationally more efficient than the existing work [35] on MNL parameterization.\n\nWEAKNESSES:\nSome details of the algorithms are not easily understood and are thus better to explain intuitively, as mentioned in questions 2-4 and 6-7 below. The regrets of both proposed algorithms are higher than $\\widetilde{O}(dH^{\\frac{3}{2}}\\sqrt{T})$ of [35]. Experiment reproducibility could be improved as shown in question 9 below.\n\nQUESTIONS:\n(1) Assumption 4 is equivalent to $\\inf _ {\\theta\\in \\mathcal{B} _ d(\\mathcal{L} _ {\\theta})} P_{\\theta}(s'|s,a)\\ge \\sqrt{\\kappa}, \\forall s,a,s'$, since $\\inf _ {s',\\widetilde{s}}P _ {\\theta}(s'|s,a)P _ {\\theta}(\\widetilde{s}|s,a)=[\\inf_{s'}P _ {\\theta}(s'|s,a)]^2$. \n\n(2) In eq. (2), how to select $L_{\\theta}$? Does eq. (2) aim to minimize the lost function $\\sum_{k=1}^N\\ell_{k,h}(\\theta)$? Why do you use Newton step instead of first-order algorithm? \n\n(3) Could you explain the intuition of eq. (3)? Is it inspired by the Hessian of the loss $\\ell_{k,h}(\\theta)$? \n\n(4) In the paragraphs about ``Stochastically optimistic value function'' after eq. (3), it seems that most words are about the drawbacks of existing works on optimistic estimated value function, and then you adapt optimistic sampling technique instead. It would be more clear to me if you introduce your eq. (4) first with more explanations, for example, what adjustments are made to the original optimistic sampling technique [7, 52, 37, 36] and why, why is $A_{j,k}^{-1}$ used instead of Euclidean norm in $\\hat{s}$ definition, and then list its advantage over existing works (maybe with less words than the current version?). Also, is your Remark 2 an advantage over relevant works like [35]? If yes, you may claim this advantage. \n\n[35] Taehyun Hwang and Min-hwan Oh. Model-based reinforcement learning with multinomial logistic function approximation. In Proceedings of the AAAI conference on artificial intelligence, pages 7971–7979, 2023.\n\n(5) In eqs. (4) and (7), is it tighter to truncate at level $H-h+1$ instead of $H$? \n\n(6) In eq. (6), is there an intuition why $\\nabla^2\\ell_{k,h}$ has coefficient $\\eta$ while $\\nabla^2\\ell_{i,h}$ ($i=1,\\ldots,k-1$) have coefficient 1? \n\n(7) Is there an intuitive explanation of $\\nu_{k,h}^{\\rm rand}(s,a)$ defined after eq. (7)? \n\n(8) You could also define $T$ in Theorem 1 to facilitate readers. The total number of steps is $T=KH$, yes? Is yes, is it better to use the equivalent sample complexity $\\kappa^{-1}d^{\\frac{3}{2}}H^2\\sqrt{K}$? \n\n(9) To ensure reproducible experiment, some hyperparameters are missing such as $M$, $\\lambda$, $\\kappa$, $\\eta$, $\\sigma_k$, $\\beta_k$, etc. \n\n(10) Could you compare your work with contemporary work [a]? \n\n[a] Li, Long-Fei, et al. \"Provably Efficient Reinforcement Learning with Multinomial Logit Function Approximation.\" ArXiv:2405.17061 (2024).\n\n(11) Is it possible to extend your work to stochastic policy and unknown reward? For example, will $\\epsilon$-greedy policy also encourage exploration more than greedy action? \n\n(12) Typo at the beginning of Section 2: {$P_h$} $_ {h=1}^H$ instead of {$P$} $_ {h=1}^H$.\n\nLIMITATIONS:\nThe final appendix section mentions the limitation of realizability assumption. I agree with the authors' claim that there is no negative societal impacts because this work focuses on theoretical aspects.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the MNL MDP setting and provides several regret bounds. The MNL setting is one where the transition probabilities are modeled as a soft-max over a linear transformation of state-action-next-state features. The work improves the dependence on an important problem parameter. The main results are presented using randomized exploration but a deterministic exploration scheme is also included in the appendix. Concretely, $\\kappa$ is a problem dependent constant that lower bounds any individual transition probability estimate. The authors explain that it relates to information gain in the transition kernel regression problem through the Fisher information matrix.\n\nSTRENGTHS:\n1. The new algorithm does not need to know $\\kappa$ (this is not the case in previous works)\n2. The obtained regret bound is improved from $\\approx \\kappa^{-1}\\sqrt{K}$ to $\\approx \\kappa^{-1} + \\sqrt{K}$\n3. the memory and computational requirements per episode are reduced from linear to constant in the number of episodes.\n\nWEAKNESSES:\n**Setting:**\nWhile the MNL model is quite intuitive, it has several shortcommings that, in my opinion, limit its usefulness.\n\n1. It does not seem to allow for a compact characterization of the value or policy. This is because these depend strongly on the size of the next state sets $\\mathcal{S}_{s,a}$. While instances where these sets are small exist this is still very limiting.\n2. The sets $\\mathcal{S}_{s,a}$ must be **known** to the algorithm.\n3. It is very hard to imagine a setting where $\\kappa^{-1}$ is not exponential in natural problem parameters.  \n\n\n**Presentation:**\n\n4. The paper is written in such a way that makes it hard to appreciate its contributions beyond the objective improvement to the regret bound. There are some discussions as to what makes the analysis more involved compared to standard TS and MNL bandits but I find it vague. Moreover, there are no proof sketches of any kind, which contributes to this issue. Overall, this makes it hard to appreciate the soundness of the paper and whether it contains meaningful technical contributions or not.\n\n\n**Results:**\n5. Most importantly, while the paper improves computational efficiency, It seems to me that it still has at least $|\\mathcal{U}|^H$ computations per round (to calculate the Q function along the policy trajectory). This exists in past algorithms as well and makes all of them inefficient (not polynomial in the natural problem parameters). Is this the case?\n6.  I do not see the randomized exploration as significant contribution. It both complicates the analysis significantly and deteriorates the regret. I appreciate the potential practical appeal but I do not think this strengthens the paper significantly. I personally think the focus of the paper should have been on the improved regret with the standard deterministic bonuses. This would also simplify the presentation.\n7. There is a recent concurrent paper ([1]) on this topic that seems to obtain similar results. While it is concurrent, I would appreciate if you provide a comparison in your response.\n\n[1] Li, Long-Fei, et al. \"Provably Efficient Reinforcement Learning with Multinomial Logit Function Approximation.\" arXiv preprint arXiv:2405.17061 (2024).‏\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning of an MDP with multinomial logistic (MNL) function approximation, which approximates the unknown transition kernel of the underlying MDP. The setting is the finite horizon episodic MDPs. Two algorithms are proposed. The first algorithm is statistically efficient with a provable frequentist regret bound, and is computationally efficient with constant-time computational cost per episode. To further improve the dependence on problem constant of the first algorithm, the second algorithm utilizes gradient information of the MNL model of the unknown transition. These two algorithms are the first randomized algorithms for the MNL transition model with both statistical and computational efficiency. Furthermore, numerical simulations show the good performance of the proposed algorithms.\n\nSTRENGTHS:\n**Significance**: \n\n**1.** The MNL transition model is a reasonable model for approximation to the unknown transition kernel and thus has potential practical usage. \n\n**2** The two algorithms are provably statistically efficient, which is further corroborated by numerical simulations. \n\n**Originality**: This paper has originality. It proposes to use multinomial distribution for modeling the transition kernel of MDP, and proposes the first efficient randomized algorithms for the MNL transition model.\n\n**Clarity**: the main text of the paper is clearly presented: algorithms are explained well (e.g. the intuition behind stochastically optimistic value function is provided); the theorems are presented with in-depth discussion of its relation to existing works, and the comparison of its proof techniques with other algorithmic guarantees in the area.\n\nWEAKNESSES:\nI did not find any major errors or technical flaws. However, the following can be improved.\n\n*Weaknesses*\n\n**1.** The theoretical analysis, especially the appendix, is a bit hard to read. This is caused by two issues: (i) there are many displayed equations that are very long; (ii) some derivations/steps are missing a proper explanation. \n\n**2.** The formatting of the equations in the appendix should be improved. There are many lines of equations which significantly exceed the length limit within a line. Probably should consider breaking the equations into combinations of shorter ones with proper explanation.\n\n*Other Minor issues*\n\n**1.** This paper so far only demonstrates the efficacy of the multinomial logistic model for approximating a synthetic MDP environment. Although the result from the numerical simulation is convincing, it would be very interesting to see how it works if applied to more realistic environment. This might be an interesting future direction.\n\nQUESTIONS:\nMy concerns are included in the **Weaknesses** session.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a computationally efficient randomized algorithm for MNL-MDPs, addressing limitations of previous exploration algorithms. It presents the RRL-MNL algorithm, focusing on online transition core estimation and stochastically optimistic value function design. This paper provides a frequentist regret analysis for MNL-MDPs with theoretical comparisons to other algorithms.\n\nSTRENGTHS:\n1. The paper is well-structured and effectively communicates complex theoretical concepts.\n\n2. The proposed algorithm, ORRL-MNL, is interesting, and it regret bound has an attractive improvement.\n\nWEAKNESSES:\n1. The focus on multinomial logistic function approximation and specific types of state transitions (inhomogeneous) may restrict the generalizability of the proposed algorithms. \n\n2. The paper lacks some discussion on the setting, MNL-MDPs to show it worth studying.\n\nQUESTIONS:\n1. Can you give a more detailed comparison between MNL-MDPs and Linear MDPs settings? Can you give a detailed discussion on the setting MNL-MDPs and show it worth studying? What is the connection between this setting and generalized linear MDPs?\n\n2. Can you summarize the theoretical challenges of randomized algorithms in MNL-MDPs compared with that in Linear MDPs?\n\n3. Can you explain the necessity of Assumption 4, which seems lack realizability motivation? I find that you mention this assumptions also use in generalized linear bandits, can you compare the assumptions in generalized linear bandit and in MNL-MDPs. Also, how about the generalized linear MDPs?\n\n4. The regret bound of ORRL-MNL is interesting. Do you have any new theoretical analysis techniques to obtain this?\n\nLIMITATIONS:\nThis paper lacks more discussions on the setting, MNL-MDPs to show it worth studying. I hope the authors can answer my questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}