Title: Provably Efficient Reinforcement Learning with Multinomial Logit Function Approximation
Venue: NeurIPS 2024
Accepted: yes
Keywords: MNL function approximation

regret analysis

reinforcement learning

Markov decision process
Abstract: We study a new class of MDPs that employs multinomial logit (MNL) function approximation to ensure valid probability distributions over the state space. Despite its significant benefits, incorporating the non-linear function raises substantial challenges in both *statistical* and *computational* efficiency. The best-known result of Hwang and Oh [2023] has achieved an $\widetilde{\mathcal{O}}(\kappa^{-1}dH^2\sqrt{K})$ regret upper bound, where $\kappa$ is a problem-dependent quantity, $d$ is the feature dimension, $H$ is the episode length, and $K$ is the number of episodes. However, we observe that $\kappa^{-1}$ exhibits polynomial dependence on the number of reachable states, which can be as large as the state space size in the worst case and thus undermines the motivation for function approximation. Additionally, their method requires storing all historical data and the time complexity scales linearly with the episode count, which is computationally expensive. In this work, we propose a statistically efficient algorithm that achieves a regret of $\widetilde{\mathcal{O}}(dH^2\sqrt{K} + \kappa^{-1}d^2H^2)$, eliminating the dependence on $\kappa^{-1}$ in the dominant term for the first time. We then address the computational challenges by introducing an enhanced algorithm that achieves the same regret guarantee but with only constant cost. Finally, we establish the first lower bound for this problem, justifying the optimality of our results in $d$ and $K$.

Provably Efficient Reinforcement Learning with Multinomial Logit Function Approximation

Abstract

We study a new class of MDPs that employs multinomial logit (MNL) function approximation to ensure valid probability distributions over the state space. Despite its significant benefits, incorporating the non-linear function raises substantial challenges in both statistical and computational efficiency. The best-known result of Hwang and Oh [2023] has achieved an ˜ O ( κ -1 dH 2 √ K ) regret upper bound, where κ is a problem-dependent quantity, d is the feature dimension, H is the episode length, and K is the number of episodes. However, we observe that κ -1 exhibits polynomial dependence on the number of reachable states, which can be as large as the state space size in the worst case and thus undermines the motivation for function approximation. Additionally, their method requires storing all historical data and the time complexity scales linearly with the episode count, which is computationally expensive. In this work, we propose a statistically efficient algorithm that achieves a regret of ˜ O ( dH 2 √ K + κ -1 d 2 H 2 ) , eliminating the dependence on κ -1 in the dominant term for the first time. We then address the computational challenges by introducing an enhanced algorithm that achieves the same regret guarantee but with only constant cost. Finally, we establish the first lower bound for this problem, justifying the optimality of our results in d and K .

1 Introduction

Reinforcement Learning (RL) with function approximation has achieved remarkable success in various applications involving large state and action spaces, such as games [Silver et al., 2016], algorithm discovery [Fawzi et al., 2022] and large language models [Ouyang et al., 2022]. Therefore, establishing the theoretical foundation for RL with function approximation is of great importance. Recently, there have been many efforts devoted to understanding the linear function approximation, yielding numerous valuable results [Yang and Wang, 2019, Jin et al., 2020, Ayoub et al., 2020].

While these studies make important steps toward understanding RL with function approximation, there are still challenges to be solved. In linear function approximation, transitions are assumed to be linear in feature mappings, such as P ( s ′ | s, a ) = ϕ ( s ′ | s, a ) ⊤ θ ∗ for linear mixture MDPs and P ( s ′ | s, a ) = ϕ ( s, a ) ⊤ µ ∗ ( s ′ ) for linear MDPs. Here P ( s ′ | s, a ) is the probability from state s to s ′ taking action a , ϕ ( s ′ | s, a ) and ϕ ( s, a ) are feature mappings, θ ∗ and µ ∗ ( s ′ ) are unknown parameters. However, the transition function is a probability distribution over states, meaning its values must lie within [0 , 1] and sum to 1. Thus, the linearity assumption is restrictive and hard to satisfy in practice. An algorithm designed for linear MDPs could break down entirely if the underlying MDP is not linear [Jin et al., 2020]. While some works explore generalized linear [Wang et al., 2021] and general function approximation [Russo and Roy, 2013, Foster et al., 2021, Chen et al., 2023], they focus on function approximation for value functions rather than transitions, hence do not tackle this challenge.

Towards addressing the limitation of linear function approximation, a new class of MDPs that utilizes multinomial logit function approximation has been proposed by Hwang and Oh [2023] recently. Such formula also aligns better with models like neural networks [LeCun et al., 2015], which inherently respect the probabilistic constraints through a softmax layer and allow for greater expressive power. However, though it offers promising benefits, the introduction of non-linear functions introduces significant challenges in both statistical and computational efficiency. Specifically, the best-known approach of Hwang and Oh [2023] has achieved an ˜ O ( κ -1 dH 2 √ K ) regret, where κ is a problemdependent quantity that measures the effective non-linearity over the entire parameter space, d is the feature dimension, H is the episode length, and K is the number of episodes. Unfortunately, as we show in Claim 1, it holds that κ -1 > U 2 , where U denotes the maximum number of reachable states, which equals to the size of the state space S in the worst case. This undermines the core motivation for function approximation, which aims to mitigate dependence on large state and action spaces. Furthermore, the method requires storing all historical data, and its time complexity per episode grows linearly with the episode count (i.e., O ( k ) at episode k ). Thus, a natural question arises:

Is it possible to design both statistically and computationally efficient algorithms for RL with MNL function approximation?

In this work, we answer this question affirmatively for the class of MNL mixture MDPs where the transition is parameterized by a multinomial logit function. Our contributions are listed as follows:

For statistical efficiency, we propose the UCRL-MNL-LL algorithm, which attains a regret bound of ˜ O ( dH 2 √ K + κ -1 d 2 H 2 ) . Our result significantly improves upon the ˜ O ( κ -1 dH 2 √ K ) rate of Hwang and Oh [2023], making the first time to achieve a κ -independent dominant term (note that the lower-order term still scales with κ -1 , but does not depend on K , making it acceptable). To achieve this, we propose a tighter confidence set based on a new Bernstein-type concentration [Périvier and Goyal, 2022] instead of the standard Hoeffding-type concentration, and exploit the self-concordant-like property [Bach, 2010] of the log-loss function to better use local information.

For computational efficiency, we propose the UCRL-MNL-OL algorithm, which enjoys the same regret bound as UCRL-MNL-LL, but with only constant storage and time complexity per episode. This is enabled by recognizing that the negative log-likelihood function is exponentially concave, which motivates the use of online mirror descent with a specifically tailored local norm [Zhang and Sugiyama, 2023] to replace the standard maximum likelihood estimation. Furthermore, we construct the optimistic value function by incorporating a closed-form bonus term through a second-order Taylor expansion, thus avoiding the need to solve a non-convex optimization problem.

We establish the first lower bound for MNL mixture MDPs by introducing a reduction to the logistic bandit problem. We prove a problem-dependent lower bound of Ω( dH √ Kκ ∗ ) for infinite action setting, where κ ∗ is an another problem-dependent quantity that measures the effective non-linearity over around the ground truth parameter. Though this does not constitute a strict lower bound for the finite action case studied in this work, it suggests that our result may be optimal in d and K . 1

Table 1 provides a comparison between our work and previous studies, focusing on regret and computational costs, including both storage and time complexity.

Organization. We introduce the related work in Section 2 and present the setup in Section 3. Then, we design a statistically efficient algorithm in Section 4. Next, we present an algorithm that achieves both statistical and computational efficiency in Section 5. Finally, we establish the lower bound in Section 6. Section 7 concludes the paper. Due to space limits, we defer all proofs to the appendixes.

Notations. We use [ x ] [ a,b ] to denote min(max( x, a ) , b ) . For a vector x ∈ R d and positive semidefinite matrix A ∈ R d × d , denote ∥ x ∥ A = √ x ⊤ A x . For a strictly convex and continuously differentiable function ψ : W ↦→ R , the Bregman divergence is defined as D ψ ( w 1 , w 2 ) = ψ ( w 1 ) -ψ ( w 2 ) -⟨∇ ψ ( w 2 ) , w 1 -w 2 ⟩ . Weuse the notation O ( · ) to indicate different types of dependencies depending on the context. For regret analysis, O ( · ) omits only constant factors. For computational costs, we use O ( · ) to solely highlight the dependence on the number of episode as this is the primary factor influencing the complexity. Additionally, we employ ˜ O ( · ) to hide all polylogarithmic factors.

2 Related Work

In this section, we review related works from both setup and technical perspectives.

RL with Generalized Linear Function Approximation. There are recent efforts devoted to investigating function approximation beyond the linear models. Wang et al. [2021] investigated RL with generalized linear function approximation. Notably, unlike our approach that models transitions using a generalized linear model, they apply this approximation directly to the value function. Another line of works [Chowdhury et al., 2021, Li et al., 2022, Ouhamma et al., 2023] has studied RL with exponential function approximation and also aimed to ensure that transitions constitute valid probability distributions. The MDP model can be viewed as an extension of bilinear MDPs in their work while our setting extends linear mixture MDPs. These studies are complementary to ours and not directly comparable. Moreover, these works also enter the computational and statistical challenges arising from non-linear function approximation that remain to be addressed. The most relevant work to ours is the recent work by Hwang and Oh [2023], which firstly explored a similar setting to ours, where the transition is characterized using a multinomial logit model. We significantly improve upon their results by providing statistically and computationally more efficient algorithms.

RL with General Function Approximation. There have also been some works that studies RL with general function approximation. Russo and Roy [2013] and Osband and Roy [2014] initiated the study on the minimal structural assumptions that render sample-efficient learning by proposing a structural condition called Eluder dimension. Recently, several works have investigated different conditions for sample-efficient interactive learning, such as Bellman Eluder (BE) dimension [Jin et al., 2021], Bilinear classes [Du et al., 2021], Decision-Estimation Coefficient (DEC) [Foster et al., 2021], and Admissible Bellman Characterization (ABC) [Chen et al., 2023]. A notable difference is that they impose assumptions on the value functions while we study function approximation on the transitions to ensure valid probability distributions. Moreover, the goal of these works is to study the conditions for sample-efficient reinforcement learning, but not focus on the computational efficiency.

Multinomial Logit Bandits. There are two types of multinomial logit bandits studied in the literature: the single-parameter model, where the parameter is a vector [Cheung and Simchi-Levi, 2017] and multiple-parameter model, where the parameter is a matrix [Amani and Thrampoulidis, 2021]. We focus on the single-parameter model, which are more relevant to our setting. The pioneering work by Cheung and Simchi-Levi [2017] achieved a Bayesian regret of ˜ O ( κ -1 d √ T ) , where T denotes the number of rounds in bandits. This result was further enhanced by subsequent studies [Oh and Iyengar, 2019, 2021, Agrawal et al., 2023]. In particular, Périvier and Goyal [2022] significantly improved the dependence on κ , obtaining a regret of ˜ O ( d √ κT + κ -1 ) in the uniform revenue setting. Most prior methods required storing all historical data and faced computational challenge. To address this issue, the most recent work by Lee and Oh [2024] proposed an algorithm with constant computational and storage costs building on recent advances in multiple-parameter model [Zhang and Sugiyama, 2023]. Their algorithm achieves the optimal regret of ˜ O ( d √ κT + κ -1 ) and ˜ O ( d √ T + κ -1 ) under uniform and non-uniform rewards respectively. However, although the underlying models of MNL bandits and MDPs share similarities, the challenges they present differ substantially, and techniques developed for MNL bandits cannot be directly applied to MNL MDPs. For example, in MNL bandits, the objective is to select a series of assortments with varying sizes that maximize the expected revenue, whereas in MNL MDPs, the goal is to choose one action at each stage to maximize the cumulative reward. Thus, it is necessary to design new algorithms tailored for MDPs to address these unique challenges.

3 Problem Setup

In this section, we present the problem setup of RL with multinomial logit function approximation.

Inhomogeneous, Episodic MDPs. An inhomogeneous, episodic MDP instance can be denoted by a tuple M = ( S , A , H, { P h } H h =1 , { r h } H h =1 ) , where S is the state space, A is the action space, H is the length of each episode, P h : S ×A×S → [0 , 1] is the transition kernel with P h ( s ′ | s, a ) is being the probability of transferring to state s ′ from state s and taking action a at stage h , r h : S × A → [0 , 1] is the deterministic reward function. A policy π = { π h } H h =1 is a collection of mapping π h , where each π h : S → ∆( A ) is a function maps a state s to distributions over A at stage h . For any policy π and ( s, a ) ∈ S × A , we define the action-value function Q π h and value function V π h as follows:

where the expectation of Q π h is taken over the randomness of the transition P and policy π . The optimal value function V ∗ h and action-value function Q ∗ h given by V ∗ h ( s ) = sup π V π h ( s ) and Q ∗ h ( s, a ) = sup π Q π h ( s, a ) . For any function V : S → R , we define [ P h V ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) V ( s ′ ) .

Learning Protocol. In the online MDP setting, the learner interacts with the environment without the knowledge of the transition kernel { P h } H h =1 . We assume the reward function { r h } H h =1 is deterministic and known to the learner. The interaction proceeds in K episodes. At the beginning of episode k , the learner chooses a policy π k = { π k,h } H h =1 . At each stage h ∈ [ H ] , starting from the initial state s k, 1 , the learner observes the state s k,h , chooses an action a k,h sampled from π k,h ( · | s k,h ) , obtains reward r h ( s k,h , a k,h ) and transits to the next state s k,h +1 ∼ P h ( · | s k,h , a k,h ) for h ∈ [ H ] . The episode ends when s H +1 is reached. The goal of the learner is to minimize regret, defined as

which is the difference between the cumulative reward of the optimal policy and the learner's policy.

Multinomial Logit (MNL) Mixture MDPs. Although significant advances have been achieved for MDPs with linear function approximation, Hwang and Oh [2023] show that there exists a set of features such that no linear transition model can induce a valid probability distribution, which limits the expressiveness of function approximation. To overcome this limitation, they propose a new class of MDPs with multinomial logit function approximation. However, their work focuses on the homogeneous setting, where the transitions remain the same across all stages (i.e., P 1 = ... = P H ). In this work, we address the more general inhomogeneous setting, allowing transitions to vary across different stages. We introduce the formal definition of inhomogeneous MNL mixture MDPs below.

Definition 1 (Reachable States) . For any ( h, s, a ) ∈ [ H ] ×S × A , we define the 'reachable states' as the set of states that can be reached from state s taking action a at stage h within a single transition, i.e., S h,s,a ≜ { s ′ ∈ S | P h ( s ′ | s, a ) > 0 } . Furthermore, we define S h,s,a ≜ |S h,s,a | and denote by U ≜ max ( h,s,a ) S h,s,a the maximum number of reachable states.

Definition 2 (MNL Mixture MDP) . An MDP instance M = ( S , A , H, { P h } H h =1 , { r h } H h =1 ) is called an inhomogeneous, episodic B -bounded MNL mixture MDP if there exist a known feature mapping ϕ ( s ′ | s, a ) : S × A × S → R d with ∥ ϕ ( s ′ | s, a ) ∥ 2 ≤ 1 and unknown vectors { θ ∗ h } H h =1 ∈ Θ with Θ = { θ ∈ R d , ∥ θ ∥ 2 ≤ B } , such that for all ( s, a, h ) ∈ S × A × [ H ] and s ′ ∈ S h,s,a , it holds that

Remark 1. This model is consistent with models like neural networks [LeCun et al., 2015], where the feature ϕ is obtained by omitting the final layer, and θ ∗ h represents the weights of the last layer. A final softmax layer is then applied to ensure that the output forms a valid probability distribution.

For any θ ∈ R d , we define the induced transition as p s ′ s,a ( θ ) = exp( ϕ ( s ′ | s,a ) ⊤ θ ) ∑ ˜ s ∈S s,a exp( ϕ ( ˜ s | s,a ) ⊤ θ ) . We then introduce the following two key problem-dependent quantities κ and κ ∗ that measure the effective non-linearity over the entire parameter space and around the ground truth parameter respectively.

Assumption 1. There exists 0 < κ ≤ κ ∗ < 1 such that for all ( s, a, h ) ∈ S × A × [ H ] and s ′ , s ′′ ∈ S h,s,a , it holds that inf θ ∈ Θ p s ′ s,a ( θ ) p s ′′ s,a ( θ ) ≥ κ and p s ′ s,a ( θ ∗ h ) p s ′′ s,a ( θ ∗ h ) ≥ κ ∗ .

Assumption 1 is similar to the assumption in generalized linear bandit [Filippi et al., 2010] and logistic bandit [Faury et al., 2020, Abeille et al., 2021] to guarantee the Hessian matrix is non-singular.

Finally, we show the claim about the range of the magnitude of κ and κ ∗ .

Claim 1. It holds that 1 / ( U exp(2 B )) 2 ≤ κ ≤ κ ∗ ≤ 1 /U 2 .

4 Statistically Efficient Algorithm

The work of Hwang and Oh [2023] first introduced the MNL mixture MDPs and proposed an algorithm with a regret bound of ˜ O ( κ -1 dH 2 √ K ) . However, as discussed in Claim 1, it follows that U 2 ≤ κ -1 ≤ ( U exp(2 B )) 2 , which results in the regret bound scaling polynomially with the number of reachable states U . In the worst case, U can be equal to the size of the state space S , thereby undermining the motivation for function approximation, which aims to mitigate the dependence on the large state and action spaces. In this section, we address this significant issue by proposing a statistically efficient algorithm that eliminates this dependence in the dominant term of the regret.

4.1 Parameter Estimation

In this section, we first present the parameter estimation method based on the maximum likelihood estimation (MLE) for MNL mixture MDPs. Next, we review the confidence set construction based on the estimated parameters from previous work [Hwang and Oh, 2023]. Finally, we propose our new confidence set construction and highlight the improvements it offers over the previous approach.

Since the transition parameter θ ∗ h is unknown, we need to estimate it using the historical data. At episode k , we collect a trajectory { ( s k,h , a k,h ) } H h =1 , then define the variable: y k,h ∈ { 0 , 1 } S k,h where y s ′ k,h = 1 { s ′ = s k,h +1 } for s ′ ∈ S k,h ≜ S s k,h ,a k,h and S k,h = |S k,h | . We denote by p s ′ k,h ( θ ) = p s ′ s k,h ,a k,h ( θ ) . Then y k,h is a sample from the following multinomial distribution:

where the parameter 1 indicates that y k,h is a single-trial sample. Furthermore, we define the noise ϵ s ′ k,h = y s ′ k,h -p s ′ k,h ( θ ∗ h ) . It is clear that ϵ k,h ∈ [ -1 , 1] S k,h , E [ ϵ k,h ] = 0 and ∑ s ′ ∈S k,h ϵ s ′ i,h = 0 .

We estimate the parameter θ ∗ h using the MLE and construct the estimator ̂ θ k,h as follows:

where λ k is the regularization parameter. Though the MLE estimator ̂ θ k,h is the same as that of Hwang and Oh [2023], the confidence set is constructed differently. Specifically, define the gradient G k,h ( θ ) and Hessian matrix H k,h ( θ ) of the MLE loss by

Furthermore, we define the feature covariance matrix A k,h = κ -1 λ k I + ∑ k -1 i =1 ∑ s ′ ∈S i,h ϕ s ′ i,h ( ϕ s ′ i,h ) ⊤ . By demonstrating H k,h ( θ ) ⪰ κA k,h , ∀ θ ∈ Θ , Hwang and Oh [2023] construct the confidence set as

Since the radius of the confidence set depends on κ -1 , the final regret bound also exhibits a dependence on κ -1 . To eliminates this dependence, we construct a κ -independent confidence set based on

new Bernstein-like inequalities in Lemma 13, following recent advances in logistic bandits [Faury et al., 2020, Périvier and Goyal, 2022]. Specifically, we show the following lemma.

Lemma 1. For any δ ∈ (0 , 1) , set λ k = d log( kH/δ ) and define the confidence set as

Then, we have Pr[ θ ∗ h ∈ ̂ C k,h ] ≥ 1 -δ, ∀ k ∈ [ K ] , h ∈ [ H ] .

Comparison to prior work. We compare the confidence sets defined in Eqs. (2) and (3).

For the confidence set in (3), by the self-concordance property of log-loss in Lemma 10, we have:

Then, note that H k,h ( θ ) ⪰ κA k,h for all θ ∈ Θ , we have

Thus, compared to the confidence set in Eq. (2) from Hwang and Oh [2023], our confidence set in Eq. (3) provides a strict improvement by at least a factor of κ -1 / 2 . This improvement is one of the key components to eliminate the dependence on κ -1 in the dominant term of the final regret bound.

Additionally, we identify a technical issue of Hwang and Oh [2023]. Specifically, they bound the confidence set in Eq. (2) using the self-normalized concentration in Lemma 12. However, the noise is not independent, and since ∑ s ′ ∈S i,h ϵ s ′ i,h = 0 (due to the learner visiting each stage h exactly once per episode), it does not satisfy the zero-mean sub-Gaussian condition in Lemma 12. We observe similar oversights in multinomial logit contextual bandits [Oh and Iyengar, 2019, 2021, Agrawal et al., 2023], an issue that, to our knowledge, has not been explicitly addressed in prior work. This issue can be resolved with only slight modifications in constant factors by a new self-normalized concentration with dependent noises in Lemma 1 of Li et al. [2024b], a simplified version of Lemma 13.

4.2 Optimistic Value Function Construction

Given the confidence set ̂ C k,h , it is natural to follow the principle of 'optimism in the face of uncertainty' and construct the optimistic value function. Hwang and Oh [2023] constructed the optimistic value function ¯ Q k,h ( s, a ) by adding a closed-form upper confidence bound as follows:

where ¯ V k,h ( s ) = max a ∈A ¯ Q k,h ( s, a ) . Then, a naive idea to compute the optimistic value function is replacing the radius of the confidence set β k with ̂ β k and the feature covariance matrix A k,h with the Hessian matrix H k,h ( θ ∗ h ) . However, there are two issues with this approach. First, the true parameter θ ∗ h is unknown thus the Hessian matrix H k,h ( θ ∗ h ) is not computable in the algorithmic updates. Second, though ̂ β k is independent of κ and the Hessian matrix H k,h ( θ ∗ h ) captures local information, the bonus term max s ′ ∈S h,s,a ∥ ϕ s ′ s,a ∥ H -1 k,h ( θ ) remains in a global form. This term involves taking the maximum over all states s ′ ∈ S h,s,a , which prevents fully utilizing the local information.

To address these challenges, we construct the optimistic value function by directly taking the maximum expected reward over the confidence set. Specifically, we define ̂ Q k,h ( s, a ) and ̂ V k,h ( s ) as

This construction addresses the first challenge by eliminating the need for the Hessian matrix H k,h ( θ ∗ h ) and directly leveraging the local information embedded in the confidence set ̂ C k,h . For the second challenge, although we bypass this issue in the construction of the optimistic value function, we still need to address it in the analysis. To tackle this, we employ a second-order Taylor expansion, in contrast to the first-order expansion used in the analysis of Hwang and Oh [2023]. This allows for a more precise capture of local information. Further details are provided in Lemma 7 in the appendix.

Algorithm 1 UCRL-MNL-LL

4.3 Regret Guarantee

Based on the parameter estimation in Section 4.1 and the construction of the optimistic value function in Section 4.2, we propose the UCRL-MNL-LL algorithm. At each stage h of episode k , the algorithm observes the current state s k,h and selects the action that maximizes the value function, i.e., a k,h = arg max a ∈A ̂ Q k,h ( s k,h , a ) , and transits to next state s k,h +1 . After collecting the trajectory { s k,h , a k,h } H h =1 , the estimator ̂ θ k +1 ,h is updated using Eq. (1), and the confidence set ̂ C k +1 ,h is updated according to Eq. (3). Then, the value function ̂ Q k +1 ,h and ̂ V k +1 ,h are updated using Eq. (6). The detailed procedure is outlined in Algorithm 1. We show it achieves the following regret guarantee.

Theorem 1. For any δ ∈ (0 , 1) , set λ k = d log( kH/δ ) , and ̂ β k = ( B +3) √ d log( kH/δ ) . With probability at least 1 -δ , UCRL-MNL-LL algorithm (Algorithm 1) ensures the following guarantee:

Remark 2. Focusing on the dominant term, our guarantee eliminates the problematic dependence on κ -1 , in stark contrast to the ˜ O ( κ -1 dH 2 √ K ) result of Hwang and Oh [2023]. As noted in Claim 1, such undesirable dependence has polynomial scaling with the number of reachable states U , which can be as large as the entire state space S in the worst case. This renders the guarantee for function approximation-designed for settings with large state and action spaces-essentially vacuous.

5 Computationally Efficient Algorithm

While the UCRL-MNL-LL algorithm is the first statistically efficient algorithm for MNL mixture MDPs, it is computationally expensive dur the the optimization of the MLE in Eq. (1) and non-convex optimization in Eq. (6). To address these challenges, we propose a computationally efficient algorithm in this section, which attains the same regret but with constant computational costs per episode.

5.1 Efficient Online Parameter Estimation

In this section, we focus on estimating the unknown parameter θ ∗ h in a computationally efficient manner. We first discuss the storage and time complexities of the MLE optimization in Eq. (1). Next, we introduce an efficient online parameter estimation based on online mirror descent that provides similar guarantees to the MLE, but with constant storage and time complexity per episode.

For the storage complexity, the optimization problem defined in Eq. (1) requires storing all historical data, resulting in a storage complexity of O ( k ) at episode k . In terms of time complexity, the problem does not have a closed-form solution and can only be solved to within an ε -accuracy, such as using projected gradient descent. As discussed in Faury et al. [2022], optimizing the MLE typically requires O (log(1 /ε )) iterations to achieve an ε -accurate solution. Since the loss function is defined over all historical data, each gradient step incurs a query complexity of O ( k ) . As ε is usually chosen as 1 /k for episode k , the total time complexity is O ( k log k ) at episode k . Consequently, both storage and time complexities scale linearly with the episode count, which is computationally expensive.

To improve the computational efficiency, the basic idea is to estimate the unknown parameter with the online mirror descent (OMD) update instead of the MLE as defined in Eq. (1). To this end, we first define per-episode loss function ℓ k,h ( θ ) , gradient g k,h ( θ ) and Hessian matrix H k,h ( θ ) as

Then, the design of the OMD algorithm can be conceptually divided into two parts: the approximation of the past losses and the approximation of the current loss. We provide the details of each below.

Approximate the past losses. To integrate historical information from previous iterations while avoiding the use of MLE in Eq. (1), we construct the estimator ¯ θ k +1 ,h using the implicit OMD form:

where η is a step size and ¯ H k,h ≜ ¯ H k,h ( ¯ θ k +1 ,h ) = ∑ k -1 i =1 H i,h ( ¯ θ i +1 ,h ) + λ k I . The optimization problem can be decomposed in two terms. The first term is the instantaneous log-loss ℓ k,h ( θ ) , which accounts for the information of the current episode. The second is a regularization term that ensures the current model remains close to the previous one, ¯ θ k,h , thereby incorporating the historical information acquired so far. The most critical aspect in the above is the design of the local norm ¯ H k,h , which intentionally approximate the per-episode Hessian matrix by H i ( ¯ θ i +1 ,h ) at a look ahead point ¯ θ i +1 ,h . Such a Hessian matrix, originally introduced by Faury et al. [2022], effectively captures the local curvature of the loss function and is crucial for ensuring statistical efficiency.

The update rule in Eq. (8) is storage efficient, as it only requires storing the Hessian matrix ¯ H k,h , which can be updated incrementally, resulting in an O (1) storage cost. In terms of time complexity, the optimization problem in Eq. (8) suffers an O (log k ) time complexity at episode k , since the loss function is defined only over the current episode. While this represents a significant improvement over the O ( k log k ) time complexity of the MLE in Eq. (1), there is still a need to reduce the cost further to O (1) per episode, particularly given the potentially large number of episodes.

)

Approximate the current loss. To achieve O (1) time complexity per episode, we can further approximate the current loss with a second order approximation. Drawing inspiration from Zhang and Sugiyama [2023], we define the second-order approximation of the original loss function ℓ k,h ( θ at ˜ θ k,h as ˜ ℓ k,h ( θ ) = ℓ k,h ( ˜ θ k,h ) + ⟨∇ ℓ k,h ( ˜ θ k,h ) , θ -˜ θ k,h ⟩ + 1 2 ∥ θ -˜ θ k,h ∥ 2 H k,h ( ˜ θ k,h ) , where ˜ θ k,h is the current estimate. Then, we can replace ℓ k,h ( θ ) with its second-order approximation ˜ ℓ k,h ( θ ) in the optimization problem in Eq. (8). This leads to the following approximate optimization problem:

where η is the step size, ˜ H k,h = H k,h + ηH k,h ( ˜ θ k,h ) and H k,h = ∑ k -1 i =1 H i,h ( ˜ θ i +1 ,h ) + λ k I . Then, Eq. (9) can be solved with a single projected gradient step with the following equivalent formulation:

Thus, Eq. (9) is computationally efficient, as it only suffers an O (1) storage and time complexity. Notice that the update rule in Eq. (9) is actually a standard online mirror descent (OMD) formula,

where the regularizer is ψ k ( θ ) = 1 2 ∥ θ ∥ 2 ˜ H k,h and D ψ k ( · , · ) is the induced Bregman divergence. Therefore, we can construct the confidence set building upon the modern analysis of OMD [Orabona, 2019, Zhao et al., 2024]. Specifically, we can construct the κ -independent confidence set as follows. √

where ˜ β k = O ( √ d log U log( kH/δ )) . Then, we have Pr[ θ ∗ h ∈ ˜ C k,h ] ≥ 1 -δ, ∀ k ∈ [ K ] , h ∈ [ H ] .

Remark 3. Compared to the confidence set in Lemma 1, the radius ˜ β k in Lemma 2 includes an additional log U factor. This is due to our approximation of the original MLE using the OMD update.

Algorithm 2 UCRL-MNL-OL

Input: Step size η , regularization parameter λ , confidence width ˜ β k , confidence parameter δ .

1: Initialization: H 1 ,h = λI, ̂ θ 1 ,h = 0 for all h ∈ [ H ] .

2: for k = 1 , . . . , K do

3: Compute ˜ Q k,h ( · , · ) in a backward way as in Eq. (11).

4: for h = 1 , . . . , H do

5: Observe state s k,h , select action a k,h = arg max a ∈A ˜ Q k,h ( s k,h , a ) .

6: Update ˜ H k,h = H k,h + ηH k,h ( ˜ θ k,h ) .

7: Compute ˜ θ k +1 ,h = arg min θ ∈ Θ ⟨∇ ℓ k,h ( ˜ θ k,h ) , θ -˜ θ k,h ⟩ + 1 2 η ∥ θ -˜ θ k,h ∥ 2 ˜ H k,h .

8: Update H k +1 ,h = H k,h + H k,h ( ˜ θ k +1 ,h ) .

9: end for

10: end for

5.2 Efficient Optimistic Value Function Construction

Although the optimistic value function in Eq. (6) preserves local information effectively and provides strong theoretical guarantees, it is computationally intractable due to the need to solve a non-convex optimization problem. To address this challenge, we propose an efficient method in this section.

The key idea is to use a second-order Taylor expansion to derive a closed-form bonus term, which replaces the operation of taking the maximum over the non-convex confidence set. While this idea has been used in bandit settings, fundamental challenges arise when applying it in the MDP setting. Specifically, Zhang and Sugiyama [2023] studied the multi-parameter MLogB bandit, where each outcome is associated with a distinct parameter vector. In contrast, MNL mixture MDPs involve a single shared parameter vector across all outcomes. This distinction leads to a more complex Hessian matrix, necessitating a more sophisticated analysis. A direct use of their analysis will leads to a polynomial dependence on the number of reachable states U , which is undesirable in the MDP setting. Lee and Oh [2024] focused on the single-parameter MNL bandit, which is more closely related to our setting. However, they construct the optimistic value function by directly taking the maximum over the confidence set, a computationally intractable approach in the MDP setting. As a result, they can apply a second-order Taylor expansion around the ground truth parameter θ ∗ h in their analysis, while we must apply it around the estimated parameter ˜ θ k,h to construct the bonus term explicitly.

, it holds

For MDPs, we show the value difference arising from the transition estimation error as follows. Lemma 3. Suppose Lemma 2 holds. For any V : S → [0 , H ] and ( h, s, a ) ∈ [ H ] ×S × A

where

Based on Lemma 3, we construct the optimistic value function as follows:

where ˜ V k,h ( s ) = max a ∈A ˜ Q k,h ( s, a ) . In contrast to the value function in Eq. (5), which incorporates the term max s ′ ∈S h,s,a ∥ ϕ s ′ s,a ∥ H -1 k,h , the refined value function in Eq. (11) replaces it with ϵ fst s,a + ϵ snd s,a . This modification better preserves local information, offering a more accurate estimation error bound.

5.3 Regret Guarantee

The overall algorithm UCRL-MNL-OL is similar to UCRL-MNL-LL, but with the estimator and optimistic value function updated in a computationally efficient manner. The detailed algorithm is presented in Algorithm 2. We provide the guarantee of UCRL-MNL-OL in the following theorem.

Theorem 2. For any δ ∈ (0 , 1) , set ˜ β k = O ( √ d log U log( kH/δ )) , η = 1 2 log(1 + U ) + ( B +1) and λ = 84 √ 2 η ( B + d ) , with probability at least 1 -δ , UCRL-MNL-LL (Algorithm 2) ensures

Remark 4. UCRL-MNL-OL attains the same regret as UCRL-MNL-LL, but with constant computational cost per episode. This is achieved by constructing an efficient online estimation based on OMDand an optimistic value function by closed-form bonus instead of the non-convex optimization.

6 Lower Bound

In this section, we establish the lower bound for MNL mixture MDPs by presenting a novel reduction, which connects MNL mixture MDPs and the logistic bandit problem.

Consider the following logistic bandit problem [Faury et al., 2020]: at each round t ∈ [ T ] , the learner selects an action x t ∈ X and receives a reward r t sampled from Bernoulli distribution with mean µ ( x ⊤ θ ∗ ) = (1 + exp( -x ⊤ θ ∗ )) -1 , where θ ∗ ∈ { θ ∈ R d , ∥ θ ∥ 2 ≤ B } is the unknown parameter. The learner aims to to minimize the regret: Reg LogB ( T ) = max x ∈X ∑ T t =1 µ ( x ⊤ θ ∗ ) -∑ T t =1 µ ( x ⊤ t θ ∗ ) .

Theorem 3. For any logistic bandit problem B , there exists an MNL mixture MDP M such that learning M is as hard as learning H/ 2 independent instances of B simultaneously.

Corollary 1 (Lower Bound) . For any problem instance { θ ∗ h } H h =1 and for K ≥ d 2 κ ∗ , there exists an MNL mixture MDP with infinite action space such that Reg( K ) ≥ Ω( dH √ Kκ ∗ ) .

Remark 5. Corollary 1 also implies a problem-independent lower bound of Ω( dH √ K ) directly. Corollary 1 can be proved by combining Theorem 3 and the Ω( d √ Tκ ∗ ) lower bound for logistic bandits with infinite arms by Abeille et al. [2021]. To the best of our knowledge, a lower bound for logistic bandits with finite arms has not been established, which is beyond the scope of this work. This absence leaves the lower bound for MNL mixture MDPs with a finite action space open through this reduction. However, after the submission of our work to arXiv [Li et al., 2024a], a follow up work by Park et al. [2024] proposed a new reduction that bridges MNL mixture MDPs with linear mixture MDPs by approximating MNL functions to linear functions. Leveraging this new reduction, they established a problem-independent Ω( dH 3 / 2 √ K ) lower bound for the finite action setting. This achievement confirms that our result is optimal in d and K , only loosing by an O ( H 1 / 2 ) factor.

Dependence on H . By the discussion in Remark 5, we note that our result is optimal with respect to d and K , but loosing by an O ( H 1 / 2 ) factor. We discuss the challenges in improving the dependence on H . Notably, MNL mixture MDPs can be viewed as a generalization of linear mixture MDPs [Ayoub et al., 2020, Zhou et al., 2021]. The pioneering work by Ayoub et al. [2020] achieved a regret bound of ˜ O ( dH 2 √ K ) for linear mixture MDPs, which matches our results in Theorem 1, differing only on the lower-order term. Later, Zhou et al. [2021] enhanced the dependence on H and attained an optimal regret bound of ˜ O ( d √ H 3 K ) . This was made possible by recognizing that the value function in linear mixture MDPs is linear, allowing for direct learning of the value function while incorporating variance information . In contrast, the value function for MNL mixture MDPs does not conform to a specific structure, posing a significant challenge in using the variance information of value functions. Thus, it remains open whether similar improvements on H are attainable for MNL mixture MDPs.

7 Conclusion and Future Work

In this work, we addressing both statistical and computational challenges for MNL mixture MDPs, which leverage MNL function approximation to ensure valid probability distributions. Specifically, we propose a statistically efficient algorithm that achieve a regret of ˜ O ( dH 2 √ K + κ -1 d 2 H 2 ) , eliminating the dependence on κ -1 in the dominant term for the first time. Then, we introduce a computationally enhanced algorithm that achieves the same regret but with only constant cost. Finally, we establish the first lower bound for this problem, justifying the optimality of our results in d and K .

There are several interesting directions for future work. First, there still exists a gap between the upper and lower bounds. How to close this gap remains an open problem. Besides, we focuses on stationary rewards in this work, extending MNL mixture MDPs to the non-stationary settings and studying the dynamic regret [Wei and Luo, 2021, Zhao et al., 2022, Li et al., 2023] is also an important direction.

Acknowledgments

This research was supported by National Science and Technology Major Project (2022ZD0114800) and NSFC (U23A20382, 62206125). Peng Zhao was supported in part by the Xiaomi Foundation.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems 24 (NIPS) , pages 2312-2320, 2011.

Marc Abeille, Louis Faury, and Clément Calauzènes. Instance-wise minimax-optimal algorithms for logistic bandits. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 3691-3699, 2021.

Priyank Agrawal, Theja Tulabandhula, and Vashist Avadhanula. A tractable online learning algorithm for the multinomial logit contextual bandit. European Journal of Operational Research , 310(2): 737-750, 2023.

Sanae Amani and Christos Thrampoulidis. Ucb-based algorithms for multinomial logistic regression bandits. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 2913-2924, 2021.

Alex Ayoub, Zeyu Jia, Csaba Szepesvári, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 463-474, 2020.

Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics , 4: 384-414, 2010.

Nicolò Campolongo and Francesco Orabona. Temporal variability in implicit online learning. In Advances in Neural Information Processing Systems 33 (NeurIPS) , 2020.

Zixiang Chen, Chris Junchi Li, Huizhuo Yuan, Quanquan Gu, and Michael I. Jordan. A general framework for sample-efficient function approximation in reinforcement learning. In Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023.

Wang Chi Cheung and David Simchi-Levi. Thompson sampling for online personalized assortment optimization problems with multinomial logit choice models. Available at SSRN 3075658 , 2017.

Sayak Ray Chowdhury, Aditya Gopalan, and Odalric-Ambrym Maillard. Reinforcement learning in parametric mdps with exponential families. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1855-1863, 2021.

Simon S. Du, Sham M. Kakade, Jason D. Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in RL. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 2826-2836, 2021.

Louis Faury, Marc Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 3052-3060, 2020.

Louis Faury, Marc Abeille, Kwang-Sung Jun, and Clément Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 546-580, 2022.

Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature , 610(7930):47-53, 2022.

Sarah Filippi, Olivier Cappé, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems 23 (NIPS) , pages 586-594, 2010.

Dylan J. Foster, Sham M. Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. ArXiv preprint , 2112.13487, 2021.

Taehyun Hwang and Min-hwan Oh. Model-based reinforcement learning with multinomial logistic function approximation. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI) , pages 7971-7979, 2023.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Proceedings of the 33rd Conference on Learning Theory (COLT) , pages 2137-2143, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman Eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 13406-13418, 2021.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Joongkyu Lee and Min-hwan Oh. Nearly minimax optimal regret for multinomial logistic bandit. In Advances in Neural Information Processing Systems 36 (NeurIPS) , page to appear, 2024.

Gene Li, Junbo Li, Anmol Kabra, Nati Srebro, Zhaoran Wang, and Zhuoran Yang. Exponential family model-based reinforcement learning via score matching. In Advances in Neural Information Processing Systems 35 (NeurIPS) , 2022.

Long-Fei Li, Peng Zhao, and Zhi-Hua Zhou. Dynamic regret of adversarial linear mixture MDPs. In Advances in Neural Information Processing Systems 36 (NeurIPS) , pages 60685-60711, 2023.

Long-Fei Li, Yu-Jie Zhang, Peng Zhao, and Zhi-Hua Zhou. Provably efficient reinforcement learning with multinomial logit function approximation. ArXiv preprint , 2405.17061, 2024a. URL https://arxiv.org/abs/2405.17061v1 .

Long-Fei Li, Peng Zhao, and Zhi-Hua Zhou. Improved algorithm for adversarial linear mixture MDPs with bandit feedback and unknown transition. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 3061-3069, 2024b.

Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. In Advances in Neural Information Processing Systems 32 (NeurIPS) , pages 3145-3155, 2019.

Min-hwan Oh and Garud Iyengar. Multinomial logit contextual bandits: Provable optimality and practicality. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI) , pages 9205-9213, 2021.

Francesco Orabona. A modern introduction to online learning. ArXiv preprint , 1912.13213, 2019.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. In Advances in Neural Information Processing Systems 27 (NIPS) , pages 1466-1474, 2014.

Reda Ouhamma, Debabrota Basu, and Odalric Maillard. Bilinear exponential family of mdps: Frequentist regret bound with tractable exploration & planning. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI) , pages 9336-9344, 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 27730-27744, 2022.

Jaehyun Park, Junyeop Kwon, and Dabeen Lee. Infinite-horizon reinforcement learning with multinomial logistic function approximation. ArXiv preprint , 2406.13633, 2024.

Noémie Périvier and Vineet Goyal. Dynamic pricing and assortment under a contextual MNL demand. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 3461-3474, 2022.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems 26 (NIPS) , pages 2256-2264, 2013.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , pages 484-489, 2016.

Quoc Tran-Dinh, Yen-Huan Li, and Volkan Cevher. Composite convex minimization involving selfconcordant-like cost functions. In Proceedings of the 3rd International Conference on Modelling, Computation and Optimization in Information Systems and Management Sciences , pages 155-168, 2015.

Yining Wang, Ruosong Wang, Simon Shaolei Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , 2021.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 4300-4354, 2021.

Lin Yang and Mengdi Wang. Sample-optimal parametric Q-learning using linearly additive features. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 69957004, 2019.

Yu-Jie Zhang and Masashi Sugiyama. Online (multinomial) logistic bandit: Improved regret and constant computation cost. In Advances in Neural Information Processing Systems 36 (NeurIPS) , pages 29741-29782, 2023.

Canzhe Zhao, Ruofeng Yang, Baoxiang Wang, and Shuai Li. Learning adversarial linear mixture Markov decision processes with bandit feedback and unknown transition. In Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023.

Peng Zhao, Long-Fei Li, and Zhi-Hua Zhou. Dynamic regret of online Markov decision processes. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 2686526894, 2022.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and non-stationarity: Problemdependent dynamic regret for online convex optimization. Journal of Machine Learning Research , 25(98):1 - 52, 2024.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvári. Nearly minimax optimal reinforcement learning for linear mixture Markov decision processes. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 4532-4576, 2021.

A Notations

In this section, we collect the notations used in the paper in Table 2.

B Properties of Multinomial Logit Function

This section collects several key properties of the multinomial logit function used in the paper.

Without loss of generality, we assume ∀S h,s,a , ∃ ˙ s h,s,a ∈ S h,s,a such that ϕ ( ˙ s | s, a ) = 0 . Otherwise, we can always define a new feature mapping ϕ ′ ( s ′′ | s, a ) = ϕ ( s ′ | s, a ) -ϕ ( s ′′ | s, a ) for any s ′′ ∈ S h,s,a such that ϕ ′ ( s ′ | s, a ) = 0 and the transition kernel induced by ϕ ′ is the same as that induced by ϕ . Furthermore, We denote the set ˙ S h,s,a = S h,s,a \{ ˙ s h,s,a } .

First, we introduce the definition of self-concordant-like functions and demonstrate that the MNL loss function is self-concordant-like.

Definition 3 (Self-concordant-like function, Tran-Dinh et al. [2015]) . Aconvex function f ∈ C 3 ( R m ) is M -self-concordant-like function with constant M if:

for s ∈ R and M > 0 , where ψ ( s ) := f ( a + s b ) for any a , b ∈ R m .

Proposition 1. The per-episode MNL loss ℓ k,h ( θ ) and the cumulative MNL loss L k,h ( θ ) are both 3 √ 2 -self-concordant-like for all k ∈ [ K ] , h ∈ [ H ] .

Proof. By proposition B.1 in Lee and Oh [2024], the per-episode MNL loss function ℓ k,h ( θ ) is 3 √ 2 -self-concordant-like. Then, the cumulative MNL loss function L k,h ( θ ) is the sum of selfconcordant-like functions and a quadratic function, it is also 3 √ 2 -self-concordant-like. ■

Then, we show the Hessian of the MNL loss function is positive semi-definite.

Lemma 5. The following statements hold for any k ∈ [ K ] , h ∈ [ H ] :

Proof. First, note that

Then, we have

where the last inequality holds by the definition of κ in Assumption 1. This finishes the proof. ■

Next, we show several concentration inequalities commonly used in the analysis.

Lemma 6. Suppose λ k ≥ 1 , for any k ∈ [ K ] , h ∈ [ H ] , for the quantities in Table 2 and define

Then, the following statements hold:

Proof. We prove the five statements individually.

Proof of statement (I). By the definition of H k,h ( θ ) , we have

Thus, we have H i,h ( θ ∗ h ) ⪰ ∑ s ′ ∈S i,h p i,h ( θ ∗ h )( ¯ ϕ s i,h )( ¯ ϕ s i,h ) ⊤ . Then, we get

As a result, we have

Since λ ≥ 1 , we have ∑ s ′ ∈S i,h p i,h ( θ ∗ h ) ∥ ¯ ϕ s ′ i,h ∥ 2 H -1 i,h ( θ ∗ h ) ≤ 1 . Using the fact that z ≤ 2 log(1 + z ) for any z ∈ [0 , 1] , we get

where the last inequality holds by the determinant inequality in Lemma 14.

Proof of statement (II). The proof is same as that of (I), except that we replace θ ∗ h with ˜ θ i +1 ,h .

Proof of statement (III). By Lemma 5, we have H k,h ( θ ) ⪰ ∑ s ′ ∈ ˙ S k,h p s ′ k,h ( θ ) p ˙ s k,h k,h ( θ ) ϕ s ′ k,h ( ϕ s ′ k,h ) ⊤ . The remaining proof is the same as the proof of statement (I).

Proof of statement (IV). By Lemma 5, we have ∀ θ ∈ Θ , it holds that H k +1 ,h ( θ ) ⪰ H k,h ( θ ) + κ ∑ s ′ ∈ ˙ S k,h ϕ s ′ k,h ( ϕ s ′ k,h ) ⊤ . Since λ ≥ 1 , we have κ max s ′ ∈S i,h ∥ ϕ s ′ i,h ∥ H -1 i,h ( θ ) ≤ κ . Using the fact that z ≤ 2 log(1 + z ) for any z ∈ [0 , 1] . By a similar analysis as the statement (I), we have

This finishes the proof of statement (IV).

Proof of statement (V). By (12), we have

Thus, we have

Then, the remaining proof is similar to the proof of statement (III).

■

C Useful Lemmas for MNL Mixture MDPs

In this section, we present some useful lemmas that are commonly used in the analysis.

C.1 Useful Lemmas

Lemma 7. For any θ 1 , θ 2 ∈ R d and positive semi-definite matrix Λ , suppose ∥ θ 1 -θ 2 ∥ Λ ≤ β . Then, for any V : S → [0 , H ] and ( h, s, a ) ∈ [ H ] ×S × A , it holds

where

Lemma 8. Suppose ∀ ( k, h, s, a ) ∈ K × [ H ] ×S ×A and ̂ θ k,h ∈ R d , it holds that θ ∗ h ∈ ̂ C k,h where

Define or,

where ̂ V k,h ( s ) = max a ∈A ̂ Q k,h ( s, a ) . Select the action as a k,h = arg max a ∈A ̂ Q k,h ( s k,h , a ) . Then, for any δ ∈ (0 , 1] , then it holds that

Lemma 9. Suppose Lemma 8 holds. Then, it holds that

C.2 Proof of Lemma 7

Proof. By the second-order Taylor expansion at θ 1 , there exists ¯ θ = νθ 1 + (1 -ν ) θ 2 for some ν ∈ [0 , 1] , such that

The gradient of p s ′ s,a ( θ ) is given by

For the first-order term, we have

where in the first inequality, we denote S + h,s,a as the subset of S h,s,a such that ( ϕ s ′ s,a ) ⊤ ( θ 2 -θ 1 ) -∑ s ′′ ∈S h,s,a p s ′′ s,a ( θ 2 )( ϕ s ′′ s,a ) ⊤ ( θ 2 -θ 1 ) is non-negative, the second inequality holds by the Holder's inequality, and the third inequality is by the condition ∥ θ 1 -θ 2 ∥ Λ ≤ β .

For the second-order term, let u s ′ s,a ( θ ) = ( ϕ s ′ s,a ) ⊤ θ and p s ′ s,a ( u ) = exp( u s ′ s,a ) 1+ ∑ s ′′ exp( u s ′′ s,a ) , further define

Then, we have

where the inequality holds by V ( s ) ∈ [0 , H ] , ∀ s .

According to Lemma 17, we have (omit the subscript S h,s,a for simplicity):

̸

To bound the first term, by applying the AM-GM inequality, we obtain

̸

Plugging (18) into (17), we have

where the last inequality holds by the condition ∥ θ -θ ∥ ≤ β

1 2 Λ Finally, combining (16) and (19) finishes the proof.

C.3 Proof of Lemma 8

Proof. First, we prove the left-hand side of the lemma. We prove this by backward induction on h . For the stage h = H , by definition, we have ̂ Q k,H ( s, a ) = r H ( s, a ) = Q ∗ H ( s, a ) , ̂ V k,H +1 ( s ) = 0 = V ∗ H +1 ( s ) . Suppose the statement holds for h + 1 , we show it holds for h . By definition, if ̂ Q k,h ( s, a ) = H , this holds trivially. Otherwise, we consider two cases:

For ̂ Q k,h ( s, a ) defined in (14), we have

where the first inequality is by the induction hypothesis, and the second inequality is due to θ ∗ ∈ C .

h k,h For ̂ Q k,h ( s, a ) defined in (15), we have

. ■

where the first inequality is by the induction hypothesis, and the second inequality is by (13). Then, we prove the right-hand side of the lemma.

For ̂ Q k,h ( s, a ) defined in (14), we have

where the inequality is by (13).

For ̂ Q k,h ( s, a ) defined in (15), we have

where the inequality is by (13).

C.4 Proof of Lemma 9

Proof. By the definition that Reg( K ) = ∑ K k =1 V ∗ 1 ( s k, 1 ) -∑ K k =1 V π k 1 ( s k, 1 ) , we have for any δ ∈ (0 , 1] , with probability at least 1 -δ , it holds that

where the first inequality is by Lemma 8 and θ ∗ h ∈ ̂ C k,h with probability at least 1 -δ , and the second inequality is by action selection a k,h = arg max a ∈A ̂ Q k,h ( s k,h , a ) .

By the right-hand side of Lemma 8, we have

Define M k,h = P h ( ̂ V k,h +1 -V π k h +1 )( s k,h , a k,h ) -( ̂ V k,h +1 -V π k h +1 )( s k,h +1 ) . Applying this recursively, we have

Summing over k , we have

where the inequality holds by the Azuma-Hoeffding inequality as M k,h is a martingale difference sequence with M k,h ≤ 2 H . This finishes the proof. ■

D Omitted Proofs for Section 3

D.1 Proof of Claim 1

Proof. First, by the definition of MNL mixture MDP, we have ∀ ( s, a ) ∈ S × A and s ′ ∈ S h,s,a , it holds that p s ′ s,a ( θ ) ≥ exp( -B ) / ( U exp( B )) , ∀ θ ∈ R d , thus κ ∗ ≥ κ ≥ 1 / ( U exp(2 B )) 2 . Next, consider the state-action pair ( s, a ) at stage h with the maximum number of reachable states U , it is clear that ∑ s ′ ∈S h,s,a p s ′ s,a ( θ ∗ h ) = 1 . This implies that ∑ s ′ ∈S h,s,a ∑ s ′′ ∈S h,s,a p s ′ s,a ( θ ∗ h ) p s ′′ s,a ( θ ∗ h ) = 1 . Applying the pigeonhole principle, there exists s ′ , s ′′ ∈ S h,s,a such that p s ′ s,a ( θ ∗ h ) p s ′′ s,a ( θ ∗ h ) ≤ 1 /U 2 . Thus, we conclude that κ ≤ κ ∗ ≤ 1 /U 2 . This finishes the proof. ■

E Omitted Proofs for Section 4

E.1 Useful Lemma

Proof. By the multivariate mean value theorem, we have

Hence, we have

where G k,h ( θ 1 , θ 2 ) = ∫ 1 0 ∇ 2 L k,h ( θ 2 + t ( θ 1 -θ 2 ))d t . By self-concordant-like property of L k,h in Proposition 1, we have H k,h ( θ 1 ) ⪯ (1 + 3 √ 2) G k,h ( θ 1 , θ 2 ) . As a result, we have √

This finishes the proof.

E.2 Proof of Lemma 1

Proof. Since ̂ θ k,h minimizes L k,h ( θ ) , we have G k,h ( ̂ θ k,h ) = 0 . Thus, we have

Therefore, since ∥ θ ∗ h ∥ ≤ B and H k,h ( θ ∗ h ) ⪰ λ k I , for any δ ∈ (0 , 1] , with probability at least 1 -δ H ,

where the last inequality holds by the Bernstein-type concentration inequality in Lemma 13. Then, by the determinant inequality in Lemma 14, we have det( H k,h ( θ ∗ h )) ≤ ( λ k + k d ) d . Thus, we obtain

By the configuration that λ k = d log( kH/δ ) and applying the union bound for h ∈ [ H ] , we have with probability at least 1 -δ , for all h ∈ [ H ] simultaneously, it holds that

This finishes the proof.

■

E.3 Proof of Theorem 1

Proof. By Lemma 10, with probability at least 1 -δ , it holds that

where the last inequality holds the confidence set ̂ C k,h in Lemma 1. Then, by Lemma 7, we have

where

Next, we bound

Bounding ϵ 1st k,h . By statement (I) of Lemma 6, we have

Bounding ϵ 2nd k,h . By statement (IV) of Lemma 6, we have

where the first inequality holds by sum k i =1 max s ′ ∈S i,h ∥ ϕ s ′ i,h ∥ 2 H -1 i,h ( θ ) ≤ 2 κ d log ( 1 + k λ k d ) , ∀ θ ∈ Θ in Lemma 6 and the second inequality holds by the configuration of ̂ β k . Combining (20), (21), and (22), we have with probability at least 1 -δ ,

This finishes the proof.

■

F Omitted Proofs for Section 5

F.1 Useful Lemma

Lemma 11. For any k ∈ [ K ] , h ∈ [ H ] , define the second-order approximation of the loss function ℓ k,h ( θ ) at the estimator ˜ θ k,h as ˜ ℓ k,h ( θ ) = ℓ k,h ( θ k,h )+ ⟨∇ ℓ k,h ( ˜ θ k,h ) , θ -˜ θ k,h ⟩ + 1 2 ∥ θ -˜ θ k,h ∥ 2 H k,h ( ˜ θ k,h ) . Then, for the following update rule

it holds that

Proof. Based on the analysis of (implicit) OMD update (see Lemma 16), for any i ∈ [ K ] , we have

〈 ∇ ˜ ℓ i,h ( ˜ θ i +1 ,h ) , ˜ θ i +1 ,h -θ ∗ h 〉 ⩽ 1 2 η ( ∥ ˜ θ i,h -θ ∗ h ∥ 2 H i,h -∥ ˜ θ i +1 ,h -θ ∗ h ∥ 2 H i,h -∥ ˜ θ i +1 ,h -˜ θ i,h ∥ 2 H i,h ) According to Lemma 4, we have

where ζ = log( K +1) + 4 . Then, by combining the above two inequalities, we have

We can further bound the first term of the right-hand side as:

where the second equality holds by the mean value theorem, the first inequality holds by the selfconcordant-like property of ℓ i,h ( · ) in Proposition 1, and the last inequality holds by ˜ θ i +1 ,h and θ ∗ h belong to Θ = { θ ∈ R d , ∥ θ ∥ 2 ≤ B } , and ∇ 2 ℓ i,h ( ξ i +1 ) ⪯ I d .

Then, by taking the summation over i and rearranging the terms, we obtain

where the last inequality holds by ∥ ˜ θ 1 ,h -θ ∗ h ∥ 2 H 1 ,h ≤ λ ∥ ˜ θ 1 ,h -θ ∗ h ∥ 2 2 ≤ 4 λB . Plugging ζ = 2 η finishes the proof. ■

F.2 Proof of Lemma 2

Proof. According to Lemma 11, we have

We bound the right-hand side of the above lemma separately in the following. The most challenging part is to bound the term ∑ k i =1 ℓ i,h ( θ ∗ h ) -∑ k i =1 ℓ i,h ( ˜ θ i +1 ,h ) . At first glance, this term appears straightforward to control, as it can be observed that θ ∗ h = arg min θ ∈ R d ¯ ℓ h ( θ ) ≜ E y i,h [ ℓ i,h ( θ )] , where the instantaneous loss ℓ i,h ( θ ) serves as an empirical observation of ¯ ℓ h ( θ ) . Consequently, the loss gap term seemingly can be bounded using appropriate concentration results. However, a caveat lies in the fact that the update of the estimator ˜ θ i +1 ,h depends on the information ℓ i,h , or more precisely y i,h , making it difficult to directly apply such concentration results.

To address this issue, we decompose the loss gap into two components by introducing an intermediate term. Specifically, we define the softmax function as [ σ k,h ( z )] s = exp([ z ]) s 1+ ∑ s ∈ ˙ S k,h exp([ z ]) s , ∀ s ∈ S k,h .

Using this definition, the loss function can be rewritten as:

Define a pseudo-inverse function of σ k,h ( · ) as [ σ -1 k,h ( p )] s ′ = log ( [ p ] s ′ 1 -∥ p ∥ 1 ) , ∀ p ∈ { p ∈ [0 , 1] S k,h | ∥ p ∥ 1 < 1 } . Then, the loss gap term can be decomposed into two parts as follows.

where z k,h = σ -1 k,h ( E θ ∼ P k,h [ σ k,h (( ϕ s ′ k,h ) ⊤ θ ) s ′ ∈ ˙ S k,h ]) , P k,h ≜ N ( ˜ θ k,h , (1 + c H -1 k,h )) is the Gaussian distribution with mean ˜ θ k,h and covariance (1 + c H -1 k,h ) where c is a constant to be specified later.

The design of the intermediate term was originally proposed by Zhang and Sugiyama [2023] in their study of the multiclass logistic bandit problem and was subsequently applied to the multinomial logit bandits problem by Lee and Oh [2024]. Notably, the intermediate loss is independent of the information contained in y i,h , enabling the application of concentration results. Specifically, based on Lemma F.2 and Lemma F.3 of Lee and Oh [2024], we obtain the following upper bounds for them.

For term (a), let δ ∈ (0 , 1] and λ ≥ max { 2 , 72 cd } , for all k ∈ [ K ] , h ∈ [ H ] , with probability at least 1 -δ , we have term ( a )

For term (b), for all k ∈ [ K ] , h ∈ [ H ] , we have

Combining term (a) and term (b), we have

where the second inequality holds by setting c = 7 η/ 6 and λ ≥ max { 84 √ 2 ηB, 84 dη } , we have

Thus, by setting η = 1 2 log( U +1) + ( B +1) , λ = 84 √ 2 η ( B + d ) , we have

This finishes the proof.

F.3 Proof of Lemma 3

Proof. Lemma 3 follows directly by substituting the confidence set ̂ C k,h defined in Lemma 2, into Lemma 7. This finishes the proof. ■

F.4 Proof of Theorem 2

Proof. Combining Lemma 3 and Lemma 9, we have

where

Next, we bound ϵ fst k,h and ϵ snd k,h respectively.

Bounding ϵ fst k,h . For simplicity, we denote

■

Then, we have

We bound these terms separately in the following.

For the first term ( c ) , we have

where and the fourth inequality is because by Lemma 15 and the fact ˜ H k,h ⪰ H k,h ⪰ λI d , we have

Therefore, we have

where the last inequality holds by ∑ k i =1 max s ′ ∈S i,h ∥ ϕ s ′ i,h ∥ 2 H -1 i,h ≤ 2 κ d log ( 1 + k λ k d ) in Lemma 6. For the term ( d ) , by similar analysis, we have

where the third inequality holds by the AM-GM inequality. Thus, we have

where the last inequality holds by ∑ k i =1 max s ′ ∈S i,h ∥ ˜ ϕ s ′ i,h ∥ 2 H -1 i,h ≤ 2 κ d log ( 1 + k λ k d ) in Lemma 6. Finally, we bound the term ( e ) as follows.

where the first inequality holds by the Cauchy-Schwarz inequality and the last holds by Lemma 6. Thus, combining (23), (24), and (25), we have

For the second-order term, by Lemma 6, we have

where the last inequality holds by ∑ k i =1 max s ′ ∈S i,h ∥ ϕ s ′ i,h ∥ 2 H -1 i,h ≤ 2 κ d log ( 1 + k λ k d ) in Lemma 6. Combining (26) and (27), we have

This finishes the proof.

G Omitted Proofs for Section 6

G.1 Proof of Theorem 3

Proof. Our proof is similar to adversarial linear mixture MDPs with the unknown transition in Zhao et al. [2023]. We prove this lemma by reducing MNL mixture MDPs to a sequence of logistic bandits.

We use each three layers to construct a block. Note that the third layer of block i is also the first layer of block i +1 and hence there are total H/ 2 blocks. In each block, both the first and third layers of this block only have one state, and the second layer has two states. Here we take block i as an example. The first two layers of this block are associated with transition probability P i, 1 and P i, 2 . Denote by s i, 1 the only state in the first layer of this block. In the second layer of the block i , we assume there exist two states s ∗ i, 2 and s i, 2 . Let s i, 3 be the only state in the third layer of this block. Further, for any a ∈ A , let ϕ ( s i, 1 , a, s i, 2 ) = 0 . The transition probability is defined as follows:

For the second layer, it satisfies ∀ s = s ∗ i, 2 , s i, 2 , and a ∈ A , P i, 2 ( s i, 3 | s, a ) = 1 . The reward satisfies r k ( s i, 1 , a ) = 0 for the first layer and r k ( s ∗ i, 2 , a ) = 1 , r k ( s i, 2 , a ) = 0 for the second for all a ∈ A .

Then, consider the logistic bandit problem where a learner selects action x ∈ R d and receives a reward r k sampled from the Bernoulli distribution with mean µ ( x ⊤ θ ∗ ) = (1 + exp( x ⊤ θ ∗ )) -1 . By this configuration, we can see that learning in each block of MDP can be regarded as learning a d -dimensional logistic bandit problem with A arms, where the arm set is ϕ ( s ∗ i, 2 | s i, 1 , a ) and the expected reward of each arm is ρ a . Thus, learning this MNL mixture MDP equals to learning H/ 2 logistic bandit problems. This finishes the proof. ■

■

H Supporting Lemmas

In this section, we provide several supporting lemmas used in the proofs of the main results.

Lemma 12 (Abbasi-Yadkori et al. [2011, Theorem 1]) . Let {F t } ∞ t =0 be a filtration. Let { η t } ∞ t =1 be a real-valued stochastic process such that η t is F t -measurable and η t is conditionally zero-mean R -sub-Gaussian for some R ≥ 0 i.e. ∀ λ ∈ R , E [ e λη t | F t -1 ] ≤ exp ( λ 2 R 2 / 2 ) . Let { X t } ∞ t =1 be an R d -valued stochastic process such that X t is F t -1 -measurable. Assume that V is a d × d positive definite matrix. For any t ≥ 1 , define

Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ , for all t ≥ 1 ,

Lemma 13 (Périvier and Goyal [2022, Theorem 4]) . Let {F t } ∞ t =0 be a filtration. Let { δ t } ∞ t =1 be an R N -valued stochastic process such that δ t is F t -measurable one-hot vector. Furthermore, assume E [ δ t |F t -1 ] = p t and define ε t = p t -δ t . Let { X t } ∞ t =1 be a sequence of R N × d -valued stochastic process such that X t is F t -1 -measurable and ∥ X t,i ∥ 2 ≤ 1 , ∀ i ∈ [ N ] . Let { λ t } ∞ t =1 be a sequence of non-negative scalars. Define

Then, for any ζ ∈ (0 , 1) , with probability at least 1 -ζ , for all t ≥ 1 ,

Lemma 14 (Abbasi-Yadkori et al. [2011, Lemma 10]) . Suppose x 1 , . . . , x t ∈ R d and for any 1 ≤ s ≤ t , ∥ x s ∥ 2 ≤ L . Let V t = λI d + ∑ t s =1 x s x ⊤ s for λ ≥ 0 . Then, we have det( V t ) ≤ ( λ + tL 2 /d ) d . Lemma 15 (Orabona [2019, Lemma 6.9]) . Let Z be a positive define matrix and W be a convex set, define w t +1 as the solution of

Lemma 16 (Campolongo and Orabona [2020, Proposition 4.1]) . Define w t +1 as the solution of

where V ⊆ W ⊆ R d is a non-empty convex set. Further supposing ψ ( w ) is 1 -strongly convex w.r.t. a certain norm ∥ · ∥ in W , then there exists a g ′ t ∈ ∂ℓ t ( w t +1 ) such that

for any u ∈ W .

Lemma 17 (Lee and Oh [2024, Lemma D.3]) . Define Q : R K → R , such that for any u = ( u 1 , . . . , u K ) ∈ R K , Q ( u ) = ∑ K i =1 exp( u i ) v + ∑ K k =1 exp( u k ) . Let p i ( u ) = exp( u i ) v + ∑ K k =1 exp( u k ) . Then, for all i ∈ [ K ] , we have

̸

Then we have

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of this work in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present the assumption in Assumption 1 and provide detailed proofs for all theoretical results in the appendices.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theoretical paper and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theoretical paper and does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theoretical paper and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theoretical paper and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theoretical paper and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper adheres fully to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical paper and there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical paper and does not release any data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a theoretical paper and does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This is a theoretical paper and does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers MDPs employing the MNL function for transition probability, following Hwang and Oh [2023]. The authors suggest efficient algorithms based on online Newton steps, inspired by [Hazan et al., 2014; Zhang et al., 2016; Oh and Iyengar, 2021]. Furthermore, to improve $\\kappa$ dependency, they provide algorithms employing local learning with mirror descent inspired by [Zhang and Sugiyama, 2023; Lee and Oh, 2024]. The algorithms achieve $1/\\sqrt{\\kappa}$ or even detach the dependency of $\\kappa$ from the leading term.\n\nSTRENGTHS:\nThe suggested algorithms are computationally efficient and show improvement in $\\kappa$ compared to the previous work of Hwang and Oh [2023].\n\nWEAKNESSES:\n- Their suggested algorithms do not seem novel because they are based on previously proposed methods for logistic or MNL bandits. Specifically, the online Newton update is widely studied for MNL or logistic bandits [Oh and Iyengar, 2021; Zhang and Sugiyama, 2023].\n\n- Furthermore, the improvement on $\\kappa$ is based on the mirror descent algorithm proposed in [Zhang and Sugiyama, 2023; Lee and Oh, 2024], and the proofs seem to follow the steps in [Zhang and Sugiyama, 2023; Lee and Oh, 2024] in the appendix.\n\n- Lastly, the MNL model for transition probability may have an inherent weakness: the number of achievable states for each (k,n) must be finite, and it is required to know the state space of $S_{k,n}$.\n\n[1] Faury, Louis, et al. \"Jointly efficient and optimal algorithms for logistic bandits.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\nQUESTIONS:\nAre there any non-trivial technical novelties in utilizing the online mirror descent method for MDPs?\n\nLIMITATIONS:\nThe authors discuss some interesting future work regarding regret bound. Additionally, I believe, as mentioned in Weaknesses, the MNL model for transition probability has an inherent weakness: the number of achievable states for each (k,n) must be finite, and it is required to know the state space of $S_{k,n}$.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author analyzes a Markov Decision Process (MDP) model with non-linear function approximation. Specifically, in the finite-time horizon inhomogeneous episodic MDPs setting, the transition dynamics are unknown but the reward function is known. The author proposes using a multinomial logit (MNL) function approximation to estimate transition dynamics, which is superior to the linear function approximation if the model is misspecified in \\cite{hwang2023model}. Additionally, the author proposes *UCRL-MBL-OL*, which adapts the previous work that is model-based and has large computational and storage complexity, to an online style that only consumes constant computation and storage resources. Moreover, the author has proven that the regret bound of *UCRL-MBL-OL* matches the state-of-the-art in Theorem 1. Its regret bound achieves $\\tilde{O}(\\kappa^{-1} dH^2\\sqrt{K})$, where $H$ is the time horizon length, $K$ is the number of total episodes and $\\kappa$ is considered as a parameter to control the sparsity of the transition dynamics and $d$ is the hidden dimensionality. Ignoring the logarithmic factor and $\\kappa$, such a regret bound has only a $\\sqrt{H}$ gap compared to the lower bound. After that, with additional assumption, the author utilizes the local information to propose another two algorithms, *UCRL-MNL-LL* and *UCRL-MNL-LL+* to remove the dependence on $\\kappa$ and get a tighter regret bound as well as maintain good properties of *UCRL-MNL-OL*.\n\nSTRENGTHS:\n1. This paper is well-written. The author makes a clear improvement point compared to the literature.\n\n2. The algorithm proposed by the author enjoys an online learning style that does not need to maintain a large historical set.\n\nWEAKNESSES:\n1. Although this paper focuses on reducing the computation complexity, I am curious about the sample complexity of *UCRL-MNL-OL*.\n    \n2. Since the algorithm builds up the estimation of the transition dynamics by using MNL function approximation, is it considered a model-based algorithm? More specifically, does it require storing the transition dynamics for each state-action pair in every step?\n\nQUESTIONS:\nPlease see the above \"Weaknesses\"\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
This work studies the MNL function approximation inhomogeneous RL, achieves the $O(1)$ computation cost, and improves the regret guarantee with regard to $\kappa$. To improve the computation cost, this work employs the online Newton step instead of MLE estimation to estimate $\theta$. Then, they design a novel confidence set by making full use of local information to improve the dependence of $\kappa$.

STRENGTHS:
1.	The use of local information instead of a uniform $\kappa$ is novel and useful to improve the dependence of $\kappa$.
2.	The UCRL-MNL-LL+ removes the $\kappa$ dependence on the lower-order term and almost matches the optimal regret results by using high-order Taylor expansion.

WEAKNESSES:
1.	[1] also use the online Newton step to improve the computation cost in the logit contextual bandits setting. It would be better to discuss the novelty of UCRL-MNL-OL.

[1] Oh, M. H., & Iyengar, G. (2021, May). Multinomial logit contextual bandits: Provable optimality and practicality. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 10, pp. 9205-9213).

QUESTIONS:
Question 1: This work achieves great results in the stochastic reward setting. Can you discuss the challenge when extending to the adversarial reward setting?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe problem considered in this paper is online learning in MDPs where transition probabilities are modelled with a log-linear model (with \"multinomial logit function approximation\"). The finite horizon, time-inhomogenous setting is considered. The problem is motivated by allowing a nonlinear transformation in modeling the MDP and yet maintaining both computational and information theoretic tractability. Inspired by results in the analogous bandit problems and algorithms developed for them, a number of gradually more complex, but (statistically) better performing algorithms are considered. In particular, while naive approaches give a poor dependence on a problem parameter $\\kappa$ that characterizes the \"strength\" of nonlinearity, by adopting previous ideas to the MDP setting, new algorithms are designed that eliminate this poor dependence. A lower bound is also established, which nearly matches the upper bound (but considers infinite action spaces, while the main paper considers finite action spaces).\n\nSTRENGTHS:\nThis is a reasonable problem setting; and the approach is also reasonable. It is nice to have a lower bound, even if there is a mismatch between the settings. It is nice to see that ideas that were developed for the bandit setting generalize to the MDP setting.\n\nWEAKNESSES:\n1. The novelty is limited by that we have seen the same story, same ideas playing out nicely in the closely related bandit setting. \n2. A new parameter, U, the number of next states that are reachable with positive probability in the worst case, appears in the analysis and will appear in the bounds.\n3. It is an unpleasant surprise for the reader to discover this dependence only through carefully reading the paper, rather than being told upfront. It is not good that the opportunity to discuss whether this quantity needs to enter the regret bound, and that this quantity needs to be small for the algorithm to be tractable, is missed.\n4. Line 83 and onward: The work of Uohamma is discussed but is mischaracterized. My reading of this work is that they do establish that their algorithm runs in polynomial time. It remains unclear why the exponential family model is incomparable with the one considered here; an explanation (with examples) is missing.\n5. The paper could use some extra proofreading (e.g., the upper indices in the bottom of page 5, in the displayed equation are not correct); in line 149, in the definition of $U$, $|\\cdot|$ is missing.\n\nQUESTIONS:\n1. Can you confirm that the regret and compute cost depend on U, the worst-case number of next states that one can transition to with positive probability? Do you think such dependencies are necessary? Are there any interesting examples where it is reasonable to expect that U is small, independently of the size of the state space?\n2. What was the most challenging aspect of extending the bandit ideas to the MDP framework?\n\nLIMITATIONS:\nn.a.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the recently proposed MDPs that use multinomial logit function approximation for state distribution validness. The results and algorithms improve the prior work of Hwang and Oh [2023] in multiple aspects, including computation efficiency, storage, and statistical dependence on the problem-dependent quantity $\\kappa$ that can be exponentially small. In addition, the authors establish a matching lower bound on $d$, the feature space dimension, and $K$, the number of episodes.\n\nSTRENGTHS:\n- The paper is well-written and has clear logic flows. Readers can see how the authors approach the MDP problem and tackle the challenges. In particular, Table 1 is quite useful for demonstrating the advancements in the work.\n- The improvements in both computation and storage efficiencies are essential for practical applications. In Theorem 2, the authors also improve the dependence on $kappa$ to $\\sqrt{\\kappa}$ without affecting efficiency. The enhancement seems significant, especially since the parameter can be exponentially small. \n- The lower bound established in the paper is the first to demonstrate the optimality of the authors' algorithms in the $d$-$K$ dependence. Per my understanding, it also confirms the results' optimality of Hwang and Oh [2023].\n\nWEAKNESSES:\n- The primary high-level techniques and tools (seem to) come from existing works and relevant fields, such as MNL contextual bandits. The authors should put more effort into highlighting the technical challenges and novelties besides the previous comparisons. \n- It would be beneficial to include experiments on synthetic and real-world datasets and compare the results to existing baselines and relevant works. In particular, the new algorithms seem more involved than prior ones, which may affect their stability and adaptiveness.\n- There is still a significant gap between the lower and upper bounds. Besides, I wonder how often $\\kappa$ could be exponentially small in practical settings, though it's definitely of theoretical interest to approach the lower limits on parameter dependency.\n\nQUESTIONS:\nOverall, I think the paper makes reasonable contributions to the problem, and I have no additional questions/comments besides the above.\n\nLIMITATIONS:\nThe authors have made various comparisons and discussed the limitations of the results, which I'm satisfied with. I do not see any potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-08-05"}