Adaptive HL-Gaussian: A Value Function Learning Method with Dynamic Support Adjustment

Authors: Chen Chen, Daming Shi, Yixiu Mao, Xiangyang Ji
Primary area: reinforcement learning
Keywords: value function learning, HL-Gaussian

Abstract:
Recent research indicates that using cross-entropy (CE) loss for value function learning surpasses traditional mean squared error (MSE) loss in performance and scalability, with the HL-Gaussian method showing notably strong results. However, this method requires a pre-specified support  for representing the categorical distribution of the value function, and an inappropriately chosen  interval for the support may not match the time-varying value function, potentially impeding the learning process. To address this issue, we theoretically establish that  HL-Gaussian  inherently introduces a projection error during the learning of the value function, which is dependent on the support interval. We further prove that an ideal interval should be sufficiently broad to reduce truncation-induced projection errors, yet not so  excessive as to counterproductively amplify them. Guided by these findings, we introduce the Adaptive HL-Gaussian (AHL-Gaussian) approach. This approach starts with a confined support interval and dynamically adjusts its range by minimizing the projection error. This ensures that the interval's size stabilizes to adapt to the learning value functions without  further expansion. We integrate AHL-Gaussian into several classic value-based algorithms and evaluate it on Atari 2600 games and Gym Mujoco. The results show that AHL-Gaussian significantly outperforms the vanilla baselines and standard HL-Gaussian with a static interval across the majority of tasks.

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729150064739

thread_type:
official_review

<thread_object>
creator: reviewer_gzrk

<content>
summary:
This paper proposed to strengthen the HL-Gaussian method by having dynamic interval on which the value function is projected into a categorical distribution. The existing works use a pre-specified fixed-length interval $[v_\text{min}, v_\text{max}]$ which can be hard to estimate. The authors conducted theoretical analysis and revealed that this projection incurs two losses:  projection loss and truncation loss. Based on the analysis, the authors proposed a learnable dynamic interval scheme in an attempt to minimize truncation error and projection error. The method was plugged into DQN and SAC and tested on standard benchmark environments.

strengths:
The paper is overall well-written and sufficiently motivated. The preliminary section introduced the background and related work in an accessible way, paving the way for later analysis and proposal. Theoretical analyses are novel and intuitive, indicating an adaptive interval could provide an edge in minimizing the truncation and projection error. The proposed learnable scheme was combined with popular base algorithms to show quantitative improvements.

weaknesses:
My concern for the paper is that its scope is limited and might not be interesting to the community, considering that the adaptive HL-Gaussian is an incremental improvement on the HL-Gaussian, and no significant higher scores can be seen from Figure 4-6. Maybe because the learning horizon was not suffiicently long, considering the original C51 algorithm ran 200m frames.

Another concern is the results seem to be limited as well. Central to the proposal, it would be more convincing to show how the dynamic interval $\zeta$ would change as learning proceeds, e.g. $\zeta$ can shrink drastically in some cases but being wide in the other. Consider those environments in figure 5 and 6 where no substantial difference was observed, was it because of $\zeta$ converged/flucturating around a number? And how was the loss of learning $\zeta$?  A question related to this is: if $\zeta$ did not change drastically but rather stayed around a fixed value, could we simply normalize the reward into $[0, 1]$, and replace the interval $[v_\text{min}, v_\text{max}]$ with $[0, \frac{1}{1-\gamma}]$? This reward normalization trick is frequently used and we do not need a separate learning scheme then.

questions:
Please refer to the above for my questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730558658309

thread_type:
official_review

<thread_object>
creator: reviewer_uvay

<content>
summary:
The paper introduces a task-agnostic, value-function-aware mechanism for adaptively adjusting the intervals of HL-Gaussian, an existing approach that replaces the mean squared error loss function with cross-entropy loss in reinforcement learning. To achieve this, the paper provides a theoretical analysis of the errors arising from discretization and truncation in HL-Gaussian, guiding the development of a solution to minimize these errors. Experiments on Atari 2600 games and Gym Mujoco demonstrate the advantages of the proposed approach.

strengths:
The paper addresses a novel problem: adaptively setting intervals for HL-Gaussian, distinguishing it from existing approaches that rely on fixed, predetermined intervals. This adaptive approach is motivated by a theoretical analysis of the discretization and truncation errors in HL-Gaussian.

weaknesses:
The paper's major weaknesses include unclear notation and an insufficient connection between the theoretical analysis and the proposed solution.

Regarding notation issues, it may be partly due to my background, but the paper's notations are quite confusing, making it challenging to verify the accuracy of the theoretical analysis. Some specific examples include:

1. Equation (1) says $Q\^\*(s\_t,a\_t) = (T\^\* Q\^\*)(s\_t,a\_t)$. It seems to be incorrect as it means $T\^*$ is an identity function. The same issue is in Equation (2).

2. In lines 129–130, it appears that $z$ is treated as a random variable, yet the paper describes it as a distribution. Additionally, this distribution is said to be taken from a set of discrete locations, which seems unusual.

3. In Equation (3), $\delta_{z_i}$ is undefined. If it is the Dirac delta function, then is $Z(s_t,a_t)$ a probability? But if $Z(s_t,a_t)$ a probability, then then $Q(s_t,a_t)$ in Equation (3) is clearly incorrect.

4. In lines 138-139, why does $p_i(s_t,a_t)$ exist if the true Q value function does not belong to the interval $[v_{min}, v_{\max}]$?

5. Why does $p_i(s_t,a_t)$ in Equation (6) satisfy the condition in line 139?

6. In the proof of Proposition 3.1: What is "the distributional value function Q"? Why does $h(x)$ is supported in the range $[v_{min}, v_{max}]$? Why is $p_\mu$ independent of the learned variable $x:=(s,a)$ even though $p_{\mu} = [p_1(s,a),\dots,p_m(s,a)]$? There are also errors in lines 704, 711.

7. In line 198, $\delta$ is defined when there is a bin containing $\mu$. However, in line 212, it is used even for the case when $\mu$ is not in the interval (no bin containing $\mu$).
   
8. In line 212, $\mathcal{E}_{discretization} \le 0$, what does this mean?

9. In the theoretical analysis, the paper uses the same notation $Q$ and $\hat{T}$ for both the action-value functions computed using a continuous range and those computed using a truncated discrete range. However, these two procedures are different, and the error between them accumulates over the Markov chain in the Bellman equation. It appears that the paper does not address this error accumulation. 

10. Line 777 is not acceptable in stating that a certain term is constant and that its value is $o(1)$ by "empirically verified". Additionally, the meaning of $o(1)$ in this context is unclear.

Regarding the solution, it appears to be largely disconnected from the theoretical analysis. Specifically, there is no assessment of the algorithm’s projection error. Furthermore, while this error depends on both the number of bins and the interval size, the proposed solution optimizes only the interval size while keeping the number of bins fixed. More importantly, the solution does not incorporate any findings from the theoretical analysis.

questions:
Please see the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667894922

thread_type:
official_review

<thread_object>
creator: reviewer_kkwb

<content>
summary:
The paper introduces Adaptive HL-Gaussian (AHL-Gaussian), an approach to learn categorical distributions for value functions in reinforcement learning. AHL-Gaussian improves upon the previous HL-Gaussian approach, which uses a fixed interval for the categorical distribution's support. The primary limitation of HL-Gaussian is that a poorly chosen interval can restrict the value function to an inaccurate range, leading to significant truncation and discretization errors.

To address this, the authors conduct a theoretical analysis of how interval selection affects error rates, clarifying the failure cases of the standard HL-Gaussian. Based on these insights, they propose AHL-Gaussian, which dynamically adjusts the support interval to align with the evolving value function. Experimental results demonstrate that AHL-Gaussian outperforms the static approach, advancing state-of-the-art performance.

strengths:
The main strength of the paper is the theoretical analysis of the dependency between projection error and the support interval. The derivations are easy to follow and are supported by explanatory figures. The section concludes with a very clear and concise summary of the main findings.

The experimental section is also well-structured and the experiments' goals are clearly explained. The authors use Atari and Mujoco and apply AHL-Gaussian to DQN, TD3, and SAC.

weaknesses:
In my eyes, the experiments are not fully convincing. I.e., the improvement over other methods seems to be environment dependent and pretty minor overall. Besides that, I doubt that the comparison is fair, as all baselines use fixed interval [-10, 10] for all environments.

- It is unclear why the interval of [-10, 10] is used for all non-adaptive methods for all environments. It looks like an unfair comparison to me. I would want to see how AHL-Gaussian compares to baselines with more adequate intervals.

- In Section 4.3, auhtors compare their method against non-learning approaches for updating the interval. As $1.1 * \text{Max Target}$ performs better than $1 * \text{Max Target}$, I would expect to see experiments with larger values. Besides, it seems natural to test other non-learning approaches. e.g., setting interval as $[\frac{r_{min}}{1-\gamma},\frac{r_{max}}{1-\gamma}]$, where $r_{min}, r_{max}$ are min/max rewards observed during training.

questions:
My main question, as explained, is related to practical usability of the method. For which problems do the authors recommend using AHL-Gaussian as opposed to treating the support interval as a tunable hyperparameter?

Typos:
Line 486: Typo ``TTo’’.

Line 711: Missing $\mathbb{E}$
</content>
</thread_object>

</thread>
<thread>
creation date:
1731124685208

thread_type:
official_review

<thread_object>
creator: reviewer_eger

<content>
summary:
This paper introduces a novel adaptive method for value based RL using cross entropy loss with discretized value space. Prior works have shown that using cross entropy loss instead of mean squared error improves the performance of RL algorithms but it also leads to suboptimal performance if value space is poorly discretized. The authors show that static discretization in HL-Gaussian leads to a projection error which increases with increase in interval size. The then propose an adaptive method that dynamically adjusts the support interval of value function during training by minimizing the projection error. They conduct extensive experiments to demonstrate the efficacy of their method over other baselines on various benchmark environments.

strengths:
1. The paper theoretically characterizes the deficiencies of the standard HL Gaussian method based on projection error.
    
2. It proposes a novel adaptive algorithm that dynamically adapts the value range and corresponding discretization by minimizing the projection error.
    
3. Their method can be integrated with various RL baselines like DQN, SAC, TD3 and it shows nearly consistent improvement in performance when compared to standard RL training.

weaknesses:
1. The proposed algorithm lacks theoretical insights on why quantization should help in training. There are no theoretical guarantee on performance of the algorithm.
2. The experiments are mostly conducted on simple RL environments and it is not yet clear if the method would work well on more complex real world environments like Dota, Minecraft etc.

questions:
1. How does the choice of bin quantization(number and varying size) affect the performance of the algorithm? Why not adaptively optimize them as well?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731725345115

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Brief Introduction to the Revisison

comment:
First, we would like to thank the reviewers for taking the time to review our paper and provide valuable feedback. These suggestions have greatly contributed to improving the quality of our work. Below, we briefly summarize the changes made in the revision, followed by individual responses to each reviewer’s comments:

1. We have added Section B.3 in the appendix to discuss the cumulative effect of the projection error generated by the HL-Gaussian algorithm on the final performance. This section emphasizes the importance of reducing the projection error during each policy evaluation step.

2. We have introduced three new non-learning-based methods as baselines in Section F of the appendix to further compare with AHL-Gaussian. This comparison highlights the versatility and consistency of AHL-Gaussian. Additionally, in Section G, we have added experiments on two more complex control tasks to showcase the advantages of AHL-Gaussian.

3. We have included training curves for $\xi$ and projection error in Section E of the appendix, which correspond to the main experimental results. These curves help visualize the dynamic process of these variables.

4. We have corrected typos and clarified points where the reviewers had misunderstandings, both in the main text and the appendix.

All modifications and new additions have been highlighted in blue.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732446556096

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to Continuing the Discussion

comment:
Dear Reviewers,  

We hope this message finds you well. Thank you once again for your time and thoughtful feedback on our paper. We deeply value the opportunity to engage with you during the rebuttal phase, as your insights are instrumental in improving the quality of our work.  

As the discussion phase approaches its deadline, we wanted to kindly check if there are any additional comments or questions you might have. We would greatly appreciate the chance to address any remaining concerns or explore further clarifications.  

Please don’t hesitate to share your thoughts—we remain eager to collaborate in refining the paper.  

Thank you for your support, and we look forward to hearing from you.  

Best regards,  
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
Thank you for your submission to ICLR. This paper presents an adaptive method for value function learning, called AHL-Gaussian, which extends the HL-Gaussian method by introducing a dynamic support interval for the categorical distribution of the value function.

This is a borderline submission. Reviewers agree that this paper provides a good characterization of deficiencies in the typical HL-Gaussian method, proposes a novel algorithm, and shows a thorough set of empirical results. However, the reviewers also had a number of concerns. In particular, some of the reviewers were not convinced of the scope and significance of the work, in comparison with simpler, non-adaptive baselines (especially when factoring in the differences in empirical performance between these methods, in certain cases). In addition, there were also concerns about the clarity of presentation in the connections between the theoretical results and the presented algorithm, which led to disagreement about the soundness of the theory. By the end of the rebuttal, a majority of the authors were unconvinced and had remaining concerns. Therefore, I recommend this paper for rejection.