$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization

## Abstract

Abstract The abstract paragraph should be indented 1/2 inch (3 picas) on both left and
right-hand margins. Use 10 point type, with a vertical spacing of 11 points.
The word Abstract must be centered, in small caps, and in point size 12. Two
line spaces precede the abstract. The abstract must be limited to one
paragraph.

## 1 Submission of conference papers to ICLR 2025

ICLR requires electronic submissions, processed by
[https://openreview.net/](https://openreview.net/). See ICLR’s website for more instructions.

If your paper is ultimately accepted, the statement \iclrfinalcopy should be inserted to adjust the
format to the camera ready requirements.

The format for the submissions is a variant of the NeurIPS format.
Please read carefully the instructions below, and follow them
faithfully.

### 1.1 Style

Papers to be submitted to ICLR 2025 must be prepared according to the
instructions presented here.

Authors are required to use the ICLR LaTeX style files obtainable at the
ICLR website. Please make sure you use the current files and
not previous versions. Tweaking the style files may be grounds for rejection.

### 1.2 Retrieval of style files

The style files for ICLR and other conference information are available online at:

[http://www.iclr.cc/](http://www.iclr.cc/)

The file iclr2025_conference.pdf contains these
instructions and illustrates the
various formatting requirements your ICLR paper must satisfy.
Submissions must be made using LaTeX and the style files
iclr2025_conference.sty and iclr2025_conference.bst (to be used with LaTeX2e). The file
iclr2025_conference.tex may be used as a “shell” for writing your paper. All you
have to do is replace the author, title, abstract, and text of the paper with
your own.

The formatting instructions contained in these style files are summarized in
sections 2, 3, and 4 below.

## 2 General formatting instructions

The text must be confined within a rectangle 5.5 inches (33 picas) wide and
9 inches (54 picas) long. The left margin is 1.5 inch (9 picas).
Use 10 point type with a vertical spacing of 11 points. Times New Roman is the
preferred typeface throughout. Paragraphs are separated by 1/2 line space,
with no indentation.

Paper title is 17 point, in small caps and left-aligned.
All pages should start at 1 inch (6 picas) from the top of the page.

Authors’ names are
set in boldface, and each name is placed above its corresponding
address. The lead author’s name is to be listed first, and
the co-authors’ names are set to follow. Authors sharing the
same address can be on the same line.

Please pay special attention to the instructions in section 4
regarding figures, tables, acknowledgments, and references.

There will be a strict upper limit of 10 pages for the main text of the initial submission, with unlimited additional pages for citations.

## 3 Headings: first level

First level headings are in small caps,
flush left and in point size 12. One line space before the first level
heading and 1/2 line space after the first level heading.

### 3.1 Headings: second level

Second level headings are in small caps,
flush left and in point size 10. One line space before the second level
heading and 1/2 line space after the second level heading.

#### 3.1.1 Headings: third level

Third level headings are in small caps,
flush left and in point size 10. One line space before the third level
heading and 1/2 line space after the third level heading.

## 4 Citations, figures, tables, references

These instructions apply to everyone, regardless of the formatter being used.

### 4.1 Citations within the text

Citations within the text should be based on the natbib package
and include the authors’ last names and year (with the “et al.” construct
for more than two authors). When the authors or the publication are
included in the sentence, the citation should not be in parenthesis using \citet{} (as
in “See Hinton06 for more information.”). Otherwise, the citation
should be in parenthesis using \citep{} (as in “Deep learning shows promise to make progress
towards AI (Bengio+chapter2007).”).

The corresponding references are to be listed in alphabetical order of
authors, in the References section. As to the format of the
references themselves, any style is acceptable as long as it is used
consistently.

### 4.2 Footnotes

Indicate footnotes with a number(^1^11Sample of the first footnote) in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2 inches
(12 picas).(^2^22Sample of the second footnote)

### 4.3 Figures

All artwork must be neat, clean, and legible. Lines should be dark
enough for purposes of reproduction; art work should not be
hand-drawn. The figure number and caption always appear after the
figure. Place one line space before the figure caption, and one line
space after the figure. The figure caption is lower case (except for
first word and proper nouns); figures are numbered consecutively.

Make sure the figure caption does not get separated from the figure.
Leave sufficient space to avoid splitting the figure and figure caption.

You may use color figures.
However, it is best for the
figure captions and the paper body to make sense if the paper is printed
either in black/white or in color.

Figure: Figure 1: Sample figure caption.

### 4.4 Tables

All tables must be centered, neat, clean and legible. Do not use hand-drawn
tables. The table number and title always appear before the table. See
Table 1.

Place one line space before the table title, one line space after the table
title, and one line space after the table. The table title must be lower case
(except for first word and proper nouns); tables are numbered consecutively.

**Table 1: Sample table title**
| PART | DESCRIPTION |
| --- | --- |
| Dendrite | Input terminal |
| Axon | Output terminal |
| Soma | Cell body (contains cell nucleus) |

## 5 Default Notation

In an attempt to encourage standardized notation, we have included the
notation file from the textbook, Deep Learning
goodfellow2016deep available at
[https://github.com/goodfeli/dlbook_notation/](https://github.com/goodfeli/dlbook_notation/). Use of this style
is not required and can be disabled by commenting out
math_commands.tex.

Numbers and Arrays

| $\displaystyle a$ | A scalar (integer or real) |
| --- | --- |
| $\displaystyle{\bm{a}}$ | A vector |
| $\displaystyle{\bm{A}}$ | A matrix |
| $\displaystyle\tA$ | A tensor |
| $\displaystyle{\bm{I}}_{n}$ | Identity matrix with $n$ rows and $n$ columns |
| $\displaystyle{\bm{I}}$ | Identity matrix with dimensionality implied by context |
| $\displaystyle{\bm{e}}^{(i)}$ | Standard basis vector $[0,\dots,0,1,0,\dots,0]$ with a 1 at position $i$ |
| $\displaystyle\text{diag}({\bm{a}})$ | A square, diagonal matrix with diagonal entries given by ${\bm{a}}$ |
| a | A scalar random variable |
| $\displaystyle{\mathbf{a}}$ | A vector-valued random variable |
| $\displaystyle{\mathbf{A}}$ | A matrix-valued random variable |

Sets and Graphs

| $\displaystyle{\mathbb{A}}$ | A set |
| --- | --- |
| $\displaystyle\R$ | The set of real numbers |
| $\displaystyle\{0,1\}$ | The set containing 0 and 1 |
| $\displaystyle\{0,1,\dots,n\}$ | The set of all integers between 0 0 and $n$ |
| $\displaystyle[a,b]$ | The real interval including $a$ and $b$ |
| $\displaystyle(a,b]$ | The real interval excluding $a$ but including $b$ |
| $\displaystyle{\mathbb{A}}\backslash{\mathbb{B}}$ | Set subtraction, i.e., the set containing the elements of ${\mathbb{A}}$ that are not in ${\mathbb{B}}$ |
| $\displaystyle{\mathcal{G}}$ | A graph |
| $\displaystyle\parents_{\mathcal{G}}({\textnormal{x}}_{i})$ | The parents of ${\textnormal{x}}_{i}$ in ${\mathcal{G}}$ |

Indexing

| $\displaystyle{a}_{i}$ | Element $i$ of vector ${\bm{a}}$, with indexing starting at 1 |
| --- | --- |
| $\displaystyle{a}_{-i}$ | All elements of vector ${\bm{a}}$ except for element $i$ |
| $\displaystyle{A}_{i,j}$ | Element $i,j$ of matrix ${\bm{A}}$ |
| $\displaystyle{\bm{A}}_{i,:}$ | Row $i$ of matrix ${\bm{A}}$ |
| $\displaystyle{\bm{A}}_{:,i}$ | Column $i$ of matrix ${\bm{A}}$ |
| $\displaystyle{\etens{A}}_{i,j,k}$ | Element $(i,j,k)$ of a 3-D tensor $\tA$ |
| $\displaystyle\tA_{:,:,i}$ | 2-D slice of a 3-D tensor |
| $\displaystyle{\textnormal{a}}_{i}$ | Element $i$ of the random vector ${\mathbf{a}}$ |

Calculus

| $\displaystyle\frac{dy}{dx}$ | Derivative of $y$ with respect to $x$ |
| --- | --- |
| $\displaystyle\frac{\partial y}{\partial x}$ | Partial derivative of $y$ with respect to $x$ |
| $\displaystyle\nabla_{\bm{x}}y$ | Gradient of $y$ with respect to ${\bm{x}}$ |
| $\displaystyle\nabla_{\bm{X}}y$ | Matrix derivatives of $y$ with respect to ${\bm{X}}$ |
| $\displaystyle\nabla_{\tX}y$ | Tensor containing derivatives of $y$ with respect to $\tX$ |
| $\displaystyle\frac{\partial f}{\partial{\bm{x}}}$ | Jacobian matrix ${\bm{J}}\in\R^{m\times n}$ of $f:\R^{n}\rightarrow\R^{m}$ |
| $\displaystyle\nabla_{\bm{x}}^{2}f({\bm{x}})\text{ or }{\bm{H}}(f)({\bm{x}})$ | The Hessian matrix of $f$ at input point ${\bm{x}}$ |
| $\displaystyle\int f({\bm{x}})d{\bm{x}}$ | Definite integral over the entire domain of ${\bm{x}}$ |
| $\displaystyle\int_{\mathbb{S}}f({\bm{x}})d{\bm{x}}$ | Definite integral with respect to ${\bm{x}}$ over the set ${\mathbb{S}}$ |

Probability and Information Theory

| $\displaystyle P({\textnormal{a}})$ | A probability distribution over a discrete variable |
| --- | --- |
| $\displaystyle p({\textnormal{a}})$ | A probability distribution over a continuous variable, or over<br>a variable whose type has not been specified |
| $\displaystyle{\textnormal{a}}\sim P$ | Random variable a has distribution $P$ |
| $\displaystyle\mathbb{E}_{{\textnormal{x}}\sim P}[f(x)]\text{ or }\mathbb{E}f(x)$ | Expectation of $f(x)$ with respect to $P({\textnormal{x}})$ |
| $\displaystyle\Var(f(x))$ | Variance of $f(x)$ under $P({\textnormal{x}})$ |
| $\displaystyle\Cov(f(x),g(x))$ | Covariance of $f(x)$ and $g(x)$ under $P({\textnormal{x}})$ |
| $\displaystyle H({\textnormal{x}})$ | Shannon entropy of the random variable x |
| $\displaystyle\KL(P\|Q)$ | Kullback-Leibler divergence of P and Q |
| $\displaystyle\mathcal{N}({\bm{x}};{\bm{\mu}},{\bm{\Sigma}})$ | Gaussian distribution over ${\bm{x}}$ with mean ${\bm{\mu}}$ and covariance ${\bm{\Sigma}}$ |

Functions

| $\displaystyle f:{\mathbb{A}}\rightarrow{\mathbb{B}}$ | The function $f$ with domain ${\mathbb{A}}$ and range ${\mathbb{B}}$ |
| --- | --- |
| $\displaystyle f\circ g$ | Composition of the functions $f$ and $g$ |
| $\displaystyle f({\bm{x}};{\bm{\theta}})$ | A function of ${\bm{x}}$ parametrized by ${\bm{\theta}}$.<br>(Sometimes we write $f({\bm{x}})$ and omit the argument ${\bm{\theta}}$ to lighten notation) |
| $\displaystyle\log x$ | Natural logarithm of $x$ |
| $\displaystyle\sigma(x)$ | Logistic sigmoid, $\displaystyle\frac{1}{1+\exp(-x)}$ |
| $\displaystyle\zeta(x)$ | Softplus, $\log(1+\exp(x))$ |
| $\displaystyle||{\bm{x}}||_{p}$ | $\normlp$ norm of ${\bm{x}}$ |
| $\displaystyle||{\bm{x}}||$ | $\normltwo$ norm of ${\bm{x}}$ |
| $\displaystyle x^{+}$ | Positive part of $x$, i.e., $\max(0,x)$ |
| $\displaystyle\bm{1}_{\mathrm{condition}}$ | is 1 if the condition is true, 0 otherwise |

## 6 Final instructions

Do not change any aspects of the formatting parameters in the style files.
In particular, do not modify the width or length of the rectangle the text
should fit into, and do not change font sizes (except perhaps in the
References section; see below). Please note that pages should be
numbered.

## 7 Preparing PostScript or PDF files

Please prepare PostScript or PDF files with paper size “US Letter”, and
not, for example, “A4”. The -t
letter option on dvips will produce US Letter files.

Consider directly generating PDF files using pdflatex
(especially if you are a MiKTeX user).
PDF figures must be substituted for EPS figures, however.

Otherwise, please generate your PostScript and PDF files with the following commands:

### 7.1 Margins in LaTeX

Most of the margin problems come from figures positioned by hand using
\special or other commands. We suggest using the command
\includegraphics
from the graphicx package. Always specify the figure width as a multiple of
the line width as in the example below using .eps graphics

or

for .pdf graphics.
See section 4.4 in the graphics bundle documentation ([http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps](http://www.ctan.org/tex-archive/macros/latex/required/graphics/grfguide.ps))

A number of width problems arise when LaTeX cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the \- command.

#### Author Contributions

If you’d like to, you may include a section for author contributions as is done
in many journals. This is optional and at the discretion of the authors.

#### Acknowledgments

Use unnumbered third level headings for the acknowledgments. All
acknowledgments, including those to funding agencies, go at the end of the paper.

## Appendix A Appendix

You may include other additional sections here.

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. How would iREPO perform on state-of-the-art models? Since only outdated models were used, testing iREPO on LLaMA 3 or similar state-of-the-art models would be crucial for determining its broader impact.
2. Why were offline methods chosen as primary baselines? A more robust comparison with other online methods would provide a fairer evaluation of iREPO’s capabilities. The authors should address the rationale for focusing on offline baselines.
3. Does iREPO maintain effectiveness under distribution shifts? Given that distributional shifts are common in real-world scenarios, an exploration of iREPO’s resilience to these changes would enhance the study's practical relevance.
soundness: 2
strengths: 1. Theoretical Exploration: The theoretical exploration around implicit reward pairwise differences is well-motivated, and the authors provide proofs and assumptions to support their claims.
2. Clarity in Model Steps: The procedural flow in the algorithm (iREPO) is clearly outlined, making the technical contributions easy to follow for researchers familiar with preference optimization.
summary: The paper presents iREPO, an empirical preference optimization framework for language model alignment, grounded in theoretical exploration with an emphasis on iterative feedback mechanisms. While the paper is structured well and includes clear algorithmic steps, the experimental design is weak, as iREPO, an online method, is primarily compared to offline baselines. This choice undermines the validity of the results, as offline methods generally face limitations in out-of-distribution scenarios, which are not an issue for online methods. Furthermore, the use of non-state-of-the-art models like Phi-2 and Mistral-7B weakens the contribution, making it difficult to assess iREPO’s potential on current high-performing models such as LLaMA 3. The approach lacks novelty, offering incremental rather than transformative improvements to preference optimization. Key questions remain, particularly regarding the rationale for comparing iREPO with offline methods, how it would perform on state-of-the-art models, and its robustness in real-world settings with distributional shifts. Overall, the paper falls short in demonstrating strong empirical evidence and addressing practical concerns for broader applicability.
weaknesses: 1. Comparative Analysis: The choice to compare an online method to several offline methods (e.g., DPO, IPO) is questionable and potentially misleading, as offline methods generally face limitations in out-of-distribution scenarios that online approaches can better handle. This mismatch in methodology reduces the validity of the experimental claims.
2. Model Selection: The experiments are conducted only on Phi-2 and Mistral-7B, which are not state-of-the-art models. To demonstrate iREPO’s robustness and generalizability, it would be more insightful to test on current high-performing models like LLaMA 3. Without this, the results are less relevant to ongoing research in language model alignment.
3. Limited Real-World Applicability: Given the reliance on certain assumptions (e.g., no data distribution disparity), the applicability of iREPO in real-world, dynamic settings remains uncertain.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Do you need always need a dense $$H_e$$ matrix. This would imply that you need to observe at least D choose 2 comparison * number of annotators? Since, iREPO filters down comparison to strongest vs weakest comparison, could you reduce the annotation burden?
- Since you are computing the logit to regress to, why not regress to the logit for all pairs of comparisons instead of filtering down to the comparison of the strongest and weakest completion pair? For other methods where strength of preference cannot be determined, filtering makes sense as they are treated similarly (with a hard preference label). However, since you have notion of "strength" of preference that you are regressing to, I'd be curious to see the results where you do not filter preference pairs.
- UltraFeedback and other datasets are based on GPT-4 labels, does labeling with DeBERTa introduces a mismatch in the preference function, especially with MT-Bench with uses GPT-4 to evaluate model preferences again?
- Could you elaborate how the new labels collected, during iteration 1 and 2? How are the labels computed for iter-DPO and iter-IPO? I am worried the comparison isn't fair.
- Could you provide more detail on how the number of annotators change the preference labels/performance of iREPO?
soundness: 2
strengths: The paper makes an interesting contribution for (a) computing the strength of preference when given two completions and (b) providing a principled framework to compute such relative preference strength. The writing is reasonably clear, and the paper provides substantial theory to go along with their algorithm.
summary: The paper contributes a new method for aligning language models with human/AI feedback. Assuming multiple responses can be compared, the model proposes to compute a quantity indicating how strongly humans prefer one answer over the other, and use a DPO/IPO like objective regress onto the strength. The paper provides a principled approach for computing the strength of preference, and shows some empirical evidence showing the utility of this.
weaknesses: - Currently, the annotation scheme is rather expensive. Collecting a single preference pair requires sampling D completions, making D choose 2 pair and annotating each pair multiple times. After computing the logit corresponding to the strength of the preference, all other completions are thrown away.
- The experimental setup can be improved and made fairer (questions below). First, the main contribution of the paper is a new offline objective for alignment. A more informative experiment would be to compare the method first in a completely offline setting as follows: Take a SFT model, sample D completions per prompt, get preference labels for each pair. Use iREPO to regress onto the strength of preference (with and without filtering down to weakest/strongest pair), but there are many ways to compute ranking here and you could consider putting in all the preferences into DPO/IPO too (as they are implicitly ranking losses).
- More experimental evidence would help, as a lot of these evaluations are fairly noisy.
- In general, the paper needs to justify why expending this additional effort on collecting preference label is justified. The claim can be backed by showing better performance / unit of annotation effort. Or you could show that these preference labels result in higher signal-to-noise ratio for preferences, and can be scaled better.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can you elaborate "Conventional approaches often approximate this reward model through reinforcement learning techniques (Williams, 1992; Ahmadian et al., 2024; Schulman et al., 2017)." at line 200? I don't get how these references apply RL to approximate the reward model.
2. In Table 3, iREPO-2 is compared to Iter-DPO & Iter-IPO. Which version of Iter-DPO/IPO is used? Could you give a full results similar to Table 2, showing different stages of Iter-DPO/IPO?
3. Which specific nine LLM rankers did you adopt to simulate human preferences?
soundness: 2
strengths: * The presentation of the paper is clear and easy to follow.
* Empirical results validate the proposed algorithm.
* Theoretical analysis is provided for the proposed method.
summary: This paper proposes iREPO, which leverages the Zermelo’s model to construct empirical human preference. With this scalar preference target, iREPO regresses the implicit reward pairwise difference towards the logit of empirical human preference, allowing effective LLM alignment. The proposed method is supported with theoretical analysis and empirical results.
weaknesses: * Correctness. 
  * Line 042: online and offline alignment. Online alignment does not necessarily involve RLHF based on PPO. In fact, typical RLHF approaches are instances of offline RL, where a reward model is learned using offline dataset then an RL algorithm learns the policy (LLM) using that learned reward.
* Motivation of Zermelo's Model. Zermelo's algorthm is used to estimate the target regression values of implicit reward (eq16). In my opinion, it's also possible to leverage any reward model (trained with eq2) to provide such regression target. Could you elaborate why Zermelo's model is proposed and what is its advantages?
* Minor issues. Grammar or spelling errors: line 011 (Large), 108 (LLM Alignment), 253, 852.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Have you considered using the raw probability generated using annotators: $P(y_s > y_l | x) = 1/h * \sum_{H} I[ y_s > y_l ] $
soundness: 4
strengths: 1. The empirical results show that iRepo outperforms both DPO and IPO.
2. The ablation studies show that leveraging logits does lead to better alignment. Further ablation studies also show the effect of the number of annotators used to generate the logits.
3. The theoretical analysis is clear even though the assumptions and conditions are likely too specific.
summary: The paper proposes a regression based loss function for LLM alignment. This loss function leverages the probability of human annotators ranking one response higher than another to construct soft lables. Through empirical experiments, the authors demonstrate that the regression based loss function leads to better alignment in comparison to DPO and IPO. The authors use Phi-2 and Mistral-7B as the base model. They perform experiments using the LLM-eval-harness and Multi-turn evaluation frameworks. The authors also theoretically demonstrate that their method achieves optimal results under certain contributions
weaknesses: 1. The authors do not address the additional computational costs to update the weights $w_i$ in each iteration. The comparison between DPO/IPO and iRepo should take into account this increase in time complexity
2. A fundamental limitation of this method is that it requires multiple annotators for each pair of output. This is not applicable to DPO/IPO.
3. There seems to be a disconnect between the description of DPO and IPO across multiple sections. Until section 3, the authors refer to vanilla DPO/IPO. In the experiments, the authors use iterative DPO/IPO as the baseline without describing the iterative version of the algorithm.
4. It seems like simply increasing the number of annotators does not lead to better alignment. Infact, this seems like a strange hyperparameter to tune. Given the marginal improvement over the baselines, Figure 2(b) needs further explanation/investigation.
5. I would encourage the authors to add stronger baselines apart from DPO/IPO: SPPO, KTO, SimPO etc.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see weakness
soundness: 2
strengths: - This paper is well written and easy to follow with
- The authors provide some of the theoretical justification for the performance of the algorithms
- Empirical validation shows the performance of the algorithm
summary: This paper studies a method to do the policy optimization for language models. The authors leverage an empirical estimation of the win rate $w_{ij}$ and use the square loss to optimize the policy function. The authors showed both theoretical analysis and empirical algorithm with validation.
weaknesses: My major concern is the contribution of the algorithm and the theoretical analysis listed as below

- In section 4.1, the delivered message, specifically $w_i / (w_i + w_j) = h_{ij} / (h_{ij} + h_{ji})$ is somewhat trivial. The rationale behind highlighting this equivalence is unclear, as it does not seem to directly address any core technical challenges. A stronger connection to the problem statement or contribution would help clarify the authors’ intent here.
- In section 4.2, The proposed method employs the square loss to approximate the binary distribution, defined as $L = E[(p - p^)^2]$. The authors do not provide a clear reason for selecting this loss function over the more conventional KL divergence, $L = E(\mathrm{KL}(p \parallel p^))$, which is typically a “soft” version of cross-entropy loss. It would be beneficial for the authors to clarify why KL divergence was deemed unsuitable for this application and how the square loss better serves the purpose of approximating the distribution $p$.
- In addition, the algorithm requires repeated evaluations of $h$ for each individual preference pair $(i, j)$. The authors should clarify why this $h$-step data cannot be directly used within DPO but instead must be aggregated into a squared loss function. For fair comparison, given that $h = 5, 9, 15, 21$ were chosen in the appendix, the iterative DPO dataset should ideally leverage these same values ($h = 5, 9, 15, 21$) rather than a single annotator in vanilla iterative DPO. A clarification on this point is necessary.
- In section 4.3, the theoretical justification here requires further detail. For example, Assumption 4.1 cannot hold universally for any $P_h$. The authors may want to define the releasability on the ground truth $P^$, indicating that $P_h \to P^$ as $h \to \infty$ (per Assumption 4.2). However, this assumption may be also impractical, as it requires a significantly large $h$. Moreover, the theoretical analysis as presented is insufficient. For example, equation (18) essentially states that if $L = (p - q)^2 = 0$, then $D_{TV}(p, q) = 0$, which is self-evident. Similarly, equation (19) mirrors the DPO analysis. Furthermore, the role of $C_t$ in Theorem 4.5 is not well-explained, especially within the online iterative framework. A more thorough explanation of these theoretical points would strengthen the analysis.
- The authors should explicit show the selection of $h$ in the main paper. 

### Regarding the proof in the appendix, here're a few issues I'm suggesting to revise.
- When applying the mean-value theorem, the correct expression is $|\sigma(x_1) - \sigma(x_2)| = \Big|\sigma’(x_0)\Big| \times  |x_1 - x_2|$ (with absolute value of $|\sigma'(x_0)|$ rather than simply $\sigma’(x_0)|x_1 - x_2|$, though this does not affect the correctness of the proof.
- In line 810, this is a concentration inequality rather than the law of large numbers. The scaling factor of $O(1 / \sqrt{n})$ relies on certain conditions, which should be explicitly verified by the authors (e.g. Hoeffding's inequality with R-sub-Gaussian). The law of large numbers only guarantees convergence to zero but does not provide scale guarantees.