## 3 Materials and Method
### Proposed Method
#### 3.3.2 Reward Estimation Module  
The reward estimation module, serving as the core component for value evaluation in reinforcement learning, is responsible for estimating the contribution of each pricing action to long-term returns. Given that feedback in real-world platforms is often significantly delayed, sparse, and incomplete, a reward-modeling structure integrating multi-step return and dynamic discounting has been designed to enhance both the accuracy of return estimation and the stability of training. Architecturally, the reward estimation module adopts a multi-layer, one-dimensional convolutional neural network (1D-CNN) tailored for temporal modeling, which is motivated by the need to capture local temporal dependencies (e.g., short-term revenue trends or clustered sales fluctuations) in reward sequences with lower computational overhead. The input consists of the state representation vectors from the policy-learning module (i.e., the transformer output), shaped as \((B,T,d)\), where \(B\) denotes the batch size, \(T\) is the temporal window length, and \(d=256\) is the feature dimension. The first convolutional layer employs a kernel size of \(k=3\) with \(C_{1}=128\) channels, resulting in an output dimension of \((B,T-2,128)\); the second layer uses a kernel size of \(k=3\) with \(C_{2}=64\) channels, yielding \((B,T-4,64)\). A global average pooling layer compresses the temporal dimension into a single vector. This is followed by two fully connected layers with widths of 128 and 1, respectively, producing the value estimate \(Q(s_{t},a_{t})\) for each state-action pair. This structure is not only computationally lightweight but also enhances local dependency modeling among adjacent states through the sliding nature of convolution kernels. From a modeling perspective, the objective function of this module is based on a hybrid multi-step return, which combines Monte Carlo return and temporal difference return through weighted fusion. The learning objective is to minimize the Bellman error as follows:  
\[\mathcal{L}_{\text{value}}=\mathbb{E}_{(s_{t},a_{t},r_{t+1},\cdots)}\bigg{[} \Big{(}Q(s_{t},a_{t})-R_{t}^{(n)}\Big{)}^{2}\bigg{]}, \tag{19}\]  
where \(n\) denotes the number of backward steps and \(R_{t}^{(n)}\) represents the \(n\)-step cumulative return, defined as  
\[R_{t}^{(n)}=\sum_{i=0}^{n-1}\gamma^{i}r_{t+i}+\gamma^{n}\cdot\max_{a^{\prime}} Q(s_{t+n},a^{\prime};\theta^{-}), \tag{20}\]  
where \(\gamma\) denotes the dynamic discount factor, which is designed as a time-decaying function:  
\[\gamma(t)=\gamma_{0}\cdot\exp(-\lambda t). \tag{21}\]  
This formulation allows the model to assign moderate but diminishing attention to long-term rewards, thereby balancing short-term and long-term strategy optimization. In experiments, \(\gamma_{0}\) is set to 0.99 and \(\lambda\) to 0.01, which have been empirically determined to be optimal. This module is jointly trained with the policy-learning module, forming an actor-critic architecture. The policy network \(\pi_{\theta}\) outputs pricing actions, while the value network \(Q_{\phi}\) estimates the returns for those actions under given states. The total loss function used for joint optimization is  
\[\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{policy}}+\beta\cdot\mathcal{L}_ {\text{value}}, \tag{22}\]  
where \(\beta\) is the weighting coefficient for value loss, which is set to 0.5 to maintain a balance between policy learning and value estimation. The policy loss \(\mathcal{L}_{\text{policy}}\) is defined as the cross-entropy between predicted and high-reward actions, encouraging the network to prefer actions with higher expected returns. This reward estimation module offers several advantages in the pricing task. First, the convolutional structure demonstrates strong capabilities in modeling local behavior sequences, which aligns well with the needs of e-commerce pricing scenarios characterized by high correlation and low latency.