SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Changhyeon Lee and Seulki Lee
UNIST (Ulsan National Institute of Science and Technology)
{changhyeon,seulki.lee}@unist.ac.kr
In this paper, we propose to approximate the softmax output, which is the key product of the attention mechanism, to reduce its activation memory usage when training attention-based networks (aka Transformers). During the forward pass of the network, the proposed softmax output approximation method stores only a small fraction of the entire softmax output required for back-propagation and evicts the rest of the softmax output from memory. Then, during the backward pass, the evicted softmax activation output is approximated to compose the gradient to perform back-propagation for model training. Considering most attention-based models heavily rely on the softmax-based attention module that usually takes one of the biggest portions of the network, approximating the softmax activation output can be a simple yet effective way to decrease the training memory requirement of many attention-based networks. The experiment with various attention-based models and relevant tasks, i.e., machine translation, text classification, and sentiment analysis, shows that it curtails the activation memory usage of the softmax-based attention module by up to  $84\%$  ( $6.2 \times$  less memory) in model training while achieving comparable or better performance, e.g., up to  $5.4\%$  higher classification accuracy.
Recently, many attention-based networks, aka Transformers [50], are widely used in many natural language processing tasks such as machine translation [50, 13], sentiment analysis [52, 29, 34], text classification [52, 54], and question/answering [47, 53, 20]. Due to their superior performance, they also have become the backbone of many large language and general task-agnostic models, e.g., GPT series [41, 5], and are actively expanding to other domains ranging from computer vision (Vision Transformers [12, 49, 38]) to protein structure predictions (AlphaFold [44, 22]). However, the stellar performance of attention-based networks comes with significant resource requirements in training, especially massive memory (RAM) usage. For example, the small-size GPT-2 model [41] is known to require  $27.5\mathrm{GB}$  of peak memory for its training with a mini-batch of size eight [3]. Hence, the memory bottleneck problem makes training of attention-based models challenging when massive memory capacity is not available, e.g., BERT [11] takes 16 TPUs [21], each equipped with 32 TB of built-in memory, for training, and the regular GPT-2 [41] is reported to be trained with 32 TPUs.
During deep model training, it is required to store various components in memory, i.e., model parameters, input data, optimizer state, etc., along with activation output, i.e., the intermediate output of each layer. As the activation output of each layer must be stored in memory for gradient-based back-propagation [42], the memory capacity required to train a model increases proportionally to the data sequence length (width), depth of the network, and mini-batch size of the model. Consequently, as the model gets huge and the training data gets bigger, the activation easily becomes to take up more memory than other components. Especially, it is more evident in NLP (natural language processing) tasks, as provided in Figure 1(a) [25], which shows the memory composition required for training four different attention-based model configurations, ranging from 22 billion to 1 trillion parameters.
For GPT-3 [5] that has 175B parameters, the activation output requires approximately  $1.45 \times$  more memory than the parameter and optimizer state, and for MT-NLG [45] that has 530B parameters, it is about  $3.6 \times$  more. Therefore, to popularize such large models in practice, it is necessary to reduce the massive amount of activation memory required in model training.
(a)
Figure 1: (a) The amount of memory required by 1) parameters and optimizer state and 2) activation memory [25] in training attention-based models (GPT-3 [5] with 22B and 175B parameters and MT-NLG [45] with 530B and 1T parameters). The dashed red line represents 80GB of memory capacity available in NVIDIA A100 GPU [35]. (b) The detailed memory composition when training an attention-based network (blue boxes). The softmax output approximation reduces the activation memory usage by storing only a small subset of them (e.g.  $s_1, s_n, \alpha, \beta$ ) in memory (red boxes).
In this paper, we propose an approximation method of the softmax output [4] produced in the attention module [50] to reduce its activation memory requirement in model training, as shown in Figure 1(b). Instead of storing the entire elements of the softmax output vector that represents attention scores, it stores only a small subset of the elements and evicts the rest from memory during the forward pass of the network. Later, during the backward pass, the discarded softmax output elements are approximated to construct the gradients for back-propagation. As only a small fraction of the softmax activation output is stored during the forward pass, a substantial amount of memory can be saved. The proposed method is not to officienate the computation complexity of the attention module, such as [9, 48, 31, 51], but to reduce the amount of activation memory used in training by storing only selected parts of the attention score, i.e., softmax output, in memory, not all of them. Thus, it is applicable to a softmax output computed from any form of the attention module [9, 48, 31, 51] in a model-agnostic way. To the best of our knowledge, this is the first and unique attempt to curtail the activation memory usage of the attention module in training by approximating the softmax output.
The efficacy of approximating the softmax output is substantial because it is the key component of the attention block used to compute the normalized attention score [48]. Since many attention-based models repeatedly use multi-head attentions in their encoders and/or decoders [50], the softmax output can be accounted as the dominant activation in the network. For instance, the softmax operation, i.e.,  $\mathrm{Softmax}(QK^T / \sqrt{d_k})$ , takes up  $80\%$  of the attention module itself, and  $64.7\%$  and  $62.2\%$  of the entire layer activation output during the forward pass of the classic Transformer [50] and BERT [11] model, respectively. Therefore, streamlining the activation memory requirement of the repeatedly-used softmax-based attention module is a simple yet effective way of improving the training memory efficiency of attention-based networks, e.g., the reduced peak-memory usage and memory footprint.
The experiment results demonstrate that the softmax output approximation drastically decreases the training memory required by the attention module by up to  $84\%$  while achieving competitive or improved performance of various attention-based models, i.e., basic Transformer [50], XLNet [52], and ALBERT [30], e.g., up to  $5.4\%$  higher accuracy, on machine translation [13], text classification [54], and sentiment analysis [34, 46] tasks. It validates that the softmax output approximation 1) can be readily applied to any existing attention-based models without modifying the network architecture, 2) does not significantly increase computation time for activation memory saving, and 3) enables models to learn longer sequences with a larger batch size when given the same memory budget. We implement the proposed softmax output approximation with PyTorch and make it public<sup>1</sup>.
<https://github.com/eai-lab/SoftmaxOutputApproximation>
2 Training of Softmax Operation
This section discusses the training procedure of the softmax operation and its memory usage.
Softmax. Let  $\vec{z} = [z_1, z_2, \ldots, z_n]$  be a vector in  $\mathbb{R}^{1 \times n}$ , where  $n$  denotes the sequence length. The vector  $\vec{z}$  is obtained from the rows of  $Z \in \mathbb{R}^{n \times n}$ , which is the result of the query/key multiplication  $(Z = QK^T / \sqrt{d_k})$ . Then, the element  $s_i$  in the softmax vector  $\vec{s} = [s_1, s_2, \ldots, s_n]$  is defined as [4].
$$
s _ {i} = \frac {e ^ {z _ {i}}}{\sum_ {j = 1} ^ {n} e ^ {z _ {j}}} \text {f o r a l l} i = 1, 2,.., n \text {s . t .} 0 \leq s _ {i} \leq 1 \text {a n d} \sum_ {i = 1} ^ {n} s _ {i} = 1 \tag {1}
$$
Given the definition of softmax in Equation 1, let us assume there is a  $l$ -layer DNN (deep neural network) having a softmax operation at the  $k$ -th layer  $(k < l)$ . To train the network via the gradient descent [42], the gradient of the loss  $L(\vec{x})$  is computed with the input  $\vec{x}$  w.r.t. the weight matrix  $W_{m}$ , i.e.,  $\partial L(\vec{x}) / \partial W_{m}$ , at the  $m$ -th layer of the network  $(m < k)$ . By using the chain rule, we get:
$$
\frac {\partial L (\vec {x})}{\partial W _ {m}} = \frac {\partial L (\vec {x})}{\partial \vec {z} _ {l}} \frac {\partial \vec {z} _ {l}}{\partial \vec {z} _ {l - 1}} \frac {\partial \vec {z} _ {l - 1}}{\partial \vec {z} _ {l - 2}} \dots \frac {\partial \vec {s}}{\partial \vec {z} _ {k - 1}} \dots \frac {\partial \vec {z} _ {m}}{\partial W _ {m}} \tag {2}
$$
where  $\vec{z}_l$  is the output of the  $l$ -th layer,  $\vec{z}_k = \vec{s}$  and  $\vec{z}_{k-1}$  in the middle is the output and input of the softmax operation at the  $k$ -th layer, respectively. Thus, to get  $\frac{\partial L(\vec{x})}{\partial W_m}$ , the gradient of  $\vec{s}$  w.r.t.  $\vec{z}_{k-1}$ , i.e.,  $\frac{\partial \vec{s}}{\partial \vec{z}_{k-1}}$ , should be computed.
Gradient Matrix. To obtain  $\partial \vec{s} / \partial \vec{z}_{k-1}$  in Equation 2, the partial derivative of the  $i$ -th softmax element  $s_i$  w.r.t. its  $j$ -th input element  $z_{k-1,j}$  is computed as follows:
$$
\frac {\partial s _ {i}}{\partial z _ {k - 1 , j}} = s _ {i} \left(\mathbb {1} _ {i = j} - s _ {j}\right) \tag {3}
$$
where  $\mathbb{1}_{i = j}$  is the indicator function. Since it only involves  $s_i$  and  $s_j$ , not  $z_{k - 1,j}$ , the gradient matrix of the softmax vector  $\vec{s}$  w.r.t. its input vector  $\vec{z}_{k - 1}$  is given by:
$$
\frac {\partial \vec {s}}{\partial \vec {z} _ {k - 1}} = \left[ \begin{array}{c c c c} s _ {1} (1 - s _ {1}) & - s _ {1} s _ {2} & \dots & - s _ {1} s _ {n} \\ - s _ {2} s _ {1} & s _ {2} (1 - s _ {2}) & \dots & - s _ {2} s _ {n} \\ \vdots & \vdots & \ddots & \vdots \\ - s _ {n} s _ {1} & - s _ {n} s _ {2} & \dots & s _ {n} (1 - s _ {n}) \end{array} \right] \tag {4}
$$
which allows the back-propagation of  $\partial L(\vec{x}) / \partial W_{m}$  in Equation 2 if  $\partial \vec{z}_l / \partial \vec{z}_{l - 1}$  for all  $l$  is available.
Memory Requirement. Thus, to compute  $\partial \vec{s} / \partial \vec{z}_{k-1}$  in the back-propagation step, the softmax vector output  $\vec{s}$  needs to be stored for each input  $\vec{x}$  as an activation in memory during the forward pass of the network. Considering that 1) many DNNs, e.g., Transformers [50] and convolutional networks [26, 18], have deep and wide layers, and 2) back-propagation is usually performed with multiple examples (mini-batch), the total amount of activation memory required in training increases linearly to the number and length of softmax layers and the mini-batch size. For example, a network with  $l$  softmax layers of length  $n$  trained with a mini-batch consisting of  $b$  examples requires a total of  $l \cdot n \cdot b$  of activation memory for a single step of back-propagation.
Memory Saving. Motivated by this observation, we reduce the amount of activation memory required to back-propagate through a softmax layer by storing only a small subset of softmax output elements  $\vec{s} = [s_1,s_2,\dots,s_n]$  in memory, while evicting the rest of the elements from memory during the forward pass of the network. During back-propagation, the evicted elements are approximated such that the gradient matrix (Equation 4) obtained from the approximated softmax output minimizes the expected loss of training performance. The amount of activation memory saved by the softmax output approximation is multiplied by the number of softmax layers  $l$  and the mini-batch size  $b$ , i.e.,  $l\cdot (n - m)\cdot b$ , where  $n$  is the total number of softmax output elements (the length of the attention score), and  $m$  is the number of elements to be stored in memory. Also, the input of the softmax,  $\vec{z}$ , i.e., query/key multiplication  $QK^T / \sqrt{d_k}$ , which is the output of the previous layer, can be obtained from the approximated softmax output from the inverse of Equation 1 as:
$$
z _ {i} = \ln \left(s _ {i}\right) + \ln \left(\sum_ {j = 1} ^ {n} e ^ {z _ {j}}\right) \tag {5}
$$
Thus,  $\vec{z}$  of length  $n$  can also be removed from the memory, which saves additional memory of  $l\cdot n\cdot b$ .
3 Softmax Output Approximation
3.1 Approximation Strategy
Given a softmax output vector  $\vec{s} \in [0,1]^n$ , its approximation,  $\vec{s}'$ , is computed with the following steps. First, among all softmax output elements of  $\vec{s}$ , the  $m$  highest and  $m$  lowest elements are stored in memory during the forward pass of the network. Then, the remaining  $n - 2m$  elements are discarded from memory and approximated later in the backward pass. Lastly, using the stored and approximated elements together, the gradient matrix (Equation 4) is constructed for back-propagation (Equation 2).
Approximated Softmax. Given the softmax output vector  $\vec{s}$ , its approximation  $\vec{s}'$  is defined as:
$$
\vec {s} ^ {\prime} = \left[ \underbrace {h _ {1} h _ {2} \dots h _ {m}} _ {m \text {h i g h e s t s}} \underbrace {s _ {1} ^ {\prime} s _ {2} ^ {\prime} \dots s _ {n - 2 m} ^ {\prime}} _ {n - 2 m \text {a p p r o x i m a t i o n s}} \underbrace {\ell_ {m} \ell_ {m - 1} \dots \ell_ {1}} _ {m \text {l o w e s t s}} \right] \tag {6}
$$
where  $h_1, h_2, \ldots, h_m$  and  $\ell_m, \ell_{m-1}, \ldots, \ell_1$  are the  $m$  highest and  $m$  lowest elements, respectively, that are stored intact in memory, and  $s_1', s_2', \ldots, s_{n-2m}'$  are the rest  $n-2m$  elements that approximate the non-stored elements,  $s_1, s_2, \ldots, s_{n-2m}$ . Figure 2 depicts an example of softmax output approximation.
Figure 2: Among a total of  $n = 21$  softmax output elements sorted in descending order, the  $m = 3$  highest elements  $(h_1, h_2, h_3)$  and the  $m = 3$  lowest elements  $(\ell_3, \ell_2, \ell_1)$  are stored in memory. The remaining  $n - 2m$  elements  $(s_1', \dots, s_{n - 2m}')$  are not stored but approximated by the softmax output approximation function described below.
3.2 Element Selection for Memory Storing
Keeping the  $m$  highest and  $m$  lowest softmax output elements in memory minimizes the possible loss of training performance caused by the softmax output approximation. The following shows doing so minimizes the difference between the gradient matrix constructed from the original softmax  $\vec{s}$  and the gradient matrix constructed from the approximated softmax  $\vec{s}'$ , so that the weight update computed from the approximated softmax becomes close to the original ones.
Gradient Error. Given a  $n \times n$  softmax gradient matrix  $\partial \vec{s} / \partial \vec{z}_{k-1}$  in Equation 4 and its approximate gradient matrix  $\partial \vec{s}' / \partial \vec{z}_{k-1}$  obtained from the approximated softmax vector  $\vec{s}'$ , the gradient error between them is defined as  $\epsilon(\vec{s}, \vec{s}') = |\frac{\partial \vec{s}}{\partial \vec{z}_{k-1}} - \frac{\partial \vec{s}'}{\partial \vec{z}_{k-1}}|$ . We first decompose the gradient error  $\epsilon(\vec{s}, \vec{s}')$  into each  $i$ -th row-wise error  $\epsilon_i(\vec{s}, \vec{s}')$  consisting of 1)  $i$ -th column element's error and 2) the sum of the remaining  $j \neq i$  column element's errors in the  $i$ -th row. Then, the row-wise error  $\epsilon_i(\vec{s}, \vec{s}')$  for all  $n$  rows are summed up to obtain the total gradient error, i.e.,  $\epsilon(\vec{s}, \vec{s}') = \sum_{i=1}^{n} \epsilon_i(\vec{s}, \vec{s}')$  as:
$$
\epsilon \left(\vec {s}, \vec {s} ^ {\prime}\right) = \sum_ {i = 1} ^ {n} \epsilon_ {i} \left(\vec {s}, \vec {s} ^ {\prime}\right) = \sum_ {i = 1} ^ {n} \left(\underbrace {\left| s _ {i} \left(1 - s _ {i}\right) - s _ {i} ^ {\prime} \left(1 - s _ {i} ^ {\prime}\right) \right|} _ {i \text {- t h e l e m e n t ' s e r r o r}} + \underbrace {\sum_ {j = 1 \backslash i} ^ {n} \left| s _ {i} s _ {j} - s _ {i} ^ {\prime} s _ {j} ^ {\prime} \right|}\right) \tag {7}
$$
Error Bound. The second term  $\sum_{j=1\backslash i}^{n}|s_{i}s_{j}-s_{i}'s_{j}'|$  in Equation 7 is calculated in two cases. The set  $S_{1}$  comprises indices where  $s_{i}s_{j}-s_{i}'s_{j}'$  for  $j\neq i$  is positive, while  $S_{2}$  consists of the opposite.
$$
\sum_ {j = 1 \backslash i} ^ {n} | s _ {i} s _ {j} - s _ {i} ^ {\prime} s _ {j} ^ {\prime} | = \left\{ \begin{array}{l l} | s _ {i} (1 - s _ {i}) - s _ {i} ^ {\prime} (1 - s _ {i} ^ {\prime}) | + 2 \sum_ {j \in S _ {1}} s _ {i} s _ {j} - s _ {i} ^ {\prime} s _ {j} ^ {\prime} & \text {i f} s _ {i} (1 - s _ {i}) <   s _ {i} ^ {\prime} (1 - s _ {i} ^ {\prime}) \\ | s _ {i} (1 - s _ {i}) - s _ {i} ^ {\prime} (1 - s _ {i} ^ {\prime}) | - 2 \sum_ {j \in S _ {2}} s _ {i} s _ {j} - s _ {i} ^ {\prime} s _ {j} ^ {\prime} & \text {i f} s _ {i} (1 - s _ {i}) \geq s _ {i} ^ {\prime} (1 - s _ {i} ^ {\prime}) \end{array} \right. \tag {8}
$$
If  $s_i s_j - s_i' s_j'$  for all  $j \neq i$  belongs to either  $S_1$  or  $S_2$ , the second term of Equation 8 is bounded by its maximum value. In the case of  $S_1$ , it becomes:
$$
2 \sum_ {j \in S _ {1}} s _ {i} s _ {j} - s _ {i} ^ {\prime} s _ {j} ^ {\prime} = 2 \left(s _ {i} \sum_ {j \in S _ {1}} s _ {j} - s _ {i} ^ {\prime} \sum_ {j \in S _ {1}} s _ {j} ^ {\prime}\right) = 2 \left(s _ {i} \left(1 - s _ {i}\right) - s _ {i} ^ {\prime} \left(1 - s _ {i} ^ {\prime}\right)\right) \leq 0. 5 \tag {9}
$$
We denote the second term of Equation 8 as  $\delta$ . From Equations 7, 8, and 9, the gradient error  $\epsilon (\vec{s},\vec{s^{\prime}})$  becomes to be bounded by:
$$
\epsilon \left(\vec {s}, \vec {s} ^ {\prime}\right) = 2 \sum_ {i = 1} ^ {n} \left| s _ {i} \left(1 - s _ {i}\right) - s _ {i} ^ {\prime} \left(1 - s _ {i} ^ {\prime}\right) \right| + \delta \leq 2 \sum_ {i = 1} ^ {n} \max  \left(s _ {i} \left(1 - s _ {i}\right), 0. 2 5 - s _ {i} \left(1 - s _ {i}\right)\right) + 0. 5 \tag {10}
$$
where the inequality comes from the property of the softmax, i.e.,  $0 \leq s_i(1 - s_i) \leq 0.25$ .
Error Minimization. Hence, to minimize the gradient error  $\epsilon (\vec{s},\vec{s}^{\prime})$  in Equation 10, we select a set of  $s_i$  as close as possible to some constants, i.e.,  $\{0,1\}$  for the first term,  $s_i(1 - s_i)$ , and 0.5 for the second term,  $0.25 - s_{i}(1 - s_{i})$ . First, by keeping the highest  $m$  and the lowest  $m$  elements in  $\vec{s}$ , we can select elements closest to the first two constants,  $\{0,1\}$ . Second, similarly, when selecting the highest element  $h_1$  from  $\vec{s}$ , it becomes the one closest to 0.5 among all softmax elements since the remaining  $n - 1$  elements are at most  $1 - h_{1}$  from the properties of softmax (Equation 1). Thus, their distances from 0.5 become at least  $|h_1 - 0.5|$ , i.e.,  $|0.5 - h_i| = |0.5 - (1 - h_1)|\geq |h_1 - 0.5|$  for all  $i\neq 1$ . It shows that selecting the  $m$  highest and  $m$  lowest softmax elements minimizes the gradient approximation error,  $\epsilon (\vec{s},\vec{s}^{\prime})$ , leading to the minimization of possible performance loss in training.
3.3 Approximation Method
During the back-propagation, the elements in the middle of Equation 6, i.e.,  $s_1', s_2', \ldots, s_{n-2m}'$ , approximate the corresponding original elements,  $s_1, s_2, \ldots, s_{n-2m}$ , that are evicted from memory.
Exponential Distribution. We use the exponential distribution [43], i.e.,  $\lambda e^{-\lambda k}$ , for the softmax approximation because 1) it naturally models a softmax in descending order well, 2) it can properly represent the non-linear changes of softmax as its slope changes rapidly; its derivative is  $-\lambda^2 e^{-\lambda k}$ , and 3) it can be efficiently computed with a small number of parameters. Unlike the original exponential distribution using a single parameter  $\lambda$ , we introduce two separate parameters  $\alpha$  and  $\beta$  to better represent the characteristics of the softmax operation in our approximation.
Approximation Function. Given the non-stored original softmax output elements sorted in descending order as  $s_1 \geq s_2 \geq \ldots \geq s_{n-2m}$  that are computed but discarded from memory during the forward pass of the network, its  $k$ -th element  $s_k$  is approximated as  $s_k'$  in the backward pass as:
$$
s _ {k} ^ {\prime} = \alpha e ^ {- \beta \cdot k} \text {f o r} k = [ 0, 1, \dots , N - 1 ] \tag {11}
$$
where  $N = n - 2m + 2$  is the number of softmax output elements to be approximated, including  $h_m$  and  $\ell_{m}$  as the first and last reference values of the approximation, i.e., we let  $s_0^{\prime} = h_m$  and  $s_{N - 1}^{\prime} = \ell_{m}$ . The two parameters  $\alpha$  and  $\beta$  are given by:
$$
\alpha = h _ {m}, \quad \beta = \frac {1}{N} \frac {1}{1 - h _ {m}} \left(\ln \frac {h _ {m}}{\ell_ {m}} + 2 \left(h _ {m} - h _ {m + 1}\right)\right) \tag {12}
$$
By using Equations 11 and 12,  $s_k'$  for  $k = 0, 1, \dots, N - 1$  can be computed to construct the gradient matrix in Equation 4 during back-propagation, while having only  $2m$  values in activation memory, i.e., the  $m - 1$  highest and  $m - 1$  lowest softmax elements plus the two parameters  $\alpha$  and  $\beta$ .
As mentioned above, the two parameters  $\alpha$  and  $\beta$  are defined such that the first and last approximation value becomes to be the same as  $h_m$  and  $\ell_m$ , respectively, i.e.,  $s_0' = h_m$  and  $s_{N-1}' = \ell_m$ . The parameter  $\alpha$  can be easily obtained as  $\alpha = h_m$  since  $s_0' = \alpha e^{-\beta \cdot 0} = h_m$ . The parameter  $\beta$  consists of three components, i.e., 1) normalization factor  $\frac{1}{N} \frac{1}{1 - h_m}$ , 2) range value  $\sigma$ , and 3) rate value  $\hat{\beta}$  as:
$$
\beta = \frac {1}{N} \frac {1}{1 - h _ {m}} \cdot \sigma \cdot \hat {\beta} \text {w h e r e} \sigma = - \frac {1}{\hat {\beta}} \ln \frac {1}{\alpha^ {2} \hat {\beta} ^ {2}} \text {a n d} \hat {\beta} = \sqrt {\frac {1}{h _ {m} \ell_ {m}}} \cdot e ^ {h _ {m} - h _ {m + 1}} \tag {13}
$$
Range value  $(\sigma)$ . The range value  $\sigma$  in Equation 13 is determined to map the domain of the approximation function, i.e.,  $k = [0,N - 1]$ , to a new domain  $k^{\prime} = [0,\sigma ]$ . Since the continuous domain of exponential distribution ranges from 0 to  $\infty$ , we define a new finite range of domain from which the corresponding approximations (codomain) are computed. To determine the range  $\sigma$ , we set the slope of the softmax approximation function in Equation 11 at  $k = N - 1$  as symmetrical to the slope at  $k = 0$  to provide the length-invariant domain given an arbitrary length of softmax. Given the differentiation of  $s_k^\prime = \alpha e^{-\beta \cdot k}$  w.r.t.  $k$  as  $\partial s_k^\prime /\partial k = -\alpha \beta e^{-\beta k}$ , the slope at  $k = 0$  becomes  $\partial s_0^\prime /\partial k = -\alpha \beta e^{-\beta \cdot 0} = -\alpha \beta$ . By making the slope at  $k = N - 1$  as symmetrical to  $-\alpha \beta$ , i.e.,  $\partial s_{N - 1}^{\prime} / \partial k = -\alpha \beta e^{-\beta \cdot (N - 1)} = -\frac{1}{\alpha\beta}$ , replace  $N - 1$  with new domain  $\sigma$ , the range  $\sigma$  becomes:
$$
\sigma = - \frac {1}{\beta} \ln \frac {1}{\alpha^ {2} \beta^ {2}} \tag {14}
$$