SlowGuess/NeurIPS_2023_OCR huggingface dataset:
System Implementation. Here we present how we implement WTA-CRS in Equation (6) in practice. For the linear layer, as we analyzed, we only replace GEMM in Equation (1c) with its approximated version. In this case,  $\mathbf{X}$  and  $\mathbf{Y}$  in Equation (6) are activation  $\mathbf{H}^{\top}$  and output gradient  $\nabla \mathbf{Z}$ , respectively. Given the total column-row pair budget  $k$ , the first step is to build the deterministic index set  $\mathcal{C}$ , where each element is summed explicitly without sampling. Note that  $\mathcal{C}$  is a set of indices with the highest probabilities in Equation (3). Thus, to build  $\mathcal{C}$ , we only need to determine its size, denoted as  $|\mathcal{C}|$ , which minimizes the variance of the estimator. As Theorem 2 suggested, we set  $|\mathcal{C}| = \min_{|\mathcal{C}| \in \{0, \dots, k\}} \frac{1 - \sum_{c \in \mathcal{C}} p_c}{k - |\mathcal{C}|}$ . The second step is to sample  $k - |\mathcal{C}|$  column-row indices from the remaining distribution  $\mathcal{P}^{\mathcal{D} \setminus \mathcal{C}}$  to obtain the set  $\mathcal{C}_{\mathrm{stoc}}$ , where  $|\mathcal{C}_{\mathrm{stoc}}| = k - |\mathcal{C}|$ . The third step is to build sub-sampled  $\mathbf{H}'$  with only rows from  $\mathcal{C} \cup \mathcal{C}_{\mathrm{stoc}}$ . Note that for rows in  $\mathbf{H}'$  from  $\mathcal{C}_{\mathrm{stoc}}$ , we need to normalize it by  $\frac{1 - \sum_{c \in \mathcal{C}} p_c}{k - |\mathcal{C}|}$  according to Equation (6). We illustrate the above process in Figure 4. The pseudocode to Appendix D Algorithm 1.
Scope. Here we show which operation can be replaced with its approximation version. As shown in Figure 5, the transformer is mainly consisted of linear layer, TensorMul, and other operations (e.g., GeLU, Dropout, LayerNorm). TensorMul in Figure 5 refers to the multiplication between two four-dimensional tensors. Our WTA-CRS can be applied to Linear-Q, -K, -V, -O, -U, -D, TensorMul-1, and TensorMul-2 (in green). The activations of Dropout and GELU operations (in blue) can be losslessly compressed. The Softmax and LayerNorm operators (in gray) remain unchanged.
4 Related Work and Discussion
Due to the page limit, we discuss the related work on approximated matrix multiplication and approximation in LLM inference. Other related topics, e.g., parameter-efficient fine-tuning, activation quantization, and gradient checkpointing, can be found in Appendix A. We also discuss the limitation and potential negative social impact in Appendix A.
Approximated Matrix Multiplication. In the context of neural networks, approximated matrix multiplication methods can be broadly categorized into two main groups: (1) Butterfly-based methods [33, 34] replace dense weight matrices with butterfly matrices. We note that they focus on the weight matrix and are orthogonal to our research, as we concentrate on sub-sampling the activation matrix. (2) Column-row sampling (CRS) methods[22, 21, 31] select important rows and columns from the input matrices and perform the multiplication on the sampled matrix. Our work is closely aligned
with this second research line. [21, 31] share similarities with our research in terms of utilizing CRS for approximating matrix multiplication within neural networks. The main distinction lies in how to select the column-row pairs. Specifically, [21] deterministically selects column-row pairs without scaling, whereas our estimator divides the column-row pairs into a deterministic component and a stochastic component. As we analyzed, selecting column-row pairs deterministically is biased. Later we show that this approach may cause a significant accuracy drop ("Deterministic" in Figure 12).
Approximation in Inference There are two approximation techniques to reduce inference latency. (1) Sparse Modeling, which only involves a subset of weights during computation to reduce both computational and memory I/O demands [35, 36, 37]. (2) Quantization, which compresses the trained weight into lower numerical precision [38, 39, 40]. All these approximation techniques trade-off model quality in return for improved efficiency. Besides approximation technique itself, researchers also propose methods to recover the accuracy drop of compressed models [41, 42], e.g., by prompting compressed models. These methods can greatly improve the accuracy-efficiency trade-off of LLMs.
In this section, we design experiments to answer the following research questions: RQ1: How effective is WTA-CRS in terms of accuracy with reduced memory usage? RQ2: How sensitive is WTA-CRS affected by its key hyper-parameters? RQ3: WTA-CRS contains two parts, i.e., the deterministic summation part and the statistical sampling part. Are they both necessary? RQ4: How is the fine-tuning speed affected by WTA-CRS?
Datasets and Evaluation Protocol. Following most of the previous work, we adopt GLUE benchmark [43] to evaluate the effectiveness of different methods, including the CoLA, SST-2, MRPC, QQP, MNLI, QNLI, RTE, and STS-B datasets. To evaluate the memory usage, we report the peak GPU memory usage and compression rate during the fine-tuning process with Huggingface API [44].
Compared Methods and Adopted Models. We consider three methods to compare in this paper: Full fine-tuning (Full), LoRA [12], and Ladder Side-tuning (LST) [9]. Specifically, Full tunes all of the parameters in the model to provide an upper bound of accuracy; LoRA inserts trainable low-rank matrices into the model to parameterize the weights' changes; LST injects a trainable ladder side structure. Since WTA-CRS essentially replace the linear operation with approximated one, we emphasize that our WTA-CRS is compatible with all these three compared methods, i.e., they can be combined together towards smaller memory usage. For the backbone model, we follow the previous work [9, 14, 12] to adopt the OPT [45], Bert-Base [2], Bert-Large, T5-Base, T5-Large, and T5-3B [46] for evaluating the effectiveness of different methods.
Hyperparameter Settings. For WTA-CRS, it only has one hyperparameter  $k$ , which controls the column-row pair budget. We assign the same  $k$  to all replaceable linear operations in the model. We consider the normalized column-row pair budget  $k / |\mathcal{D}| \in \{0.3, 0.1\}$ , which are denoted as WTA-CRS @0.3 and WTA-CRS@0.1, respectively. We also consider the combination of WTA-CRS and LoRA to further reduce the memory cost of both optimizer and activations. The detailed hyperparameters are given in Appendix F. All reported results are averaged over three random trials.
5.2 Accuracy versus Memory Usage (RQ1)
To answer RQ1, we first analyze the trade-off between the model performance and memory saving. The evaluation results and peak memory usage are given in Tables 1 and 2, respectively. We observe:
WTA-CRS achieves a superior trade-off between accuracy and memory usage compared to baselines. Specifically, WTA-CRS has negligible accuracy drop, while the peak memory usage is reduced by  $2.1 \times 2.7 \times$  (when combined with LoRA).
Table 1: The GLUE benchmark results with T5 and Bert at different scales.
Model	Method	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	STS-B	AVG
BERT-Base	Full	60.9±1.89	92.2±0.34	87.9±0.46	87.8±0.01	83.7±0.05	90.7±0.14	66.4±0.36	88.1±0.27	82.2
LoRA	61.6±0.25	91.7±0.17	90.0±0.34	86.9±0.1	83.6±0.02	90.8±0.17	68.2±0.36	87.6±0.52	82.6
WTA-CRS@0.3	60.7±0.89	90.2±0.06	87.0±0.14	87.5±0.03	83.4±0.06	90.4±0.07	65.9±0.18	89.3±0.2	81.8
LoRA+WTA-CRS@0.3	61.5±1.08	89.6±0.52	89.6±0.09	86.3±0.02	82.8±0.35	90.6±0.16	67.9±0.72	87.3±0.7	81.9
T5-Base	Full	60.1±0.37	94.9±0.29	91.5±0.29	88.5±0.07	87.0±0.1	93.3±0.03	79.4±0.78	90.6±0.14	85.7
LoRA	60.6±0.94	94.6±0.05	92.2±0.31	87.4±0.06	86.2±0.06	93.4±0.03	80.6±0.74	90.7±0.05	85.7
LST	55.5±0.24	94.0±0.17	91.1±0.18	87.4±0.03	85.7±0.13	93.4±0.0	72.7±0.54	90.4±0.06	83.8
WTA-CRS@0.3	60.9±0.52	94.8±0.14	91.1±0.35	88.0±0.11	86.3±0.02	93.1±0.07	78.7±0.59	90.5±0.05	85.4
LoRA+WTA-CRS@0.3	60.0±0.51	94.4±0.16	92.0±0.38	87.3±0.04	85.6±0.08	93.2±0.01	80.1±1.02	90.4±0.06	85.4
BERT-Large	Full	66.8±0.31	93.5±0.29	89.5±0.26	88.5±0.03	86.4±0.19	92.1±0.24	72.6±0.36	90.2±0.76	85.0
LoRA	65.9±0.27	93.8±0.17	90.8±0.37	87.6±0.08	85.9±0.05	92.0±0.2	71.3±0.18	90.3±0.09	84.7
WTA-CRS@0.3	64.7±0.44	93.5±0.0	89.3±0.39	88.2±0.04	85.2±0.03	91.9±0.12	73.8±0.54	90.4±0.02	84.6
LoRA+WTA-CRS@0.3	66.0±0.33	93.3±0.29	89.7±1.32	87.6±0.02	86.0±0.07	91.9±0.14	72.4±0.17	89.7±0.04	84.6
T5-Large	Full	61.3±1.01	96.3±0.0	93.4±0.13	89.7±0.01	89.8±0.07	94.2±0.05	85.3±0.17	91.8±0.08	87.7
LoRA	63.3±0.26	96.4±0.14	93.5±0.16	88.5±0.03	89.5±0.05	94.3±0.07	84.2±0.68	91.7±0.13	87.7
LST	59.9±0.77	95.8±0.06	91.8±0.08	88.4±0.01	88.7±0.05	94.2±0.02	82.5±0.18	91.4±0.07	86.6
WTA-CRS@0.3	60.9±1.18	96.3±0.25	93.6±0.57	89.3±0.04	89.5±0.12	94.1±0.03	84.4±0.34	91.3±0.05	87.4
LoRA+WTA-CRS@0.3	62.9±1.19	96.2±0.05	93.6±0.47	88.3±0.02	89.2±0.08	94.0±0.07	83.9±0.95	91.3±0.03	87.4
T5-3B	LoRA	70.1±0.37	96.8±0.29	94.0±0.27	89.9±0.0	91.0±0.14	95.6±0.05	85.9±0.36	92.9±0.08	89.5
LoRA+WTA-CRS@0.3	71.4±0.35	96.4±0.06	94.6±0.39	90.0±0.05	91.0±0.06	95.6±0.12	86.3±0.36	92.9±0.09	89.8
Table 2: Peak memory usage (GB) and compression rate of fine-tuning T5-Base and -Large. We measure the memory usage on a single NVIDIA A100 (80GB) GPU. For T5-3B, since it is trained using multi-GPUs with data parallel. We instead report the maximum batch size in Figure 6 for it.
FP	LoRA	LST	WTA-CRS@0.3	WTA-CRS@0.1	LoRA+WTA-CRS@0.3	LoRA+WTA-CRS@0.1
T5-Base	17.66 (1×)	13.84 (1.3×)	5.50 (3.2×)	8.44 (2.1×)	7.30 (2.4×)	6.50 (2.7×)	5.44 (3.2×)
T5-Large	45.85 (1×)	36.83 (1.2×)	14.85 (3.1×)	21.58 (2.1×)	18.46 (2.5×)	17.44 (2.6×)	14.64 (3.13×)
As we analyzed, LoRA mainly reduces the memory of optimizer states. Thus, although it has negligible accuracy drop, it can only achieve  $\sim 1.3\times$  peak memory saving. LST can reduce memory usage up to  $3\times$ , but its accuracy drop is much larger than LoRA and WTA-CRS. Since WTA-CRS executes at the operation level and focuses on the activations, we further combine LoRA with WTA-CRS to reduce memory usage more aggressively. When combing with LoRA, WTA-CRS achieves  $2.7\times$  memory usage saving with almost no accuracy drop. To fully tune T5-3B, it requires 66.4GB of memory for in a mini-batch size of 32, which relies on A100 GPUs with a capacity of 80GB. On the other hand, LoRA+WTA-CRS only requires 21.6GB of memory for finetuning with a mini-batch size of 32, which can run on a GPU with 24GB memory, e.g. RTX3090Ti or A5000. We have experimentally confirmed this
conclusion. Under the same hardware, WTA-CRS enables the tuning of larger models, resulting in improved down-streaming task performance. Thus as shown in Figure 1,  $②$  under the similar memory budget, WTA-CRS outperforms the other methods in terms of the accuracy.
Also, according to Figure 6, for T5-3B, LoRA itself can enable  $1.9 \times$  larger batch size. When combined with LoRA, WTA-CRS enables  $4.8 \times (k = 0.3|\mathcal{D}|)$  to  $6.4 \times (k = 0.1|\mathcal{D}|)$  larger batch-size.
Influence of Row-column Pairs Budget (RQ2). As we analyzed in Section 3.2, WTA-CRS only have one hyperparameter, i.e., the total column-row pair budgets  $k$ . We conduct the ablation study with different budget  $k$  in Figure 7. We observe that ④ It has almost no accuracy drop when  $k = 0.3|\mathcal{D}|$ . And the accuracy drop is about  $1\%$  when  $k = 0.1|\mathcal{D}|$ . Notably, for T5-3B, the accuracy drop is only  $0.4\%$  when  $k = 0.1|\mathcal{D}|$ , which is much smaller than T5-Base and T5-Large. This suggests that larger models are more compressible because they have more redundant activations, which is consistent with previous observations [47].
5.3 Ablation Study (RQ3 and RQ4)
To answer RQ3, WTA-CRS is compared with two compositional methods to demonstrate its superiority. Namely, (1) the Deterministic method selects row-column pairs with top  $k$  probability of Equation (3). We note that this is the estimator proposed in [21]. (2) The CRS method follows Equation (3) to
Fig. 6. Peak memory usage versus the maximum batch size of T5-3B. More similar results are shown in Appendix E.2.
(a) T5-Base
(b) T5-Large
Fig. 7. Average validation results on GLUE dataset of wTA-CRS with varying budgets.
(c) T5-3B
sample the row-column pairs. All methods are deployed to GEMM in the backward pass, while leaving the forward one unchanged. The experiments are conducted on the training of T5-base language model on the SST2, MNLI, and QQP datasets; The column-row pair budget takes  $k / |\mathcal{D}| = 0.1$  for all methods. The validation accuracy versus training epoch is given in Figure 12. We observe:
WTA-CRS outperforms all compared methods, especially as the training epoch grows. The deterministic selection of top  $k$  column-row pairs suffers from accumulation of bias error that ultimately results in a failure of convergence. For CRS, it also enables the unbiased weight gradient. However, as we theoretically and experimentally analyzed in Theorem 7 and Figure 3, it is worse than WTA-CRS due to larger variance. In summary, both the deterministic and stochastic parts contribute to the effectiveness of WTA-CRS, which is consistent with our theoretical analysis.
The speed of WTA-CRS (RQ4). The configuration of computational infrastructure is given in Appendix F.1. We note that WTA-CRS does not add any extra parameters to the model. Thus, WTA-CRS only affects the fine-tuning speed, without affecting the inference speed. Below we analyze how the fine-tuning speed affected by WTA-CRS. As we analyzed in Appendix A limitation, the current implementation is not heavily optimized and thus the execution time of WTA-CRS is still slower than the original linear operation (details are shown in Appendix E.2). However, under the same
(a) T5-Large
Fig. 8. Batch size versus training throughput (sentences/sec) with different methods, where the sequential length is 128. The hardware is one single NVIDIA-A100 (80GB).
(b) T5-3B
hardware, a reduction in activation memory enables the use of larger batch sizes, thereby improving training speed due to increased GPU utilization [48, 9]. As we analyzed in Figure 6, WTA-CRS can enlarge the available batch size by up to  $4.8 \times$  larger. This enhancement is expected to result in a acceleration of the training speed. To illustrate, Figure 8 presents a visualization of batch size against training throughput (sentences per second) for both T5-Large and T5-3B models. We observe that WTA-CRS enables faster training speed under the same hardware. Specifically, on the T5-Large model, WTA-CRS@0.1 shows  $1.08 \times$  higher training throughput; and on the T5-3B model, WTA-CRS @0.3 and WTA-CRS@0.1 achieve  $1.14 \times$  and  $1.21 \times$  higher training throughput, respectively.
The authors thank the anonymous reviewers for their helpful comments. The work is in part supported by NSF grants NSF IIS-1849085, IIS-2224843, and NSF Awards 2117439 and 2112606. This work made use of the High Performance Computing Resource in the Core Facility for Advanced Research Computing at Case Western Reserve University.
In this paper, we propose wTA-CRS, a new unbiased estimator for matrix production with reduced variance. We theoretically and experimentally show when and why the estimator is better than the traditional unbiased estimator in terms of the variance. In the context of adapting transformers, it almost has no accuracy drop while reducing the peak memory usage by up to  $2.7 \times$ , and it enables a  $6.4 \times$  larger batch size, which in return resulting in  $1.2 \times$  higher training throughput.
[49] Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, and Christopher Re. MONGOOSE: A learnable LSH framework for efficient neural network training. In International Conference on Learning Representations (ICLR), 2021.
[50] Zirui Liu, Kaixiong Zhou, Fan Yang, Li Li, Rui Chen, and Xia Hu. EXACT: Scalable graph neural networks training via extreme activation compression. In International Conference on Learning Representations, 2022.
[51] Guanchu Wang, Zirui Liu, Zhimeng Jiang, Ninghao Liu, Na Zou, and Xia Hu. Towards memory efficient training via dual activation precision. arXiv preprint arXiv:2208.04187, 2022.
[52] Guanchu Wang, Zaid Pervaiz Bhat, Zhimeng Jiang, Yi-Wei Chen, Daochen Zha, Alfredo Costilla Reyes, Afshin Niktash, Gorkem Ulkar, Erman Okman, Xuanting Cai, et al. Bed: A real-time object detection system for edge devices. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 4994–4998, 2022.
A Extended Related Work and Discussion
Parameter-Efficient Fine-tuning. Parameter-efficient tuning methods select a small subset of parameters or insert a few parameters to a pre-trained network. Then they only update the small subset of parameters, while keeping others fixed [8, 9, 10, 11, 12, 13, 14, 49]. For example, Adapters [14, 13] insert a small module into the transformer blocks and only update it. Similarly, prompt tuning [8] introduces a small vector that is concatenated with the input embeddings. BitFit [11] only tunes the bias term of the model. LoRA [12] injects trainable rank decomposition matrices into the transformer block. Although these methods are "parameter-efficient", they actually cannot reduce the memory usage of the model itself. This is because we still needs to build the computation graph for the whole model. Instead, the memory usage of optimizer states will be significantly reduced, which is in proportional to the number of trainable parameters [15].
Activation Quantization. The activation quantization methods focus on quantizing the activation into low numerical precision numbers, e.g., 8-bit integers [16, 30, 50, 51, 52]. Our work also compresses the activation, but in a different way. We emphasize that our work is orthogonal to activation quantization in the sense that our work essentially reduces the dimension of activation. This distinction allows our method to be readily combined with activation quantization techniques, offering the potential for even more aggressive compression.
Gradient Checkpointing. Gradient checkpointing helps decrease activation memory usage by saving only a selection of activations. However, it demands additional computation during the backward pass, as discarded activations must be recalculated [18, 17]. According to the report of Checkmate[17], it achieves "a 2.3x memory reduction when training a BERT model with Checkmate optimizations (at 1x extra overhead for rematerialization)".
Limitations Although WTA-CRS significantly reduces the computation of the backward pass in a hardware-friendly way i.e., dropping entire rows/columns in the tensor, the current implementation still hampers the execution time of linear operations. This is because the extra sampling process and data movement counteract the acceleration. However, we note that (1) the overhead can be greatly reduced with better implementation, e.g., using prefetch and operation-fusion technique [30]; (2) the existing implementation can still yield a large speedup when employing larger batch sizes (Figure 8).
Potential Negative Societal Impacts. Our research primarily focuses on reducing the memory requirement of fine-tuning Language Models (LMs). The carbon emissions produced by LM fine-tuning may pose environmental issues. Our next step is to further improve the efficiency of LM fine-tuning, particularly on hardware with lower energy consumption.
B Unbiasedness of Weight Gradient
This part we directly follow the proof of Theorem 1 in ActNN [16]. For completeness, we provide the proof sketch here that is short and easy to follow. Specifically, here we use ReLU as the activation function for illustration convenience. We note that the conclusion in this section holds for any non-linear activation function. Specifically, the forward pass of ReLU-Linear at the  $l^{\mathrm{th}}$  layer is
$$
\mathbf {Z} ^ {(l + 1)} = \mathbf {H} ^ {(l)} \mathbf {W} ^ {(l)},
$$
$$
\mathbf {H} ^ {(l + 1)} = \operatorname {R e L U} (\mathbf {Z} ^ {(l + 1)}),
$$
and the backward pass of ReLU is:
$$
\begin{array}{l} \mathbb {E} \left[ \nabla \mathbf {Z} ^ {(l + 1)} \right] = \mathbb {E} \left[ \mathbb {1} _ {\mathbf {Z} ^ {(l + 1)} > 0} \odot \nabla \mathbf {H} ^ {(l + 1)} \right] \\ = \mathbb {1} _ {\mathbf {Z} ^ {(l + 1)} > 0} \odot \mathbb {E} [ \nabla \mathbf {H} ^ {(l + 1)} ], \\ \end{array}
$$
<sup>3</sup>https://github.com/parasj/checkmate/issues/153
where  $\odot$  is the element-wise product and  $\mathbb{1}$  is the indicator function. The element-wise product is linear operation and  $\mathbb{1}_{\mathbf{Z}^{(l + 1)} > 0}$  is only related to the pre-activation  $\mathbf{Z}^{(l + 1)}$  in the forward pass. We only apply the approximation during the backward pass so  $\mathbb{1}_{\mathbf{Z}^{(l + 1)} > 0}$  can be extracted from the expectation. We know that for the last layer  $L$ , we have  $\mathbb{E}[\nabla \mathbf{H}^{(L)}] = \mathbf{H}^{(L)}$  since we do not apply activation at the output layer. We then can prove by induction that  $\mathbb{E}[\nabla \mathbf{H}^{(l + 1)}] = \mathbf{H}^{(l + 1)}$  and  $\mathbb{E}[\nabla \mathbf{W}^{(l)}] = \mathbf{W}^{(l)}$  for any layer  $l$ .
C Proof
C.1 Derivation of Equation (3)
Let  $\mathbf{X} \in \mathbb{R}^{n \times m}$ ,  $\mathbf{Y} \in \mathbb{R}^{m \times q}$  be two matrices. The matrix multiplication  $\mathbf{XY}$  can be estimated as
$$
\mathbf {G E M M} (\mathbf {X}, \mathbf {Y}) = \sum_ {i = 1} ^ {m} \mathbf {X} _ {:, i} \mathbf {Y} _ {i,:} \approx \sum_ {t = 1} ^ {k} \frac {1}{k p _ {i _ {t}}} \mathbf {X} _ {:, i _ {t}} \mathbf {Y} _ {i _ {t,:}} = \mathbf {X} ^ {\prime} \mathbf {Y} ^ {\prime},
$$
Equation (3) shows the approximation error  $\mathbb{E}[||\mathbf{X}\mathbf{Y} - \mathbf{X}'\mathbf{Y}'||_F]$  is minimized when the probabilities
$$
p _ {i} = \frac {| | \mathbf {X} _ {: , i} | | _ {2} | | \mathbf {Y} _ {i , :} | | _ {2}}{\sum_ {j = 1} ^ {m} | | \mathbf {X} _ {: , j} | | _ {2} | | \mathbf {Y} _ {j , :} | | _ {2}}.
$$
Proof. Let  $f(i) = \frac{\mathbf{X}_{:i}\mathbf{Y}_{:i}}{p_i} \in \mathbb{R}^{n\times q}$ . We note that  $f(i)$  is an unbiased estimation of  $\mathbf{X}\mathbf{Y}$ . Namely,
$$
\mathbf {X} ^ {\prime} \mathbf {Y} ^ {\prime} = \frac {1}{k} \sum_ {t = 1} ^ {k} f \left(i _ {t}\right), \tag {8}
$$
where  $i_1, \dots, i_t$  are the index of the sampled column-row pairs at  $t^{\text{th}}$  random trials. For each  $i_t$ , its variance is
$$
\begin{array}{l} \operatorname {V a r} \left[ f \left(i _ {t}\right) \right] = \operatorname {V a r} \left[ \frac {\mathbf {X} _ {: i _ {t}} \mathbf {Y} _ {i _ {t}}}{p _ {i _ {t}}} \right] \\ = \mathbb {E} [ \frac {\mathbf {X} _ {: i _ {t}} ^ {2} \mathbf {Y} _ {i _ {t}:} ^ {2}}{p _ {i _ {t}} ^ {2}} ] - \mathbb {E} ^ {2} [ \frac {\mathbf {X} _ {: i _ {t}} \mathbf {Y} _ {i _ {t}:}}{p _ {i _ {t}}} ] \\ = \mathbb {E} [ \frac {\mathbf {X} _ {: i _ {t}} ^ {2} \mathbf {Y} _ {i _ {t}:} ^ {2}}{p _ {i _ {t}} ^ {2}} ] - (\mathbf {X Y}) ^ {2}. \\ = \sum_ {t = 1} ^ {m} \frac {\mathbf {X} _ {: t} ^ {2} \mathbf {Y} _ {t :} ^ {2}}{p _ {t}} - (\mathbf {X Y}) ^ {2}. \tag {9} \\ \end{array}
$$
where the first step follows from the fact that  $\operatorname{Var}[\mathbf{x}] = \mathbb{E}[\mathbf{x}^2] - \mathbb{E}^2[\mathbf{x}]$ .
$$
\begin{array}{l} \mathbb {E} [ | | \mathbf {X Y} - \mathbf {X} ^ {\prime} \mathbf {Y} ^ {\prime} | | _ {F} ] = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {q} \mathbb {E} [ (\mathbf {X Y} - \mathbf {X} ^ {\prime} \mathbf {Y} ^ {\prime}) _ {i j} ^ {2} ] \\ = \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {q} \operatorname {V a r} [ (\mathbf {X} ^ {\prime} \mathbf {Y} ^ {\prime}) _ {i j} ]. \\ \end{array}
$$
By combining Equation (8) and Equation (9) into the above equation, we have
$$
\begin{array}{l} \mathbb {E} [ | | \mathbf {X Y} - \mathbf {X} ^ {\prime} \mathbf {Y} ^ {\prime} | | _ {F} ] = \frac {1}{k} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {q} \sum_ {t = 1} ^ {m} \frac {\mathbf {X} _ {i t} ^ {2} \mathbf {Y} _ {t j} ^ {2}}{p _ {t}} - \frac {1}{k} \| \mathbf {X Y} \| _ {F} ^ {2}. \\ = \frac {1}{k} \sum_ {t = 1} ^ {m} \frac {\| \mathbf {X} _ {: , t} \| _ {2} ^ {2} \| \mathbf {Y} _ {t , :} \| _ {2} ^ {2}}{p _ {t}} - \frac {1}{k} \| \mathbf {X Y} \| _ {F} ^ {2}. \\ \end{array}
$$