SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3 Clustered Compositional Embeddings with  $c$  columns and  $2k$  rows
1: class CCE:  
2: method Initialize:  
3: for  $i = 1$  to  $c$  do:  
4:  $h_i, h_i' \gets \text{i.i.d.} \sim \text{random functions from } [d_1]$  to  $[k]$   
5:  $M_i, M_i' \gets \text{i.i.d.} \sim N(0,1)^{k \times d_2 / c}$   
6:  
7: method GetEmbedding(id):  
8: return CONCAT  $(M_i[h_i(id)] + M_i'[h_i'(id)]$  for  $i = 1$  to  $c$ )  
9:  
10: method Cluster(itemss):  
11: for  $i = 1$  to  $c$  do:  
12:  $T \gets [M_i[h_i(id)] + M_i'[h_i'(id)]$  for  $id \in [d_1]$   
13: centroids, assignments  $\leftarrow$  K-MEANS(T)  
14:  $h_i \gets \text{assignments}$   
15:  $M_i \gets \text{centroids}$   
16:  $h_i' \gets \text{random function from } [d_1]$  to  $[k]$   
17:  $M_i' \gets 0^{k \times d_2 / c}$
In line 12 we likely don't want to actually compute the embedding for every id in the vocabulary, but instead use mini batch K-Means with oracle access to the embedding table. In practice we follow the suggestion from FAISS K-means[Johnson et al., 2019] and just sample  $256k$  ids from  $[d_1]$  and run K-means only for this subset. The assignments from  $id$  to nearest cluster center are easy to compute for the full vocabulary after running K-means. The exact number of samples per cluster didn't have a big impact on the final performance of CCE.
We have shown the feasibility of compressing embedding tables at training time using clustering. Our method, CCE, outperforms the state of the art on the largest available recommendation data sets. While there is still work to be done in expanding our theoretical understanding and testing the method in more situations, we believe this is an exciting new paradigm for dynamic sparsity in neural networks and recommender systems in particular.
Previous studies have presented diverging views regarding the feasibility of compressing embedding tables. Our belief is that Figure 4a, Figure 4b, and Figure 4c shed light on these discrepancies. At standard learning rates and with one epoch of training, it's challenging to make significant improvements over the DLRM baseline, corroborating the findings of Naumov et al. [2019]. However, upon training until convergence, it's possible to achieve parity with the baseline using a thousand times fewer parameters than typically employed, even with the straightforward application of the hashing trick. Nevertheless, in the realm of practical recommendation systems, training to convergence isn't a common practice. Our experiment, as illustrated in Figure 4a, proposes a potential reason: an excessive size of embedding tables may lead to overfitting for most methods. This revelation is startling, given the prevailing belief that "bigger is always better" and that ideal scenarios should allow each concept to have its private embedding vectors. We contend that these experimental outcomes highlight the necessity for further research into overfitting within DLRM-style models.
In this paper we analyzed only the plain versions of each algorithm. There are a number of practical and theoretical improvements one may add. All methods are naturally compatible with float16 and float8 reduced or mixed precision. The averaging of multiple embeddings may even help smooth out some errors. It is also natural to consider pruning the vocabularies. In particular in an offline setting we may remove very rare values, or give them a smaller weight in the clustering an averaging. However, in an online setting this kind of pruning is harder to do, and it is easier to rely on hash collisions and SGD to ignore the unimportant values. In our experiments we used 27 embedding tables, one for each categorical feature. A natural compression idea is to map all features to the same embedding table (after making sure values don't collide between features.) We didn't experiment with this, but it potentially could reduce the need for tuning embedding table sizes separately. A later paper by Coleman et al. [2023] report good results with this method. For things we tried, but didn't work, see Appendix A.
We would like to thank Ben Ghaemmaghami from the University of Texas at Austin for fruitful discussions of learned hash functions. Thanks to Chunyin Siu for helpful discussions on the convergence of multi-step CCE. And finally to Michael Shi, Peter Tang, Summer Deng, Christina You, Erik Meijer, and the rest of the Probability group at Meta for helpful discussions of table embeddings and earlier versions of this manuscript.
Hao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, and Jiyan Yang. Compositional embeddings using complementary partitions for memory-efficient recommendation systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 165-175, 2020.
Dan Tito Svenstrup, Jonas Hansen, and Ole Winther. Hash embeddings for efficient word representations. Advances in neural information processing systems, 30, 2017.
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th annual international conference on machine learning, pages 1113-1120, 2009.
David P. Woodruff. Sketching as a tool for numerical linear algebra. Found. Trends Theor. Comput. Sci., 10(1-2):1-157, 2014. doi: 10.1561/0400000060. URL https://doi.org/10.1561/0400000060.
Donna Xu, Ivor W Tsang, and Ying Zhang. Online product quantization. IEEE Transactions on Knowledge and Data Engineering, 30(11):2185-2198, 2018.
Chunxing Yin, Bilge Acun, Xing Liu, and Carole-Jean Wu. Tt-rec: Tensor train compression for deep learning recommendation models. CoRR, abs/2101.11714, 2021. URL https://arxiv.org/abs/2101.11714.
The backbone recommendation model, DLRM by Naumov et al. [2019], has an open-source PyTorch implementation available on Github which includes an implementation of CE. For CCE you need a fast library for K-means. We recommend the open-sourced implementation by Johnson et al. [2019] for better performance, but you can also use the implementation in Scikit-learn [Pedregosa et al., 2011]. The baseline result should be straightforward to reproduce as we closely follow the instructions provided by Naumov et al. [2019]. For the CE methods, we only need to change two functions in the code: create_emb and apply_emb. We suggest using a class for each CE method; see Figure 3. For the random hash function, one could use a universal hash function or numpy.random.randint.
Measuring the Embedding Compression factor The most important number from our experimental section is the "Embedding Compression" factor in Table 1. We measure this by training the model with different caps on of parameters in the embedding tables (See e.g. the x-axis in Figure 4a. E.g. if the Criteo Kaggle dataset has categorical features with vocabularies of sizes 10, 100 and  $10^{6}$ , we try e.g. to cap this at 8000, using a full embedding table for the small features, and a CCE table with  $8000 / 16 = 500$  rows (since each row has 16 parameters). This corresponds to a compression rate of  $(10 + 100 + 10^{6}) / (10 + 100 + 500) \approx 1639.5$ . Or if we measure only the compression of the largest table,  $10^{6} / 500 = 2000$ . Unfortunately there's a discrepancy in the article, which we only found after the main deadline, that uses the second measure in the introduction (hence the number 11,000x compression) where as Figure 4a uses the first measure (and thus the lower number  $8,5000x$ ).
For the experiments where we only train for 1 epoch, some methods never reach baseline BCE within the number of parameters we test. Hence the Compression Rates we report are based on extrapolations. For each algorithm we report a range, e.g. 127-155x for CE with Concatenation on Criteo Kaggle 1 epoch. Since the loss graphs tend to be convex, the upper bound (155x) is based on a linear interpolation (being optimistic about when the method will hit baseline BCE) and the lower bound (127x) is based on a quadratic interpolation, which only intersects the baseline at a higher parameter count.
K-means For the K-means from FAISS, we use max_points_per_centroid=256 and niter=50. The first parameter sub-samples the number of points to 256 times the number of cluster centroids  $(k)$ , and is the recommended rate after which "no benefit is expected" according to the library maintainers. In practice we predict the right value will depend on the dimensionality of your data, so using the split into lower dimensional columns is beneficial. For niter we initially tried a larger value (300), but found it didn't improve the final test loss. We found on Kaggle for PQ, niter=50, BCE=0.455540 and niter=300, BCE=0.455537. For CCE (single epoch, single clustering at half an epoch), niter=50 gave BCE=0.45928 and niter=300 gave BCE=0.45905, so a very slight improvement, but not enough to make up for the extra training time.
Datasets For our experiments, we sub-sampled an eighth of the Terabyte dataset and pre-hashed them using a simple modulus to match the categorical feature limit of the Kaggle dataset. For both Kaggle and Terabyte dataset, we partitioned the data from the final day into validation and test sets. Using the benchmarking setting of the DLRM code, the Kaggle dataset has around 300,000 batches while the Terabyte dataset has around 2,000,000 batches.
Early stopping Early stopping is used when running the best of 10 epochs on the Kaggle dataset. We measure the performance of the model in BCE every 50,000 batches (around one-sixth of one epoch) using the validation set. If the minimum BCE of the previous epoch is less than the minimum BCE of the current epoch, we early stop.
Deep Hash Embeddings We follow Kang et al. [2021] in using a fixed-width MLP with Mish activation. However, DHE is only described in one version in the paper: 5 layers of 1024 nodes per layer. For our experiments, we need to initialize DHE with different parameter budgets. We found that in general, DHE performs better with fewer layers when the number of parameters is fixed. However, we cannot use just a single layer, since that would be a linear embedding table, not an MLP. As a compromise, we fix the number of hidden layers to 2 and set the number of hashes to be the same as the dimension of the hidden layers.
For example, if we were allowed to use 64,000 parameters with an embedding dimension of 64, then by solving a quadratic equation we get that the number of hashes and the dimension of the hidden layers are both 136. This gives us
n_hashes  $\cdot$  hidden_dim  $+2*$  hidden_dim  $^2 +$  hidden_dim  $\cdot$  embedding_dim  $= 64192$  parameters.
A What didn't work
Here are the ideas we tried but didn't work at the end.
Using multiple helper tables It is a natural idea use more than one helper table. However, in our experiments, the effect of having more helper tables is not apparent.
Circular clustering Based on the CE concat method, the circular clustering method would use information from other columns to do clustering. However, the resulting index pointer functions are too similar to each other, meaning that this method is essentially the hashing trick. We further discuss this issue in Appendix H.
Continuous clustering We originally envisioned our methods in a tight loop between training and (re)clustering. It turned out that reducing the number of clusterings didn't impact performance, so we eventually reduced it all the way down to just one. In practical applications, with distribution shift over time, doing more clusterings may still be useful, as we discuss in Section 3.
Changing the number of columns In general, increasing the number of columns leads to better results. However the marginal benefits quickly decrease, and as the number of hash functions grow, so does the training and inference time. We found that 4 columns / hash-functions was a good spot.
Residual vector quantization The CCE method combines Product Quantization (PQ) with the CE concat method. We tried combining Residual vector quantization (RVQ) with the Hash Embeddings method from Tito Svenstrup et al. [2017]. This method does not perform significantly better than the Hash Embeddings method.
Seeding with PQ We first train a full embedding table for one epoch, and then do Product Quantization (PQ) on the table to obtain the index pointer functions.
We then use the index pointer functions instead of random hash functions in the CE concat method. This method turned out performing badly: The training loss quickly diverges from the test loss after training on just a few batches of data.
Here are some variations of the CCE method:
Earlier clustering We currently have two versions of the CCE method: CCE half, where clustering happens at the middle of the first epoch, and CCE, where clustering happens at the end of the first epoch. We observe that when we cluster earlier, the result is slightly worse. Though in our case the CCE half method still outperforms the CE concat method.
More parameters before clustering The CCE method allows using two number of parameters, one in Step 1 where we follow the CE hybrid method to get a sketch, and one in Step 3 where we follow the CE concat method. We thought that by using more parameters at the beginning, we would be able to get a better set of index pointer tables. However, the experiment suggested that the training is faster but the terminal performance is not significantly better.
Smarter initialization after clustering In Algorithm 3 we initialize  $M_{i}$  with the cluster centroids from K-means and the "helper table"  $M_{i}^{\prime} \gets 0$ . We could instead try to optimize  $M_{i}^{\prime}$  to match the residuals of  $T$  as well as possible. This could reduce the discontinuity during training more than initializing to zeros. However, we didn't see a large effect in either training loss smoothness or the ultimate test score.
B Proof of the main theorem
$$
\left[ \begin{array}{c c} 0. 4 1 7 & 0. 7 2 0 \\ 0. 0 0 0 & 0. 3 0 2 \\ 0. 1 4 7 & 0. 0 9 2 \\ 0. 1 8 6 & 0. 3 4 6 \\ 0. 3 9 7 & 0. 5 3 9 \\ 0. 4 1 9 & 0. 6 8 5 \\ 0. 2 0 4 & 0. 8 7 8 \end{array} \right] \approx \left[ \begin{array}{c c c c} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \end{array} \right] \left[ \begin{array}{c c} 0. 4 1 1 & 0. 6 4 8 \\ 0. 0 9 3 & 0. 3 2 4 \\ 0. 2 0 4 & 0. 8 7 8 \\ 0. 1 4 7 & 0. 0 9 2 \end{array} \right]
$$
Figure 5: K-means as matrix factorization. A central part of the analysis of CCE is the simple observation that K-means factors a matrix into a tall sparse matrix and a small dense one. In other words, it finds a sparse approximation the column space of the matrix.
Let's remind ourselves of the "Dense CCE algorithm" from Section 3: Given  $X \in \mathbb{R}^{n \times d_1}$  and  $Y \in \mathbb{R}^{n \times d_2}$ , pick  $k$  such that  $n > d_1 > k > d_2$ . We want to solve find a matrix  $T^*$  of size  $d_1 \times d_2$  such that  $\| XT^* - Y \|_F$  is minimized – the classical Least Squares problem. However, we want to use memory less than the typical  $nd_1^2$ . We thus use this algorithm:
Dense CCE Algorithm: Let  $T_0 = 0 \in \mathbb{R}^{d_1 \times d_2}$ . For  $i = 1$  to  $m$ :
Sample  $G_{i}\sim N(0,1)^{d_{1}\times (k - d_{2})};$
Compute  $H_{i} = [T_{i - 1}\mid G_{i}]\in \mathbb{R}^{d_{1}\times k}$
$$
M_{i} = \operatorname *{arg  inf}_{M}\| XH_{i}M - Y\|_{F}^{2}\in \mathbb{R}^{k\times d_{2}}.
$$
$$
T _ {i} = H _ {i} M _ {i}
$$
We will now argue that  $T_{m}$  is a good approximation to  $T^{*}$  in the sense that  $\| X T_{m} - Y\|_{F}^{2}$  is not much bigger than  $\| X T^{*} - Y\|_{F}^{2}$ .
Let's consider a non-optimal choice of  $M_{i}$  first. Suppose we set  $M_{i} = \left[ \begin{array}{c} I_{d_{2}} \\ M_{i}' \end{array} \right]$  where  $M_{i}'$  is chosen such that  $\| H_{i}M_{i} - T^{*}\|_{F}$  is minimized. By direct multiplication, we have  $H_{i}M_{i} = T_{i-1} + G_{i}M_{i}'$ . Hence in this case minimizing  $\| H_{i}M_{i} - T^{*}\|_{F}$  is equivalent to finding  $M_{i}'$  at each step such that  $\| G_{i}M_{i}' - (T^{*} - T_{i-1})\|_{F}$  is minimized.
In other words, we are trying to estimate  $T^{*}$  with  $\sum_{i} G_{i} M_{i}'$ , where each  $G_{i}$  is random and each  $M_{i}'$  is greedily chosen at each step. This is similar to, for example, the approaches in Barron et al. [2008], though they use a concrete list of  $G_{i}$ 's. In their case, by the time we have  $d_{1}/k$  such  $G_{i}$ 's, we are just multiplying  $X$  with a  $d_{1} \times d_{1}$  random Gaussian matrix, which of course will have full rank, and so the concatenated  $M$  matrix can basically ignore it. However, in our case we do a local, not global optimization over the  $M_{i}$ .
Recall the theorem:
Theorem B.0. Given  $X \in \mathbb{R}^{n \times d_1}$  and  $Y \in \mathbb{R}^{n \times d_2}$ . Let  $T^* = \arg \min_{T \in \mathbb{R}^{d_1 \times d_2}} \| XT - Y \|_F^2$  be an optimal solution to the least squares problem. Then
$$
\mathrm {E} \left[ \| X T _ {i} - Y \| _ {F} ^ {2} \right] \leq (1 - \rho) ^ {i (k - d _ {2})} \| X T ^ {*} \| _ {F} ^ {2} + \| X T ^ {*} - Y \| _ {F} ^ {2},
$$
where  $\rho = \| X\|_{-2}^{2} / \| X\|_{F}^{2}$
Here we use the notation that  $\| X\|_{-2}$  is the smallest singular value of  $X$
Corollary B.1. In the setting of the theorem, if all singular values of  $X$  are equal, then
$$
\operatorname {E} \left[ \| X T _ {i} - Y \| _ {F} ^ {2} \right] \leq e ^ {- i \frac {k - d _ {2}}{d _ {1}}} \| X T ^ {*} \| _ {F} ^ {2} + \| X T ^ {*} - Y \| _ {F} ^ {2}.
$$
Proof of Theorem B.1. Note that  $\| X\| _F^2$  is the sum of the  $d_{1}$  singular values squared:  $\| X\| _F^2 = \sum_i\sigma_i^2$ . Since all singular values are equal, say to  $\sigma \in \mathbb{R}$ , then  $\| X\| _F^2 = d_1\sigma^2$ . Similarly in this setting,  $\| X\|_{-2}^{2} = \sigma^{2}$  so  $\rho = 1 / d_{1}$ . Using the inequality  $1 - 1 / d_{1}\leq e^{-1 / d_{1}}$  gives the corollary.
Proof of Theorem B.0. First split  $Y$  into the part that's in the column space of  $X$  and the part that's not,  $Z$ . We have  $Y = XT^{*} + Z$ , where  $T^{*} = \arg \min_{T}\|XT - Y\|_{F}$  is the solution to the least
squares problem. By Pythagoras theorem we then have
$$
\operatorname {E} \left[ \| X T _ {i} - Y \| _ {F} ^ {2} \right] = \operatorname {E} \left[ \| X T _ {i} - (X T ^ {*} + Z) \| _ {F} ^ {2} \right] = \operatorname {E} \left[ \| X (T _ {i} - T ^ {*}) \| _ {F} ^ {2} \right] + \| Z \| _ {F} ^ {2},
$$
so it suffices to show
$$
\operatorname {E} \left[ \| X (T _ {i} - T ^ {*}) \| _ {F} ^ {2} \right] \leq (1 - \rho) ^ {i (k - d _ {2})} \| X T ^ {*} \| _ {F} ^ {2}.
$$
We will prove the theorem by induction over  $i$ . In the case  $i = 0$  we have  $T_{i} = 0$ , so  $\operatorname{E}[\| X(T_0 - T^*)\|_F^2] = \operatorname{E}[\| XT^*\|_F^2]$  trivially. For  $i \geq 1$  we insert  $T_{i} = H_{i}M_{i}$  and optimize over  $M_{i}'$ :
$$
\begin{array}{l} \operatorname {E} [ \| X (T _ {i} - T ^ {*}) \| _ {F} ^ {2} ] = \operatorname {E} [ \| X (H _ {i} M _ {i} - T ^ {*}) \| _ {F} ^ {2} ] \\ \leq \mathrm {E} [ \| X \left(H _ {i} [ I \mid M _ {i} ^ {\prime} ] - T ^ {*}\right) \| _ {F} ^ {2} ] \\ = \mathrm {E} [ \| X ((T _ {i - 1} + G _ {i} M _ {i} ^ {\prime}) - T ^ {*}) \| _ {F} ^ {2} ] \\ = \operatorname {E} [ \| X \left(G _ {i} M _ {i} ^ {\prime} - \left(T ^ {*} - T _ {i - 1}\right)\right) \| _ {F} ^ {2} ]. \\ = \operatorname {E} \left[ \left\| X \left(G _ {i} M _ {i} ^ {\prime} - \left(T ^ {*} - T _ {i - 1}\right)\right) \right\| _ {F} ^ {2} \mid T _ {i - 1} \right] \\ \leq (1 - \rho) ^ {k - d _ {2}} \mathrm {E} [ \| X (T ^ {*} - T _ {i - 1}) \| _ {F} ^ {2} ] \\ \leq (1 - \rho) ^ {i (k - d _ {2})} \| X T ^ {*} \| _ {F} ^ {2}, \\ \end{array}
$$
where the last step followed by induction. The critical step here was bounding
$$
\operatorname {E} _ {G} [ \inf _ {M} \| X (G M - T) \| _ {F} ^ {2} ] \leq (1 - \rho) ^ {k - d _ {2}} \| X T \| _ {F} ^ {2},
$$
for a fixed  $T$ . We will do this in a series of lemmas below.
We show the lemma first in the "vector case", corresponding to  $k = 2, d_2 = 1$ . The general matrix case follows below, and is mostly a case of induction on the vector case.
Lemma B.2. Let  $X \in \mathbb{R}^{n \times d}$  be a matrix with singular values  $\sigma_1 \geq \dots \geq \sigma_d \geq 0$ . Define  $\rho = \sigma_d^2 / \sum_i \sigma_i^2$ , then for any  $t \in \mathbb{R}^d$ ,
$$
\mathrm {E} _ {g \sim N (0, 1) ^ {d}} \left[ \inf  _ {m \in \mathbb {R}} \| X (g m - t) \| _ {2} ^ {2} \right] \leq (1 - \rho) \| X t \| _ {2} ^ {2}.
$$
Proof. Setting  $m = \langle Xt, Xg \rangle / \| Xg \|_2^2$  we get
$$
\begin{array}{l} \| X (g m - t) \| _ {2} ^ {2} = m ^ {2} \| X g \| _ {2} ^ {2} + \| X t \| _ {2} ^ {2} - 2 m \langle X g, X t \rangle (1) \\ = \left(1 - \frac {\langle X t , X g \rangle^ {2}}{\| X t \| _ {2} ^ {2} \| X g \| _ {2} ^ {2}}\right) \| X t \| _ {2} ^ {2}. (2) \\ \end{array}
$$
We use the singular value decomposition of  $X = U\Sigma V^T$ . Since  $g \sim N(0,1)^d$  and  $V^T$  is unitary, we have  $V^T g \sim N(0,1)^d$  and hence we can assume  $V = I$ . Then
$$
\begin{array}{l} \frac {\langle X t , X g \rangle^ {2}}{\| X t \| _ {2} ^ {2} \| X g \| _ {2} ^ {2}} = \frac {(t ^ {T} \Sigma U ^ {T} U \Sigma g) ^ {2}}{\| U \Sigma t \| _ {2} ^ {2} \| U \Sigma g \| _ {2} ^ {2}} (3) \\ = \frac {\left(t ^ {T} \Sigma^ {2} g\right) ^ {2}}{\| \Sigma t \| _ {2} ^ {2} \| \Sigma g \| _ {2} ^ {2}} (4) \\ = \frac {\left(\sum_ {i} t _ {i} \sigma_ {i} ^ {2} g _ {i}\right) ^ {2}}{\left(\sum_ {i} \sigma_ {i} ^ {2} t _ {i} ^ {2}\right) \left(\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}\right)}, (5) \\ \end{array}
$$
where Equation (4) follows from  $U^T U = I$  in the SVD. We expand the upper sum to get
$$
\begin{array}{l} \mathrm {E} _ {g} \left[ \frac {\left(\sum_ {i} t _ {i} \sigma_ {i} ^ {2} g _ {i}\right) ^ {2}}{\left(\sum_ {i} \sigma_ {i} ^ {2} t _ {i} ^ {2}\right) \left(\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}\right)} \right] = \mathrm {E} _ {g} \left[ \frac {\sum_ {i , j} t _ {i} t _ {j} \sigma_ {i} ^ {2} \sigma_ {j} ^ {2} g _ {i} g _ {j}}{\left(\sum_ {i} \sigma_ {i} ^ {2} t _ {i} ^ {2}\right) \left(\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}\right)} \right] (6) \\ = \mathrm {E} _ {g} \left[ \frac {\sum_ {i} t _ {i} ^ {2} \sigma_ {i} ^ {4} g _ {i} ^ {2}}{(\sum_ {i} \sigma_ {i} ^ {2} t _ {i} ^ {2}) (\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2})} \right]. (7) \\ \end{array}
$$
Here we use the fact that the  $g_{i}$ 's are symmetric, so the cross terms of the sum have mean 0. By scaling, we can assume  $\sum_{i} \sigma_{i}^{2} t_{i}^{2} = 1$  and define  $p_{i} = \sigma_{i}^{2} t_{i}^{2}$ . Then the sum is just a convex combination:
$$
(7) = \sum_ {i} p _ {i} \mathrm {E} _ {g} \left[ \frac {\sigma_ {i} ^ {2} g _ {i} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}} \right]. \tag {8}
$$
Since  $\sigma_{i}\geq \sigma_{d}$  and  $g_{i}$  's are IID, by direct comparison we have
$$
\mathrm {E} _ {g} \left[ \frac {\sigma_ {i} ^ {2} g _ {i} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}} \right] \geq \mathrm {E} _ {g} \left[ \frac {\sigma_ {d} ^ {2} g _ {d} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}} \right]
$$
$$
(7) \geq \mathrm {E} _ {g} \left[ \frac {\sigma_ {d} ^ {2} g _ {d} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}} \right] \sum_ {i} p _ {i} = \mathrm {E} _ {g} \left[ \frac {\sigma_ {d} ^ {2} g _ {d} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}} \right].
$$
It remains to bound
$$
\mathrm {E} _ {g} \left[ \frac {\sigma_ {d} ^ {2} g _ {d} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2} g _ {i} ^ {2}} \right] \geq \frac {\sigma_ {d} ^ {2}}{\sum_ {i} \sigma_ {i} ^ {2}} = \rho , \tag {9}
$$
but this follows from a cute, but rather technical lemma, which we will postpone to the end of this section. (Theorem B.4.)
It is interesting to notice how the improvement we make each step (that is  $1 - \rho$ ) could be increased to  $1 - 1 / d$  by picking  $G$  from a distribution other than IID normal.
If  $X = U\Sigma V^T$ , we can also take  $g = V\Sigma^{-1}g'$ , where  $g' \sim N(0,1)^{d_1 \times (k - d_2)}$ . In that case we get