SlowGuess/NAACL_2022_OCR huggingface dataset:
Yoonseok Yang*, Kyu Seok Kim*, Minsam Kim*, Juneyoung Park†
Riiid AI Research
{yoonseok.yang, kyuseok.kim, minsam.kim, juneyoung.park}
@riiid.co
Content-based collaborative filtering (CCF) predicts user-item interactions based on both users' interaction history and items' content information. Recently, pre-trained language models (PLM) have been used to extract high-quality item encodings for CCF. However, it is resource-intensive to train a PLM-based CCF model in an end-to-end (E2E) manner, since optimization involves back-propagating through every content encoding within a given user interaction sequence. To tackle this issue, we propose GRAM (GRadient Accumulation for Multi-modality in CCF), which exploits the fact that a given item often appears multiple times within a batch of interaction histories. Specifically, Single-step GRAM aggregates each item encoding's gradients for back-propagation, with theoretic equivalence to the standard E2E training. As an extension of Single-step GRAM, we propose Multi-step GRAM, which increases the gradient update latency, achieving a further speedup with drastically less GPU memory. GRAM significantly improves training efficiency (up to  $146\times$ ) on five datasets from two task domains of Knowledge Tracing and News Recommendation. Our code is available at https://github.com/yoonseok312/GRAM.
Collaborative filtering (CF) is a popular technique used for mining relationships between items and users. Recently, CF has been successfully applied to various tasks including Knowledge Tracing and Recommender Systems (Smith and Linden, 2017; Bennett et al., 2007; Melville et al., 2002). However, conventional CF only considers item-user interactions, and disregards any item-/user-specific information. This leads to the so-called cold-start problem (Lam et al., 2008), where the CF model
fails to make predictions for unseen users/items, even when they resemble observed users/items.
To remedy this issue, Content-based Collaborative Filtering (CCF) incorporates items' content information into the item encoding. This not only addresses the cold-start problem, but also leads to significant performance improvement (Wu et al., 2019, 2021a; Lu et al., 2018). Specifically, large pre-trained language models (PLM) (Devlin et al., 2019; Brown et al., 2020) have shown great potential for extracting items' content information. However, fine-tuning PLMs for CCF requires prohibitive computational resources in terms of (1) training time and (2) GPU memory footprint.
This issue arises due to CCF's multi-modal nature, where item representations are learned from both tabular user records and their textual information. As a given item appears multiple times within a batch of users' records, its textual encoding needs to be computed every time it appears within the batch. Moreover, the need to store intermediate activations for hundreds of millions of parameters in PLM requires high GPU memory footprint (Wang et al., 2020).
To that end, we propose Single-step GRAM, GRadient Accumulation for Multi-modality in CCF, which alternately trains the task specific module and content encoder module. Accumulating gradients for repeated items in a training step, Single-step GRAM provides 4 times faster training while being theoretically equivalent to standard E2E training.
As a natural extension of Single-step GRAM, we also propose Multi-step GRAM which accumulates gradients across multiple training steps. Multi-step GRAM (1) provides an even higher ratio of acceleration without significant performance loss and (2) consumes less than  $40\%$  GPU memory of E2E. Results show that the computational speed can increase up to  $146\times$ $(45\times$  on avg.) with less than  $1\%$  drop in AUC.
This author is the corresponding author.
Human Language Technologies, pages 839 - 851
- We present GRAM (Single-step & Multi-step) which accelerate training of CCF by accumulating the gradients of redundant item encodings.
- We empirically show that GRAM, especially Multi-step GRAM, noticeably reduces GPU memory footprint compared to E2E training.
- We evaluate GRAM in a variety of settings on 5 real-world datasets in two task domains, News Recommendation and Knowledge Tracing.
2.1 Collaborative Filtering and Content-based Filtering
Collaborative filtering (CF) (Su and Khoshgoftaar, 2009) attempts to predict user-item interaction based on past history. CF alone disregards any user-specific or item-specific information, leading to the inability to extract useful features from user information or item content. This especially leads to low performance on cold-start users and items. Various content-based filtering methods (Van Meteren and Van Someren, 2000; Basilico and Hofmann, 2004) have been proposed to mitigate these issues through labeled meta-data. It uses raw textual features of item, instead of requiring other users' data during a user's recommendation like CF. However, these approaches require manual labelling and lack the extensive usage of the content itself.
2.2 Content-based Collaborative Filtering
Content-based Collaborative Filtering (CCF) incorporates content into CF in order to unify the strengths of CF and content-based filtering. It consists of two major components: content-encoder (CE) and collaborative filter (CF) components connected in an end-to-end fashion. Major task domains where items' raw textual content may significantly aid CF include:
News Recommendation (NR), a task of predicting whether a user will click an article among others, provided with the user's past interactions.
Knowledge Tracing (KT), a task of predicting whether a user will correctly responds to a question or not based on the user's past responses.
In this section, we briefly review the widely-used approaches to utilize content information in CCF, with a focus on these two domains.
2.2.1 Training Content-Encoder (CE) in CCF
Existing works mainly train or fine-tune CE module in an E2E fashion to obtain a useful content representations for the given CF task. In NR, NRMS (Wu et al., 2019) applied Glove (Pennington et al., 2014) word representation and Multi-head Self-attention (MHSA) (Vaswani et al., 2017) to encode the article's text to the item representation. Similarly in KT, EERNN (Su et al., 2018) used BiLSTM (Huang et al., 2015) to process Word2Vec (Mikolov et al., 2013) representations of question text into question representation. The representation is then fed into another LSTM layer to make final prediction on the user response.
While aforementioned methods directly use content-encoder's output as item vectors, there has also been work to use CE's output to regularize item vectors. Topical Attention Regularized Matrix Factorization (TARMF) (Lu et al., 2018) uses Matrix Factorization as CF and attention-based GRU network as CE to incorporate review data in predicting user-item ratings. Alternatively training CF and CE module, it uses the CE output to regularize the item representations in CF.
Most recently, researchers started to fine-tune large Pre-trained Language Models (PLMs) with generic language understanding as a CE module for better content representation. In NR, NRMS-PLM (Wu et al., 2021a) fine-tunes BERT (Devlin et al., 2019) in an E2E manner, achieving meaningful performance gain.
2.3 Efficient Fine-tuning of Large PLMs
While PLMs show powerful performance as a content-encoder in CCF, fine-tuning PLMs is known to be inefficient (Houlsby et al., 2019) as it includes updating billions of parameters. Although it is possible to use PLM's output as fixed features for downstream tasks, numerous studies (Devlin et al., 2019; Reimers et al., 2019) emphasize such feature-based approach cannot match the performance of E2E fine-tuning. Thus, researchers have considered fine-tuning a subset of the PLM architecture (Devlin et al., 2019) and adding task-specific parameters (Houlsby et al., 2019) to reduce cost and performance degradation.
However, the computational complexity deteriorates even more under multi-modal settings like CCF. In such cases, PLM is called numerous times for a single-user, adding a new dimension of computational load, making E2E training almost impos
sible. SpeedyFeed (Xiao et al., 2021) was proposed to accelerate the fine-tuning of PLM for news recommendation through combination of several methods. However, they are mainly engineering heavy implementations with domain dependencies. For a general training scheme in CCF, we propose a novel method that can be applied in an orthogonal manner to aforementioned techniques like (Houlsby et al., 2019) and (Xiao et al., 2021), while achieving remarkable speed boost in such multi-modal CCF settings.
Notation	Description
Iu	Interaction sequence of user u
ci	Content of item i
Inu = (enu, ru)	n-th interaction of user u
enu = CE(cu)	Embedding of content cn from CE
B(t)	Mini-batch B at update time-step t
[B1, Bu]	Set of unique [items, users] in B
[lt, lI]	Length of [tokens, interactions] in [c, I]
d	Content embedding dimension
L	CCF minimization objective
In this section, we formally setup CCF framework and its notations for efficient discussion. CCF framework consists of two major components: content-encoder (CE) and collaborative filter (CF) components connected in an end-to-end fashion.
3.1 CF component
CF component predicts user  $u$  's response  $r$  to an arbitrary item based on the user's past interactions  $I^{u} = (I_{1}^{u},I_{2}^{u},\dots,I_{|I^{u}|}^{u})$  where each  $n$  -th interaction  $I_n^u = (e_n^u,r_n^u)$  is represented as a tuple of item representation  $e_n^u$  and the user's response to the item  $r_n^u$  . In other words, the CF module aims to estimate the probability:
$$
C F (I ^ {u}; e _ {[ I ^ {u} | + 1} ^ {u}) = P (r _ {[ I ^ {u} | + 1} ^ {u} \mid I _ {1} ^ {u}, I _ {2} ^ {u}, \dots , I _ {[ I ^ {u} ]} ^ {u}; e _ {[ I ^ {u} | + 1} ^ {u})
$$
3.2 CE component
CE component outputs the item representation  $e$ :  $e_n^u = CE(c_n^u)$  where  $c_{n}^{u}$  is the token sequence of the corresponding item. The model parameters  $\theta_{CE}$  and  $\theta_{CF}$  of the CE-CF pipeline is then trained in an end-to-end fashion based on cross-entropy loss for response prediction. The summary of notation used for CCF is provided in Table 1. The existing approaches to tackle CCF are formatted and presented in Table 2, along with the pipeline we adopt for our later experiments.
Table 1: Notations for Content-based CF(CCF)
Task	Model	CE	CF
NR	NRMS	Glove, MHSA	MHSA
NRMS-PLM	BERT	MHSA
Our Experiments	BERT	MHSA
KT	EERNN	W2V, BiLSTM	LSTM
Our Experiments	BERT	LSTM
Table 2: CE-CF Pipeline Choice for CCF
3.3 Inefficiency of E2E in CCF
In CCF, end-to-end fine-tuning of the CE (PLM) suffers from cubic computational complexity in terms of sequence length, due to the data multimodality. Let's assume average text token length of  $l_{t}$  and average interaction record length of  $l_{I}$  with each mini-batch  $B$  containing users  $B_{u}$ .
Attention-based CE module would be called  $l_{I}$  times, producing forward/backward-pass computational complexity of  $O(|B_u| \cdot l_I(l_t^2 d + l_t d^2))$  where  $d$  represents embedding dimension.
Under similar average sequence length of per-item tokens and per-user interactions  $l_{t} \approx l_{I}$ , the resulting cubic complexity in terms of sequence length significantly increases space and time complexity of model training and becomes the limiting bottleneck factor.
For efficient training, we propose GRadient Accumulation for Multi-modality in CCF (GRAM) with two variants: Single-step GRAM and Multi-step GRAM.
4.1 Single-step GRAM
Single-step GRAM trains CE module and CF module separately. To update the CE module, Single-step GRAM accumulates gradients of redundant items, effectively reducing the number of gradient computation for each step. As shown in Figure 1, Single-step GRAM can accelerate training by limiting the number of gradient calculation to the number of unique items in the batch. With this, Single-step GRAM can reduce the algorithmic complexity of CE to  $O(|B_I| \cdot (l_t d^2 + l_t^2 d))$  per mini-batch update, from E2E's  $O(|B_u| \cdot l_I (l_t^2 d + l_t d^2))$ .
Proposition. Given parameters  $(\theta_f, \theta_g)$ , suppose a neural network in separable form with  $y = g(h; \theta_g), h = f(x; \theta_f)$ , and loss  $L(\theta_f, \theta_g)$ . Define pseudo-target  $h$  as
$$
\tilde {h} := f (x; \theta_ {f}) - \left. \frac {\partial L}{\partial h} \right| _ {h = f (x; \theta_ {f})}, \tag {1}
$$
Figure 1: Comparison to E2E Training: Let's assume a mini-batch  $B$  of 12 interactions consisting of 3 users and 5 unique items. For the content encoder, E2E computes 12 gradients from each interaction, while GRAM computes 5 only, based on accumulated gradient signal in pseudo-target  $\tilde{h}_i$  from each item  $i$ .
and pseudo-loss  $\tilde{L} (\theta_f)$  for another network  $f(x;\theta_f^{\prime})$  as
$$
\tilde {L} \left(\theta_ {f}\right) := \frac {1}{2} \left(\tilde {h} - f \left(x; \theta_ {f} ^ {\prime}\right)\right) ^ {2}. \tag {2}
$$
Then, the gradient of  $\theta_{f}$  w.r.t  $L$  is equivalent to the gradient of  $\theta_f^\prime$  w.r.t  $\tilde{L}$
$$
i. e. \quad \left. \frac {\partial L}{\partial \theta_ {f}} = \frac {\partial \tilde {L}}{\partial \theta_ {f} ^ {\prime}} \right| _ {\theta_ {f} ^ {\prime} = \theta_ {f}}. \tag {3}
$$
While GRAM trains CF module and CE module separately, it guarantees a theoretically equivalent parameter update with E2E as shown in the Proposition (proof in Appendix A). Yielding identical outcome of E2E back-propagation with fewer gradient computation, Single-step GRAM enables efficient training under hierarchical multi-modal setting like CCF by accumulating gradient buffer in the pseudo-target for items that are updated multiple times in a mini-batch.
4.2 Multi-step GRAM
Instead of a single update, we can accumulate gradient buffers for multiple mini-batch updates, gaining additional speed boost. In other words, CF / CE modules may separately update multiple times across multiple mini-batches, further elaborated in the following section. However, this relaxation of GRAM's alternating period no longer guarantees equivalence with E2E back-propagation.
As the CE module can use multiple steps to update item representations, Multi-step GRAM can
also reduce GPU memory footprint drastically compared to E2E.
4.3 Training Scheme of GRAM
Algorithm 1: GRAM
Input :  $\{B_I^{(t)}\}_{t = 0}^{T - 1}$  : mini-batch,  $\theta^{(0)}$  : initial parameters
Output:  $\theta^{(T)}$ : resulting parameters
while  $t = \{0,\dots ,T - 1\}$  do
(1. Produce content representations)
$$
h _ {i} ^ {(t)} \leftarrow C E \left(c _ {i}; \theta_ {C E} ^ {(t)}\right), \forall i \in B _ {I} ^ {(t)} \tag {4}
$$
(2. Update CF parameters)
$$
\theta_ {C F} ^ {(t + 1)} \leftarrow \theta_ {C F} ^ {(t)} - o p t \left(\nabla \mathcal {L} \left(\theta_ {C F} ^ {(t)}\right)\right) \tag {5}
$$
(3. Update content representations)
$$
\tilde {h} _ {i} ^ {(t)} \leftarrow h _ {i} ^ {(t)} - \nabla \mathcal {L} \left(h _ {i} ^ {(t)}\right), \forall i \in B _ {I} ^ {(t)} \tag {6}
$$
if  $t$  mod  $N = 0$  then
(4. Update CE parameters)
$$
\theta_ {C E} ^ {(t + 1)} \leftarrow \theta_ {C E} ^ {(t)} - o p t \left(\nabla \tilde {\mathcal {L}} \left(\theta_ {C E} ^ {(t)}\right)\right) \tag {7}
$$
$$
t \gets t + 1
$$
See Algorithm 1 for the pseudo-code of GRAM. We denote gradient accumulation step as  $N$ . The
algorithm becomes Single-step GRAM for  $N = 1$  and it becomes Multi-step GRAM for  $N > 1$ .
First, CE produces content representation  $h_{i}^{(t)}$  for CF module to complete the forward-pass for unique items  $i$  in the mini-batch  $B^{(t)}$ . Then, we update the CF module's parameters using standard back-propagation with a normal CF training objective  $\mathcal{L}$  in Eq.(8), while simultaneously updating the content representation (output from CE module) by treating it as trainable embedding  $\tilde{h}_{i}^{(t)}$ .
$$
\mathcal {L} := \sum_ {u \in B _ {u} ^ {(t)}} L \left(I ^ {u}, \left\{h _ {i} ^ {(t)} \mid \forall i \in B _ {I} ^ {(t)} \right\}; \theta_ {C F} ^ {(t)}\right) \tag {8}
$$
Lastly, for items of which embeddings are modified in Eq.(6), the CE module is trained to follow/regress towards the modification with respect to pseudo-target  $\tilde{h}_i^{(t)}$  and pseudo-loss  $\tilde{L}$  in Eq.(9).
$$
\tilde {\mathcal {L}} := \frac {1}{2} \sum_ {i \in B _ {I} ^ {(t)}} \left(\tilde {h} _ {i} ^ {(t)} - C E \left(c _ {i}; \theta_ {C E} ^ {(t)}\right)\right) ^ {2} \tag {9}
$$
Based on updated content representations, we repeat the process with  $t \gets t + 1$ . Note that for Eq.(6), Stochastic Gradient Descent with a learning rate of 1 should be used to guarantee the theoretical equivalence with E2E. For Eq.(5) and Eq.(7), choice of a optimizer (e.g. Adam) doesn't impact the equivalence with E2E.
4.4 GRAM's Speed Boost ratio  $\mathcal{R}$
Given a mini-batch  $B$  of  $|B_u|$  users' interaction sequences, the standard E2E back-propagation updates CE module for  $\sum_{u \in B_u} |I^u|$  (i.e. number of total interactions), while Single-step GRAM updates CE module for  $|B_I|$  (i.e. number of unique items) times only. Since PLM based CE modules are usually significantly larger than the head (CF module) attached for downstream task such as News Recommendation, the following ratio of speed boost for CE module applies in a close to directly proportionate manner for the entire training procedure.
$$
\mathcal {R} := \frac {\sum_ {u \in B _ {u}} \left| I ^ {u} \right|}{\left| B _ {I} \right|} = \frac {\# \text {i n t e r a c t i o n s} (B)}{\# \text {i t e m s} (B)} \tag {10}
$$
The ratio  $\mathcal{R}$  monotonically increases as minibatch size becomes larger. Thus, larger mini-batch
size would yield larger efficiency boost via Single-step GRAM. This is why Multi-step GRAM can achieve even more speed boost compared to Single-step GRAM. If the gradient accumulation latency becomes 1 epoch for Multi-step GRAM, the speed boost ratio  $\mathcal{R}$  becomes:
$$
\mathcal {R} := \frac {\text {t o t a l i n d a s e t}}{\text {t o t a l i n d a s e t}} \tag {11}
$$
Considering there are significantly less number of items compared to the total number of interactions in real-world datasets, GRAM with high enough update latency can achieve remarkable speed boost. However, we can also expect that longer accumulation latency would hurt model performance and convergence. In the following section's experiments, (i) training efficiency boost and (ii) performance degradation from different alternating frequency are closely monitored on various GRAM alternating periods (Single-Step, 10-Step, Half-EPOCH, Full-EPOCH).
While GRAM utilizes gradient accumulation across duplicate item representations to boost training, the resulting speed boost is orthogonal with traditional gradient accumulation as it focuses on increasing effective batch size under limited computational resource.
We first define the scope of tasks and metrics used in the experiments. Detailed description of datasets and methods are provided in section 5.1 and section 5.2, respectively.
Tasks: Experiments are conducted on two major task domains of CCF: Knowledge Tracing(KT) and News Recommendation(NR), where models predict whether a student/reader will solve/click an question/article, as a classification task.
Metrics: Overall AUC and cold-start item AUC (CSAUC) are measured for KT. AUC, MRR, nDCG@5, nDCG@10 are measured for NR. As cold start problem is intrinsically abundant in news recommendation environment (Wu et al. (2019)), we did not measure CSAUC separately. Wall-clock training time until convergence is reported for all experiments. For fair comparison of training time, all models are run on equivalent device (NVIDIA A100 GPU) in an isolated manner.
Dataset	Users	Total Items	Total Interactions	CS Items	CS Interactions	Average lt
Spanish	2,643	4,628	279,747	200	3,191	5.32
French	1,202	4,078	174,749	200	1,970	5.24
POJ	22,110	2,597	898,384	200	10,523	271.34
TOEIC	1,240,955	9,336	94,264,845	684	321,933	147.47
MIND	750,434	104,150	3,760,125	11.52(639.57)
Table 3: Dataset Information. For MIND, both average  ${l}_{t}$  of title alone and  ${l}_{t}$  of title + abstract + body are reported.
To evaluate our model, we used five real-world datasets: four datasets in KT, and one dataset in NR, on which both textual content data and user interaction data are available. Experiments on Duolingo French and Spanish dataset are done with single NVIDIA A100 GPU, and those on POJ, TOEIC, and MIND are done with eight NVIDIA A100 GPUs, in distributed data parallel training. The results were shown to be statistically significant (p  $< 0.05$ ).
Detailed per-dataset description is written below, and specification is in Table 3.  ${}^{1}$  Dataset collection methods are mentioned in Appendix F.
(KT) Duolingo (Spanish, French): Language Translation (Settles et al., 2018) contains questions and responses for Duolingo users. Following (Srivastava and Goodman, 2021), we collapsed the original token level mistakes to question level binary labels. We used Spanish and French dataset.
(KT) POJ: Computer Programming was collected from Peking online platform and consists of computer programming questions.
(KT) TOEIC: Language Comprehension from EdNet (Choi et al., 2020) is the largest publicly available benchmark dataset in education domain consisting of student interaction logs.
(NR) MIND: Microsoft News Dataset is one of the largest English dataset for monolingual news recommendation. MIND dataset provides news articles' title, abstract, and body text for news content modeling. For comparison with other models, we only utilized the news title, following Wu et al. (2019, 2021a). As Multi-step GRAM's significant speed boost allowed us to utilize more features, we also provide results on utilizing all textual features for Multi-step GRAM as well.
5.2.1 E2E & GRAM
To fairly compare GRAM's training methodology against the standard E2E, we apply GRAM/E2E on an identical model architecture, defined for KT and NR respectively. Model choices are shown in Table 2. To the best of our knowledge, this work is the first study to fine-tune BERT for KT. Detailed model architectures are described in Appendix D.
Based on this model architecture, we compare: E2E training, single-step GRAM 1S, 10-Step GRAM 10S, 0.5-EPOCH GRAM 0.5E, and 1-EPOCH GRAM 1E on the aforementioned metrics. We also provide benchmarks for other existing approaches, as elaborated in the following section 5.2.2. Note that E2E training in NR has the same structure as NRMS-PLM in (Wu et al., 2021a).
5.2.2 Other Baselines
In addition to the shared CCF model architecture defined above, we also include other approaches for extensive comparison.
NoFinetune approach directly adopts fixed item representation encoded from PLM without finetuning. Only the CF component is trained, receiving PLM's fixed output as the input.
NoContent approach does not incorporate any textual content at all. Each item representation is randomly initialized before being trained along with the standalone CF component. For KT, we used DKT (Piech et al., 2015).
(KT) Content Regularized CF (CRCF) is our implementation of the proposed regularization in TARMF with equivalent CF and CE modules of GRAM. As TARMF's content-encoder and user feature vector should go through additional iterations of optimization in sequential recommendation, we adopt hierarchical user encoder like E2E setting to eliminate the need of model retraining.
(KT) LM-KT formulates KT as auto-regressive modeling task to fine-tune pre-trained GPT-2. The method's major bottleneck to other sequential recommendation domains is that the model's sequence
1Train / Test set were randomly split in 8:2 ratio.  $10\%$  of train dataset were randomly chosen for validation set.
length has to increase in a multiplicative fashion:  $O(l_{t} \times l_{I})$  in Table 1.
(KT) EERNN is a specific instance of CE-CF pipeline where Bi-directional LSTM (CE) encodes question text's W2V representation into question embedding. CF consists of another LSTM layer.
(NR) NRMS is another specific instance of CE-CF pipeline where CE uses Glove word embedding and Multi-Head Self-Attention layers. Its CF component is also based on MHSA layers.