SlowGuess/NAACL_2022_OCR huggingface dataset:
(NR) SpeedyFeed used (i) auto-regressive modeling, (ii) BusLM, (iii) Dynamic Batching, and (iv) Cache mechanism to speedup PLM-based news recommendation. For fair comparison, we used equivalent CE module for both GRAM and SpeedyFeed.
Further experimental details are in Appendix D.
6.1 Knowledge Tracing
Performance Comparison: In general, E2E and GRAM in Table  $4^{3}$  shows best performance across datasets. Difference among GRAM variants highlights how Multi-step GRAM's performance gradually deviates from the E2E baseline as alternating period increases. As claimed in the previous section 4.1, GRAM(1S)'s performance most closely matches the E2E baseline. Confidence intervals of the two methods are reported in Appendix E.
NoFinetune and NoContent achieve significantly worse performances on both metrics on all datasets, as compared to the full CCF setting with both CE and CF fine-tuned properly. Also, NoContent does not show any inference power on cold-start items, reporting AUC values around 50. LM-KT and EERNN showed lower performance in AUC and CSAUC than E2E or GRAM(1S), respectively.
As shown in Figure 2, we empirically confirm that standard E2E training converged to better local minima than CRCF, at all regularization hyperparameter values used in (Lu et al., 2018). GRAM 1S even converges faster than all variants of CRCF while maintaining E2E performance. CRCF also showed larger degradation in CSAUC. We noticed that as the degree of regularization increases, time for convergence increases notably.
Speed Comparison: We first highlight the reduction in training time via GRAM. Across 4 datasets,
Dataset	Method	AUC	CSAUC	Speed-up
TOEIC	E2E	75.7	64.2	1(135hr)
GRAM 1S	76.0	63.0	5.7
GRAM 10S	75.8	65.1	1.7
GRAM 0.5E	75.7	64.7	26
GRAM 1E	75.3	64.6	146
EERNN	75.8	62.3	10
NoFinetune	69.1	64.3	343
NoContent	74.4	49.4	2547
POJ	E2E	69.0	65.4	1(123m)
GRAM 1S	69.0	65.5	4.5
GRAM 10S	69.1	65.0	3.8
GRAM 0.5E	68.7	64.7	9.1
GRAM 1E	68.8	64.5	12.5
EERNN	68.6	64.0	1.3
NoFinetune	68.3	65.8	41.0
NoContent	63.8	50.9	30.8
Spanish	E2E	75.1	68.7	1(74m)
GRAM 1S	75.1	68.3	3.5
GRAM 10S	74.4	67.2	2.1
GRAM 0.5E	74.4	67.0	2.8
GRAM 1E	74.7	67.0	5.7
CRCF 1S	74.9	66.2	1.2
CRCF 1E	74.3	66.3	2.1
EERNN	74.3	66.3	1.6
LM-KT	74.6	68.7	0.5
NoFinetune	72.5	66.2	24.7
NoContent	67.0	49.3	37.0
French	E2E	74.8	74.7	1(39m)
GRAM 1S	75.0	75	3.3
GRAM 10S	74.2	73.2	3.5
GRAM 0.5E	74.1	73.2	4.3
GRAM 1E	74.4	73.3	7.8
CRCF 1S	74.7	73.0	1.0
CRCF 1E	74.4	72.9	1.8
EERNN	74.0	71.3	1.2
LM-KT	74.3	74.7	0.3
NoFinetune	71.4	69.4	9.8
NoContent	67.0	49.5	13.0
Table 4: Prediction Performance / Speed-up in Knowledge Tracing. Training time is reported for E2E training, and best results among content finetuning methods are marked in bold.
GRAM 1S achieves  $4.3 \times$  speedup of the E2E baseline, while GRAM 1E achieves acceleration of  $43 \times$ . We observe GRAM 1E achieves most significant training time acceleration as expected, since the boost ratio  $\mathcal{R}$  of Eq.(11) is the largest in epochwise alternation. The power of GRAM 1E multiplies as the size of dataset increases, achieving remarkable  $146 \times$  speed up for the largest dataset, TOEIC. Among all datasets, TOEIC has the largest boost ratio  $\mathcal{R}$  based on Table 3 which explains the largest efficiency gain.
GRAM 1E also out-speeds EERNN at all datasets despite the fact that EERNN uses W2V embeddings and a single LSTM layer for its CE. In terms of GRAM's alternating period, we note
$^{2}$ LM-KT baseline was only tested on Duolingo datasets as other datasets' large average token length prevents LM-KT from considering more than a few items per each user.
For CRCF, best result is reported among hyper-parameter ablations in Figure 2.
Figure 2: Comparison between GRAM, E2E, and CRCF. For CRCF, regularization parameter is marked as well.
that the impact on speed boost is not monotonic on some datasets, potentially due to increased variance of optimization switching between the CE component and the CF component.
Based on GCP GPU cost for on-demand A100 ( $2.93/hr), training E2E model in TOEIC costs$ 3,161. GRAM 1E drastically reduces the training cost to $21.
Method	AUC	MRR	nDCG @5	nDCG @10	Speed-up
(Title-only)
E2E	68.9	33.3	36.8	43.2	1(10.4hr)
GRAM 1S	69	33.5	37.1	43.4	2.5
GRAM 10S	68.6	33.7	37.3	43.4	1.9
GRAM 0.5E	68.7	32.9	36.2	42.7	13.5
GRAM 1E	68.7	33.1	36.6	42.8	17.3
NRMS	67.2	33.3	35.5	42	13.9
SpeedyFeed	68.3	33.4	36.6	43	2.0
NoFineTune	66.8	32.4	35.7	41.9	33.5
(Title+Body)
E2E	1(*202hr)
GRAM 0.5E	69.6	34	37.6	44	45
GRAM 1E	69.3	34.1	37.8	44	56
Table 5: Prediction Performance and Training Speed on MIND Dataset. Training time is reported for E2E training. Overall best results among content finetuning methods are marked in bold, and best results utilizing title only are underlined.
6.2 News Recommendation
Performance Comparison: As shown in Table 5, Single-step GRAM matches performance of E2E training, and Multi-step GRAM shows less than  $0.5\%$  performance loss. Multi-step GRAM's capability to incorporate abstract and body of the news article (Title+Body) significantly improved the performance beyond all methods relying on title information alone.
SpeedyFeed (Xiao et al., 2021) shows worse performance than all GRAM methods. This may due to SpeedyFeed's cache mechanism, as it fails to optimize news representations that were generated in recent time steps, unlike GRAM. We also noticed that SpeedyFeed's performance is highly sensitive to its hyper-parameters on the method's cache policy. Increasing max cache step hyper-parameter for faster training easily caused the training loss to spike, deteriorating the model convergence.
An ensemble of Single-step GRAM and Multi-step GRAM is currently ranked 4th in the MIND official leaderboard provided by Microsoft. Even without state-of-the-art CF module (Fastformer (Wu et al., 2021b)) and CE module (UniLM (Dong et al., 2019)), the ability to encode the body of the news article with GRAM shows a comparable performance with state-of-the-art News Recommendation models.
Speed Comparison: Multi-step GRAM shows consistent speed boost in MIND dataset, where
<sup>4</sup>https://msnews.github.io/
Title-only GRAM 1E is  $17.3 \times$  faster than E2E. SpeedyFeed's acceleration, on the other hand, was lower than that of GRAM 1S. Although SpeedyFeed's central batching collects unique items in the forward pass, it still requires gradient computations for PLM's weights on every interaction during backpropagation, having limited speed gain. Also, SpeedyFeed's auto-regressive formulation, the most significant factor of speed boost, was not applicable to MIND dataset, which does not provide negative samples (news impression list) per each positive interaction (news click) step.
While Wu et al. (2019) reported that using all textual information increases model performance significantly in a small version of MIND dataset, baseline results from Wu et al. (2019, 2021a) only utilize news title due to computational complexity. With eight A100 GPUs, E2E training with all textual features is estimated to require 202 hours ( $4,735 of training cost) to converge. For this reason, we were unable to produce the result for E2E with all textual information. In contrast, GRAM 1E requires only 3.6 hrs ($ 84) to converge, requiring  $56 \times$  less training time compared with E2E.
Method	CE Batch Size	GPU Memory in %, (Gb)
TOEIC	MIND
E2E	N/A	95.2, (38.6)	95.1, (38.4)
GRAM 1E	8	12.1, (4.8)	12.5, (5.1)
32	16.0, (6.5)	34.9, (14.1)
Table 6: GPU Memory Consumption, with CF batch size of 4 in single NVIDIA A100. E2E doesn't have CE batch size as CE module naturally receives all items included in CF batch size as input.
6.3 GPU Memory Consumption
In E2E training, the entire computational graph as well as activations of all layers should be stored, resulting in a large GPU memory footprint (Wang et al., 2020). In this perspective, splitting CF module and CE module in GRAM brings down the required GPU memory during computation. As (1) the model size of the CF module is relatively small (single LSTM/MHSA) and (2) CE module of Single-step GRAM updates all item representations in a given batch in one step, the memory reduction is not significant in Single-step GRAM.
However, Multi-step GRAM can bring down
the memory requirement significantly as its CE module uses multiple steps to update item representations. Table 6 compares the memory consumption between E2E and GRAM 1E with content encoder batch size of 8 and 32, respectively. Overall, GRAM 1E consumes less than  $40\%$  of memory compared to E2E. While only GRAM 1E is compared, Multi-step GRAM in general consumes similar memory given the same content encoder batch size.
In this paper, we proposed GRAM as an efficient method to train content-based collaborative filtering models. Single-step GRAM splits the CE module and CF module during training, accumulating gradients for items appearing repeatedly in a batch. This effectively reduces the number of CE module's gradient computation and negates the need to store the intermediate activations for both of the modules at once. Extending Single-step GRAM, we increase the gradient accumulation latency for Multi-step GRAM, gaining additional training speed boost and memory footprint reduction.
GRAM was empirically evaluated on 5 different tasks to demonstrate its efficiency and comparable prediction power. Utilizing GRAM's efficiency, Knowledge Tracing model trained with GRAM 1E will be deployed in Santa<sup>6</sup>, an AI-powered English learning platform with 4 million users.
A limitation of our method is that we introduce an additional hyperparameter of gradient accumulation latency for Multi-step GRAM. We expect a more sophisticated gradient accumulation scheme may adaptively choose the gradient accumulation latency. Another potential extension of our research is to scale up GRAM for CCF tasks involving higher-dimensional inputs, such as images and videos.
We thank the anonymous reviewers of ACL Rolling Review, Seunghyun Lee (Riijid AI Research), and Suyeong An (Riijid AI Research) for their helpful feedback. We also thank Wansoo Kim (Riijid Infra) for helping us scale up our experiments to the Google Cloud Platform.
Estimated time is calculated based on per epoch time for E2E with all texts and number of epochs to converge for Title-only E2E.
<sup>6</sup>https://www.aitutorsanta.com/
Justin Basilico and Thomas Hofmann. 2004. Unifying collaborative and content-based filtering. In Proceedings of the twenty-first international conference on Machine learning, page 9.
James Bennett, Stan Lanning, et al. 2007. The netflix prize. In Proceedings of KDD cup and workshop, volume 2007, page 35. New York, NY, USA.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. 2020. Ednet: A large-scale hierarchical dataset in education. In International Conference on Artificial Intelligence in Education, pages 69-73. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Xiaqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4163-4174.
Xuan Nhat Lam, Thuc Vu, Trong Duc Le, and Anh Duc Duong. 2008. Addressing cold-start problem in recommendation systems. In Proceedings of the 2nd international conference on Ubiquitous information management and communication, pages 208-211.
Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, 33(1):100-115.
Yichao Lu, Ruihai Dong, and Barry Smyth. 2018. Coevolutionary recommendation model: Mutual learning between ratings and reviews. In Proceedings of the 2018 World Wide Web Conference, WWW '18, page 773-782, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Prem Melville, Raymond J Mooney, Ramadass Nagarajan, et al. 2002. Content-boosted collaborative filtering for improved recommendations. Aaiiaiai, 23:187-192.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in Neural Information Processing Systems, 28:505-513.
Nils Reimers, Iryna Gurevych, Nils Reimers, Iryna Gurevych, Nandan Thakur, Nils Reimers, Johannes Daxenberger, Iryna Gurevych, Nils Reimers, Iryna Gurevych, et al. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Burr Settles, Chris Brust, Erin Gustafson, Masato Hagiwara, and Nitin Madnani. 2018. Second language acquisition modeling. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pages 56-65.
Brent Smith and Greg Linden. 2017. Two decades of recommender systems at amazon.com. *IEEE internet computing*, 21(3):12-18.
Megha Srivastava and Noah Goodman. 2021. Question generation for adaptive education. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 692-701, Online. Association for Computational Linguistics.
Xiaoyuan Su and Taghi M Khoshgoftaar. 2009. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009.
Yu Su, Qingwen Liu, Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Chris Ding, Si Wei, and Guoping Hu. 2018. Exercise-enhanced sequential modeling for student performance prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Robin Van Meteren and Maarten Van Someren. 2000. Using content-based filtering for recommendation. In Proceedings of the machine learning in the new information age: MLnet/ECML2000 workshop, volume 30, pages 47-56.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. 2020. Revisiting locally supervised learning: an alternative to end-to-end training. In International Conference on Learning Representations.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019. Neural news recommendation with multi-head self-attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6389-6394.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021a. Empowering news recommendation with pre-trained language models. arXiv preprint arXiv:2104.07413.
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2021b. Fastformer: Additive attention can be all you need.
Shitao Xiao, Zheng Liu, Yingxia Shao, Tao Di, and Xing Xie. 2021. Training large-scale news recommenders with pretrained language models in the loop.
$$
\frac {\partial L}{\partial \theta_ {f}} = \frac {\partial L}{\partial h} \frac {\partial h}{\partial \theta_ {f}} = \frac {\partial L}{\partial y} \frac {\partial y}{\partial h} \frac {\partial h}{\partial \theta_ {f}} \tag {12}
$$
$$
\begin{array}{l} \left. \frac {\partial \tilde {L}}{\partial \theta_ {f} ^ {\prime}} \right| _ {\theta_ {f} ^ {\prime} = \theta_ {f}} = - \frac {\partial f}{\partial \theta_ {f} ^ {\prime}} \cdot \left(f (x; \theta_ {f}) - \frac {\partial L}{\partial y} \frac {\partial y}{\partial h} - f \left(x; \theta_ {f} ^ {\prime}\right)\right) (13) \\ = \frac {\partial L}{\partial y} \frac {\partial y}{\partial h} \frac {\partial h}{\partial \theta_ {f}} (14) \\ \end{array}
$$
B Hierarchical illustration of GRAM
Figure 3 is the hierarchical illustration of GRAM. Refer to Section 4.3 for more details.
Figure 3: Gradient on item representations are accumulated on Pseudo-target to mimic E2E training.
C Validation Performance
Figure 4 shows the validation performance per training time for E2E and GRAM variants. As the graph includes 10 epochs for early stopping patience, some overfitting is witness at the end of the training curve, such as the sudden drop of GRAM 1S in Spanish dataset after marking best performance.
D Experiment Detail
All experiments are ran 3 times and averaged results are reported except TOEIC E2E and MIND E2E, due to high computational cost mentioned in the result section. Across all experiments, CE component uses TinyBERT(Jiao et al., 2020), a
distilled BERT with on-par performance. The architecture contains 6 MHSA layers of dimension 768. CF component for KT uses 2-layer LSTM network, following Piech et al. (2015). In NR, we use a single MHSA layer, following Wu et al. (2019, 2021a).
In both domains, learning rate of 1e-4 was used for CF module and CE module after learning rate ablation in the scope of [1e-3, 1e-4, 1e-5]. Adam with Noam scheduler was used as the optimizer. Test metrics were measured by the best validation checkpoint after early stopping of patience 10 epochs. As E2E training consumes different amount of memory based on item token length, different batch sizes were used across datasets. Batch sizes per datasets are the following: 32 for Duolingo, POJ, and TOEIC, and 256 for MIND. As Multi-step GRAM requires much less memory compared with E2E, higher batch size was able to be used for large datasets such as TOEIC and MIND. Details are mentioned in Section C.
For NoFinetune experiment, pre-computed item representations from CE were initialized to CF's item embeddings to boost training speed.
Authors of EERNN also proposed EKT(Liu et al., 2019), which explicitly models the student's knowledge state for different knowledge concepts. As knowledge concept labels are not available in most datasets, we do not test EKT.
D.1 Knowledge Tracing
Mean-pooling was used to extract question representation from contextualized token embedding. For relatively small KT datasets (Duolingo and POJ), it was challenging to secure meaningful number of cold-start items (questions) in the test split. For these datasets, additional cold-start questions were randomly picked and interactions on those questions in the training split were removed to secure meaningful number of cold start interactions. Items with token length over max seq len (512) were truncated. For TOEIC dataset, passage, question, and choices were concatenated as content token sequence for CE component.
D.2 News Recommendation
For NR, we follow (Wu et al., 2021a) to use additive attention based pooling to extract news article representation. Title, abstract, body were concatenated for Setting B, with max len of 24 for title, 50 for abstract, and 400 for body. Items with token length over max seq len were truncated.
Figure 4: . Validation AUC convergence and training time until early-stopped checkpoint, across different GRAM alternation periods. Color schemes are synchronized across all subplots.
E Result Detail
E.1 E2E vs Single-step GRAM Confidence Interval
Metrics	E2E	GRAM (1S)
French Total AUC	0.748 (0.001)	0.750 (0.0006)
Spanish Total AUC	0.751(0.0005)	0.751(0.0006)
POJ Total AUC	0.654 (0.0009)	0.655 (0.0011)
As TOEIC and MIND E2E result is from a single run due to high computational complexity, Test AUCs on remaining three datasets (Duolingo French, Spanish, POJ) are reported with  $95\%$  CIs in Table 7
F Dataset Collection Methods
Duolingo: gathered from 2018 Duolingo Shared Task on Second Language Acqui-sition Modeling.
POJ: publicly available question texts and interaction logs were scraped from their public website.
TOEIC: content materials for corresponding question IDs in the dataset were collected privately.
MIND: collected from website
Duolingo (French, Spanish), POJ, MIND datasets are free to download for research purposes under respective terms. Interaction data for TOEIC is available as well for research purposes.