Where to Go Next for Recommender Systems? ID- vs. Modality-based recommender models revisited
Authors: Zheng Yuan
Fajie Yuan
Yu Song
Youhua Li
Fei Yang
Yunzhu Pan
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recommender models that utilize unique identities (IDs for short) to represent distinct users and items have been the state-of-the-arts and dominating the recommender system (RS) literature for over a decade. In parallel, the pre-trained modality encoders, such as BERT and ResNet, are becoming increasingly powerful in modeling raw modality features, e.g., text and images. In light of this, a natural question arises: whether the modality (a.k.a, content) only based recommender models (MoRec) can exceed or be on par with the ID-only based models (IDRec) when item modality features are available? In fact, this question had been answered once a decade ago, when IDRec beat MoRec with strong advantages in terms of both recommendation accuracy and efficiency.
We aim to revisit this `old' question and systematically study MoRec from several aspects. Specifically, we study several sub-questions: (i) which recommender paradigm, MoRec or IDRec, performs best in various practical scenarios, including regular, cold and new item scenarios?  does this hold for items with different modality features? (ii) will MoRec benefit from the latest technical advances in corresponding communities, for example, natural language processing and computer vision? (iii) what is an effective way to leverage item modality representations, freezing them or adapting them by fine-tuning on new data? (iv) are there any other factors that affect the efficacy of MoRec. To answer these questions, we conduct rigorous experiments for item recommendations with two popular modalities, i.e., text and vision. We provide empirical evidence that MoRec with standard end-to-end training is highly competitive and even exceeds IDRec in some cases. Many of our observations imply that the dominance of IDRec in terms of recommendation accuracy does not hold well when items' raw modality features are available. We promise to release all related codes & datasets upon acceptance.

Under review as a conference paper at ICLR 2023
WHERE TO Go NEXT FOR RECOMMENDER SYSTEMS? ID- VS. MODALITY-BASED RECOMMENDER MODELS REVISITED
Anonymous authors Paper under double-bling review
ABSTRACT
Recommender models that utilize unique identities (IDs for short) to represent distinct users and items have been the state-of-the-arts and dominating the rec- ommender system (RS) literature for over a decade. In parallel, the pre-trained modality encoders, such as BERT (Devlin
al., (018)) and ResNet (He al.) 2016), are becoming increasingly powerful in modeling raw modality features, e.g., text and images. In light of this, a natural question arises: whether the modal- ity (a.k., content) only based recommender models (MoRec) can exceed or be par with the ID-only based models (IDRec) when item modality features are avail- able? In fact, this question had been answered once decade ago, when IDRec beat MoRec with strong advantages terms both recommendation accuracy and efficiency.
We aim revisit this 'old' question and systematically study MoRec from sev- eral aspects. Specifically, we study several sub-questions (i) which recommender paradigm, MoRec IDRec, performs best various practical scenarios, includ- ing regular, cold and new item scenarios? does this hold for items with different modality features? (ii) will MoRee benefit from the latest technical advances corresponding communities, for example, natural language processing and com- puter vision? (iii) what is an effective way to leverage item modality representa- tions, freezing them adapting them by fine-tuning any other factors that affect the efficacy of MoRec. To answer these questions, we conduct rigorous experiments for item recommendations with two popular modal- ities, i.e., text and vision. We provide empirical evidence that MoRec with stan- dard end-to-end training is highly competitive and even exceeds IDRec in cases. Many of our observations imply that the dominance of IDRec in terms of recommendation accuracy does not hold well when items' raw modality features are available. We promise to release all related codes & datasets upon acceptance.
new data? (iv) there
INTRODUCTION
Recommender systems (RS) model the historical interactions users and items and recommend items that users may interact with in the future. RS are playing key role search engines, adver- tising systems, e-commerce websites, video and music streaming services, and various other Internet platforms. Mainstream recommender models usually use unique IDs represent items, which can be broadly categorized into two classes: two-tower based architectures (Rendle et al. 2012 Huang al. 2013) and sequence session- bassed neural architectures (Hidasi al. 2015 Yuan et al.,
2019
Kang & McAuley) 2018 Sun et al. 2019). These ID-only or ID-based recommender models (IDRec) are well-established and have been dominating the RS field for over decade.
Despite their popularity and success, there are also key weaknesses that should not be ignored. First, IDRec highly rely on the ID interactions, which fail to provide recommendations when users and items have few interactions (Yuan al. (2020), a.k.a. the cold-start setting. Second, pre-trained IDRec are not transferable across platforms given that user IDs and item IDs are in general not shareable in practice. This issue seriously limits the development of big & general-purpose RS models (Ding et al.) 202I]; Bommasani et al., 2021 Wang et al.) (022), an emerging paradigm in other deep learning application areas. Third, IDRec represent items mainly by IDD embedding

---

Under review as a conference paper at ICLR 2023
features, ignoring the inherent content features and thus prone to achieving sub-optimal perfor- mance. Moreover, maintaining a large and frequently updated ID embedding matrix for users and items remains key challenge in industrial applications (Sun al. 2020). Beyond these issues, ID-only recommender models cannot benefit from advances in other communities, such as powerful representation models developed in NL. (natural language processing) and CV (computer vision) areas. Last but not the least, recommender models leveraging ID features have obvious drawbacks in terms of interpretability, visualization and evaluation.
In contrast to IDRec, content-based recommender models (CoRec) rely heavily on item features, i... characteristics of the item such as the color of an object, authors of book, and keywords in an article. While intuitive and interpretable, they have been far less prevalent than IDRec over the past decade. A key reason for this could be that the content-basen item encoders are not as expressive as the standard item II) embedding, therefore leading to unsatisfactory performance. Nevertheless, we believe that given the recent extraordinary success of deep representation learning, to revisit the critical comparison between CoRec and IDRec. In particular, BERT (Devlin et al. 2018), GPT-3 (Brown al., 2020) and Vision Transformers (Dosovitskiy et al., 2020, Liu et al. 2021 have revolutionized the NLP and CV fields in terms of representing the text and vision features. Whether the item representations learned by these backbone models are better suited for recommender systems than ID embeddings remains largely unknown until now.
time
In this paper, we intend to rethink the potential of CoRec and study a key question: should we still stick the ID-based recommender paradigm? We concentrate on item recommendation based on the text and vision modalities - the two most common modalities in literature. To differentiate from traditional attribute-based CoRec, refer recommender models directly encoding items' raw modality features as MoRec. To be concise, we attempt to address the following sub-questions:
Q(i:: Equipped with strong modality encoders (ME), can MoRec perform comparably or bet- ter than IDRec in various recommendation scenarios? To answer this question, we conduct em- pirical studies by taking into account the two most representative recommender architectures (i.e.. two-tower based DSSM (Huang et al. 2013 Rendle et al. (0200) and session- based SASRec (Kang) & McAuley] 2018)) equipped with four powerful ME evaluated on three large-scale recommenda- tion datasets with two modalities (text and vision) and three recommendation scenarios (regular, cold & new item settings).
Q(i): If Q(i is yes, can the recent technical advances developed in NLP and CV fields be translated into accuracy improvement for MoRec when they utilize text and vision features? We address this question by performing three experiments. First, evaluate MoRec by compar- ing modality-based item encoders (e.g. BERT and ResNet (He et al. 2016)) with vs without pre- training on corresponding NLP and CV datasets; second, evaluate MoRec by comparing weaker vs stronger ME where weaker and stronger are determined by NLP and CV tasks; third, we evaluate MoRec by comparing smaller vs larger ME given that ME with larger model sizes tend to perform better than their smaller counterparts in various downstream tasks.
Qiii): How we effectively employ item modality representations derived from NIP or CV encoder network? Is the end-to-end (E2E) fine-tuned representation largely superior to the frozen representation given that the E2E training fashion requires much more compute and training time? The de facto practice for industrial recommender systems is to first extract item modality representations through some ME as ooff-hhe-heelf features and then incorporate them into recommender model (McAuley et al.) 2015 Covington al. 2016). often referred to the two- stage (TS) paradigm. While such TS paradigm is architecturally flexible, easy-to-implement and requires less compute and training time, we show that there is substantial accuracy loss compared to the E2E paradigm.
Beyond these key questions, we also identify several other factors that affect the training of MoRec in practice. To serve as a foundation for further research of MoRec, we will publish all our codes and datasets, including large-scale real-world video recommendation dataset (collected by ourselves) containing over 4 million user-video interactions with around 128K video thumbnails and 400K users!!
"K' is short for thousand.
2

---

Under review as a conference paper at ICLR 2023
2
IDREC & MOREC
I
ID
ID,
item ID Emnbediing Macrix
DSSM SASRec
IDRec
Transformer Slocks
Bie
2, m.
The best-selling Nintendo Switch gaames
trained Text Encoder
OSSM SASRiC
E
SASPcc
V
DT-Layer
MoRec
MiPs
MiPSs
Pre trained Image Encoder
OSSM SASRet
E
Encoder
Embedding
DT-layer
DT: Dimension Transfornation
Figure 1: Illustration of IDRec vs MoRec. Vi and T; denote raw features of vision and text modal- ities. E, is the item representation vector fed into the recommender model. The only difference between IDRec and MoRec is the item encoder. IDRec use an item ID embedding matrix as the item encoder, whereas MoRec use the pre-trained ME (followed by a dense layer for the dimension transformation, denoted by DT-layer) as the item encoder.
One core function of recommender model is to represent items and users. Denote I (of size [Z] and U (of size (u() as the set of items and users respectively. For an item i E I, we can represent it either by its unique ID i its modality content, such text and image features. Likewise, for user u E 24 we can represent her either by the unique ID u or the profile of u, where a profile can be the demographic information or a sequence of interacted items.
For IDRec, ID embedding matrix X R is constructed the item encoder for all items in I, where d is the embedding size. During training and inference, IDRec retrieve X7, Rd from x as the embedding of item and then feed it to the recommender network.
In MoRec, items are assumed to contain modality information beyond their IDs. For item i, MoRec use the modality encoder (ME) to generate the representation for the raw modality input of i and use it to replace the ID embedding vector in IDRec. For instance, in the news recommendation scenario, can the pre- pre-trained BERT or RoBERTa (Liu et al. 2019) text ME and represent piece of news by the output textual representation of its title. Similarly, when items contain visual features, can simply use pre-trained ResNet Vision Transformer as vision ME.
In this paper, we perform rigorous empirical studies on the two most commonly adopted recom- mender paradigms: DSSM (Huang et al.) (2013) and SASRec (Kang & McAuley 2018). The DSSM model is a two-tower based architecture where users/items are encoded by their own encoder net- works with user and item IDs as input. SASRec based on multi-head self-attention (Vaswani et al., 2017) which describes a user by her interacted item ID sequence. As mentioned before, by replacing the ID embedding matrix with an item modal- ity encoder, we obtain the MoRec version of both DSSM and SASRec. We illustrate IDRec and MoRee in Figure and provide training details in Appendix A
well-known sequential recommender model
3 EXPERIMENTAL SETUPS
3.1 DATASETS
We evaluate IDRec and MoRec on three real-world datasets, namely, the MIND dataset from the Microsoft news recommendation platform (Wu et al. 2020). the HM clothing purchase dataset from the H&M platforuf] and the Bili dataset (see Appendix C for data collection details) from an online video recommendation platform For MIND, the item, while for HM & Bili, we represent items by their associated images (one image per item). To fully exploit the capabilities of MoRec, the dataset used should ensure that the user's decisions
use the title of each article to represent
3

---

Under review as a conference paper at ICLR 2023
(whether to interact with item not) are determined only by the content features of the item. We notice that neither HM nor Bili meets this assumption since, in addition to the appearance of the item, users' purchase decisions HM are also influenced by the price and textual description of the product, while users' commenting behavior on Bili is mainly affected by the video (with hundreds of images) and audio signals (not only the cover image). Appendix Figure (6). That is, HM and Bili are not well suited for MoRec when using only an image to represent the item."
Table 1: Dataset characteristics. and m denote the numbers of users and items respectively. [ train, [Rvakd], |R [Roold] and |R denote the number of interactions of the training set, validation set, testing set, cold items, and new items respectively. [R|/ represents density.
Datasel
R tramm
[R
[R Behavior Types
MIND
630K
79,707
8,407
630K
630A
32.246
13.133
0.0167%
clicks purchases
HM
500K
86,733
5,500K
500K
500K
37,087
14,498
0.0127%
Bili
400
127,625
4,400K
400K'
400A
39,331
5,030
0.0086%
comments
To construct the datasets, we randomly select around 400K, 500K and 600K users from Bili, HM, and MIND, respectively. Then, we perform basic pre-processing by setting the size of all images to 224 and the title of all news articles to a maximum of 30 tokens (covering 99% of descriptions). For MIND, select the latest 23 items for each user to construct the interaction sequence. For HM and Bili, we choose the 13 most recent interactions since encoding images requires much larger GPU memory (especially with the SASRec architecture). Following (Rendle al.] 2012), we remove users with less than 5 interactions, simply because we do not consider cold user settings in this paper. Moreover, we do consider the cold item setting. Specifically, we count the interactions of all items in the training set (data split is described below) and regard those that appear less than 10 times as cold items and those that never appear as new items. Then we select additional user sequences out of the training dataset that contain such cold or new items to perform evaluation We report the statistics of all processed datasets in Table T..
3.2 EVALUATIONS
We split the datasets into training, validation, and testing sets by adopting the standard strategy. Specifically, the latest interaction of each user was used for evaluation, while second-to- last was used as validation for hyper-parameter searching, and all others are used for training. We evaluate all models using two popular top-N ranking metrics: HR @N (Hit Ratio) and NDCG@N (Normalized Discounted Cumulative Gain) (Yuan et al. 2019), where N is set to 10. We rank the ground-trutt target item by comparing it with all the other items in the item pool. Finally, we report the results on the testing set by choosing checkpoints with the best validation performance.
3.3 COMPARISON SETTINGS
For a fair comparison, we ensure that IDRec and MoRec have exactly the same network architecture except the item encoder. For both text and vision encoders, we pass their output item representations to a DT-llaye (see Figure T for dimension transformation. Regarding the hyper- setting, our principle is to ensure that IDRec are always thoroughly tuned, including learning rate 7, em- bedding size d, layer number I, dropout p. etc. While for MoRec, we attempt to first use the same set of syper-parameters. as IDRec and then perform some basic searching around the best choices. Thereby, without specially mentioning we do not guarantee that MoRec are reported with the best re- sults as searching all possible syper-parameten for MoRec is super expensive and cime-consu.ing. We report detailed settings in the Appendix G Table 12
4 COMPARATIVE STUDIES (Q(1)
We perform evaluation on three RS scenarios, namely, the canonical scenario (CANO-SC ACNO-SC) with both warm, cold and new items, cold item scenario (COLD-SC) and new item scenario (NEW-SC).
Nevertheless, there is no ideal publicly available image recommendation dataset (with raw image features) where a user's interaction decisions are only or mostly determined hy the image itself.
This is simply because the original testing set has too few cold items for evaluation.
4

---

Under review as a conference paper at ICLR 2023
Table 2: Accuracy (%) comparison of IDRec and MoRec using DSSM and SASRec in CANO-SC. MoRec with different ME are directly denoted by their encoder names for clarity. The best results for DSSM and SASRec are bolded. 'Improv.' is the relative improvement of the best MoRec compared with the best IDRec. All results of MoRec are obtained by fine-tuning their whole parameters including both the item encoder and user encoder. Swin-T and Swin-B are Swin Transformer with different model sizes, where T is tiny and B is base. ResNet50 is a 50-layer ResNet variant.
DSSM
SASRec
Datasel
Metries
Improv.
IDRcc
BERT
RoBERT
IDRec
BERT
BERT
RoBERT
MIND
H 10 NDCG@ 10
3.58
2.68
3.07
17.71
18.50
18.23
18.68
+5.48%
1.69
1.21
1.35
9.52
9.94
9.73
10.02
+5.25%
IDRec
ResNet50
Swin
IDRec
RexNet50
Swin-T
Swin-B
HR@ 10
4.93
1.49
1.87
6.84
6.67
6.97
7.24
+5.85%
HM
NDCG@ 10
2.93
0.75
0.94
4.01
3.56
3.80
3.98
-0.75%
BUI
HR
1.14
0.38
3.03
3.18
3.28
+8.25%
NDCU@ 10
0.56
0.18
0.27
1.63
1.45
1.59
1.66
+1.84%
4.1
MOREC vs IDREC ON CANO-SC
Here, we evaluate IDRec and MoRec with the two most important recommender architectures, i.e.. DSSM and SASRec. We use pre-trained BERT and RoBERTa as ME when items are of text fea- tures, and use pre-trained ResNet and Swin Transformer (Liu et al. (021)) when items are of visual features. We provide details (model size & download urls) of these ME in A Appendix Tablee Note for BERT and RoBERTa, we add the DT--layer on the final representation of the "ICLS]" token. We report results on the testing set in Table [ and convergence curves on Appendix
B Figure (and
Appendix F22 Figure 10 with session length of 23 for vision recommendations).
First, we observe that DSSM always substantially underperforms SASRec, regardless of the item encoding strategy used. For instance, MasSeec-based IDRec is around 4.9 better than DSSM-based IDRec in terms of HR@ 10 for news recommendation although their training, validation, and testing sets are kept the same. The performance gap for image recommendation is relatively small, around 1.4x and 2.7x, on HM and Bili respectively. This is expected since much prior literature (Kang & McAuley 2018 Sun al. 2019) has revealed that properly representing users with their interacted item sequence e.g., by an autoregressive training manner (Kang & McAuley 2018 Yuan et al., 2019 in general more powerful than dealing them as individual user IDs. Besides, DSSM neither explicitly, nor implicitly, takes into account interaction orders which reflect user's dynamic preference, thereby leading inferior results for time-award aware item recommendations.
Second, we notice that with the DSSM architecture, MoRec perform much worse than IDRec in all three datasets even with the SOTA ME, in particular for the visual recommendation scenarios. By contrast, with the SASRec architecture, MoRec consistently achieve better results than IDRec on MIND using any of the three text encoders, i.e., BERT small: BERT buse and RoBERTabsee For in- stance, MoRec outperform IDRec by over 5% the two evaluation metrics with the RoBERT text encoder. Meanwhile, MoRec perform comparably to IDRec when using Swin Transformer as ME but perform relatively worse when using ResNet50. The performance disparity of MoRec between DSSM and SASRec potentially implies that a powerful recommender architecture (SAS- Rec vs DSSM) is required to fully harness the strengths of the modality-basee item encoder
4.2 MOREC VS IDREC ON COLD-SC & NEW-SC
MoRec are a natural fit for cold item recommendation as their ME module is specifically developed model the raw modality features of item, whether it is cold or not. To validate this, we evalu- ate IDRec and MoRec in the two scenarios, i.e., COLD-SC and NEW-SC. We report the results in Appendix D Table 7 As clearly shown in Table 7 MoRec consistently and substantially improve IDRec on all three datasets for both text and vision modalities in both COLD-SC and NEW-SC. In particular, the HR @ 10 on MIND rises from 0.0036% to 3.0637% COLD-SC and from 0.0125%
'Given MoRec's poor results with DSSM, we only focus the SASRec architecture in the following.
5

---

Under review as a conference paper at ICLR 2023
to 0.5899% in NEW-SC" Similar observations also be made for the image and video recom- mendation on HM and Bili. The superiority of MoRec comes from the powerful representations of ME which is first pre-trained on large-scale text and image datasets and then fine-tuned to adapt the recommendation objectives.
The above results shed the following insights: (1) the recommender architecture matters lot for the performance of MoRec; (2) the item encoder network influences the performance of MoRec as well; (3) (Answer for Q(i) equipped with the most powerful ME, MoRec clearly beat the IDRec counterpart for text recommendation and is on par with IDRec for vision recommendation with the sequential recommender architecture. However, it seems that there is little chance for MoRec to replace IDRec with the typical two-tower DSSM paradigm in the canonical scenario, in particular for items with vision modality. (4) Without any doubt, MoRec markedly outperform IDRec in the star-start and new item settings. is worth mentioning again that Bili and HM datasets are somewhat unfair for MoRec since their interacted items are not mostly determined by image features (e.g., it is nearly impossible to learn the price audio features from only a raw image). Suggested by reviewr jqDE & (NVZ: We also present the cross-domain recommendation task of MoRec in Appendix !
CAN MOREC INHERIT ADVANCES IN MULTIMEDIA COMMUNITIES? (Q(I))
One key advantage of MoRec is that it promises to introduce the strong representation learning techniques from other communities, e.g., NLP NLP and CV, to the recommendation task. Hence, we ask: can the latest advances NLPP and CV be transferred the tasks of recommender systems? We aim to answer this question from the following perspectives.
Table 3: Pre-trained (PE) ME vs TFS on the testing set regarding HR @ 10 (%). BERT are used as text ME, and ResNet50 and Swin-T are used as vision ME. 'Improv.' indicates the relative improvement of PE over TFS. We report the convergence behaviors on the validation set Appendix
317.5
17.0
BERT
BERT
BERT
OPT
BERT
RoBERTa
(a) MIND.
4
ResNet18
ResNet34
ResNet50
Swin-T
Swin-B
(b) HM.
Figure
Dataset
Enceder
TFS
PF
Improv.
ResNet18
ResNet33
ResNet50
Swin-T
Swin-B
MIND BERT
17.78
18.23
+2.53%
(c) Bili.
ResNer50
6.67
Accuracy with different pre-trained ME in MoRec. Parameters of the pre pre-trained encoder network all fine-tuned on the recommendation task.
Figure 2:
HM
Swin-T
6.27
6.97
+11.16%
Re
2.93
+9.74%
Bili
Swin-T
2.83
3.18
+12.37%
First, we investigate whether a larger pre-trained ME enables better recommendation accuracy since in NLP and CV larger pre- tre-trained models tend to offer higher performance in corresponding down- stream tasks. As shown in Figure 2 a larger vision item encoder always achieves better image rec- ommendation accuracy, i.e., 8 MoRec <, ResNet34-baseds MoRec ResSobbb-bdseed MoRec, and Swin-T based MoRec < Swin-B based MoRec. Similarly, we find that BERT BERRTiny-bbe -baeed MoRec < BERT small mediun-- BERThsse based MoRec. One difference is that BERT medium and - based MoRec do not outperform BERT -based MoRec although the latter has a smaller-size BERT variant. We conclude that in general larger ME tend to improve the recommen- dation accuracy but this may not strictly hold in all cases.
Second, we investigate whether a more powerful encoder network enables better recommenda- tions (see Appendix G Table 9 for more details about these ME). For example, it is recognized that RoBERTa outperforms BERT (Liu et al. 2019), and BERT outperforms the unidirectional
We find that the accuracy of IDRec on MIND in COLD-SC is worse than the random strategy (HR @ N zz x N) used for new item recommendation (e.g.. 0.0036 vs. 0.0125). This is reasonable since cold items have fewer chances he ranked higher after training IDRee, sometimes even worse than basic random sampler.

---

Under review as a conference paper at ICLR 2023
GPT (Radford et al. 2018), e.g., OPT, for most text understanding (but not generative) tasks with similar model size, and that Swin Transformer often outperforms ResNet in many CV tasks (Liu al. 202T). As shown in Figure MoRec's performance keeps consistent with the findings in NLP and CV, i.e., RoBERTanase Tanuse-based MoRec > BERT BERThaa---asee MoRec > 12T12ss-based MoRec, and Swin-T based MoRec > Res Ret50-bb- MoRec (Swin-T has similar model size to ResNet50, the same for RoBERTaNes- BERT and OPT
Third, we investigate whether the pre-trained ME produces higher recommendation accuracy than its training-from-scree the pre-trained BERT, ResNet, and Swin largely improve corresponding NLP and CV tasks against their TFS versions. We report the recommendation results on the testing set in Table 3 and the convergence behaviors on the validation set Appendix E Figure 7 It can be clearly seen that pre-trained MoRec obtain better convergence and final results. In particular, MoRec achieve around 10% improvements with the pre-trained ME (ResNet and Swin) on HM and Bili, which also aligns with findings in NLP and CV domains.
(TFS) version (i.e., with random initialization). There is no doubt that
According to the above experiments, we conclude that (Answer for Q(i)) MoRec build connec- tions for RS and other multimedia communities, and can in general inherit the latest advances from the NLP and CV fields. In other words, MoRee have more chances to be improved in the future long new breakthrough happens in corresponding research fields.
END-TO-END (E2E) VS TWO-STAGE (TS) TRAINING (Q(I))
E2E-based MoRec incur a significant computational overhead due to forward and backward prop- agation for each item in given sequence of user interactions. The common approach in practice is usually based on a TS manner: first extracting offline features by ME and then adding them into a recommender network (He & McAuley, 2016b a) (see Figure 3). Such TS pipeline especially popular for industrial applications (e.g.. Deep Crossing (Shan et al. (0166) and YouTube (Cov-) ington et al. 2016)) considering that there are often hundreds of millions of training examples. Here, we evaluate whether this widely adopted strategy with fixed features leads to ideal recommendation.
As shown in Table 4 we find that bSsba MoRec show surprisingly poor results, compared to IDRec and E2E-based MoRec. In particular, with ResNet, it achieves only 60% and 25% perfor- mance of E2E-based MoRec on HM and Bili respectively. The results indicate that the modality features pre-trained by these NLP and CV tasks are not universal enough, which thereby results in inferior recommendation results. MoRec achieve the best results by performing further parameter fine-tuning, which needs the E2E training manner (Suggested by reviewer jqDE: For comprehen- sive comparison, we also add several additional neural network layers on the top of the extracted item representation so as see whether such strategy will better align these fixed representation with the current recommender backbone network. We report the results in Appendix D. Thereby, we want to remind researchers and practitioners that (Answer for Q(ii)) the de facto two-stage recommender regime causes severe performance degradation, which should not be ignored in practice. Meanwhile, we advocate more efforts on optimizing the training cost and efficiency of the E2E-based MoRec since it cause around 100x-1000x large compute & training time than IDRec (see Appendix Table 12).
Floer
Table 4: E2E vs TS in terms of HR @ 10 (%).
Nestrusen
Encoder
lows rgemaziion
Dataset
IDRec
Encoder
TS
H2F
SAStec
Pyetroiec Feoodde
Zanable Aeprrsrtittoon
Di crier
MIND
17.71
BERThae
13.93
18.23
TR
ResNe150
4.03
6.67
HM
6.84
Swin-T
3.45
6.97
ResNet50
0.72
2,93
Figure 3: Illustration of MoRec with TS and E2E.
Bili
3.03
Swin-T
0.79
3.18
For the online inference stage, MoRec as fast as IDRec since ME can be calculated by offline manner.
7

---

Under review as a conference paper at ICLR 2023
7 BAG OF OTHER OBSERVATIONS
E2F-bbaed MoRec have been less studied before, especially for visual recommendation. Here, we present several other findings (including both positive and negative results) towards practical MoRec.
A second round of pre-training on ME. Performing a second round of pre-training for ME using the downstream dataset (without using labels) sometimes works well in the DL literature (Guru- rangan et al., 2020). Here, we explore whether it offers improved recommendations for MoRec. Following the pre-training of BERT, we adopt the "masked language model" (MLM) objective to train the text encoder of MoRec (denoted by BERT -MLMM) on MIND and report results in S As shown, BERT bM.. gains higher accuracy than BERT base for both the TS and E2E models. Similarly, we explore whether it holds for the vision encoder. Note that ResNee and Swin Trans- former used in previous experiments are pre-trained a supervised manner, but neither HM nor Bili contains supervised image labels. To this end, we turn to use MAE (He et al., (022), SOTA image encoder pre-trained in an unsupervised manner, similar to MLM. We find M MMLM clearly improves the standard MAE on HM with the TS model, but obtains marginal gains with the E2E model. By contrast, no accuracy improvements are observed on Bili. By examining image cases in Errenixx[FFiguuree
we find that pictures in Bili have very diverse topics and than HM (with only very simple fashion elements). Our conclusion is that the effectiveness of the second round of pre-training depends on individual datasets; more importantly, it seems very difficult to obtain larger accuracy gains for the E2E strategy.
challenging
Trade-off between accuracy and efficiency. While fine-tuning full parameters in ME produces better accuracy than frozen features, it also imposes greater computational burden. Here. we attempt investigate whether all parameters need adaptation. As shown in Table 6 MoRec with BERT-based ME show the optimal results when fine-tuning the top 6 blocks (approximately half of all parameters). For the two visual datasets, fine-tuning all layers yields slightly better results than fine-tuning the top 2 out of 4 blocks. According to this, we conclude that in practice, fine-tuning the top half of ME is worth considering for the balance between accuracy and efficiency.
Besides, we have examined separate parameter searching for item & user encoders, and report the results in Appendix(H) H Our conclusion is searching hy byprpaarameters (e.g., learning rate & weight decay) separately for ME and user encoder matters for obtaining better MoRec results.
Table 5: Comparison of HR @ 10 (%) w/ and
Table 6: Comparison of HR 10 (%) when
w/o extra pre-training with the MLM objective
fine-tuning different blocks of the pre-trained ME. 'n/m' means fine-tuning the top n blocks of ME containing a total of m blocks.
using the TS and E2E training strategy. 'Im-
prov.' means the relative improvement of w/
MLL compared to w/o MLM.
muni-tunnng blocks
Dataset
Encoder
Datasel
Encoder
Manner
MLM
MLM
Improv.
0/12
2/12
6/12
12/12
MIND
BERT
13,93
17.87
18.51
18.23
13.93
14.68
+5.38%
MIND
BERT
F2F
18.23
18.63
+9%
0/4
1/4
4/4
HM
MAP
E2E
TS
2.50
7.03
7.07
2.79
+11.60%
+0.57%
IIM
ResNet50
Swin-T
4,03
6.18
6.46
6.59
6.67
6.97
Bili
MAB
E2E
T3
3.18
0.57
0.57
3.17
-0.31%
0.00%
Bili
ResNe110
Swin-T
0.72
2.88
2,83
2,89
3.00
3.18
2.93
RELATED WORK
ID-based recommender systems (IDRec). There are thousands of recommender models purely based on user/item ID) as inputs, ranging from early item-to-item collaborative filtering (Linden ct al., 2003). latent factorization models (Koren et al., 22099] Rendle 2010). to deep learning (DL) models (He et al.) 2017 Hidasi et al. 2015). According to whether explicitly modeling users' tem- poral preferences, they can be roughly divided into two types: two-tower based models (TTRM) and
We have not reported such results since it can be easily analyzed from Appendix Table 12 that both training time, compute resources and GPU memory consumption will he largely reduced by fine-tuming only few layers.
8

---

Under review as a conference paper at ICLR 2023
sequential neural models (SRM). TTRM (Rendle al. 2012 Huang et al. 2013 He & McAuley 2016b] Covington et al.] 2016) assign a unique user ID to represent the user and posses both user encoder and item encoder. Despite the remarkable success of DL. TTRM in literature still has very shallow layers (Yang et al., 2020 Rendle et al. 2020). By contrast, SRM uses the user-item interac- tion sequence to represent user, which can be stacked with deeper layers (Yuan et al.) 2019 Wang et al., 2021, Sun et al.) 2020) and is typically more powerful than TTRM. The most representative sequential RM include GRU4Rec (Hidasi al. 2015), NextItNet (Yuan et al. 2019). SASRec and BERT4Rec Sun et al. 2019) with RNN, CNN, Transformer and BERT as the backbone respectively, among which SASRec often performs the best in the literature (Fischer et al. (022))
Modality-based recommender systems (MoRec). MoRec, falls into the direction of content- based RS, and focuses on modeling item's modality features, such as text (Wu et al.. 2020). im- ages (McAuley et al. 20155) videos (Deldjoo et al. (016), voice (Van den Oord et al. 2013) and text-image pairs (Wu et al. 202Tb). Previous work tended to adopt the two-stage regime by first extracting item modality features by ME and then incorporate these fixed features into the recom- mender model (McAlley et al. 2015] He & McAuley 2016b|a Shan et al.] 2016 Lee & Abu- El-Haija 2017 Tang et al.j 2019: Wei et al. (019). More importantly, most of these works only used modality information as side features but with IDs as main features. The unpopularity of the E2E-style MoRec can be attributed into several reasons: (1) the two-stage regime is architecturally flexible for industrial applications and requires much less compute and training cost; (2) there is no high-quality publicly available dataset with raw item modality features until the recently released MIND and HM (still not ideal for such research); (3) ME in the past literature are not very expres- sive even by E2E training; and (4) researchers had lost confidence in content-basee cold-start settings) since a decade ago. Some recent works started to explore the based MoRec, however, most of them focus on text recommendation. For example, [Wu et al.) [Shin et al.) Yu al.) Yang et al.) Xiao et al.) Hou et al.J applied different types of pre-trained ME for news recommenda- tion; Elsayed et al. introduced ResNet to fashion- based recommendation and co-trained it with ID embeddings. However, to our knowledge, none of them performed formal, purposeful and com- prehensive study towards the comparison of IDRcc and MoRec under a fair experimental setting and particularly for the non cold-start setup. A more clear comparison of related work is shown in Appendix K
RS (unless for
CONCLUSION, LIMITATIONS, AND FUTURE WORKS
In this paper, we studied an important yet unexplored question regarding whether IDRec would continue to dominate the RS community. Obviously, this question cannot be thoroughly addressed by one paper and need more efforts from the RS community. Yet, one major finding here that with the SOTA ME, MoRec could already perform on par or better than IDRec with the typical recommender architecture (i.e., Transformer) even in the non-cold item recommendation setting 10 Moreover, MoRec can largely benefit from the technical advances in the NLP and CV fields, which implies that has larger room for accuracy improvements in the future. Given this, we believe our research is meaningful and would potentially inspire more studies on MoRec, for example, develop- ing more powerful recommender architectures, more expressive & generalized item encoders, better item & user fusion strategies and more effective optimizations to reduce the compute & memory costs and the longer training time. We also envision that in the long run the prevailing paradigm of RS may have a chance shift from IDRec MoRec when raw modality features are available.
As mentioned above, this study is only a preliminary of MoRec and has several limitations: (1) we considered RS scenarios with only text and vision, whereas MoRec's behaviors with other modali- ties, e.g., voice and vidco, remain unknown; (2) we consider only single-modal item encoders, while the behaviors of multimodal MoRec are unknown; (3) we considered only a very basic approach to fusing ME into recommender models, thereby MoRec may achieve sub-optimal performance; (4) we only considered pure IDRec or MoRec, whereas the performance of a hybrid E2E- rec- ommender model with both ID and modality features is unknown (note in this case, some merits of MoRec will no longer hold because of the ID features); (5) it remains to be seen whether the key findings still hold if we scale up training data to 100x or 1000 as in many industrial systems.
We emphasize again that since the evaluation settings are very unfair to MoRec, including unfavorable recommendation datasets (see Section[3.]) sub-optimal paramee settings (see Section B.3) and not the largest or strongest item encoders, it might perform considerably better than IDRcc on more fair settings.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang. Zheng Liu, and Xing Xie. Neural news rec- ommendation with long-and short-term user representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 336-345, 2019.
Qiwei Bi, Jian Li, Lifeng Shang, Xin Jiang, Qun Liu, and Hanfang Yang. Mtrec: Multi-task learning
over hert for news recommendation. In Findings the Association for Computational Linguisties: ACI 2022, pp. 2663-2669, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108. 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Yashar Deldjoo, Mehdi Elahi, Paolo Cremonesi, Franca Garzotto, Pietro Piazzolla, and Massimo Quadrana. content-basee video recommendation system based on stylistic visual features. Jour- nal on Data Semantics, 5(2):99 113,99111,,1 2016.
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. Zero-shot recommender sys- tems. arXiv preprint arXiv: 100.083/ 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
Shereen Elsayed. Lukas Brinkmeyer, and Lars Schmidt-Thieme End-to-end image- based fashion recommendation. arXiv preprint 22X5::2005.0292. 2022.
Elisabeth Fischer, Daniel Zoller, and Andreas Hotho. Comparison of transformer-based sequential product recommendation models for the coveo data challenge. 2021.
Jon Atle Gulla, Lemei Zhang, Peng Liu, Özlem Özgöbek, and Xiaomeng Su. The adressa dataset for news recommendation. In Proceedings of the international conference on web intelligence, pp. 1042-1048, 2017.
Zhiqiang Guo, Guohui Li, Jianjun Li, and Huaicong Chen. Topicvae: Topic-aware disentangle- ment representation learning for enhanced recommendation. In Proceedingy of the 30th ACM International Conference on Multimedia, pp. 511-520, 2022.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't 't stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
Mengyue Hang. Ian Pytlarz, and Jennifer Neville. Exploring student check-in behavior for improved point-of- interestt prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 321-330, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
10

---

Under review as a conference paper at ICLR 2023
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517, 2016a.
Ruining He and Julian McAuley. Vbpr: visual bayesian personalized ranking from implicit feed- back. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016b.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural col- laborative filtering. In Proceedings of the 26th international conference world wide web, pp- 173-182, 2017.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-basee rec- ommendations with recurrent neural networks. arXiv preprint arXiv: (ao:::511..99399 2015.
Min Hou, Le Wu, Enhong Chen, Zhi Li, Vincent W Zheng, and Qi Liu. Explainable fashion rec- ommendation: A semantic attribute region guided approach. arXiv preprint arXiv: /2862, 2019.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. To- wards universal sequence representation learning for recommender systems. arXiv preprint arXiv:2206.0594/ 059441 2022.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338, 2013.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender
systems. Computer, 42(8):30-31) 2009.
Joonseok Lee and Sami Abu-EI-Haija. Large-scale content-only video recommendation. In Pro- ceedings of the IEEE International Conference Computer Vision Workshops, pp. 987-995, 2017.
Chenyi Lei, Yong Liu, Lingzi Zhang, Guoxin Wang, Haihong Tang, Houqiang Li, and Chunyan Miao. Semi: sequential multi-modal information transfer network for e-commerce micro-video recommendations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discov- ery & Data Mining, pp. 3161-3171, 2021.
Jian Li, Jieming Zhu, Qiwei Bi, Guohao Cai, Lifeng Shang, Zhenhua Dong, Xin Jiang, and Qun Liu. Miner: Multi-interest matching network for news recommendation. Findings of the Association for Computational Linguistics: ACL 2022, pp. 343-352, 2022.
Greg Linden, Brent Smith, and Jeremy York. Amazon. com recommendations: Item-to-item collab- orative filtering. IEEE Internet computing, 7(1):76-80, 2003.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17X1v:1711.05101, 2017.
Xufang Luo, Zheng Liu, Shitao Xiao, Xing Xie, and Dongsheng Li. Mindsim: User simulator for news recommenders. In Proceedings of the ACM Web Conference 2022, pp. 2067-2077, 2022.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image- based rec- ommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.

---

Under review as a conference paper at ICLR 2023
Jianmo Ni. Jiacheng Li, and Julian McAuley. Justifying recommendations using distandy--abelle reviews and fine-graine reeeggained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNIPP)JCNIIP), pp. 188-197, 2019.
Rong Pan, Yunhong Zhou, Bin Cao, Nathan N Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. One-class collaborative filtering. In 2008 Eighth IEEE International Conference on Data Mining, pp- 502-511. IEER, 2008
Gustavo Penha and Claudia Hauff. What does bert know about books, movies and music? prob- ing bert for conversational recommendation. In Fourteenth ACM Conference Recommender Systems, pp. 388-397, 2020.
Tao Qi. Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. Personalized news recommendation with knowledge-aware interactive matching. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 61-70, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Steffen Rendle. Factorization machines. In 2010 IEEE International conference on data mining, pp. 995-1000. IEEE, 2010.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.200 2012.
Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. Neural collaborative filtering vs. matrix factorization revisited. In Fourteenth ACM conference recommender systems, pp. 240- 248, 2020.
Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang. Dong Yu, and JC Mao. Deep crossing: Web- scale modeling without manually crafted combinatorial features. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pp. 255-262, 2016.
Kyuyong Shin, Hanock Kwak, Kyung-Min Kim, Minkyu Kim, Young-Jin Park, Jisu Jeong, and Seungjae Jung, One4all user representation for recommender systems in e-commerce. arXiv preprint 1tti::206.000550 2021.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. Proceedings of the 28th ACM international conference information and knowledge management, pp. 1441- 1450, 2019.
Yang Sun, Fajie Yuan, Min Yang. Guoao Wei, Zhou Zhao, and Duo Liu. A generic network compres- sion framework for sequential recommender systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1299-1308, 2020.
Jinhui Tang, Xiaoyu Du, Xiangnan He, Fajie Yuan, Qi Tian, and Tat-Seng Chua. Adversarial training towards robust multimedia recommender system. IEEE Transactions on Knowledge and Data Engineering, 32(5):855 867, 2019.
Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen. Deep content- content-based music rec- ommendation. Advances in neural information processing systems, 26, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. Fine tme-grained interest matching for neural news recommendation. In Proceedings of the 58th annual meeting of the association for compu- fational linguistics, pp. 836-845, 2020.
12

---

Under review as a conference paper at ICLR 2023
Jiachun Wang, Fajie Yuan, Jian Chen, Qingyao Wu, Min Yang, Yang Sun, and Guoxiao Zhang. Stackree: Efficient training of very deep sequential recommender models by iterative stacking. In Proceedings of the 44th International ACM SIGIR conference Research and Development in Information Retrieval, pp. 357-366, 2021.
Jie Wang. Fajie Yuan, Mingyue Cheng, Joemon M Jose, Chenyun Yu, Beibei Kong, Zhijin Wang, Bo Hu, and Zang Li. Transrec: Learning transferable recommendation from mixture-of oo-uur--f-modalityy feedback. arXiv preprint arXiv:2206.0611 200,0 2022.
Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. Mmgen: Multi-modal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 1437-1445, 2019.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang. and Xing Xie. Neural news rec- ommendation with multi-head self-attention. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNLP-IJCNLP), pp. 6389-6394, 2019a.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. Empowering news recommendation with pre-trained language models. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1652-1656, 2021a.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. Mm-rec: multimodal news recommen- dation. arXiv preprint arXiv:110.007477, 2021b.
Chuhan Wu, Fangzhao Wu, Yang Yu, Tao Qi, Yongfeng Huang, and Qi Liu. Newsbert: Distilling pre-trained language model for intelligent news application. arXiv preprint trXiv:2102.04885 2021c
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, al. Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 3597-3606, 2020.
Le Wu, Lei Chen, Richang Hong, Yanjie Fu, Xing Xie, and Meng Wang. A hierarchical attention model for social contextual image recommendation. IEEE Transactions on Knowledge and Data Engineering, 22(10):185: 101118887771111 2019b.
Shitao Xiao, Zheng Liu, Yingxia Shao, Tao Di, Bhuvan Middha, Fangzhao Wu, and Xing Xie. Training large-scale news recommenders with pretrained language models in the loop. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 4225- 2022.
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaoming Wang, Taibai Xu, and Ed H Chi. Mixed negative sampling for learning two-tower neural networks in recom- mendations. In Companion Proceedings of the Web Conference 2020, pp. 441-447. 2020.
Yoonseok Yang, Kyu Seok Kim, Minsam Kim, and Juneyoung Park. Gram: Fast fine- tuning of pre-trained language models for content-based collaborative filtering. arXiv preprint D0X1v:2204.04179, 2022.
Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi. Tao Qi, and Qi Liu. Teny-newsrec: Efficient and 2021.
effective plm-based news recommendation. arXiv preprint
Fajie Yuan, Guibing Guo, Joemon M Jose, Long Chen, Haitao Yu, and Weinan Zhang. Lambdafm: learning optimal ranking with factorization machines using lambda surrogates. In Proceedings of the 25th ACM international conference on information and knowledge management, pp- 227-236, 2016.
Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M Jose, and Xiangnan He. A simple convolutional generative network for next item recommendation. In Proceedings of the twelfth ACM international conference on web search and data mining, pp. 582-590, 2019.
13

---

Under review as a conference paper at ICLR 2023
Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. Strameter-efficient transfer from sequential behaviors for user modeling and recommendation. Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp- 1469-1478, 2020.
Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. Unbert: User-news matching bert for news recommendation. In IJCAI, pp. 3356-3366 2021.
A TRAINING DETAILS
Denote R the set of all observed interactions in the training set. For each observed interaction u, SE R. we randomly draw a negative sample < u, >g R in each epoch during training. Note that some literature (Yuan et al. 2016) shows that using more advanced hard negative sampling could improve the results of recommender models, which we left for future investigation. All observed interactions and sampled negative interactions can form the training set Rraii Following (He al., 2017 Kang& & McAuley 2018), we adopt the binary cross entropy loss as the objective function for both DSSM and SASRec:
min
(loi())()ui)) + log(1 a (fuj)))
Roongii
where = =xx)=//(11 + ( is the sigmoid function.
B CONVERGENCE OF IDREC AND MOREC ON SASREC
We show the convergence of MASRec--baee IDRec and MoRec in Figure 4 Note that report the
3.5
(96)
17.
HR@ (%)
3.0
5
HR
IDRec
IDRcc
IDRec
ResNet50
2.0
ResNet50
14
BERT
Swin-T
Swin-T
RoBERTan
Swin-B
1.5
Swin-B
1.0
40
80
120
160
200
40
80
120
160
200
0
40 80
120 160 200 240 260
Training Epoch
Training Epoch
Training Epoch
(a) MIND.
(b) HM.
(c) Bili.
Figure 4: Comparison of convergence of IDRec and MoRec based on the SASRec framework on the validation data of the three datasets. We only report the convergence of SASRec since it far outperforms DSSM. Note that increasing epochs does not further improve IDRec's accuracy.
results on the testing set in Table by using the best validation checkpoints here. It can be seen that there is some performance gap between the validation set and testing set. In fact, notice in much recommendation literature, authors only use training and testing set without a validation set when choosing hyperparameters- such results might be questionable.
C IMAGE EXAMPLES AND DATASET COLLECTION
As shown in Figure randomly pick up some image examples from the ImageNet1 K dataset used for ME pre-traine preitraining and the two vision datasets HM and Bili used in our experiments.
For I dataset in Figure 5(a) there is wide variety of images from people's daily life and natural environments. Vision encoders such as ResNet and Swin Transformers are pre-trained this dataset to obtain representations of arbitrary images (He et al. 2016 Liu al. 2021).
14

---

Under review as a conference paper at ICLR 2023
(a) ImageNetIK dataset.
(b) HM dataset.
iPad
7.0
23 #aTT
JE R
H
T
D
STAY
(c) Bili dataset.
Figure 5: Image examples on HM and Bili. We have also shown some examples from ImageNet, which are used to train these vision ME.
Image:
0373 (3:0)
NO e EM Unde #7e:A
PS. M www. UUR1:
Text Description: (4M) (01 :8: 22 H.SA.....
Vicatints #
Views:
Label: Conscious choice
Comments:
Text Description:
Producer: (Anonymous for privacy)
Lined collar Denim Jacket
Publication time: 8-19
Price: $4.99
-WX-RON
6m115
(a) Item cases on HM. Each item has a picture, piece of text description, and price, and some items have "conscious choice" label for sustain- able material.
(b) Item cases on Bili. Every item has thumbnail, piece of text description, video views, video com- ments, produoer(anoyymouu for privacy), publication time and vidco duration
Figure 6: Item cases on HM and Bili datasets. We only use the raw image feature information to evaluate the performance of MoRec.
15

---

Under review as a conference paper at ICLR 2023
The HM dataset in Figure 5(b) contains clothing images from H&M, a clothing shopping platform. The images in HM dataset differ in the type, style and color clothes. During the pre-training process in ImageNetIK, the vision encoder often only needs to identify a few broad categories of clothes and does not need to distinguish the details. However, clothing detail has a critical evaluation criterion for users purchase. That might be the reason why E2E training is largely better than the two-stage freezing representations.
The Bili dataset in Figure 5(c) consists of thumbnails of videos in Bili, an online video recommen- dation platform. To build this dataset, we randomly crawled videos (with duration time less than 10 minutes) from 23 different video channels in Bili and recorded their public comment infor- mation of these items as interactions. These interactions occur from 2017 to 2022. We did not crawl or use information that would involve the privacy of interacted users, whose IDs have already been anonymous (We would be very happy to provide our datasets and codes during the paper discussion period since according to the submission guidance this makes these ma- terials only available to reviewers and AC). In total, we have collected over 2 million users and 150 thousand items with around 50 million interactions. However, performing research on such a large-scale dataset will require huge compute and time, thereby simply randomly draw 400K users with their commented items as the evaluation dataset, as shown in Table For this research, only the thumbnails represent the item since directly encoding the original videos is super computationally expensive. As shown in Figure 5(e) the images in Bili datasets contain various domain-specific knowledge (c.g., human-created video covers with complex semantics and heavy text insertion), which may confuse the recognition of visual encoders.
We also show the actual item cases of H&M and Bili platforms in Figure 6(a) and Figure 6(b) respectively. It can be found that the actual recommendation scene contains rich information. The image and other factors (e.g.. price, text description, etc.) together play a role in attracting the user's interactions. Clearly, it is impossible for MoRec to extract such features (e.g.. price and audio) from only a picture - the only input of an item encoder. Note that IDRee can indeed implicitly learn such features in latent factors by item similarity supervision (Koren et al. 2009). For example, it is recognized by the RS community that ID-based location recommender models (Hang et al. 2018) can learn some good distance information even if we do not feed them distance features.
D MOREC VS IDREC ON COLD-SC & NEW-SC
We report the results of IDRec and MoRec on COLD-SC and NEW-SC in Table Note that IDRec cannot serve new items and thereby are approximated by the simple random sampling strategy.
Table 7: HR @ 10 (%) of IDRec and MoRee for cold and new item recommendation. All results are evaluated based on the SASRec architecture.
Table 8: Pre-trained (PE) ME vs TES (ran- dom initialization of ME) regarding HR @ 10 (%) in the 50K datasets.
Dataset
Cold item
New item
Dataset
Encoder
TFS
Improv.
IDRcc
BERT
IDRee
BERT....
MIND-50K
BERT
15.04
14.35
-4.59%
MIND
0.0036
3.0637
0.0125
0.5899
HM-50K
ResNet50
2.74
3.26
118.98%
IDRec
Swin-B
IDRec
Swin-B
Swin-T
2.84
4.47
+57.39%
HM
Bili
0.3744
0.3551
0.6400
1.0965
0.0078
0.0115
0.0832
0.6846
Bili-0KK
ResNet50
Swin-T
1.07
1.08
1.20
1.46
+35.19%
+12.05%
E MORE RESULTS OF MOREC WITH PRE-TRAINED ME AND ITS TRAINING-FROM-SCRATII VERSION
We show a more detailed comparison of MoRec with pre-trained ME and its training-from-scratch (TFS) version. We report the convergence behaviors of Pre-trained ME vs TFS on the validation set in Figure 7 It can be found that MoRee with the pre- pre-trained ME have a faster convergence rate and a more stable training process. The performance of TFS is relatively poor and may fluctuate wildly during the convergence process.
16

---

Under review as a conference paper at ICLR 2023
35
(EFS
DERT
Swin-T
15
60
80
150
20 d0
160
60
100
00
60
10
30
160
Training
Spech
Tpech
Trakiig Racch
(a) MIND(B).
(b) HM(R).
(c) HM(S).
(d) Bili(R).
(e) Bili(S).
Figure 7: Convergence behaviors of pre-trained ME vs TES on the validation set. B, R, S denotes BERT, ResNet and Swin Transformer, respectively.
20
18220
18g 13000
BERT (FF5
Swin-T(TFS)
Resdiad
Swin-T(TFS)
BERT
ResNue50
Swin-T
Trainin Spech
Twihing Spech
Taining Spech
Traiing Eosch
Traiing Rocch
(a) MIND-50K(B)
(b) HLM-50K(R)
(c) HM-SOK(S).
(d) 50K)
(e) Bili- SOK.)..
Figure 8: Convergence behaviors of pre-trained ME vs TFS on the validation set with a smaller dataset by randomly drawing 50K users. Larger relative improvements can be observed by compar- ing with the above figure.
We also construct the smaller version datasets by randomly drawing 50K users from MIND, HM, and Bili. We report the recommendation results the testing set in Table g and the convergence behaviors on the validation set in Figure [ It can be seen that the advantages of pre-trained ME over TFS are more obvious on small datasets. MoRec achieve around 57% and 35% improvements with the pre-trained Swin-T on HM-50K and Bili-50K, respectively. However, we found that the pre-trained BERThase hase even worse than its TFS version on MIND-50K.
F SUPPLEMENTARY EXPERIMENTS
F.1 MIND DATASET WITH OTHER EVALUATION STRATEGY
The evaluation method we used for MIND differs from the data splitting strategy used in (An et al., 2019 Wu al. 2021a Yu et al. 2021). The MIND dataset contains both positive feedback and true negative feedback where items are exposed to the user but there is no observed user interaction. The above works used the exposed but no interacted items as true negative examples rather than the randomly sampled items. However, most recommender system datasets such as HM and Bili contain no item exposure information. For consistency, we treat all these recommender tasks as one-class collaborative filtering (Rendle et al. 2012 Pan al. 2008 99% cases in literature.
Meanwhile, we also conducted experiments in CANO-SC following the same experiment setting of (An et al. 2019 Wu et al., 2021a Yu et al. 2021). and report the result in Table 9 The result shows that MoRec significantly outperform IDRec in text recommendation as well.
F.2 INCREASING USER SEQUENCE LENGTH
We conduct additional experiments with sequence length of maximum 23 items in CANO-SC on the HM and Bili datasets, and report the convergence in Figure 10 We find that the results in such settings are consistent as reported before: IDRec and MoRec still show comparable recommendation accuracy.
17

---

Under review as a conference paper at ICLR 2023
44
8
5.0
NDCG@10 (%6)
4.5
36
HR@10, (%6
4.0
3.5
32
IDRec
IDRec
3.0
IDRec
BERT
ResNet50
2.5
ResNet50
15 Training Epoch
10
20
Swin-T
2.0
Swin-T
3
Swin-B
1.5
Swin-B
2
50
100
150
200
250
60 120
180 240 300 Training Epoch
Figure 9: Comparison of convergence of IDRec and MoRec on the MIND dataset under the same experiment setting as (An al.) 2019 Wu Yu et al. 2021).
Training Epoch
(a) HM-23L.
(b) Bili Bili-23L.
Figure 10: Comparison of convergence of IDRec and MoRec based on the SASRec framework on the validation data of HM and Bili datasets with sequence length of maximum 23 items.
ct al.
2021a la
G HYPER PARAMETERS AND TRAINING COST
For all methods, employ AdamW (Loshchilov & Hutter 2017
the default optimizer and find that the dropout rate set to 0.1 (i.e., removing 10% parameters) offers the optimal results on follow the common practice and perform
the validation set. Regarding other hyper-parameters extensive searching. For IDRec, we tune the learning rate y from (1e-3, 5e-4, le-4, 5e-5), , the em- bedding size from (64, 128, 256, 512, 1024, 2048, 4096), and batch size 1024 for DSSM and 128 for SASRec. For MoRec, we set d to 512 for both DSSM and SASRec, b to 512 and 64 for DSSM and SASRec respectively due GPU memory constraints. Given that ME (c.g.. BERT and ResNet) has already well pre-trained parameters, we use relatively smaller than other parts in the recommender model. That is, we search y from f1e-4, 5e-5, 1e-5) for the pre-trained ME networks, and set y to le-4 for other parts with randomly initialized parameters. Finally, tune the weight decay 3 from 10.1, 0.01, 0) both IDRec and MoRec. The effectiveness of such factors will be analyzed later in Appendix
For the MLPs (multilayer perceptron) used in DSSM, we initially set their middle layer size to well and search the layer number from {0, 5, but find that 0 (i.e., no hidden layers) always produces the best results. For the Transformer block used in SASRec, we set to 2 and the head number of the multi-head attention to 2 for the optimal results. All other ssperpparameters kept the same for IDRec and MoRec unless specified otherwise. We report details the pre-traines ME used in the experiments in Table 2 and all hyper-parameter details in the Table 12
Table 9: Network architecture, parameter size, and download URL of the pre-trained ME we used. L: number of Transformer blocks, H: number of multi-head attention, C: channel number of the hidden layers in the first stage, B: number of layers in each block.
Modality
Pre-trained model
Architecture
#Param
URL
BERT
Il=188
hitps://ugggingfaee face cul cufprajjwal L/bert tiny https:/huggingfaa gfaggggnnnnneegnooooo co/prajjwal /hert-small hups::/huggin glace comprajiwal
BERT
1,4, He512
29M
BERT medio
L=8, I1-12
mediumm hitps://hug gginglaee lace. corbert base-buceee uncased https:/huuging batp::/huge of lace www.boooook//oooooooooooooooooo. 125M
Text
BEKTb
L=12, H:768
109M
RoBERTa
huse
OPT
L=12, H=768
125M
ResNet18
C- 64, (-
12M
https:/download pytorch c. www..oo/g/ooooooooooooooooooow wtg.moodcls//essettt8888 8-5c 106coo 106ccdo,th
ResNet34
C=64, 6,,3,4,6,3)
22M
hitps://kowwlooe nloud pytorch wwwch.orp htwp://mmmmelleseeee44444444 335
Image
ResNet50
C- 6-
26M
brtps://dwnlood pytorch www.och..rg/moodelssseeeooo http///ooooes//eeeeet000... 19:8833377709
Swin-T
C= 2,
28M
https:/huggiingiee www.w//wcicwoooooooooooooommmmoe
7-224
Swin-R
(= 18,2)
https://ugggngff wwce www.iiioooot/////oooooooo patch4 window
MAFaue
http::/hugging
18

---

Under review as a conference paper at ICLR 2023
Table 10: Searching learning rate yM, .M and R. denote ME and the remaining modules of SAS- Rec, respectively.
Table 11: Searching same and different weight decay & combinations.
Searching 1tt und gM
7 1-mle-4, searching PM
Dataset
Encoder
g
gM
=9011
Datasct
Encoder
Male:5
5M=5e:5
Malle44
3=0
6=001
$=0.001
MIND
BERT
18.16
18.23
16.35
MIND
BERT
17.31
17.96
18.23
18.19
ResNet50
6.08
6.62
6.67
ResNet50
6.42
6.36
6.67
6.57
HM
Swin-1l
6.41
6.97
6.92
HM
Swin-T
6.51
5.88
6.81
6.97
MAE
6.26
7.03
6,96
MAB
6.30
5.72
6.63
7,03
ResNet50
2.33
2.84
2.93
ResNet50
2.76
2,58
2.93
2.74
Swin-T
2.29
3.18
3.01
Bili
Swin-T
2.73
3,18
MAE
2.78
3.05
3,18
MAEa
2.81
2.09
2.95
3.18
Table 12: The best hyper parameters and training cost. 7R. learning rate of recommendation net- work, y.M. fine-tune learning rate of modality encoder, gR; weight decay of recommendation net- work, gM, weight decay of modality encoder, d: embedding size, b: batch size, HR: HR @ 10 (%), ND: NDCG@ 10 (%), #Param: number of tunable parameters, FLOPs: computational complexity (we measure FLOPs for each model with b 1). Time/E: averaged training time for one epoch, "m" means minutes, BE: epoch with the best validation result, MU: GPU memory use, e.g., "V100- 32G(2)" means that we used 2 V100s with 32G memory. BERTh (2/12) means we only fine-tune the top 2 blocks out of the 12 blocks, similar to ResNet and Swin Transformer. Note that FI.OPs can vary signssigiiiiiii significantly depending on the implementation. For example, we can cache the representations in certain layers of the ME for all items during pre-processing, and avoid the forward&backwwad propagation on the bottom layers when lne-tuning only the top layers of ME.
Dataset
Mothod
gR
gM
d
b
HR
ND
4Param.
FLOPs
Time/E
BE
MU
GPU
IDRec
lo-4
0.1
128
17.71
9.52
47M
0.12G
120
3G
V100-32G(()
BERT iny
1c-4
5e-5
0.1
512
17.64
9.42
11M
0.63G
10m
105
4G
V100-32G(()
BERT and
1o-
le-
0.1
0
512
64
18.50
9.94
35M
1643
42m
84
13G
V100-32G()
BERT retuor
1e4
Se-5
0.1
0
512
18.39
9.88
18M
32G
83m
66
23G
V100-32G(1)
BERT....0
0.01
4G
V100-32G(1)
MIND (V12)) M
1o-4
0.1
512
64
14.68
8.00)
7M
0.146
3m
44
4G
V100-32G()
BERTs..
1644
le-4
0.1
0
512
64
17.57
9.66
21M
107G
96m
84
27G
V100-32G(1
BERT
1e-4
V100-32G(1)
BERT
1e-4
5e-5
0.01
0.01
$12
64
18.23
9.73
116M
107G
102m
75
52G
V100-32G12)
MART...MMLL.
1e4
5e-5
0.1
0
512
64
18.63
10.05
116M
107G
102m
81
52G
V100-32G(2)
5e-5
0.1
18
V100-320(2)
OPT 12M
16-4
0.1
132M
107G
100m
45G
V100-32G(2)
IDRee
5e-5
0.1
2018
128
6,84
4.01
114M
IG
1m
82
5G
V100-320(1)
ResNet18
le-4
1e44
0.01
0.01
512
6,30
3.36
18M
40G
95m
28
23G
V100-32G(L
ResNer34
16-4
0.01
0.01
512
6.40
29M
BIG
136m
29
30G
V100-12G())
ResNe150(014)
1c-4
0.1
512
64
1.03
2.12
7M
0
Im
80
4G
V100-32G()
ResNet50(14)
1o-4
1e-4
0.01
0.01
512
6.46
3.45
22M
91G
110m
29
13G
V V100-32G(1)
ResNe150(244)
1c-4
0.01
0.01
6.59
150m
V100-320(L)
ResNet50
le-4
1e-4
0.01
0.01
512
64
6.67
3.56
31M
91G
83m
35
80G
V100-32G(4)
HM
Swin-T2044
To-4
0.1
512
64
3.45
1,78
7M
0.076
64
46F
V V100-32G(1)
Swin-T(1:9)
1c-4
V100-320122
Swin-T(2/4)
1o-4
1c-4
0.1
0
512
64
6,50
3,70
33M
96G
104m
35
S4G
A100-32G(2)
Swin-T
le-4
Se-5
0.1
0
512
64
6.97
3,80
34M
96G
107m
35
157G
A100-400(()
Swin-B
le-
le-4
0.1
O
7.24
3.98
94M
333G
102m
33
308G
(00-400(8)
(011))
0.01
512
2.50
1,26
7M
0.076
2m
68
4G
(100-3ZG(1)
MAE
(0MM)MMM
le-4
0.1
512
64
2.79
1.10
7M
0.07G
2m
73
4G
V100-320(1)
MAEsa:
1o-4
le-4
0.1
512
7.03
3.83
9IM
966
86m
42
46G
V100-320(2)
MAE...MLMM.
le44
0.1
V100-32012)
IDRe
3e-5
0.1
10224
1288
3.113
1.63
72M
0.256
4m
191
4G
Y V100-32G(1
ResNett8
1c44
1e-4
0.01
0.01
512
64
2.50
1.24
18M
40G
78m
37
23G
V100-32G(1
ResNe134
1c-4
0.01
0.01
2.73
113m
V100-32G(L
ResNe150(0(4)
le-4
0
512
64
0.72
0.34
7M
0.09G
4m
51
4G
V100-32G(()
Re5Net50(
1e4
1e-4
0.01
0.01
512
64
2.83
1.41
22M
9IG
39m
34
1SG
V100-32G()
ResNo15(02244
1e-4
0.01
0.01
V100-32G(1)
ResNer50
1c-4
le-4
0.01
0.01
$12
64
2.93
L.45
MIM
91G
67m
88
S0G (100-32G(4)
Bili
Swin T(144)
Swin-T(004)
Ic-4
le-4
le-4
0.1
0.1
0
512
512
64
0.79
2.88
0.37
1.45
21M
7M
0.07G
96G
88m
3m
55
53
43G
4G
V100-320(L)
3100-32G(2)
Swin-T((/4)
1e-4
0.1
966
47
54G
1100-12G22
Swin-T
Io-4
Se-5
0.1
0
512
64
3.18
1.59
34M
964
86m
74
157G
A100-400(4)
Swin-B B
le-4
le-4
0.1
0
512
3.28
1.66
9MM
333G
81m
34
308G A100-400(B)
(112))
0.01
7M
0.07G
1G
V100-320(L)
MAE (LMM)
le-4
0.1
512
64
0.56
0.26
7M
0.07G
2m
57
JG
V100-32G(1)
MAP
164
1e4
0.1
0
512
64
3.18
1.60
92M
9643
78m
45
46G
V100-32G(2)
MAB MLM
1c-4
0.1
3.17
53
V100-320(2)
19

---

Under review as a conference paper at ICLR 2023
H LEARNING RATE AND WEIGHT DECAY SEARCHING
Separate hyperparameters for item and user encoders. MoRec typically consist of two modules: item encoders or a user encoder. Since item encoders or ME has been already well pre-trained in advance, it might be reasonable to set separate arper-parameters for ME and user encoder setting a smaller learning rate for ME. We report such results in Table TO and Table [T] As shown, MoRec in general yield better results by using different learning rates and weight decay for ME and user encoder. In particular, performs much worse on MIND when applying the same learning rate for all modules (e.g., 16.35 18.23). Similar behaviors are found for Swin Transformer the two visual datasets. Thereby, we draw the conclusion that searching hyperparameters (e.g., learning rate & weight decay) separately for ME & user encoder matters for the best MoRec results.
e.g..
I TS-BASED MOREC BY ADDING MORE MLP LAYERS
Suggested by reviewer JqDE, we apply several multi-layer perceptrons (MLPs) layers after the dense layer of the fixed representation extracted from the TS-based MoRec. We report the results in Table 13 As shown, we find that deep neural network can indeed improve the performance of TS; however, it is still obviously worse than IDRec and E2E-based MoRec.
Table 13: HR @ 10 (%) of E2E vs TS with additional MLP layers. "TS-DNN 6" denotes that TS- based MoRec with six additional MLPs layers. 'Improv.(ID) indicates the relative improvement over IDRec. 'Im))ov((TS) indicates the relative improvement over TS-based MoRec without MLPs.
Dataset
IDRec
Encoder
TS
TS-DNN
Improv.
Improv.
F2R
Improv.
Improv.
2
6
10
12
(ID)
(TS)
(ID)
(TS)
MIND
17.71
BERT
13.93
15.20
16.26
16.66
16.32
16.14
-5.93%
+19.00%
18.23
+2,94%
+30.87%
HM
6.84
ResNec50
Swin-T
4.03
3.45
4.64
4.46
5.40
5.28
5.55
3.40
5.40
5,38
5,02
-18.86%
-21.05%
+68.87%
+33.66%
6.97
6.67
+1.90%
-2.49%
+102.2%
+65.10%
Bili
3.03
ResNe150
Swin-T
0.72
0.79
1.40
1,23
181
1.62
2.10
1.28
1.95
1.64
1.24
-30.69%
46.53%
+165.8%
+125.0%
2.93
3.18
+4.95%
-3.30%
+302.5%
+306.992
J PROSS-DOMANE RECOMMENDATION EVALUATION OF MOREC
Suggested by reviewer jqDE& & tNVZ, to evaluate the transfer learn- ing ability of MoRec, we train MoRee on the three large-scale datasets used in this paper as the source domain and then per- form fine-tuning on three additional smaller datasets, i.e. the tar- get datasets. To be specific, we use Adressa (Gulla et al. (017). a Norwegian news recommendation datase 1T as the target dataset for textual MoRee pre-trained on MIND; Then, we use the Amazon clothing&shoes (Ni et al., 2019) dataset as the target dataset for vi- sual MoRee pre-trained on HM. At last, we use Kuaishou dataset (collected similarly as Bili) as the target dataset for visual MoRec pre-trained on Bili. For all these target datasets, we randomly sample 20,000 and 5,000 users for evaluation. Note that there is no over- lapped users and items between the target and source datasets. The statistics of the target datasets is shown in Table 14
Table 14: Characteristics of the target dataset for cross- domain recommendations.
Dataset
[R] fraet
20A
3,149
241K
Adressa
5K
1,633
60X
20K
14,348
129K
Amazon
5K
7,453
31K
20A
30,463
144K
Kuaishou
5K
10,544
34K
We report the convergence behavior of MoRec adapted on the target datasets with (w/) and without (w/o) pre-training in Figure [1] First, we indeed can with pre-training MoRec converges faster on all six datasets, in particular in the beginning epoches; It seems that MoRec with pre-training has more advantage on smaller target dataset (e.g. on Adressa 5K and Amazon 5K) than on larger target dataset (i.e. on Adressa 20K and Amazon 20K). Second, pre-trained MoRec does not show obviously better results than its trininn-frrmm from--cratchh version except on Adressa, which is
We translate this dataset from Norwegian to English by Google Translate. 12 hhtps://www..tr www.:/wwwww.uu.c.oomo comm////oomes
20

---

Under review as a conference paper at ICLR 2023
text recommendation dataset. The results surprise bit since we thought MoRec could perform substantially better with pre-training in advance. We guess there are two possible reasons: (1) our source dataset is still not large enough compared with the pre-trained data in NLP and CV (BERT and Swin Transformer are trained with huge compute and data sources). It is interesting to see whether MoRec could obtain larger improvements if we scale up the source data from 10x to 100x times larger. (2) The source and target datasets still have some gap (at least for image recommenda- tion) although they have the same modality. We believe this is a very interesting question for future research work. We will release all datasets and codes used for this experiment.
33-
2.5
E.
2.0
Hheto
30-
30-
3.55
24-
tra pre training
28-
pre-training pro traring
tra
pre training
20
10
20
30
26
20
40
8C
80
0.0
20
40
60
8G
100
Training Fpach
Training Epoch
Triining Fpoch
(a) Adressa (20K).
(b) Amazon (20K).
(c) Kuaishou (20K).
34
33
3.0
2.5
N10 1983
5 2.0
28-
pre-training wi pre training
pre-traring wi pre training
pre-training ml pre training
8.5
18
20
30
40
26
20
40
60
0.0
40
Traning Epach
Traning Epoch
Tra ning Epach
(d) Adressa (5K).
(e) Amazon (5K).
(I) Kuaishou (5K).
Figure 11: Comparison of convergence behavior of MoRec w/ and w/o pre-training.
K MOREC LITERATURE IN RECENT YEARS
We report the modality-only recommender systems (MoRec) literature in recent years in Table 15 (Note that, in this paper, we study the fair comparison of MoRec (use only modality features) vs IDRec, rather than ID + modality vs IDRec (which is not our focus)). As shown, so far there is no even one MoRec paper performing an explicit study between MoRec and its IDRec counterpart. Note that we do not aim to question these literature since the comparison study might not be a focus for these literature. Corrspondingly, we belive the comparison study of our paper novel.
Table 15: The modality-onke recommender systems (MoRec) literature in recent years.
Research
Feature
Fid22end
Fair comparison with IDRcc?
Reason
HASC (Wa et al_
2019b
vision
X
X
MoRec & IDRec used different network backbone for comparison
NRMS
20
text
No comparison with IDRcc
LSTUR
AB et ml.
2019
text
X
No compurison with IDRec
SAERS
[Toir 2f m.
2019
vision
X
MoRec & IDRcz used different netwoek hackhome for compurison
FIM
21.
text
compurison with IDRec
Proh-REII -
Pellra & Faruft
2020
text
X
X
No comparison with IDRee
KIM QLet
text
X
No comparison with IDRec
SEMI Zl
2012
text, vision, video
MoRee & IDRcc used different network hackhone comparison
MM-Roe
2021
text, vision
No comparison with IDRcc
MINDSim
No comparison with IDRce
Topic VAE
text
MoRce & IDRcc used different aetwork backbone for comparison
ACMLM Ni
2019
text
No comparison with IDRec compurison with IDRec
PLM-NewaRec
text
NewsBERT
(Wo et a.. ZurLrS
text
X
No comparison with IDRec
UNBERT Zhmin
No comparison with IDRe:
MTRee B puzzi
text
No comparison with IDRec
Speedy Feed
Xix efat 2012
text
No comparison with IDRcc
MINER
MoRec & IDRce used different network backbone comparison
Ours
text, vision
V
21


Reviews:
[{"review_id": "iiMmi1hZr0", "reviewer": "Reviewer_jqDE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "There are two different paradigms to represent the items in a recommendation setting: (i) IDRec which uses a unique identifier for each item (ii) MoRec which encodes the available content/modalities of the item (text description, images, etc.). IDRec has been dominating the recommender systems literature in the past decade due to its high flexibility. However, a recent trend in the recsys community is to use the recent advances in NLP and CV to build a powerful MoRec model. In particular, one of the promise of this approach is to develop of a general recommender system which can transfer its knowledge from a dataset to another. \nIn this paper, the authors propose an empirical comparison of the two approaches (IDRec vs MoRec). They compare two different model architectures (two-tower and sequential models) and several encoders (BERT, RoBERTa, Swin, etc.) on three datasets. \nThe main conclusion of this paper is that MoRec can be highly competitive with IDRec and benefits the recent advances in NLP and CV. \n\n", "summary_of_the_review": "The paper is very pleasant to read and well structured. The authors propose an empirical comparison of two paradigms (IDRec and MoRec). This comparison is useful for the recsys community since there is a recent trend in building a general recommender system. However, there is no technical novelty in the paper. ", "strength_and_weaknesses": "Strength. The authors propose an empirical comparison of MoRec and IDRec. The experiments are serious, exhaustive and well conducted. In particular, the authors chose to compare:\n- 2 model architectures: DSSM (two-tower model) and SASRec (sequential model)\n- several encoders including BERT, RoBERTa, ResNet and Swin\n- 3 datasets, including one new dataset (Bili) on video recommendation - MIND (text), H&M and Bili (images)\n- several learning strategies: end-to-end training, frozen encoders, etc.\nThis kind of empirical comparison is useful for the recsys community. \n\nWeaknesses. There is no technical novelty in the paper and the empirical novelty is somewhat limited. There is a recent trend in integrating the recent advances of NLP to build a general recommender system. It would be great to include some technical challenges to improve the technical part of the paper, or to quantify the transfer skills of the MoRec approach. Some results are expected. For example, the cold-start is a known limitation of the IDRec setting.\n\nComments and questions.\n- Chose one acronym between MoRec and CoRec.\n- It can be useful to clarify the differences between the learning techniques. Many of them are introduced incrementally and it can lead to some confusions.\n- Table 2. Split the table into two parts (one for DSSM and one for SASRec) with one improvement column for each architecture.\n- Table 3. I am not sure to understand what the numbers in the column New item/IDRec represent. IDRec cannot learn an item embedding if there is no example available in the dataset. Therefore, is the embedding used for evaluation correspond to the random initialization? \n- DT layer. End-to-end training is shown to perform better than two-stage training (but is very expensive to train). One possible explanation is that the obtained representation with the encoder is not well aligned with the recommendation model. The DT layer is only a dense layer in the experiments. Does a better DT layer (using a neutral network) can improve the results of the two-stage approach?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T_1JDpSDpi0", "reviewer": "Reviewer_j7gn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors revisited the topic: whether the modality based recommender models (MoRec) can exceed or be on par with the ID-only based models (IDRec) when item modality features are available? They examined this question from several perspective, and performed multiple experiments. Their results show that MoRec with standard end-to-end training is highly competitive and even exceeds IDRec in some cases. ", "summary_of_the_review": "The paper studies an important topic. The authors aim to compare the ID-based recommendation models and modality based recommendation models. They did the comparison in multiple fold and draw the conclusions based on their observations from multiple experiments. The logic is clear and the conclusion is helpful for creating recommendation models in industry. ", "strength_and_weaknesses": "The paper revisited an important question as they observe IDRec model are becoming more prevalent in industry. To compare it with MoRec models, they proposed several questions to examine the two models from diverse perspectives, and performed many experiments, then gave suggestion/ conclusion based on their observations. In section 8, they also provided some additional observations they had, which provided many practical suggestions especially helpful for people who consider leveraging pre-trained models in their MoRec models. \n\nQuestion:\nIn the IDRec model that the author used in their experiment, it is not clear to me are the ID embeddings randomly initialized and trained E2E in the big model, or there are some pre-trained embeddings used for the items?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Scq0MJ8kVT", "reviewer": "Reviewer_eDRW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper on recommender systems approaches the question of the utility of modality in recommendation. The authors carefully collected two large-scale datasets of multimedia. Experiments on two collected datasets and one public dataset verify that leveraging modality can benefit the recommendation performance on both sequential and non-sequential recommendation. The authors also provide some insights based on the comparisons among different feature-extraction methods (neural network models for textual input and vision input). ", "summary_of_the_review": "The paper proposes a general framework for leveraging modality for recommendation, but the technical contribution is limited. ", "strength_and_weaknesses": "Strengths.\n1. This paper is about recommender systems, which is one of the most successful applications of deep learning, and thus this paper well matches the scope of ICLR.\n2. The authors have put a lot of effort into collecting two large-scale datasets, which consist of both adequate behavioral logs and modality data. The public release of these two datasets, as promised by the authors, will benefit the community.\n3. The paper writing is good. The figures and tables are clearly presented and easy to understand.\n\nWeaknesses\n1. The technical contribution is limited. For an ICLR paper, more than just proposing a general framework is required (it is quite easy for recommendation engineers or researchers to propose the framework if they are required to utilize modality in recommendation).\n2. The experimental results are not fully explained and analyzed. For example, the performance drop in DSSM when modality is utilized is not well explained. I prefer to attribute it to the weakness of the DSSM model rather than the non-sequential setting.\n3. Some experiments are not helpful since the conclusions are apparent. For example, it is obvious that end-to-end training will have better performance than two-stage training. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8aah4DXRbl", "reviewer": "Reviewer_tNVZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper raises the research question of whether the modality-only based models (MoRec) can exceed or be par with the ID-only based models (IDRec). To answer the question, the authors study the performance of MoRec and IDRec under different scenarios, e.g., item cold-start and new item scenarios, and under different settings. Accordingly, they conducted rigorous experiments for item recommendations with two modalities based on two recommendation models. The useful conclusion is that MoRec can benefit from the modality representation techniques in NLP and CV and potentially improve recommendation performance.", "summary_of_the_review": "The paper provides extensive experiments to investigate several research questions about the contribution of the modality content to a recommendation. It contributes to the recommendation community but lacks novelty and originality and insightful conclusions.", "strength_and_weaknesses": "Strongness \n1. The paper is well-written and easy to follow. The motivation for investigating the comparison between MoRec and IDRec is intuitive and clear.\n2. Extensive experiments draw contributive conclusions and show researchers inspiring findings about MoRec. \n\nWeakness:\n1. Experiments based on DSSM and SASRec seem not sufficient since recommendation accuracy highly relies on the model details, e.g., network designs, score prediction by FFN or dot-product computation.\n2. Although extensive experiments demonstrate and conclude many useful findings, those conclusions are not surprising and insightful. The authors should provide more analyses and insights from the experimental results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2BwG0kr0E4B", "reviewer": "Reviewer_BBQN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\n\nThis paper proposes to compare Content-based RecSys, or called Modality-based RecSys in this paper, with ID-based RecSys. The methodology is to replace the item embeddings in IDRec with the pre-trained item encoder in ModalityRec where the item encoder is BERT for item texts and ResNet/Swin-T for item images. Note, the user embeddings are the same for both IDRec and ModalityRec. The paper investigates two neural architectures, one is tower-based DSSM architecture and the other is sequence-based SASRec. The experiments are conducted under three scenarios, i.e., canonical scenario, cold-item scenario, and new-item scenario.\n", "summary_of_the_review": "Comments:\n\nC1: Many choices are not given a reason, for example, the numbers 23 and 13 in the following statement, “For MIND, we select the latest 23 items for each user to construct the interaction sequence. For HM and Bili, we choose the 13 most recent interactions...”\n", "strength_and_weaknesses": "Strong & Weak:\n\nS1: It is interesting to see what happened when the item embeddings are replaced by pre-trained item encoders like BERT and ResNet.\n\nW1: The ModalityRec method proposed in this paper is actually equivalent to the IDRec variants by replacing the item embeddings with pre-trained item encoders like BERT. However, using BERT/ResNet as the item encoder to represent item content has been proposed in Ref [1][2][3]. As a result, the two components of ModalityRec, IDRec architecture + pre-trained item encoders, are both not new. This weakens the contributions of ModalityRec.\n\nReferences:\n\n[1] Zhang et al, UNBERT: User-News Matching BERT for News Recommendation, IJCAI’21.\n\n[2] Li et al, MINER: Multi-Interest Matching Network for News Recommendation, ACL’22 Findings.\n\n[3] Wu et al, MM-Rec: Multimodal News Recommendation, SIGIR’22\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject