VIP5: Towards Multimodal Foundation Models for Recommendation
Authors: Shijie Geng
Juntao Tan
Shuchang Liu
Zuohui Fu
Yongfeng Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal Foundation Model
Recommender Systems
Large Language Model
Parameter-efficient Tuning
Personalized Prompt

Abstract: Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other's advancements. With the recent development of foundation models, large language models have emerged as a potential general-purpose interface for unifying different modalities and problem formulations. In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable the processing of multiple modalities in a shared architecture for improved recommendations. To achieve this, we introduce multimodal personalized prompts to accommodate multiple modalities under a shared format. Additionally, we propose a parameter-efficient training method for foundation models, which involves freezing the P5 backbone and fine-tuning lightweight adapters, resulting in improved recommendation performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https://github.com/jeykigung/VIP5.

VIP5: Towards Multimodal Foundation Models for Recommendation
Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, Yongfeng Zhang Department of Computer Science, Rutgers University, NJ 08854, US 1sg1309, juntao.tan, shuchang.syt.lu,, zuohui.fu, yongfeng.zhang @rutgers.edu
Abstract
Historically, the technical developments for pro- cessing different types of information (such as personalization, visual, and textual) are mostly spread across different research communities. For- tunately, recent advances in Foundation Models (FMs) such as Large Language Models (LLMs) unfold promising route for building general- purpose models and unifying diverse modalities, so that one single architecture can handle visual, textual and personalized information at the same time, enabling the possible approach towards Arti- ficial General Intelligence (AGI) (Ge et al., 2023) and Artificial General Recommender (AGR) (Lin and Zhang, 2023). As pioneering work, GPT-3 (Brown et al., 2020) can perform in-context learn- ing, enabling it to solve brand-new problems given few-shot demonstration examples prompts. Sim- ilarly, CLIP (Radford al., 2021; Geng al., 2022d) maintains superior zero-shot generalization ability when shifting to an out-of-diittrbution vi- sual domain if provided with appropriate prompt. With more and more emergent abilities (Wei et al.. 2022b) revealed in foundation models, they be- come not only popular backbone to finetune downstream tasks (Alayrac et al., 2022; Sanh al., 2022; Wei et al., 2022a) but also effective train- ing scheme for unifying multiple modalities in a shared interface (Wang et al., 2022: Chen et al.. 2022; Cho et al., 2021; Jiang et al., 2022). Follow- ing the trend in language and vision domains, P5 (Geng et al., 2022c) and M6-Rec (Cui et al., 2022) put forward the concept of personalized foundation models for recommendation and propose pre- train LLMS instructional prompts accommo- date various recommendation tasks under shared model and training objective.
Computer Vision (CV), Natural Language Pro- cessing (NIP), and Recommender Systems (RecSys) are three prominent Al applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other's advancements. With the recent devel- opment of foundation models, large language models have emerged as a potential general- purpose interface for unifying different modali- ties and problem formulations. In light of this, we propose the development of multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable processing of multiple modalities in shared ar- chitecture for improved recommendations. To achieve this, introduce multimodal personal- ized prompts to accommodate multiple modal- ities under a shared format. Additionally, we propose parfmeter-efffcient training method for foundation models, which involves freezing the backbone and fine-tuning lightweight adapters, resulting in improved recommenda- tion performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https:// gi thub. com/ jeykiguuu eyykigugg/VIP5.
1 Introduction
With rapid growth, recommender systems have gradually become an indispensable element in peo- ple's daily lives. With more time spent on the Web, people reveal their interests through richer modali- ties than before. In response to the trend, current recommendation systems (Meng al., 2020; Hou et al., 2019; Zhang al., 2021a, 2017) consider more diverse contents when making recommenda- tion decisions to users.
While there large models for language (Raf- fel al., 2020; Brown et al., 2020), vision (Yu et al., 2022; Radford al., 2021), graphs (Ye al., 2023; Geng et al., 2022b) and recommendation (Geng et al., 2022c; Cui et al., 2022) domains

---

Sequential
3003
4406
5633
S634
help
Direct Rec Recommendation
Pick suttable from fullowing ing und wocemmend L90_2511
10264
VIP5
2317
Explanation Generation
Ramed the festure ward eone se, generate on repiorttion uwr.4t about thk produc:: Moxatbnn Proclads
small frannsight
Resiszance Band See with Doo Anchot Aville Stop. Bundie Chan, and Resisance Band Cacs
An example task scope of VIP5 covering three popular recommendation tasks. Based on multimodal personalized prompts (leff) that interleave language and visual tokens, VIP5 is able to transfer all task and all modalitics into a unified sequence format, and generates target outputs (right) according to certain task descriptions. VIPS treats large language models as a fixed general-purpose interface and finetunes extra visual and language processing layers to achieve the ability for handling various recommendation tasks.
Figure I:
separately, in this work, we take one step further and aim to unify the above foundation models to jointly process multi-modality information sources for personalization and recommendation. To this end, we follow the "Pretrain, Personalized Prompt and Predict Paradigm (P5)" for recommendation (Geng al., 2022c: Xu et al., 2023) and propose Multimodal Foundation Model (MFM) named VIP5 (Visual P5), which provides the following advantages for recommender systems: 1) VIP5 provides multimodal personalized prompts for sup- porting all modalities' connections to the recom- mendation foundation model. Specifically, to con- struct multimodal personalized prompts, VIP5 em- ploys mapping network to transfer features from other modalities into the corresponding tokens. By this step, multimodal features are projected to the same manifold space of the backbone foundation model. 2) The VIP5 framework provides the abil- ity of Parameter-effcient tuning rather than Pre- training in existing recommendation foundation models such as P5 (Geng et al., 2022c). Different from the pre-training step of P5 that updates all the parameters in the backbone foundation model which is impractical when the size of foundation model grows explosively VIP5 only finetunes a small proportion of extra lightweight adapter mod- ules during training while maintaining the large language model backbone fixed. 3) With the ability of multi modaliyy learning and parameter-efficiee tuning. VIP5 further improves the performance of recommendation foundation models with both less training time and less memory usage, making it easier train and deploy foundation models for recommendation. Overall, our key contributions
are outlined as follows:
We propose VIP5 framework unify CV, NLP. and RecSys foundation models and facilitate rec- ommendation with multimodal information.
We introduce multimodal personalized prompts to adapt multi-modalin information into a shared tokenized space with textual, visual and person- alization inputs.
We develop adapter-based parameter-efficient tuning for VIP5 to achieve better recommenda- tion performance and training efficiency. Based on the experimental results, VIP5 beats strong baselines on three task groups while sav- ing substantial training time and memory usage.
Related Work
Prompt Learning. Prompt learning (Liu et al., 2021) gradually emerges as a popular paradigm to control the behavior of large language models since it can effectively adapt pretrained model to downstream tasks in either zero-shot few-shot style. The success of GPT series (Radford et al., 2019; Brown et al., 2020) attracts the first wave of interests on the topic. The in-context learning capability of GPT-3 (Brown et al., 2020) inspires many efforts on automatic prompt search or genera- tion (Gao et al., 2021; Jiang et al., 2020; Shin et al., 2020; Zhang et al., 2022) to achieve higher-quality discrete prompts. However, it is naturally hard to optimize these approaches in discrete space. To solve this issue, soft prompt based approaches such as Prefix-Tuning (Li and Liang, 2021), Prompt- Tuning (Lester et al., 2021), CoOp (Zhou et al., 2022), and Visual- Prommpt Tuning (Jia et al., 2022)

---

are proposed leverage additional trainable con- tinuous embeddings as prefix to conduct finetuning on downstream tasks. While achieving better scal- ability and generalization ability, the learned soft prompts are more difficult to interpret than discrete prompts. To accommodatc all above merits, in- struction prompts that directly describe different tasks via natural language instructions are adopted by a lot of methods (Weller al., 2020; Wei et al., 2022a; Sanh et al., 2022; Aribandi et al., 2022; Mishra et al., 2022), highlighting significant im- provements on unseen tasks.
and then fuses different visual information through dual-gating module. JRL (Zhang et al., 2017) pro- poses Joint Representation Learning over multiple modalities for improved recommendation. Another stream of approaches focus on providing recom- mendations along with correlated visual explana- tions. These methods usually work in domains where visual information is important to user be- havior patterns, such fashion (Hou al., 2019; Verma et al., 2020; Chen et al., 2019), travel (Geng et al., 2022a), and food (Meng et al., 2020). Fur- thermore, several recent approaches have been pro- posed to discover the rich intra-item and inter-item semantic structures from multimodal contents to facilitate better item representations and thus en- hance recommendation performances (Zhang et al., 2021 Deldjoo 2022).
Large Recommendation Models. Motivated by the success of Large Language Models (LLMs), the RecSys community started to pay more attention to recommendation model's generalization ability and transferability (Li et al., 2023b). For example, inspired by the prompt learning paradigm, PETER (Li et al., 2021) and PEPLER (Li et al., 2022) pro- poses learn personalized continuous prompts to represent user and item IDs and generates natural language explanations to justify recommendations. In contrast, M6-Rec (Cui et al., 2022) converts all user behavior information to plain text sequences and feeds them into a Transformer encoder, and then designs a task-specific training loss for down- stream tasks and finetuming. Apart from previous efforts, P5 (Geng et al., 2022c) and OpenP5 (Xu al.. 2023) leverage not only instruction-basee finetuning on LLMs to represent personalized fields for users and items but also unifies various tasks via natural language instructions. Hence, P5 is able to unify various recommendation tasks into a shared mncoerr-decode architecture and joint training objective. P5-1D (Hua et al., 2023b) fur- ther explores different item ID creation methods for LLM-based recommendation models, such as se- quential indexing, collaborative indexing, semantic indexing, and hybrid indexing.
3 VIP5 Paradigm with Multimodal Personalized Prompts
We introduce the proposed VIP5 paradigm in this section. In Section 3.1, incorporate multimodal signals into personalized prompts. In Section 3.2, we elaborate how to conduct parameter-efficient tuning with adapters based on multimodal person- alized prompts.
3.1 Multimodal Personalized Prompts
A personalized prompt includes personalized fields for users and items (Geng et al., 2022c; Li et al.. 2022, 2023a), with formats ranging from ID num- bers detailed descriptions. In work, we de- velop foundation models as general-purposs in- terface to connect available modalities that could be helpful for cliciting user preferences. To facili- tate this end, we propose "multimodal personalized prompts". Technically, we consider textual, visual, and personalization information as three example modalities in multimodal personalized prompts (Figure 2).
Multimodal Recommendation. Current ap- proaches to multimodal recommendation can be divided into three categories. The most common usage is to leverage multimodal content as side information to assist recommendation decisions. For example, VBPR (He and McAuley, 2016) pro- poses using visual features supplement user feed- back and improve matching-based recommenda- tions. PiNet (Meng et al., 2020) proposes to cover more personalized visual preferences about users. It simultancously learns heterogeneous visual fea- tures with semantic and collaborative information
Given an item image I E Hx where H and W are the image height and width, we first adopt a visual encoder such as CLIP image branch (Radford et al., 2021) to extract its feature E IRd, where d, represents the visual feature dimension. To connect the image feature to other text-based tokens in personalized prompt, illus- trated in Figure 2(c), we design a mapping network with two linear layers to transfer the original im- age feature to image tokens: .I.....PP = f()). Then we append the image tokens to their corre-

---

Adopter A
P5 Encoder
Toar
Jick
TAX
(145
Adapter
TpTTA
SOITA
Decoder @
VPP Model
Fice the aulutta bamn 251
2317
CLIP Image Encoder
Mapsing
Vicual
Networke +
Mattroodal Personsitoed fond Prongt
An illustration the VIPS framework. VIP5 built on an deooder-decoder Transformer model that takes in textual inputs as well as image inputs to produce responses or make recommendation decisions. In the figure, fire symbol represents training with parameter update while a snow flake symbol stands for the frozen parameters.
Figure 2:
sponding item tokens to construct a multimodal personalized field MM:
reduces trainable parameters, decreasing training time and memory usage. Tuning few additional parameters addresses efficiency concerns when in- corporating visual tokens into text-based person alized prompts. More importantly, fine-tuning the entire backbone may cause over-fitting for easier tasks, whereas parameter-efficient tuning can lever- age both training efficiency and the power of large foundation models.
M : w1.--uu. Wm Pl......
(1)
item tokens image tokens
We create a collection of 29 multimodal personal- ized prompts covering three important task families sequential recommendation, direct recommenda- tion, and explanation. The full list of prompts is provided in Figure 8, and 10. Based on the collec- tion of multimodal personalized prompts, we use the multimodal personalized field M in E1. substitute the item field the prompt. It is worth noting that the prompts for sequential and direct recommendation usually contain more than one multimodal personalized fields.
Formally, if we denote the input sequence for the i-th layer of text encoder S [s1, sn). in traditional Transformer, S, will go through one self- attention block and a feed-forward network. While in VIP5, we insert adapters (Houlsby et al., 2019; Sung et al., 2022) in both the self-attention block and the feed-forward network, the exact position is after each module and before the LayerNorm (Ba et al., 2016). The whole process can be written as:
3.2 Parameter-efficient Tuning with Adapters Our VIP5 framework is shown in Figure 2(a). For tokenized multimodal sequence S, we first apply position encoding P and whole-word embedding W on S to help the model better recognize the absolute positions of input tokens and important user/item fields (c.g., "user 251" is split into 4 sep- arate tokens ["user"" the same whole- word embedding "(w7)"). Besides, we adopt an additional category embedding C to identify whether a token is textual or visual. Af- terwards, we feed the resulting sequence into the T--layered text encoder & and decoder 'D modules. Besides multimodal personalized prompts, propose parameter-efficient tuning using adapters for computation- and memory-efficient training. By inserting adapters into the foundation model backbone, freezing its parameters, and updating lightweight adapter modules, this strategy largely
S.+1 dd (FFN (Ar (Attention ( &. (2)
where W WK. Wy dddd, are weight matrices for projecting query, key, and value, respectively, dn = d/h is the dimensionality for each head. The Attention function is defined as:
"25", "1"1, but they share
QK Van.
Attention QQ K, V) = softmax
V.
(3)
Besides, FFN is a feed-forward module consisting of two fully-connected layers. A₁ and An are the feature adapters after the self-attention and feed- forward network. They are both bottleneck fully- connected layers with an activation function in be- tween. We can represent these adapters as:
= fup (o (fawn (S.))) + S.
(4)

---

where Sdown and Jup are the down-sampling and up-sampling layers of an adapter, and a is the GELU activation function (Hendrycks and Gim- pel, 2016). Similar to text encoder, we also adopt adapters between the cross-attention block and its LayerNorm layer inside text decoder.
Dataset
Clothing
Sports
Beauty
Toys
#Users
39,387
35,598
22,363
19,412
#ltems
23,033
18,357
12,101
11,924
#Reviews
278,677
296,337
198,502
167,597
#Photos
22,299
17.943
12,023
11,895
#Sparsity (%)
0.0307
0.0453
0.0734
0.0724
VIP5 utilizes the conditional token generation loss for all three recommendation tasks. After en- coding the input multimodal personalized prompts into a contextualized latent sequence with &, the text decoder D autoregressively predict next tokens conditioned on the already generated tokens y<j and the input text t. In summary, VIP5 adopts the following training objective to perform parameter- efficient tuning with adapters:
Table 1: Detailed statistics of the datasets used in our paper.
and ablation studies, namely Clothing, Shoes & Jewelry, Sports & Outdoors, Beauty, and Toys & Games. These datasets! contain user purchase records, reviews, item descriptions, and images. Table offers detailed dataset statistics.
Tasks and Metrics. In this paper, we cover three recommendation task groups: A) sequential rec- ommendation, B) direct recommendation, and C) explanation generation. We follow the same pre- processing steps and train//alidation/tess splits as in (Geng al., 2022c). For sequential recommen- dation, the last and second last items each user's interaction history are adopted as test and valida- tion ground-truths, with remaining items as training data. For direct recommendation we use sequen- tial recommendation's 's train/validation/tes. splits to generate 100 candidate lists in (Zhao et al., 2022). For explanation generation, we adopt an 8:1:1 random split and extract rating explanations using the Sentires library (Zhang et al., 2014).
y
Co -
log Po (y, Yej.t).
(5)
After training, we perform inference with VIP5 based on given multimodal personalized prompts. For sequential and direct recommendation task groups, we create a list of candidate items for rec- ommendation via beam search. For explanation task group, we simply apply greedy decoding for text generation.
4 Experiments
We evaluate sequential and direct recommenda- tion task groups using Hit Ratio (HR @ k) and Nor- malized Discounted Cumulative Gain (NDCG@K) @k), while explanation generation tasks assessed ing text generation metrics like BLEU and ROUGE. In all tables, bold numbers indicate the best ap- proach for each metric.
In this section, we provide the performance com- parison between the VIP5 framework and repre- sentative approaches for different task groups. We conduct extensive experiments and ablation studies explore the following research questions:
RQI: Does the proposed parameter-efficiest VIP5 framework perform well when compared with baseline methods across the three task groups?
Implementation Details. We utilize the pre- trained P5-small checkpoint as VIP5's backbone, as it often outperforms P5-base (Geng et al., 2022c), VIP5's encoder and decoder consist of 6 Trans- former blocks, 512-dimension embedding size, and 8 attention heads. To process visual informa- tion, we use CLIP's (Radford al., 2021) image branch as VIP5's visual encoder and pre-extract image features. Similar to P5, we employ the Sen- tencePiece (Sennrich et al., 2016) tokenizer with a 32, 100 vocabulary size to generate sub-word input tokens. By default, the mapping network serves as the image tokenizer in VIP5 and the number of image tokens is set to 2, while the adapters have reduction factor of 8 for the bottleneck dimension. For each task group, all multimodal personal-
RQ2: When conducting parameter-efficient tun- ing, which parts should we insert adapters and perform finetuning? In addition, will different adapter reduction rates affect the performance and efficiency of VIP5?
RQ3: Does visual information play an important role in multimodal personalized prompts? What if we change the number of image tokens and the type of visual encoder?
4.1 Experimental Setups
Datasets. We employ four real-word datasets collected from Amazon platform for experiments
'htt://jimcuuley.usd..c http:////aaaaaaaaaaoaiiiinsasssa

---

Sports
Beauty
Methods
HR@5
NDCG@5
HR@10
NDCG@ 10
HR@5
NDCG@5
HR@
NDCG@110
HGN
0.0189
0,0120
0.0313
0.0159
0.0325
0.0206
0.0512
0.0266
SASRec
0,0233
0.0154
0.0192
0.0387
0.0249
S*RRec
0.0251
0.0161
0.0385
0.0204
0.0387
0.0244
0.0647
0.0327
P5 (A-3)
0.0272
0.0169
0,0361
0.0198
0.0503
0,0370
0.0659
0.0421
VIPS (A-3)
0.0412
0.0345
0,0475
0,0365
0,0556
0,0427
0.0677
0,0467
P5 (A-9)
0,0258
0.0159
0,0346
0.0188
0.0490
0.0358
0,0646
0.0409
VIPS (A-9)
0,0392
0.0327
0.0456
0.0347
0.0529
0.0413
0.0655
0.0454
Clothing
Toys
Methods
HR@5
NDCG@5
HR@ 10
NDCG@1 10
HR@5
NDCG@5
HR@ 10
NDCG@10 10
HGN
0.0107
0.0071
0.0175
0.0092
0.0321
0.0221
0.0497
0.0277
SASRec
0.0107
0.0066
0.0194
0.0095
0,0463
0.0306
0,0675
0.0374
S*-Ree
0.0076
0.0045
0.0135
0.0063
0,0443
0.0294
0,0700
0.0376
(A-3)
0.0478
0.0376
0.0554
0.0401
0.0655
0.0593
VIPS (A-3)
0.0603
0.0564
0,0632
0,0873
0.0662
0.0577
0.0749
0,0604
P5 (A-9)
0.0455
0.0359
0.0534
0.0385
0.0631
0.0547
0.0701
0.0569
VIP5(AA-)
0,0569
0,0531
0,0597
0.0540
0.0641
0.0556
0,0716
0.0580
Table 2: Performance comparison on sequential recommendation.
Sports
Beautyy
Methods
@R@1
HR@5
NDCG@5
HR@10
NDCG@ 10
@R@1
@R@5
NDCG@5
HR@ 10
NDCG@ 10
BPR-MF
0.0314
0.1404
0.0848
0.2563
0.1220
0.0311
0,1426
0.0857
0,2573
0.1224
BPR-MLP
0.0351
0.1520
0.0927
0.2671
0.1296
0.0317
0,1392
0.0848
0.2542
0,1215
VBPR
0.0262
0.1138
0.0691
0,2060
0.0986
0.0380
0.1472
0.0925
0,2468
0.1245
P5 (B
0.0574
0.1503
0.1050
0.1276
0.0601
0.1611
0.1360
VIPS (B-5)
0,0606
0.1743
0.15
0.2539
0.1441
0,0580
0,1598
0,1099
0.2306
0.1327
P5 (B-8)
0.0567
0,1514
0.1049
0.2196
0,1269
0.0571
0,1566
0.1078
0.2317
0.1318
VIPS (B-8)
0.0699
0.1882
0.1304
0,2717
0.1572
0.0615
0.1655
0.1147
0.2407
0.1388
Clothing
Toys
Methods
HR@1 1
HR@5
NDCG@5
HR@ 10
NDCG@ 10
HR@ 1
HR@5
NDCG@s
6R
NDCG@
BPR-MF
0.0296
0.1280
0.0779
0,2319
0.1112
0.0233
0.1066
0.0641
0,2003
0.0940
BPR-MLP
0.1384
0.2327
0.1161
0,0252
0.1142
0.0988
VBPR
0,0352
0.1410
0.0877
0.2420
0,1201
0,0337
0.1294
0,0808
0.2199
0,1098
P5 (B-5)
0.0320
0.0986
0.0652
0.1659
0,0867
0,0418
0,1219
0,0824
0.1942
0.1056
VIPS (B-5)
0,0481
0.1287
0.0890
0.1992
0.1116
0.0428
0.12
0.0833
0. 1906
0.1055
P5 (B-8)
0.0355
0.19
0.0688
0.1722
0,0912
0.0422
0.1288
0.0858
0,2041
0.1099
VIPS (B-8)
0.1544
0.1058
0.2291
0.1297
0.0433
0.1301
0.1110
Table 3: Performance comparison on direct recommendation.
ized prompts except the last are used to train VIP5. Prompts A-3/A-9, B-5/B-8, and C-3/C-12 used for evaluation purpose, with A-3, B-5, C-3 testing model performance under seen prompts and A-9, B-8, C-12 under zero-shot unseen prompts. VIPS is trained for 10 epochs with a batch size of 36 on four NVIDIA A100 GPUs, using learning rate of 1 X 10-3 and AdamW (Loshchilov and Hut- ter, 2018) optimizer. As multimodal personalized prompts contain more image tokens, we set input token's maximum length to 1024. During infer- ence, beam size B is set 20 for sequential and direct recommendation tasks that require generat- ing a list of candidate items.
text-based personalized prompts via autoregressive language modeling loss and performs inference using greedy decoding or beam search gener- ate outputs. Additionally, we compare with task- specific approaches. For sequential recommenda- tion, baseline methods include HGN (Ma et al., 2019), SASRec (Kang and McAuley, 2018), 8), and $ ReeRec (Zhou et al., 2020). For direct recommen- dation, we compare with BPR-MF (Rendle et al., 2009), BPR-MLP, and VBPR (He and McAuley, 2016). For explanation generation, we inherent the baselines of P5: Attn2Seq (Dong et al., 2017). NRT (Li et al., 2017), and PETER (Li al., 2021). When providing a hint feature word as input, PE- TER becomes its variant PETER+, which we also use as an explanation generation bascline.
To make performance comparisons, we consider collection of baselines for each task group. For all three task groups, we include P5 (Geng et al., 2022c) as baseline com- pare with existing foundation models for recom- mendation. P5 pre-trains all tasks with predefined
Comparison Baselines.
4.2 Performance on Task Groups (RQ1)
In this section, we conduct parameter- efficcient tun- ing for VIP5 on multimodal personalized prompts

---

Sports
Beauty
Methods
BI.UFA
ROUGEI
ROUGE2
ROUGEL
BLUEE
ROUGEI
ROUGE2
ROUGEL
Attn2Seq
0,5305
12,2800
1.2107
9.1312
0,7889
12,6590
1,6820
9,7481
NKT
0,793
11,0723
1.1304
7.6674
0,8295
12.7815
1.8543
9.9477
PETER
0.7112
12.8944
1.3283
9.8635
1.1541
14.8497
2,1413
11.4143
P5 (C-3)
0.6212
1,0230
2.0761
VIP5 (C-3)
1,6639
14,8628
2.1012
11.1059
1,2850
17,7492
2.3482
12.9170
PETER+
2,4627
24.1181
5.1937
18.4105
3.2606
25,5541
5.9668
19.7168
(C-12)
1.3144
22.9182
4,9976
17.1976
1.6313
24.6267
4,9623
18.6423
(C-12)
18.6610
2,8390
26.0513
Clothing
Toys
Methods
BLUE4
ROUGEI
ROUGE2
ROUGEL
BLUE4
ROUGEI
ROUGE2
ROUGEL
Atln2Seq
0.6296
11,4888
1.2558
9.0429
1.6238
13.2245
2,9942
10.7398
NKT
0,4599
10.1480
0,9720
8.2434
1,9084
13.5231
3,6708
11.1867
PETER
0.7204
12.1836
1,9861
14.2716
3.6718
11.7010
P5 (C:3)
0.7569
12.2833
1.8116
9.6023
1.4522
12,6100
3.8144
10.
VIP5 (C-3)
1,1904
14,1685
2.0308
10.8488
2.3241
15.3009
3.689)
12.0421
PETER+
36204
28.4342
7,7994
22 22,4059
4.7919
28, 3083
9.4520
22,7017
(C-12)
1.8811
27.7922
7,3203
21,5462
2,62166
27. 27.8984
VIP5 (C-12)
3.2581
28.9059
8.5168
22,8807
3,9293
28,9225
9,5441
23.3148
Table Performance comparison on explanation generation (numbers are in percentage %)
C-12
3.10
2.00
0.30
0.33
D100
0.00
02/000
0.31 Typ
0.00
Figure 3: Performance comparison between text-based prompt and multimodal prompt.
from all the three task groups. For each task group, we select one seen and one unseen prompt for eval- uation. Performance comparisons with baselines are presented in Table 2, 3. and 4.
plied to evaluate all methods under hint feature word setup, while Prompt C-3 targets at direct explanation generation with only the given user- item pair. The experimental results indicate that VIP5 outperforms other baselines when equipped with the multimodal personalized Prompt C-3. For Prompt C-12, VIP5 achieves superior performances than P5 across all datasets in terms of all metrics and has the highest ROUGEI, ROUGE2, ROUGEL scores of the four experimental datasets.
Sequential Recommendation. As shown in Ta- ble 2, we adopt Prompt A-3 and Prompt A-9 to evaluate the performances of different approaches. From the table, we can see that VIP5 is able to achieve better performances than all sequential rec- ommendation baselines on all the four experiment datasets, among which a relatively large gap can be observed on Sports and Clothing datasets. The re- sults show that our parameter-efficient tuning strat- egy works effectively on the sequential recommen- dation task group.
4.3 Parameter-efficien Tuning (RQ2)
In this section, we discuss how to conduct parameter-efficient tuning with adapters to show the impact of different tuming choices.
Direct Recommendation. For the direction rec- ommendation task group, we evaluate different methods using Prompt B-5 and Prompt B-8 as input multimodal personalized prompts. Table 3 presents the performance comparison, showing VIP5 out- performing all baselines on Sports. While VIP5 achieves marginally lower HR@ 10 on Toys, Beauty, and Clothing datasets, still surpasses all basclines on other metrics.
How conduct parameter-efficient tuning. We first try three fine-tuning approaches: 1) inserting adapters Transformer's 's self-attention blocks and only fine-tuning them, 2) fine-tuning adapters in both self self-attention and cross aooss-attention blocks, 3) fully fine-tuning all parameters. For this ablation, conduct experiments Toys with ResNet-101 101 visual features, reduction rate of 8, and sin- gle image token in multimodal prompt. Figure 4 demonstrates that fine-tuning adapters all atten- tion blocks is necessary to achieve better (Prompt C- 12) or comparable (Prompt A-9 & B-8) results
Explanation Generation. Table illustrates the performance comparison for explanation genera- tion task group. In the table, Prompt C-12 12 are ap-

---

1.20
3.66
2.00
1,30
2.66
1.50
11:33
1.46
0.00
1.00
0.02
0604
1,40
2.03
0.51
0403
1,90
A-B
Seit-Alth
Aros
Figure 4: Performance comparison among only activat- ing adapters self-attention blocks, both self-attention and cross-attention blacks, and full finetuming.
Figure 5: Ablation on the downsample reduction rate.
0.03
0.27
2.00
0.07
with full fine-tuning. Morcover, Table 5 shows that the former saves 21.2% time and 18. 1.1% % mem- ory usage during training compared the latter, highlighting VTP5's effectiveness and efficiency. On adapter reduction rate. The reduction rate is
0.05
A-5
8-5
C-12
1mng
Plng
3lmg
51mg
Figure 6: Ablation on the image token number.
an important hypeer-paammtee for adapters. When decreasing the reduction rate, the hidden dimension of bottleneck layers will increase correspondingly. resulting in higher percentage of trainable param- eters. We select five different values of reduction rates and perform all experiments with ResNet-10 visual features and a single image token in multi- modal prompt. Figure 5, we can see that and are suitable reduction rates for all the task groups.
0.90
0.04
0.04
0.43
8:088
0.03
0.00
FONT01
VS B802
WTL//14
Figure 7: Ablation on the visual encoder rype.
Figure 6, enabling 5 image tokens in multimodal personalized prompt achieves the best performance on Prompt A-9 and Prompt B-8, while image tokens perform the best for Prompt C-12. However, longer visual prompt results in more training time (e.g.. 5 image tokens take 60.8% more time than 2 image tokens). Therefore, we choose 2 image tokens default setting considering the trade-off. On visual encoder type. The visual encoder type is another factor influencing multimodal person- alized prompt representation. We explore vari- ous CLIP visual branch architectures: ResNet50, ResNet101, ViT-B/32, ViT-B/16, ViT-L/14 (in an ascending order of visual encoder ability according to CLIP (Radford et al., 2021)). All experiments are performed on Toys with a reduction rate of 8 and a single image token. The results are reported in Figure 7. Similar to our previous conclusions, vi- sual information matters most for direct recommen- dation, with continuous performance gains when using better visual encoders. However, for sequen- tial recommendation and explanation generation, better visual encoders do not always improve per- formance. This is most likely because the purchase sequence is more crucial than visual information for predicting the next item in sequential recom- mendation, leading to similar performances under different visual encoders. For explanation genera-
4.4 Ablation on Visual Components (RQ3)
this section, we aim to explore whether visual information matters for different task groups. We also estimate the influence of the number of image tokens and the visual encoder type.
Text-based
multimodal personalized prompts. To compare text-based and multimodal personalized prompts, we set the number of image tokens to and I, respectively, and conduct experi- ments on all four datasets with a reduction rate of 8 and ResNet-101 visual features. Figure shows that introducing visual signals into personalized prompts improves all datasets for direct recommen- dation task group (Prompt B-8). This is in line with our expectation that an item's visual appearance significantly influences people's 's choices. For se- quential recommendation, visual information does not bring obvious performance improvements, indi- cating that the purchase sequence itself is more sig- nificant for predicting next items. For explanation generation, visual information positively impacts all datasets, especially for the Toys dataset.
On the number of image tokens. To examine the influence of the number of image tokens, select four different numbers (1, 2, 3, 5) and conduct ad- ditional experiments on Toys with reduction rate and ResNet-101 visual features. According to

---

Methods/Metrics
Time/Epoch
Trainable Param.
Memory Usage
is an important aspect for further investigation. 4) Efficiency of LLM: Efficiency an important fac- tor for LLMs to gain practical applications in real- world systems, because the latency should be lim- ited to small amount of time when delivering ser- vices to users. In this work, we have made an initial attempt to improve LLM efficiency by proposing the parameter- tuning approach. In the fu- ture, it is important to investigate the efficiency of LLMs on various stages of the pipeline, such as the effieney of pre-training, fine-tuning and prompt- based inference (Li ct al., 2023a). conclusion, addressing these limitations can pave the way for improved multimodal foundation models and more effective recommendations across various applica- tions and domains.
Self-Attn
10.55
2.97
27.4
Self-& Cross-rm
3.58
Tull (PS)
14.08
100
35.6
Table 5: Comparison of different training strategies in terms of trainable parameter (%). training time (min), and memory usage (GB) on the Toys dataset.
tion, hint words significantly influence generated sentences, and the compatibility between hint word and visual embedding varies across different visual encoders. However, VIP5 is still better than the best baseline under most visual encoders.
5 Conclusions
This paper presents VIP5, parameter-efficient multimodal foundation recommendation model uni- fying vision, language, and personalization in- formation. We design multimodal personalized prompts to integrate visual signals with text and personalization information, enhancing recommen- dation across diverse modalities. Our parameter- efficient tuning strategy updates small propor- tion of adapters, achieving a better trade-off be- tween recommendation performance, training ef- ficiency, and memory usage. Through extensive experiments, we show the effectiivnesss of our VIP5 framework and show that multimodality informa- tion helpful for various recommendation tasks. Future work includes further scaling up the back- bone model, incorporating even more modalities, and exploring improved prompt strategies, such as chain-of-thought prompting.
A Acknowledgement
This work was supported in part by NSF 1910154, 2007907, 2046457 and 2127918. Any opinions. findings, conclusions, or recommendations ex- pressed in this material are those of the authors and do not necessarily reflect those of the sponsors.
References
Jean- Bran-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning, arXiv preprint arXiv:2204. 14198.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao. Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2022. Ext5: Towards extreme multi-task scaling for transfer learning, In International Confer- Learning Representations.
Limitations and Future Work
Despite the promising results and advantages offered by our Multimodal Foundation Model (MFM), there are several limitations that need to be addressed 1) Bias and fairness: VIP5 relies on the quality and diversity of training data. Existing biases may lead biased and unfair recommen- dations. Future work could explore methods to mitigate biases, improve data representativennss. and promote fairness of LLMs for recommendation (Li and Zhang, 2023; Hua et al., 2023a). 2) Model transparency and interpretability: VIP5 lacks in- herent transparency, which can hinder users' trust in recommendations. Future work will aim to en- hance transparency and explainability for VIP5- generated recommendations. 3) Scalability to other modalities: Extending the VIP5 framework to other modalities, such as audio or video, remains a chal- lenge. Incorporating these modalities efficiently
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint 1607.0645
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In NewrIPS.
Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton, 2022. Pix2seq: A language model-

---

ing framework for object detection. In International Conference on Learning Representations.
Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. 2022d. Hielip: Contrastive language-image pretraining with hierarchy-aware at- tention. In The Eleventh International Conference on Learning Representations.
Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang. Yixin Cao, Zheng Qin, and Hongyuan Zha. 2019. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommenda- tion. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 765-774.
Ruining He and Julian McAuley. 2016. Vbpr: visual bayesian personalized ranking from implicit feed- back. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian error linear units (gelus). arXiv preprint 1606v:160.00844
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying vision-and- aasionaand-lagguage tasks via text genera- tion. In Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pages 1931-
Min Hou, Le Wu, Enhong Chen, Zhi Li, Vincent Zheng, and Qi Liu. 2019. Explainable fashion rec- ommendation: A semantic attribute region guided approach. arXiv preprint arXiv: 12862.
1942. PMLR.
Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. M6-rec: Generative pretrained language models are open-ended recommender sys- tems. arXiv preprint arXiv:2205.0800 08088.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter- transfer learning for nip. In In- ternational Conference on Machine Learning, pages 2790-2799 PMLR.
Yashar Deldjoo, Tommaso Di Noia, Daniele Malitesta, and Felice Antonio Merra. 2022. Leveraging content- style item representation for visual recommendation. In European Conference Information Retrieval, pages 84-98 94.92. Springer.
Wenyue Hua, Yingqiang Ge. Shuyuan Xu, Jianchao Ji, and Yongfeng Zhang. 2023a. UP5: Unbiased Foun- dation Model for Fairnes--wware Recommendation. arXiv:2305.1 12090.
Li Dong, Shaohan Huang. Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. 2017. Learning to generate product reviews from attributes. In EACL.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. 2023b. How to Index Item IDs for Recommendation Foundation Models. In Proceed- ings of 1st International ACM SIGIR Conference on Information Retrieval in the Asia Pacific (SIGIR-AP).
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACI-IICNLPP.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. 2022. Visual prompt tuning. arXiv preprint arXiv2203.12119.
Yingqiang Ge, Wenyue Hua, Kai Mei. Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang. 2023. OpenAGl: When L.M Meets Domain Experts. In Proceedings of the Thirty-Seventhy An- nual Conference on Neural Information Processing Systems (NeurIPS).
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yonggiang Dou, Yanjun Chen, Li Fei-Fei, An- ima Anandkumar, Yuke Zhu, and Linxi Fan. 2022. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210003094
Shijie Geng, Zuohui Fu, Yingqiang Ge, Lei Li, Gerard de Melo, and Yongfeng Zhang. 2022a. Improving personalized explanation generation through visual- ization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume Long Pupers), pages 245-255.
Zhengbao Jiang. Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Shijie Geng, Zuohui Fu, Juntao Tan, Yinggiang Ge, Ger- ard De Melo, and Yongfeng Zhang. 2022b. Path language modeling over knowledge graphsfor ex- plainable recommendation. In Proceedings of the ACM Web Conference 2022, pages 946-955.
Wang-Cheng Kang and Julian McAuley, 2018. Self- attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM). pages 197-206. IEEE.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-effict prompt tuning. In EMNLP.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022c. Recommendation as
language processing (rip): A unified pretrain, per- sonalized prompt & predict paradigm (p5). In Pro- ceedings of the Sixteenth ACM Conference on Rec- ommender Systems.
Lei Li, Yongfeng Zhang, and Li Chen. 2021. Person alized transformer for explainable recommendation. In Proceedings of the 59th Annual Meeting of the

---

Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 4947-4957.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMI.R
Lei Li, Yongfeng Zhang, and Li Chen. 2022. Person- alized prompt learning for explainable recommenda- tion. ACM Transactions on Information Systems.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog.
Lei Li, Yongfeng Zhang, and Li Chen. 2023a. Prompt Distillation for Efficient LLM-based Recommenda- tion. In Proceedings of the 32nd ACM International Conference Information and Knowledge Manage- ment.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, 2020. Exploring the limits of transfer learning with a unified text-to-teed transformer. Journal of Machine Learning Research. 21(140):1-67.
Lei Li, Yongfeng Zhang, Dugang Liu, and Li Chen. 2023b. Large language models for generative rec- ommendation: survey and visionary discussions. arXiv:2309.01157.
Piji Li, Zihao Wang. Zhaochun Ren, Lidong Bing, and
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Thieme. 2009. Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Un- certainty in Artificial Intelligence, UAI '09, page 452-461, Arlington, Virginia, USA. AUAI Press.
Wai Lam. 2017. Neural rating regression with ab- stractive tips generation for recommendation. In Pro- ceedings of the 40th International ACM SIGIR con- ference on Research and Development in Information Retrieval, pages 345-354,
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACI.
Victor Sanh, Alhert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Yunqi Li and Yongfeng Zhang. 2023. Faimess of Chat- GPT. arXiv:2305.1856%
Guo Lin and Yongfeng Zhang. 2023, Sparks of Arti- ficial General Recommender (AGR): Experiments with ChatGPT. Algorithmss 16(9).
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint 10Xiv:2107. 13586.
Ilya Loshchilov and Frank Hutter. 2018 Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1225.
Chen Ma, Peng Kang, and Xue Liu. 2019. Hierarchical gating networks for sequential recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 825-833.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Elliciting knowledge from language models with tomatically generated prompts. In EMNLP.
Lei Meng, Fuli Feng. Xiangnan He, Xiaoyan Gao, and Tat-Seng Chua. 2020. Heterogeneous fusion of se- mantic and collaborative information for visually- aware food recommendation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1460-3468.
Yi-Lin Sung. Jaemin Cho, and Mohit Bansal. 2022. Vl-adapter: Parameter-efffcient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 5227-5237.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. 2022. Reframing instructional prompts to gptk's language. Findings of the Association for Computational Linguistics: ACL 2022.
Dhruv Verma, Kshitij Gulati, Vasu Goel, and Rajiv Ratn Shah. 2020. Fashionist: Personalising outfit recom- mendation for cold-start scenarios. In Proceedings of the 28th ACM International Conference on Multi- media, pages 4527-4529.

---

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li. Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying ar- chitectures, tasks, and modalities through simple sequence-to-sqquence learning framework. In Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23318 --38-00 PMLR.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXi::2210.03493.
Wayne Xin Zhao. Zihan Lin, Zhichao Feng, Pengfei Wang. and Ji-Rong Wen. 2022. A revisiting study of appropriate offline evaluation for top-n recommenda- tion algorithms. ACM Transactions on Information Systems, 41(2))1-41.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022a. Finetuned language models are zero-shot learner. In International Con- ference Learning Representations.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022, Learning prompt for vision- language models. International Journal of Computer Vision, pages 1-12.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Deam, and William Fedus. 2022b. Emer- gent abilities of large language models. Transactions Machine Learning Research.
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang. Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020. S3-rec: Self-supervised learning for sequential recommendation with mutual informa- tion maximization. In Proceedings of the 29th ACM International Conference on Information Knowl- edge Management, pages 1893-1902.
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew Peters. 2020. Learning from task de- scriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361-
Shuyuan Xu. Wenyue Hua, and Yongfeng Zhang. 2023. OpenP5: Benchmarking Foundation Models for Rec- ommendation. 2r36.v2300.... 1016.
Ruosong Yc, Caiqi Zhang, Runhui Wang. Shuyuan Xu, and Yongfeng Zhang. 2023. Natural Language is All a Graph Needs. arXiv:2308.07334.
Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Ye- ung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text founda- tion models. arXiv preprint arXiv:2205.01917
Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang. and Liang Wang. 2021a. Mining latent structures for multimedia recommendation. In Pro- ceedings of the 29th ACM International Conference on Multimedia, pages 3872-3880.
Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Mengqi Zhang, Shu Wu, and Liang Wang. 2021b. Latent structures mining with contrastive modality fusion for multimedia recommendation. arXiv preprint arXiv:2111.0068.
Yongfeng Zhang, Qingyao Ai, Xu Chen, and W Bruce Croft. 2017. Joint representation learning for top- n recommendation with heterogeneous information sources. In Proceedings of the 2017 ACM Con- ference on Information and Knowledge Management, pages 1449-1458.
Yongfeng Zhang, Haochen Zhang, Min Zhang, Yiqun Liu, and Shaoping Ma. 2014. Do users rate re- view? boost phrase-level sentiment labeling with review-level sentiment classification. SIGIR.

---

Appendix
From Figure 8 to Figure 10, we provide detailed list 29 multimodal personalized prompts used our paper that covers three recommendation tasks.
Prompt A-1
Prompt A-2
Input template: Given the fo following purchase history user_((user_id)> ((purchase history)) predict next possible item to be purchased by the user?
Input template: I find the purchase history List user user_( [fuser___d__ (rrrchase_history]
I wonder which is the next item to recommend to the user. Can you help me decide?
Target template: ((next_item))
Target template: {(next_item))
Prompt A-3
Prompt A-4
Input template: Here is the purchase history list (ssr_((user_id)): wwurchase_historyye try to recommend next item to the user Target template: ((next_item))
Input template: Given the following purchase history of {[user_desc)): ((purchase_histe history)) predict next possible item for the user Target template: ((next_item))
Prompt A-5
Prompt A-6
Input template: Based on the purchase history f(user_desc])
Input template: Here is the purchase history (tuser_desc)) {(purchase history))
((purchase_ swurcchase_hstorry. Can you decide the next item likely to be purchased by the user? Target template: ((next_item))
What to recommend next for the user? Target template: ((next_item)
Prompt A-7
Prompt A-8
Input template: User ({user_id}) has the following purchase history:
Input template: According to ((user_desc)} purchase history list:
{(purchase {(purchass_histtory)) history))
{(purchase_hise history))
Does the user likely to buy {(item_id)} ffitem photo}} {(item_photo>) next?
Predict whether the user will purchase
{(item_id}) {(item satte__phtto) next?
Target template: ([anserr_chotess[[abbe]}}
Target template: {[answer_choices (yes/no)
(yes/no)
Prompt A-9
Input template: According the purchase history of [[user_desc)): ((purchase histtos_hhsttrr)
Can you recommend the next possible item to the
Target template: ((next_iten)
Figure 8: Multimodal personalized prompts for Task Group A: Sequential Recommendation.

---

Prompt B-1
Prompt B-2
Input template: will user ((user_uuaaee (idser_id))))))-- Likely interact with stem_((item_id)) [[item_photo]]]] Target template: ([answer_choices([abel))) (label)))) (yes/no)
Input template: Shall recommend item_ (fitem_(i____]) {(item_photoo> to user_ (fus(((uur__d])) Target template: ([answer_choocces[[[bbb]}] (yes/no)
Prompt B-3
Prompt B-4
Input template: For ((user_desc) do you think it good recommend ((item_title)) {{item photo))?
Input template: I would like to recommend some items user user_[fuser (euu((___iiiipp following item good choice? {(item_titles ((item_photo) Target template: (fanswer_choices([all)) [label]]] (yes/no)
Target template: [[answer_choicces(([bb]]]}} (yes/no)
Prompt B-5
Prompt B-6
Input template: Which item of the following recommend for [[user_desc)]? {[candidate_itenl Target template: ((target_item)}
Input template: Choose the item from the candidates recommend for {[user_desc))? [[candidate, (iste_items) Target template: ((target_item))
Prompt B-7
Prompt B-8
Input template: Pick the most suitable item from the following list and recommend to user (or_[[[se___]]]) [[candidate__iemms) Target template: ((target_item))
Input template: We want to make recommendation for user user_[(user_id]] Select the best item from these candidates: {fcandi Target template: {(target_item))
Figure 9: Multimodal personalized prompts for Task Group B: Direct Recommendation.

---

Prompt C-1
Prompt C-2
Input template: Generate an explanation for user_ (fur_((urr__id) about this product: (fitem_title)) ((item_photo)} Target template: ((explanation))
Input template: Given the fo following review head line [[revieewhheadlnnee can you help generate an explanation of user_((user_ion for item_(fitem_id) {(item_photol)
Target template: ((explanation))
Prompt C-3
Prompt C-4
Input template: Help user (fuser_iddsdeee generate {[star_rating))-staa explanation about this product: {(item_title)) {(item_photo)}
Input template: Generate an explanation for ((user_desc}) about this product: (fitem_title) ((iten_photo)} Target template: (fexplanation))
Target template: f(explanation))
Prompt C-5
Prompt C-6 Input template: Help [(user_desc) generate ([star_ (ittar_ratinn))sstaa explanation for item (fitem id}} fritem_photon Target template: ((explanation))
Input template: Based on the lowinwing review headline: [freview_headllie)) Generate (fuser desc)) purchase explanation about (fitem_ttit title)] {(item_pha)) {iitem_phoooo) Target template: ((explanation))
Prompt C-7
Prompt C-8
Input template: Predict use [ffeature_wordd) feature word to generate user_((user_dd] purchase explanation for (aitem_photo)] Target template: ((star_rating),, {(explanation))
rating then
Input template: What score will (fuser_desc)) rate
{(item_photes ] Then give an explanation for the rating score. (1
being lowest and 5 being highest) Target template: ((star_rating)), {(explanation))
Prompt C-9
Prompt C-10
Input template: Based the feature word ([feature_word) generate an explanation for user (ser_[(user_id] about this product: (fitem title)) (ritem_phooto) Target template: ((explanation))
Input template: Given the word (rceaturewwrd]) can you help generate exp explanation for ((user_desc)) about the product:
((item_ title)] ((item {(itempphoto)) Target template: ((explanation))
Prompt C-11
Prompt C-12
Input template: Using the word ([feature_words write a (osstr_rating))-ae explanation for user (fuser_id]) about item ((item_id)) id)} (fitem_photo)) Target template: ((explanation))
Input template: According to the feature word ((feature word),, generate a [fstar_rating rating))] star explanation for {[user_dess about item [(item_ id}) (fitem_photon Target template: ((explamation))
Figure 10: Multimodal personalized prompts for Group C: Explanation Generation.


Reviews:
[{"review_id": "oGbNlc2k5H", "reviewer": "Reviewer_gRHd", "summary": "", "questions": "na", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\nThanks for your comments and questions. We address your concerns as below:\n\nQ1: Are these differences primarily confined to minor variations in wording, and if so, what is the significance of adopting such an approach? \n\nA1: Yes. We design these multimodal personalized prompts to express similar sequential recommendation instructions using different wording. This prevents the model from overfitting to a specific instruction during training. In fact, the diversity of prompts is crucial -- having more variations of prompts allows the model’s attention to focus more on the useful information in the input, such as the user's purchase history, to generate the desired output, rather than other ordinary words in the prompts which serve as instructions. When we developed our approach, we also found that if we reduced the number of multimodal personalized prompts during training, the performance of the model would decrease as well.\n\n\nQ2: Explanation of the notably inferior performance of the 3 image tokens in Figure 6. \n\nA2: The difference in values for the 3-image tokens scenario is not as large as shown in the graph. Since the values are relatively close to each other, the gap between the Y-axis markings for NDCG@5 is very small. In Figure 6, the four bar values in the B-8 scenario are 0.0661 (1 image token), 0.0671 (2 image tokens), 0.0649 (3 image tokens),  0.0687 (5 image tokens).\n\n\nQ3:  Explanation of the contrast in performance between different visual encoders, specifically B-8 and C-12 in Figure 7.\n\nA3: As we mentioned in the paper, visual information has a positive correlation with user choices in direct recommendation situations, which is why the trend observed in Task B-8 in Figure 7 is in line with our expectations. However, when it comes to hint word based explanation generation tasks, not all aspects that users focus on are related to appearance. For instance, sometimes users need explanations about whether a tool is durable or if a game is enjoyable. It's important to note that this task is different from a pure image captioning task, as the focus is not solely on describing the visual aspects of an item. Therefore, it is quite normal for the trend not to follow the order of better visual encoders if the ground-truth explanation does not have a direct correlation with visual information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The related work is well-presented, and the illustration of the MFM framework is clear and concise.\n2.The comprehensive performance comparison showcased in this paper highlights a significant enhancement, thereby offering robust substantiation of the effectiveness of the proposed algorithm.\n", "reasons_to_reject": "1.Could the authors provide further clarification regarding the distinctions among prompts such as A-1 to A-3 in the Appendix? Are these differences primarily confined to minor variations in wording, and if so, what is the significance of adopting such an approach?\n2.Figure 6 presents challenges in terms of discerning patterns, particularly the notably inferior performance of the 3-image tokens. The paper would benefit from an explanation of this unexpected observation.\n3.The contrast in performance between different visual encoders, specifically B-8 and C-12 in Figure 7, appears striking. Relating this solely to compatibility issues, as mentioned in the text, might not offer a wholly convincing explanation.", "questions_to_authors": "na", "paper_topic_and_main_contributions": "This paper introduces the Multimodal Foundation Model (MFM) framework, which serves to integrate foundational models from Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys). The framework aims to enhance recommendations by leveraging multimodal information. The method, in particular, attains remarkable performance and operational efficiency across three distinct task groups. This achievement is attributed to the integration of features like multimodal personalized prompts and a parameter-efficient training approach.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "na", "Typos_Grammar_Style_and_Presentation_Improvements": "na", "Justification_for_Ethical_Concerns": "na"}, "extra_text": {}}, {"review_id": "712HwA3Qch", "reviewer": "Reviewer_hNgC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\nThank you for the positive comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The framework performs well in three types of recommendation tasks: sequential, direct and explanation generation", "reasons_to_reject": "I don't see any weakness", "questions_to_authors": "", "paper_topic_and_main_contributions": "The key contributions are\nA) The multimodal foundation model framework\nB) A parameter-efficient training method\nC) Experimental results", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4xFFWNEupz", "reviewer": "Reviewer_tDAe", "summary": "", "questions": "Q1: concerning architecture design, and experiment setup.\n\ni) On line 362, the number of image tokens is set to 2. Is the proposed multimodal recommendation affected by this hyperparameter?\n\nii) On line 376, set input token’s maximum length to 1024. The size 1024 may be too limited. In SDIM (Sampling Is All You Needon Modeling Long-Term User Behaviors for CTR Prediction, Cao et al, CIKM 2022), the recent 1,024 behaviors are supported as sequence length. In such cases, the token length of 1024 is not enough. How to address the issues of capturing long-term user interests? \n\niii) On line 337, for the direct recommendation, we generate 100 candidate lists. Size 100 might be useful in the refining ranking stage after we have got a small set from millions of items pool. Size 100 is not directly applicable in real-world million-scale items pool. Is the proposed multimodal recommendation affected by this hyperparameter?\n", "limitations": "", "rating": 1, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\nThank you for your helpful comments and suggestions. Here are our responses to your questions:\n\nQ1: The experimental evaluation is limited and not comprehensive. \n\ni) The datasets are small;  ii) the datasets are only from one recommendation platform; iii) The baselines are ignoring important SOTA methods. The baselines are ignoring large proportions of works on graph neural networks (GNN)-based RecSys.\n\nA1: For i), the 24 categories within the Amazon dataset are independent, with a combined total of 82M ratings, 20M users, and 9M items. In order to facilitate experiments and eliminate the influence of noisy users and items, we chose to use the 5-core version of the dataset for our study. This version ensures that the included users and items have k reviews each, which is a commonly adopted routine in the recommendation research community [2, 3]. Consequently, Table 1 presents the 5-core version statistics of the Amazon dataset sub-categories. \n\nFor ii), though the datasets are from the Amazon platform, they are four independent datasets that represent very different item categories, i.e., sports, beauty, clothing and toys, and thus users’ behavior patterns on the four datasets can be very different. By using the four different item category datasets for experiments, we aim to validate that the proposed method is robust to different recommendation scenarios.\n\n\nFor iii), we appreciate the reviewer's suggestions on incorporating additional state-of-the-art methods for comparison. In fact, our primary focus in this study is to explore the potential of language model-based generative recommendation and investigate whether multimodal information can help in improving LM-based recommendation systems.\n\nAlthough competing with SOTA methods is important, it is not the central focus of this paper. Nevertheless, we have compared our approach with the P5 method, which is a very recent baseline published in September 2022. In the P5 paper, the authors compared their method with SimpleX, which has been shown to outperform many GNN-based methods. Therefore, our comparison with P5 reflects our method's performance in the context of GNN-based approaches.\n\nWe acknowledge that there are numerous other methods and GNN-based RecSys models that could be considered as additional baselines. In future work, we will consider incorporating more methods and further validate the effectiveness of our approach.\n\n\nQ2: Some claims are not sound and seem too big. i) In the abstract, “visual information has received limited attention in recommender systems”. ii) In Contribution point 1, “framework to unify CV, NLP, and RecSys foundation models”. Is the proposed framework really indeed a unified framework for CV+NLP? Or it just adopts existing NLP+CV tools for recommendation applications?\n\nA2: For i), our main point is that in the recommendation domain, papers utilizing image modality are relatively fewer compared to those using user behavior sequence, rating, and text review information. We acknowledge that our statement may seem subjective, and we are happy to remove this sentence in the camera-ready version. Additionally, we will cite the two papers mentioned by the reviewer.\n\nFor ii), we apologize for any confusion caused by our statement. Our intention was not to claim a unified framework for CV and NLP alone, but rather a framework that combines all three domains: CV, NLP, and RecSys. Our main contribution claim is that we are the first to consider these three tasks together using a unified language modeling approach. We will revise our statement in the contribution section to better reflect this clarification.\n\n\nQ3: concerning related works, novelty, and findings. These differences and limitations are missing. It is needed to address the novelty issue of the proposed MFM model by thoroughly comparing it with existing methods.\n\nA3: Due to the 8-page limit of manuscript, we were unable to write the related work section with too much space in the paper. However, in the camera-ready version, we will have an additional page to thoroughly expand our discussion on the related work, as well as the differences and limitations of existing methods.\n\n\nQ4: The experimental results in Table 3 are unfair to the baseline P5 model. On the same three datasets (Sports, Beauty, Toys), the reported results in the P5 model’s original paper are much higher than that in this paper. Why not report the best result of P5 and explanations should be given.\n\nA4: In the original P5 paper, they used 5 task families of prompts for pretraining. However, in order to make a fair comparison with our proposed MFM model, we retrained the P5 model using prompts consistent with the 3 task groups used in our paper. This adjustment was made to ensure that both models were compared under the same conditions. We will add an explanation in the paper to clarify this point and provide a better understanding of the comparison between the MFM model and the P5 baseline. \n\nQ5: On line 362, the number of image tokens is set to 2. Is the proposed multimodal recommendation affected by this hyperparameter?\n\nA5: Indeed, the performance is affected by the number of image tokens used, which is shown in Figure 6 of our paper. In this figure, the model performance varies with different numbers of image tokens. There are trade-offs between efficiency and performance when selecting the number of image tokens. By considering these trade-offs, we have chosen 2 image tokens for our experiments, which provides a good balance between computational efficiency and recommendation performance.\n\n\nQ6: The size 1024 may be too limited. In such cases, the token length of 1024 is not enough. How to address the issues of capturing long-term user interests?\n\nA6: It is important to note that this limitation is not specific to our proposed method but a common issue for any large language model (LLM) with a fixed context length. In our experiments, we were constrained by the available GPU resources, and we found that the input length of 1024 was sufficient for the dataset we adopted. However, it is possible to increase the context length if more powerful GPUs are used, or if other techniques for handling longer sequences are employed. Furthermore, our proposed parameter-efficient method has already helped to increase the token length: by incorporating parameter-efficient methods, MFM can achieve higher training efficiency while maintaining close to the original model performance, thus supporting longer prompt sequences.\n\n\nQ7: Size 100 is not directly applicable in a real-world million-scale items pool. Is the proposed multimodal recommendation affected by this hyperparameter?\n\nA7: The size 100 is not a hyperparameter in our model but rather a widely used experimental setup in many existing research works (e.g.,  CF [1], SASRec [2], S3-Rec [3], BERT4Rec [4]). In a real-world large-scale scenario, our proposed multimodal recommendation approach can be used effectively in the refining stage. For instance, initial candidate items can first be generated using simple recommendation techniques, reducing the million-scale item pool to a smaller, more manageable size. Our approach can then be applied to further refine and rank these items for the final recommendation list, taking advantage of the multimodal information.\n\n\nQ8: Missing important references.\n\nA8: Thanks for pointing out these references. We recognize the importance of including relevant and recent works in our paper to provide a comprehensive understanding of the research landscape. We will include the references in the camera-ready version of our paper.\n\n\nReference:\n\n[1] Koren, Yehuda. \"Factorization meets the neighborhood: a multifaceted collaborative filtering model.\" Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008.\n\n[2] Kang, Wang-Cheng, and Julian McAuley. \"Self-attentive sequential recommendation.\" 2018 IEEE international conference on data mining (ICDM). IEEE, 2018.\n\n[3] Zhou, Kun, et al. \"S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization.\" Proceedings of the 29th ACM international conference on information & knowledge management. 2020.\n\n[4] Sun, Fei, et al. \"BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer.\" Proceedings of the 28th ACM international conference on information and knowledge management. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: a nice try to enable recommendation with multimodal information from the perspective of multimodal foundation models.\n", "reasons_to_reject": "W1: The experimental evaluation is limited and not comprehensive.\n\ni) The datasets are small. The proposed MFM model is evaluated on the Amazon dataset. Following the dataset link (http://jmcauley.ucsd.edu/data/amazon/links.html) given in the paper, the Amazon dataset contains 82M Ratings, 20M Users and 9M Items. However, the statistics shown in Table 1 are much smaller.\n\nii) the datasets are only from one recommendation platform, i.e., the Amazon.com. Actually, the recommendation datasets that contain multimodal information are abundant, including widely used benchmarks: The Yelp dataset (6M reviews + 200K pictures), and The Million Song related datasets (see Deep content-based music recommendation by van den Oord et al, NIPS’13). Furthermore, the backbone of the proposed model, i.e., the P5 model has also been evaluated on the Yelp dataset.\n\niii) The baselines are ignoring important SOTA methods. For sequential recommendation, IDNP: Interest Dynamics Modeling using Generative Neural Processes for Sequential Recommendation by Du et al, arXiv:2208.04600. The baselines are ignoring large proportions of works on graph neural networks (GNN)-based RecSys. See Figure 6 in [Graph Neural Networks in Recommender Systems: A Survey by Shiwen Wu et al] for more baselines. See Missing references for more works.\n\nW2: Some claims are not sound and seem too big. \n\ni) In the abstract, “visual information has received limited attention in recommender systems”. Typical previous works: Learning Image and User Features for Recommendation in Social Networks, ICCV'15. See the survey for more details: Recommender Systems Leveraging Multimedia Content, ACM Computing Surveys 2020. \n\nii) In Contribution point 1, “framework to unify CV, NLP, and RecSys foundation models”. Is the proposed framework really indeed a unified framework for CV+NLP? Or it just adopts existing NLP+CV tools for recommendation applications?\n\nW3: concerning related works, novelty, and findings. \n\ni) the related works section should highlight the limitations of previous (recommendation-focused) methods compared to the proposed MFM model. These differences and limitations are missing. It is needed to address the novelty issue of the proposed MFM model by thoroughly comparing it with existing methods.\n\nii) some findings in this paper are trivial. For example, “extensive experiments...show that multimodality information is helpful for various recommendation tasks.” \n\nW4: The experimental results in Table 3 are unfair to the baseline P5 model. On the same three datasets (Sports, Beauty, Toys), the reported results in the P5 model’s original paper are much higher than that in this paper. Why not report the best result of P5 and explanations should be given.\n", "questions_to_authors": "Q1: concerning architecture design, and experiment setup.\n\ni) On line 362, the number of image tokens is set to 2. Is the proposed multimodal recommendation affected by this hyperparameter?\n\nii) On line 376, set input token’s maximum length to 1024. The size 1024 may be too limited. In SDIM (Sampling Is All You Needon Modeling Long-Term User Behaviors for CTR Prediction, Cao et al, CIKM 2022), the recent 1,024 behaviors are supported as sequence length. In such cases, the token length of 1024 is not enough. How to address the issues of capturing long-term user interests? \n\niii) On line 337, for the direct recommendation, we generate 100 candidate lists. Size 100 might be useful in the refining ranking stage after we have got a small set from millions of items pool. Size 100 is not directly applicable in real-world million-scale items pool. Is the proposed multimodal recommendation affected by this hyperparameter?\n", "paper_topic_and_main_contributions": "This paper adopts multimodal foundation models to exploit different modalities (text, +image+interactions) information for three recommendation-related tasks (sequential recommendation, direct recommendation, and explanation generation. It bases on the P5 model (Pretrain, Personalized Prompt & Predict Paradigm) proposed by (Geng et al, Recsys’22) and extends the P5 model to include visual modality information. Experiments on the Amazon.com real-world platform are conducted.", "excitement": 1, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Missing importance references:\n\nTowards Universal Sequence Representation Learning for Recommender Systems, Hou et al, KDD'22\n\nSelective Fairness in Recommendation via Prompts, Wu et al, SIGIR'22\n\nLanguage Models as Recommender Systems: Evaluations and Limitations, Zhang et al, NeurIPS 2021 workshop on I (Still) Can’t Believe It’s Not Better\n\nPrompt Learning for News Recommendation, Zhang et al, SIGIR'23. It was uploaded to arXiv on Apr 11 while the EMNLP deadline was on Jun 23.\n"}}, {"review_id": "xcZm1QrKGN", "reviewer": "Reviewer_AFHM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer:\n\nThanks for your comments. We address your concerns as below:\n\nQ1: The novelty of the proposed work is somewhat limited; The purpose of having to use parameter-efficient training methods; Experimental comparisons of fine-tuning adapters versus fine-tuning full parameters.\n\nA1: In this paper, we are the first to introduce parameter-efficient methods to the LLM-based generative recommendation domain. The primary consideration is that the user purchase history sequences in sequential recommendation are inherently long, and the additional visual tokens will significantly impact the model's training efficiency, even for P5-small. By incorporating parameter-efficient methods, MFM can achieve higher training efficiency while maintaining close to the original model performance, thus supporting longer prompt sequences. As shown in Figure 4, we provide a performance comparison among MFM variants and full fine-tuning. In fact, our MFM is a framework in which various parameter-efficient methods can be integrated to further improve training efficiency.\n\n\nQ2: The baseline for comparison is not the latest methods.\n\nA2: In the submission, we adopt the P5 approach that was published in September 2022 as a very recent baseline to compare.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors introduce a unified form to model sequential recommendation, direct recommendation, and explanation using a multimodal foundation model.\n2. This paper is clearly presented.\n3. This paper conducts insightful ablation experiments on parameter-efficient training method and visual encoder, etc.\n", "reasons_to_reject": "1. The novelty of the proposed work is somewhat limited. The parameter-efficient training method is very similar to LoRA and llama-adapter.\n2. They use P5-small as the backbone model, which is a small model, and I don't understand the purpose of having to use parameter-efficient training method.\n3. Experimental comparisons of fine-tuning adapters versus fine-tuning full parameters are lacking.\n4. The baseline for comparison is not the latest methods.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. The authors introduce multimodal personalized prompts through a unified form to model sequential recommendation, direct recommendation, and explanation using a multimodal foundation model.\n2. They propose a parameter-efficient training method for multimodal foundation models, which further improves the performance of recommendation foundation models with less training time and less memory usage. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the incorporation of multimodal information for recommendation and the efficiency of the method in terms of performance. However, the authors, are highly encouraged to include their clarifications during the rebuttal in the final version of the paper and: 

- clarify the writing as suggested by R3: avoid any potential overclaim, revise the related work section and situate this work a bit better,

- add experiments (e.g., those suggested by R3 and R4),

- better explain the motivation for parameter efficiency.
Decision: Accept-Findings