Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond
Authors: Tianxin Wei
Bowen Jin
Ruirui Li
Hansi Zeng
Zhengyang Wang
Jianhui Sun
Qingyu Yin
Hanqing Lu
Suhang Wang
Jingrui He
Xianfeng Tang
Venue: ICLR 2024 poster (2024)
Keywords: multimodal personalization
user modeling
generative modeling
instruction tuning
large vision-language model
Primary area: general machine learning (i.e., none of the above)
TL;DR: Developing a universal model that can effectively harness heterogeneous resources and respond to a wide range of personalized needs in the e-commerce domain.

Abstract: Developing a universal model that can effectively harness heterogeneous resources and respond to a wide range of personalized needs has been a longstanding community aspiration. Our daily choices, especially in domains like fashion and retail, are substantially shaped by multi-modal data, such as pictures and textual descriptions. These modalities not only offer intuitive guidance but also cater to personalized user preferences. However, the predominant personalization approaches mainly focus on ID or text-based recommendation problems, failing to comprehend the information spanning various tasks or modalities. In this paper, our goal is to establish a Unified paradigm for Multi-modal Personalization systems (UniMP), which effectively leverages multi-modal data while eliminating the complexities associated with task- and modality-specific customization. We argue that the advancements in foundational generative modeling have provided the flexibility and effectiveness necessary to achieve the objective. In light of this, we develop a generic and extensible personalization generative framework, that can handle a wide range of personalized needs including item recommendation, product search, preference prediction, explanation generation, and further user-guided image generation. Our methodology enhances the capabilities of foundational language models for personalized tasks by seamlessly ingesting interleaved cross-modal user history information, ensuring a more precise and customized experience for users. To train and evaluate the proposed multi-modal personalized tasks, we also introduce a novel and comprehensive benchmark covering a variety of user requirements. Our experiments on the real-world benchmark showcase the model's potential, outperforming competitive methods specialized for each task.

Published as a conference paper at ICLR 2024
TOWARDS UNIFIED MULTI-MODAL PERSONALIZA- TION: LARGE VISION-LANGUAGE MODELS FOR GEN- ERATIVE RECOMMENDATION AND BEYOND
Tianxin WeizS Booen Jin Ruirui Li2, Hansi Zengus Zhengyang Wang", Jianhui Sun Qingyu Yin?, Hanqing Lu², Suhang Wang", Jingrui He Xianfeng Tang²
Unniversity of Illinois Urbana-Champa UUniversity of Virginia 'The Pennsylvania State University ftwei10, bowenj4, jiingnnnneeee (atgruijellinoolll edu, hzengêcs umass edu, wewguevrrginia..ooooo szw4 94 940psu. edu, [ruirul, zhengywa, qingyy, Luhanqin, xianft] aama zon com
2 Ammzonn University of Massachusetts Amherst
ABSTRACT
Developing unified model that can effectively harness heterogeneous resources and respond to a wide range of personalized needs has been a longstanding com- munity aspiration. Our daily choices, especially in domains like fashion and retail, are substantially shaped by multi- data, such as pictures and textual descrip- tions. The vision and language modalities not only offer intuitive guidance but also cater to personalized user preferences. However, the predominant personalization approaches mainly focus the ID text-based recommendation problem, failing to comprehend the information spanning various tasks or modalities. In this paper, our goal is establish Unified paradigm for Multi-modal Personalization sys- tems (Un iMP), which effectively leverages modtll data while eliminating the complexities associated with task- and codality-specinng customization. We argue that the advancements in foundational generative modeling have provided the flexibility and effectiveness necessary to achieve the objective. In light of this, we develop a generic and extensible personalization generative framework, that can handle a wide range of personalized needs including item recommendation. product search, preference prediction, explanation generation, and further user- guided image generation. Our methodology enhances the capabilities of founda- tional language models for personalized tasks by seamlessly ingesting interleaved dision-languae user history information, ensuring a more precise and customized experience for users. To train and evaluate the proposed multi-modal personalized tasks, we also introduce a novel and comprehensive benchmark covering variety of user requirements. Our experiments on the real-world benchmark showcase the model's potential, outperforming competitive methods specialized for each task.
INTRODUCTION
With rapid growth, personalization systems have emerged as a key factor in meeting the user's ex- pectations for tailored experiences that align with their unique needs and preferences. In today's digitally driven landscape, individuals engage with diverse data types, such as ratings, images, de- scriptions, and prices, especially in domains like fashion and retail (Kang et al.) 2017 Hwangbo ctt al. 2018) where visuals and text are essential for decision- makoon.. Given the profound influence of these multi-modal stimuli, there exists pressing need for systems that can seamlessly integrate and harness these diverse data streams for improved personalization. Moreover, users often require variety of information systems meet their different needs. For example, e-commerce platforms commonly offer both search and recommendation functionalities to users (Luo et al. (022). The combination of accurate preference prediction, persuasive recommendation explanation and intu- itive visual guidance could further enhance the user experience. In response to these evolving de- mands, there's 's an emerging imperative to develop the Unified Multi-mode Personalization system
Work was done while Tianxin, Bowen, Hansi, Jianhui interning Amazon.

---

Published as a conference paper at ICLR 2024
(UniMF). Such system aims to cater to individualized preferences that are often hidden in the com- plex interplay between different modalities and data types, streamlining engineering processes and uniting research efforts working on various aspects of personalized user modeling.
Decades of research have been dedicated to bolster the personalization capability of models for each specialized scenario. Standard and well-established methods include matrix factorization and neural networks (CNN/RNN/Seff-atten attenttvv models). They convert user consumption logs into sequences of item IDs and create embedding tables for encoding. In order to augment the context information, some approaches also incorporate side features such as item categories (Zhou et al. 2020; Hou et al.) 2022) into the framework. Within carefully designed network, the final representation can then be employed for specific tasks, such as matching the items that are likely to pique user interests.
Despite their effectiveness, traditional methods suffer from several key flaws, First, these mod- els struggle to process multi-modal personalization data from user histories. User histories often comprise a mixture of visual, textual, Product ID and tabular data, arranged in arbitrary sequences. Effectively utilizing such heterogenous data input poses significant challenges to the models. Sec- ondly, while some research works (Geng et al. (023) employ pre-trained models like BERT (Devlin) et al. 2018), ResNet (He et al. 2016), or VIT (Dosovitskiy et al. 2020) to derive textual or visual representations, these models are usually trained on a general corpus, often differing in language or visual contexts. In light of this, the model cannot discern the importance of various textual at- tributes and visual elements, hindering its ability to understand fine-grained user preferences for personalization tasks. Moreover, loosely integrating the multi-modal representation fails to grasp the complex interactions of the moos-tyy data during the user modeling process. Lastly, it of- ten requires careful per-task network architecture design and hyper-parameter tuning. The process is resource-intensive and also hinders knowledge transfer across various tasks, Even the recent works (Geng et al. 2023 2022) address only a relatively narrow range of use cases and don't 't tackle the multi-takk learning optimization dilemma.
Recently, pre-trained generative models with instructional tuning (Chung et al.] 2022, Liu et al. 2023) have demonstrated remarkable generalization abilities on a variety of tasks. In this project, we propose the Unified Multi-modal Personalization system (UniMP) utilize and extend the ca- pabilities of the generative language model, facilitating multi-modal understanding and enhancing the system's personalization capabilities to address the aforementioned limitations. First, instead of restricting to specific data types, UniMP introduces unified data format which is flexible enough to seamlessly incorporate various types of user history information, including images, texts, IDs, etc. Each user consumed items can include varying attributes contextual elements. Second, to better understand multi-modal information interactions and their sequential transitions, we propose an in- novative user modeling architecture based on Alaaraa et al.]] (2022). in which vision model extracts the visual elements and a language model is responsible for reasoning and generation based on user history. Visual information of each item is conditioned on the corresponding textual representation throughout the layers of the language model, allowing for fine-graine multi-modal information extraction and alignment. Moreover, inspired by prior examination of information retrieval and fil- tering systems (Belkin & Crofi 1992), which suggests that joint modeling and optimization can lead to higher generalization, we further broaden the hypothesis to encompass multiple multi-modal tasks. Leveraging the power of foundation models, we propose to conceptualize each molti-modal task as an instruction, integrating multiple personalization task learning within a token generation framework to learn transferable knowledge. To address the dilemma of the multi-task optimization, we introduce token-level re- weightitnngg to enable -ne-grained prioritization of samples across diverse tasks, and employ context reconstruction improve training efficacy. Notably, our flexible frame- work not only accommodates multi-modal input but also supports multi-modal output generation. This is achieved by quantizing images into visually-coherent tokens Esser al. (2021).
To evaluate Uni iM,, we present a novel and comprehensive henchmark tailored for a variety of user requirements. The proposed UniMP facilitates multi-task multi-modal personalized learning, and efficient fine-tuning on each specific task can further enhance the overall performance. Our contribution can be summarized as follows:
We introduce unified data format to seamlessly ingest various types of user history information. Our flexible framework not only accommodates multi-modal user inputs but also facilitates multi- modal output generation to fulfill personal needs.
2

---

Published as a conference paper at ICLR 2024
Suggestion
Assistance
Content Creation
Preference Level: 4
user-friendly interface, affering seanless inlegration with your other Apple devices
1_3
Languaae Model Layer
Cross-attentio Module
Vision
2023 MariBook
Encoder
[MG]
Tite
Brand
Apple
Gategory
Electroniz
Flatten
Title
MacBook
2023
Category
Electronics
Brand
Apple
Our proposed UniMP framework operates as follows: Item contextual data streamlined into a user sequence, which is then processed through fine- innegrriined cross-modal fusion. Depending on the instructions, the output tailored produce diverse response types.
Figure I:
Our innovative user modeling architecture is designed for fine-grained multi-modal information extraction and alignment, ensuring precise user preference predictions. We also present an effec- tive mulli--assk optimization approach that bolsters the generalization capabilities of Un UP.MP. We propose unify and benchmark representative personalization tasks evaluate diverse per- sonal needs and heterogeneous knowledge sources. Extensive experiments are conducted to show the effectiveness UMi.P. Results show the model can outperform competitive methods special- ized for each task and substantially improve the model's transfcrability.
2 PROPOSED METHOD
In this section, we present our proposed UniMP. We first introduce how to seamlessly ingest het- erogeneous user history information into large language model. Subsequently, we illustrate the architecture of the multi-modal user modeling framework, facilitating fine-graines user preference understanding. Lastly, we propose to formulate personalization tasks as multi-modal prompts, ef- fectively leveraging generative language model to build general-purpose e-commerce system. To overcome challenges in multi-task optimization, we employ context reconstruction to improve train- ing efficacy and introduce token-level re-weighting to automatically prioritize optimization across different contexts, samples and tasks. The overview of our proposed model is shown in Figure
2.1 INCORPORATE HETEROGENEOUS USER HISTORY INFORMATION
For the multi-modal personalized tasks, (J1, Je...., .....,,y, } in temporal order where I, denotes the contextual information of item and v is the length of un. In many prior works on sequential recommendation, each interacted item i solely associated with a unique item ID. In real-world scenario, I. of item in un contains rich heteroge- nous information including image, category, brand, description, price, etc. Based on Mh, we seek to introduce unified input format to harness the multi-modal heterogeneous content. Toward this goal, we construct an attribute dictionary D. for each corresponding item e, which consists of het- erogenous key-value attribute pairs Le D. {(" (""imaee". .x), (ki, v1), (k2, U2) ...., (kim Um) J. Here denotes an attribute name (e.g., Brand), u is the corresponding value (e.g., Apple) and x is both described by natural languages and contain words where x and w" are words of and v from shared vocabu-
given user's interaction history uh.
the visual input of the item. & and (k, v) = {wf.... w w..... ... w") lary in the language model and denotes the truncated length of text. The unified attribute dictionary
3

---

Published as a conference paper at ICLR 2024
De can include all kinds of available item information such as images, titles, descriptions, colors, etc.
In order to harness the dictionary D. with the model, flatten key- attribute pairs into de Flatten(4.) = {[MG], k1, U1, k2, U2,..., kim, Um, [EOC]}
A special token [IMG] in plain text inserted into the item sequence denote the location of the item image, setting the stage for subsequent processing of æ. Following AAaayaac et al. (2022). another special token [EOC] (end of chunk) is used denote the end of the textual and visual
description of each item. The sequences of items interacted by the user are combined to form the user sentence denoted as
th ||CLS], 11 22, .n.
We then replace the user history with the user sentence, ensuring no loss of information, and incor- porate special token [CLS] the beginning of the sequence. Notably, unlike previous personalized systems (Geng et al. 2022 Li et al. 2023b) using only text or item IDs. in this study, we aim to handle multi-modal heterogeneous user sequence. This sequence including visual information isn immediately compatible with the language model.
2.2 FINE-GRAINED MULTI-MODAL USER MODELING
To solve the problem, here we present a multi-modal model designed to process the multi-modal user sequence consisting of both text and interleaved images. The key architectural components shown in Figure are chosen to leverage pre-trained vision and language models (LMs) and bridge them effectively. To effectively process the modalities beyond text. The visual input is fed into the pre-trained vision encoder fo to obtain the features. Following (Alayrac et al.) (022), the vision sampler receives visual features from the vision encoder and design learnable query vectors attend to the features and output a fixed number of visual embeddings 2. R Hxxd with the same hidden dimension as the text encoder. H is the number of images the user sequence, C the number of extracted visual embeddings and d is the hidden dimension. The user sentence in plain text form simultaneously fed into the pretrained language model. These visual tokens are then used to condition the frozen LM in fine-grained manner using cross-attention mechanism. For the cross- attended layer j. denote the textual token embedding of the user sentence as f/ Rg-xd where N the number of tokens in the sequence. The cross-attention mechanism can be described as:
(QKT Vd
+ = Cross Att(Q = 24, KV = Tv, .) = Softmax
M MM
(3)
g
+
(4)
where the query vector is the output textual embedding at j-th layer, and the key as well as the value vectors the extracted vision embeddings. The mask matrix, denoted M, is constructed using special tokens to indicate the correspondence between textual tokens and images. Its purpose is to ensure that the vision information exclusively conditions on the corresponding textual information during cross-modal fusion. Subsequently, the fused information undergoes learnable gating mech- anism denoted as g (Hochreiter & Schmidhuber [997) and residual connection (He et al.) 2016). These components enable fiexible control over the integration of visual information. The architec- ture offers an expressive and fine-grained way for the LM to incorporate supplementary historical visual information for user understanding.
2.3 INTEGRATION OF PERSONALIZATION TASKS
UniMP proposes to integrate multiple personalization tasks within a token generation framework, leveraging the power of foundation models. Based the requirement of various tasks, we can formulate each multi-modal personalized task as the following next-token prediction objective:
(5) where V>n is the prediction token target based on the input user and task information, T is the task description and usn is the set of preceding tokens of the user history sequence with length n. For example, recommendation task is to predict the relevant item IDs that the user will be interested
min log p(y u) = log (y>n
Ugn = (un, T})
4

---

Published as a conference paper at ICLR 2024
Recommendation
User History
Whata fro rexl kon.
Prudict
Geagany Brand
prdfuanso tien
Lawel_ 4
Explosation Ganeration
Exile wey brrms
inathon avariees inkggation.
Irage Selection
UniMP
well foo US
Canepoyy Tha
Nam
Parmonsltoed Search
Ouary fine some
me
User-guided Image Generation
Blnu tuclion anoltar: Stow pore
Figure 2: Through multi-task, multi-modal instruction tuning, the model can adapt to range of user requirements. By altering the instructions, it can generate diverse responses to suit user needs. For in-depth and exact information task instructions and input formats, please see Appendix A.2
in. To unify and optimize multiple tasks together, each task can be described by an instruction T. For example, the recommendation task can be described as: "What's the next item that the user may like?" By conceptualizing all the tasks as textual instructions, and combining it with the user history information, we can unify multiple tasks together with the unified instruction format, The framework is shown in Figure 2 An intuitive solution is continuous learning from all tasks. However, we observe it is hard to design the order of task learning and the prior knowledge will easily be forgetton. Thus, we formulate the final objective as joint multi-task learning:
min
Xa B((y)))
Σ log (yen ugn = {u,, TaF)
(6)
where Aa is the trade-off loss coefficient for Task a, is the number of tasks, Ta and To rep- resents the task description and task instruction data of a. Though the above loss looks promis- ing, the naive multi-task learning faces several problems. First, the prediction objective based on multi-sample/mmage user history does not match with the pre-trained task which performs single- sample/image prediction. This discrepancy raises concerns about the model's ability to effectively leverage the pre-trained knowledge when it is required to make prediction based whole user his- tory. Second, difficult to identify the importance of various tasks. Manually tuning Xa is labor intensive and only achieves coarse coarse-grained weight adjustment, leading to sub-optimal performance. To address the above problems, we propose the following strategies.
Context Reconstruction. Our previous model only optimize the loss for the task response lusk based on complete user history Uhen of multiple samples. Our goal is to reconcile this by integrating modifications that align the prediction objective more closely with the pre-training task of sample- level prediction. Thus, we propose optimizing the context reconstruction loss terms jointly. This can be easily included in the optimization without much extra computational cost, and boosts training effectiveness. The revised objective is given
i = busk + Lountext
log p@y(2>n lign = {un, T})
log p(u,. = 12,,,2, ...).)
where the two loss corresponds to the token prediction loss for the task response and context of each historical item.
Token-level Reiwettgting. Then, to prioritize the samples from different tasks and contexts, we propose to reshape the loss function to re-weight the token prediction loss. Inspired by (Lin et al, 2017), down-weight easy tokens and thus focus training hard tokens by incorporating a mod- ulating factor (1 - 7 to the likelihood, where y is the tunable focusing parameter and p is the standard token prediction likelihood. The process of re-weighting becomes particularly meaningful
5

---

Published as a conference paper at ICLR 2024
when optimizing both the task loss and context loss concurrently, given that contextual informa- tion tends to more predictable. Without proper re- re-weighting, the model may be susceptible to converging into local minima during training. The new token probability Pio is defined as:
Pvr (1 p)7 logp
The re-weighted likelihood offers a fine- grained prioritization of token loss optimization, balancing the contributions of different tasks and samples, thereby enhancing effectiveness.
Retrieval-augmenter Generation beyond Text. Beyond text, our framework can also be adapted to generate other data types, such as images. Building on (Esser et al. (02I), images can be quan- tized into sequences of visually coherent tokens, which the transformer can then learn generate. These tokens can subsequently be translated back to the original images using the CNN decoder from (Esser et al., 2021). Such quantization capitalizes term modeling of realistic images. Here we formulate the task of User-guided Image Generation, aimed real-time generation of personalized image content. This facilitates users in locating and exploring intriguing items, enhancing their experience and addressing varied informational needs. However, generating images with precision can largely hampered by the interference of noisy historical data. To address this, we' ve introduced an intermediate search phase to identify relevant items. These items then help guide the accurate generation of image tokens, as delineated below:
the language model's prowess in long-
S = Beamlppy>> Ugn T.H),, G Beamlp(y>>n Ugn (,),
where T, denotes the retrieval task instruction based the user query, Beam represents generation process from beam search algorithm, is the set of items retrieved by the model and Ty. G are image generation instruction and output tokens. The tokens will be fed into quantization decoder to get the actual image.
3 EXPERIMENTS
In this section, we empirically evaluate the performance of our proposed UniMP, intending to an- swer the following research questions:
RQI: Does UniMP enable effective learning for various multi-modal personalization tasks?
RQ2: How do different components in our framework contribute to the performance?
RQ3: How well does our proposed UniMe generalize across different scenarios, such as with new users and in unfamiliar domains?
RQ4: Can UniMP generate personalized content to user's query?
RQ5: How robust is UniMP when faced with noisy multi-modal input scenarios? (Appendix A.3
3.1
EXPERIMENTAL. SETUP
Datasets. We assessed the model's personalization capabilities with multi-modal input by con- ducting experiments several domains from the publicly available Amazon review dataset (He & McAuley) 2016a Ni et al. 2019). These domains include: Amazon-Baby, Amazon-Beauty. Amazon-Clothing, Amazon- Amazon-Grocery, Amazon-Sports, Amazon-Toys and Amazon-Office. Of these, Amazon-Office was reserved for testing the model's generalizability to new domains, while the others served training data and in-domain evaluation. We combined all the datasets create a more realistic and comprehensive dataset. Only users and items with at least five interactions were included for training. For each user, the latest interaction served the test sample, the second-to- last as the validation sample, and all earlier interactions were used for training. Detailed information about the dataset provided in Table 5 in Appendix A. To provide comprehensive evaluation, introduced new benchmark that encompasses several multi-modal tasks, showcasing the multi- modal system's 's personalization potential. These tasks encompass personalized image generation, multi-modal explanation, and multi-modal personalized search. Collectively, these tasks represent wide range of current user needs.
Implementation Details. For visual encoder, utilize the CLIP ViT-L model (Radford et al. 2021). For the pre-trained large language model, we adopt the instruction-tun -uneed 3B model (To- gether.ai 2023). During model training, by default, train model for 10 epochs with learning

---

Published as a conference paper at ICLR 2024
rate of 2e-4 with AdamW optimizer (Loshchilov & Hutter 2017) and batch size 24 on an 8 A100 40G machine. We utilize the cosine learning scheduler to achieve better convergence. We compare with the representative and state-of-the-art baselines of each task. For recommendation, baselines into four categories: general recommenders, sequential recommenders, generative model- based recommenders and multi-modal recommenders. More details about implementation, tasks and baselines can be found in Appendix A.2 A.2
split the
Table 1: Experimental Comparisons on Product Recommendation.
HR@3
NDCG@3
MRR@3
HR@5
NDCG@5
MRR@5
MF
0.0105
0,0077
0,0065
0.0165
0.0093
0,0078
MACR
0.0110
0,0080
0.0068
0.0170
0.0105
0.0091
LightGCN
0,0142
0.0103
0.0088
0.0206
0.0129
0.0094
UlraGCN
0.0151
0.0111
0.0095
0.0215
0.0134
0.0102
HGN
0.0167
0.0113
0.0113
0.0231
0.0153
0.0117
GRU4Rec
0,0132
0.0101
0.0086
0.0201
0.0128
0.0096
SASRec
0.0189
0.0124
0.0102
0.0276
0,0175
0,0126
R-eRec
0.0205
0.0149
0.0133
0.0311
0.0193
0.0156
BERT<Rec
0.0121
0.0092
0.0079
0.0198
0.0127
0,0105
UniSRec
0.0201
0.0146
0.0135
0.0281
0.0191
0.0158
P5
0.0086
0.0056
0.0042
0.0124
0.0074
0.0061
VIP5+
0.0175
0.0125
0.0108
0.0262
0.0163
0.0127
VBPR
0.0114
0.0084
0,0071
0.0181
0.0102
0.0086
CausalRoe
0.0143
0.0105
0.0088
0.0229
0.0146
0.0121
MMGCL
0,0151
0.0112
0.0095
0.0241
0.0159
0.0130
MMSSL
0.0181
0.0132
0.0112
0.0281
0.0194
0.0164
UniMP (Ours)
0.0248
0.0194
0.0176
0.0337
0.0231
0.0196
Table 2: Personalized Task Performance Evaluation.
BLEU
ROOGH
METEOR
BERTSCORE
MAE
RMSE
P5
0,0822
0.0624
0.0321
0.723
P5
1.12M
1.8756
VIPS-
0.0967
0.0733
0.0383
VIPS-
MRG
0.1101
0.0754
0.0466
0.812
SANCL
0.122
1,5781
UmMP (Ouns)
0.1423
0.1080
0.0678
0.853
UniMP (Oum)
0.7049
1.1611
(a) Personalized Explaination Generation
(b) Personalized Preference Prediction
Recall
Precision
F1
HRes
NDCG@5
MRR65
P5
0.6882
0.6111
0.6.26
RM25
0.0513
0.0378
0.0114
VIPS+
0.7625
0.6952
0.7044
HEE
0.0654
0.0474
0.0368
S" Rec
0.7856
0.613
0.6572
ZAM
0,0842
0.0615
0.0542
MMSSL
0.6211
RIM
0,0006
0.0887
UniMP (Ouns)
0,9111
UnIMP (Ouns
0.1769
0.1389
0,1158
(c) Personalized Multi-modal Selection
(d)
Personalized Multi-modal Search
3.2 RESULTS (RQ1)
Performance comparisons across various personalization tasks for different methods can be found in Tables [ 2a 2b 2c and 2d Several key observations emerge from these tables: Our method
substantially and consistently surpasses other competitive basclines across all tasks, outperforming those specifically designed for each respective task. Notably, in the challenging recommendation scenario where prior information is available, our approach still outperforms the baselines by considerable margin. This performance difference widens further for other tasks where additional information is provided by user. An intriguing pattern we noticed is the method's enhanced perfor- mance at higher positions. Specifically, the improvement in HR @3 3 and NDCG@3 3 over baselines is substantially greater than the improvement seen in HR @//DDG@@.
3.3 ABLATION STUDY (RQ2)
In Table 3
we present ablation study for UniMP. The following notations are used: "Vision": Utilization of vision information during the user modeling process. :rinn-grained": Implementa- tion of the layer-wise cross-attention mechanism, rather than simplistic integration of visual and
7

---

Published as a conference paper at ICLR 2024
Table 3: Ablation study of UniMP on Recommendation Task.
HR@5
NDCG@5
MRR@5
w/o Vision
0.0284
0.0205
0.0169
w/o Fine-grained
0.0257
0.0192
0.0161
w/o Context & Rew
0.0311
0.0214
0.0184
w/o Context
0.0318
0.0225
0.0188
w/o Rew
0.0302
0.0212
0.0178
UniMP (Ours)
0.0337
0.0231
0.0196
textual embeddings. "Context"; Incorporation of the contextual reconstruction component. "Rew": Token-level re-weighting mechanism. The results indicate that each component plays a crucial role in the model's performance. A drop in performance is observed when any of these components removed. Particularly, the vision information and fine-gre tneegrained modeling emerge as pivotal factors in achieving optimal performance. Moreover, note that excluding the re- re-weighting mechanism results in a lower performance than when both contextual reconstruction and re-weighting are re- moved. This observation underscores the importance of e-wigghting, especially when optimizing both task and context losses, given the predictability of contextual information.
3.4 GENERALIZATION ABILITY (RQ3)
Table 4: Experiments of Generalization Ability of UniMP.
HR@5
NDCG@5
MRR@5
HR@S
NDCG@S
MRR@S
P5
0,0119
0.0079
0.0062
0,0081
0.0062
VIPS
0.0217
0.0143
0.0124
VIPS
0.0235
0.0167
0.0148
S9-Rec
0,0228
0,0140
0.0123
S9-Rcc
0.0226
0.0152
0.0128
UniSRec
0.0245
0.0152
0.0131
UniSRec
0.0278
0.0184
0.0175
UniMP (Ours)
0.0364
0.0263
0.0228
UniMP (Ours) 0.0433
0.0289
0.0242
New User Recommendation
(b) New Domain Recommendation
In Table 4a and Table 4b we assess the generalization capabilities of different methods when eval- uated on new users and new domain that are not present in the training set. Note that most baselines can not handle the new users introduced after training as they need to optimize the new user em- beddings from scratch. It evident that UniMP reaps greater advantages the transfer learning contexts, primarily attributed to its efficient exploitation of heterogeneous user histories.
3.5 VISUALIZATION OF CONTENT GENERATION. (RQ4)
Query: Low top smeaker
Query: top
Figure 3: Visualization of the generated images.
we present visualization of the images generated by UniMP. Initially, it evident that the generated images possess a high degree of realism and clarity. Moreover, when examining the two generated images, it becomes apparent that even when provided with the identical query "low top sneaker," " our model has the capability to produce diverse and pertinent images tailored to individual user history.
In Fig.
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
Multi-modal Personalization. The development of multi-modal personalization systems is cru- cial due to the prevalence of multi-modal information on the Web. The most common usage is to leverage multi-modal content as side information to assist recommendation decisions. For exam- ple, VBPR (He & McAuley 2016b) and AMR (Tang et al., 2019) propose using visual features to support collaborative filtering computation. MMGCN (Wei et al. 2019) designs multi- Graph Convolution network framework, which can yield modal-specific representations users and micro-videos to better capture user preferences. CausalRee (Qiu et al.] 2021) develops causal in- ference framework to effectively make use of the visual feature and in the meanwhile remove the visual bias. MMSSL (Wei al., 2023b) proposes modality-awan interactive structure learning paradigm via adversarial perturbations to generate robust representations, However, all the above methods only focus on single personalized recommendation tasks and fail to model the mixture of visual, textual, and ID data, arranged in arbitrary sequences.
General- purpose Model. Language an incredibly powerful tool that can effectively describe a wide range of concepts. With the advancements in modern language models, language has evolved into a versatile interface that can be applied to various tasks and modalities. Several methods have emerged to connect different tasks and modalities within a unified language model. VL-T5 (Cho) al. )02T) and Pix2Seq (Chen et al. (202T) approach vision-language tasks and object detection by formulating them as toreence-t--equenee problems based on conditional language generation. To solve the text-to-image generation task, DALL-E (Ramesh et al., 202T), on the other hand, tackles the text-to- -itttto-image generation task by converting images into latent tokens and employing Trans- former decoder to model both text and image tokens in an autoregressive manner. In a similar vein, foun-
OFA (Wang al. 2022) has contributed to the development of comprehensive dision-languee dation models that excel in generating text from images. Flamingo (Alayrac et al.] 2022 Awadalla al. 2023), BLIP Li al. 2023c) and Otter (Li al. 2023a) further advance multi-moda founda- tion models, exploring robust modal alignment and instruction tuning mechanisms, Moving towards personalization, P5 (Geng et al. 2022) has proposed a unified framework that integrates various rec- ommendation tasks, while VIP5 (Geng et al. (023) extends this framework to a multi-modal setting. However, they fail to fully harness the potential of raw data and do not effectively capture the in- teractions among them. They lack the required flexibility to effectively handle the diverse input and output requirements inherent in multi-task learning.
Large language model for personalization. Personalization has been well-studied for information access problems. Recently, large language models (LLMs) have shown great performance in vari- ous personalization tasks. Some works explored personalization for fundamental language modeling with the openly available user data on Reddit (Welch et al. 2022). Facebook, and Twitter (Soni et al,, 2022). (Soni al. 2022) also explore applying a personalized language model for downstream tasks in stance classification and demographic inference. Similarly, other work has explored personalized sentiment prediction tasks on publicly available Yelp and IMDB data (Mireshghallah et al.) 2022 Zhong et al.] 2021). The large language model has demonstrated remarkable success in learning unified representations and has the potential to enhance the personalization of recommendation sys- tems (Geng et al. 2022 Li et al.) [0233b] Hou et al. 2022) as well. (Salemi et al. (2023) introduces the LaMP benchmark and highlights the importance of personalization for LLMs. However, most previous methods mainly focus on the text and fail to model the multi-modal data.
CONCLUSION
In the paper, we developed a unified data format designed for multi-modal user history integra- tion. Our flexible framework is uniquely positioned to handle multi-modal user inputs and generate corresponding multi-modal outputs tailored individual needs. Our proposed user modeling archi- tecture is optimized for meticulous predictions of user preferences. Through our multi-task optimization strategy, we further enhanced the generalization prowess of UniMP. Additionally, we undertook the initiative to benchmark sev- eral multi multi-modal personalized tasks including recommendation. retrieval, prediction, and generation, catering to diverse set of user requirements. Comprehensive evaluation affirmed the superiority of UniMP, demonstrating its capability to outperform specialized competitor methods across tasks and largely enhance its transferability.
data extraction and alignment, ensuring accurate

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by National Science Foundation under Award No. IIS-21 and Agricul- ture and Food Research Initiative (AFRI) grant no. 1020-67021-32799/rrojeee accession no. no.1022178 from the USDA National Institute of Food and Agriculture. The views and conclusions are those of the authors and should not be interpreted representing the official policies of the funding agencies or the government.
REFERENCES
Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W Bruce Croft. Learning hierarchical embedding model for personalized product search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 645-654, 2017.
Qingyao Ai, Daniel N Hill, SVN Vishwanathan, and W Bruce Croft. A zero attention model for personalized product search. In Proceedings of the 28th ACM International Conference on Infor- mation and Knowledge Management, pp. 379-388. 2019.
Cean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 2023.
Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. EE-net: Exploitationnexploraiio neural networks contextual bandits. International Conference on Learning Representations, 2022.
URL httphttsss//ppereeieeeeee
net Eorummidd-a X ch3VrNSRg
Wenxuan Bao, Tianxin Wei, Haohan Wang, and Jingrui He. Adaptive test-time personalization for
federated learning, Advances Neural Information Processing Systems, 36, 2024.
Nicholas Belkin and W Bruce Croft. Information filtering and information retrieval: Two sides of the same coin? Communications of the ACM, 35(12):29-38, 1992.
Keping Bi, Qingyao Ai, and W Bruce Croft. Learning fine fine-grained review-based transformer model for personalized product search. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 123-132, 2021.
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852 2021.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference Machine Learning, pp. 1931-1942 PMLR, 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fineture language mod- els. arXiv preprint arXiv:2210.1 1416, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0440 048000 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12873-12883 2021.
10

---

Published as a conference paper at ICLR 2024
Dongqi Fu, Zhe Xu, Bo Li, Hanghang Tong, and Jingrui He. A view-adversarial framework for multi-view network embedding. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2025-2028, 2020.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, pp. 299-315, 2022.
Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. Vip5: Towards multi- modal foundation models for recommendation. arXiv preprint arXiv:2305. /4302, 2023.
Wei Han, Hui Chen, Zhen Hai, Soujanya Poria, and L. Lidong Bing. Sancl: Multimodal review help- fulness prediction with selective attention and natural contrastive learning. In Proceedings of the 29th International Conference on Computational Linguistics, pp- 5666-5677, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Ruining He and Julian McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517, 2016a.
Ruining He and Julian McAuley. Vbpr: visual bayesian personalized ranking from implicit feed- back. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016b.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639 648, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 585-593, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Hyunwoo Hwangbo, Yang Sok Kim, and Kyung Jin Cha. Recommendation system development for fashion retail e-commerce. Electronic Commerce Research and Applicutions, 28:94-101, 2018.
Dietmar Jannach and Malte Ludewig. When recurrent neural networks meet the neighborhood for session-based recommendation. In Proceedings of the eleventh ACM conference on recommender systems, pp. 306-3063333 300, 2017.
Bowen Jin, Hansi Zeng, Guoyin Wang, Xiusi Chen, Tianxin Wei, Ruirui Li, Zhengyang Wang. Zheng Li, Yang Li, Hanqing Lu, et al. Language models semantic indexers. arXiv preprint arXiv:2310.07815, 2023.
WWag-Chhen Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Wang-Cheng Kang, Chen Fang, Zhaowen Wang, and Julian McAuley. Visually fashion rec- ommendation and design with generative image models. In 2017 IEEE international conference on data mining (ICDM), pp. 207-216. IEEE, 2017.
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37 2009.

---

Published as a conference paper at ICLR 2024
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint 230.0:2306.05425,, 2023a.
Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley, Text all you need: Learning language representations for sequential recommendation, arXiv preprint WrXiv:2305.1373:3 2023b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learming, pp. 19730-19742. PMLR, 2023c.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pD- 2980-2988, 2017.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 034-5 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Chen Luo, William Headden, Neela Avudaiappan, Haoming Jiang, Tianyu Cao, Qingyu Yin, Yifan Gao, Zheng Li, Rahul Goutam, Haiyang Zhang, et al. Query attribute recommendation amazon search. In Proceedings of the 16th ACM Conference on Recommender Systems, pp. 506-508, 2022.
Chen Ma, Peng Kang, and Xue Liu. Hierarchical gating networks for sequential recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 225-833, 2019.
Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiuqiang He. Ultragen: ultra simplification of graph convolutional networks for recommendation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1253-1262, 2021.
Fatemehsadat Mireshghallah, Vaishnavi Shrivastava, Milad Shokouhi, Taylor Berg- King-Kirkpatrick,, Robert Sim, and Dimitrios Dimitriadis. Userldentifier: Implicit user representations for sim- ple and effective personalized sentiment analysis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3449-3456, Seattle, United States, July 2022. Association for Computational orr/
Linguistics. doi: 10.1806..00111 2022. www.claaaaa... in. 252
main.222..2 URL
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labeled reviews and fine ine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)) pp. 188-197, 2019.
Yunzhe Qi, Yikun Ban, and Jingrui He. Graph neural bandits. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1920-1931, 1931, 2023.
Yunzhe Qi, Yikun Ban, Tianxin Wei, Jiaru Zou, Huaxiu Yao, and Jingrui He. Meta-learning with neural bandit scheduler. Advances in Neural Information Processing Systems, 36, 2024.
Ruihong Qiu, Sen Wang, Zhi Chen, Hongzhi Yin, and Zi Huang. Causalrec: Causal inference for vi- sual debiasing in visually-aware recommendation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3844-3852, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghumandan H Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q Tran, Jonah Samost, et al. Recommender systems with generative retrieval. arXiv preprint arXiv:2305.05065, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-to-image generation. In International Conference on Machine PMLR, 2021.
Learning, pp. 8821-883
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint
2012.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-389) 2009.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv preprint arXiv:2304. /4406, 2023.
Nikita Soni, Matthew Matero, Niranjan Balasubramaniaan, and H. Andrew Schwartz. Human lan- guage modeling. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 622-636, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ v1/2022. fin-0222finninss-ee acl.52. URL httpp::/aclaatthhlllaaa [2
org/2022. http:/2222.....ngggfeee.em Findings-acl.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1441- 1450, 2019.
Jianhui Sun, Xidong Wu, Heng Huang, and Aidong Zhang. On the role of server momentum in federated learning, arXiv preprint arXiv:2312.12677, 2023.
Jinhui Tang, Xiaoyu Du, Xiangnan He, Fajie Yuan, Qi Tian, and Tat-Seng Chua. Adversarial training towards robust multimedia recommender system. IEEE Transactions Knowledge and Data Engineering, 32(5):855-888 867, 2019.
Together.ai. Releasing 3b and 7b redpajamaincite family of models including base, instruction-tuned chat models. 2023. URL. http://wwwwttttt.rr.to.o. httttteerrrr.//boo////ooo
redpa janaajamaa.. ama a mondass-odlll
Quoc-Tuan Truong and Hady Lauw. Multimodal review generation for recommender systems. In The World Wide Web Conference, pp. 1864-1874, 2019.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through learning framework. In International Conference Machine Learning, pp. 13318-23340. PMLR, 2022.
simple eqquence-t--squeee
Tianxin Wei and Jingrui He. Comprehensive fair meta-leamed recommender system. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1989- 1999, 2022.
Tianxin Wei, Ziwei Wu, Ruirui Li, Ziniu Hu, Fuli Feng, Xiangnan He, Yizhou Sun, and Wei Wang. Fast adaptation for cold-start collaborative filtering with meta-learning. In 2020 IEEE Interna- tional Conference on Data Mining (ICDM), pp. 661-670. IEEE, 2020.
Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. Model-agnostic counterfactual reasoning for eliminating popularity bias in recommender system. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1791-1800, 2021.
Tianxin Wei, Zeming Guo, Yifan Chen, and Jingrui He. auk-appooinmating mlp fusion for effi- cient language model fine-tuning. In International Conference on Machine Learning, pp. 36821 36838. PMLR, 2023a.
13

---

Published as a conference paper at ICLR 2024
Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. Multi-modal self-supervised learning for recommendation. Proceedings of the ACM Web Conference 2023, pp. 100-000 2023b.
Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. Mmgen: Multi-modal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM international conference on multimedia, pp. 1437-1445, 2019.
Charles Welch, Chenxi Gu, Jonathan K Kummerfeld, Verónica Pérez-Rosas, and Rada Mihalcea. Leveraging similar users for personalized language modeling with limited data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1742-1752, 2022.
Xidong Wu, Jianhui Sun, Zhengmian Hu, Aidong Zhang, and Heng Huang. Solving class of non- convex minimax optimization in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
Zixuan Yi, Xi Wang, Iadh Ounis, and Craig Macdonald. Multi-modal graph contrastive learning for micro-video recommendation. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1807-1811, 2022.
Hansi Zeng. Chen Luo, Bowen Jin, Sheikh Muhammad Sarwar, Tianxin Wei, and Hamed Zamani. Scalable and effective generative information retrieval. arXiv preprint arXiv:2311.091 2023.
Wanjun Zhong, Duyu Tang, Jiahai Wang, Jian Yin, and Nan Duan. Useradapter: Few-shot user learning in sentiment analysis. In Findings of the Association for Computational Linguisties: ACL-IJCNLP 2021, pp. 1484-1488, 2021.
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM international conference on informa- tion & knowledge management, pp. 1893-1902, 2020.
Rong Zhou and Eric A Hansen. Beam-stack search: Integrating backtracking with beam search. In ICAPS, pp. 90-98, 2005.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
The code and model weights will be open-sourced after the review procedure.
A.1 DATASET STATISTICS
Table 5: Statistics of datasets
# Users
Items
Interactions
Sparsity
Baby
19,822
7,776
163,856
99.89%
Beauty
25,837
16,893
227,920
99.95%
Clothing
58.197
44,310
422,474
99.98%
Grocery
16.318
11,581
165,893
99.91%
Sports
40,358
24,766
334,238
99.97%
Toys
24,314
18,906
209,281
99.95%
Office
6,913
4,775
68,306
99.79%
Table displays the dataset's statisties. The dataset spans several domains, including Amazon- Baby. Alazon--Bautt Amazon-Clove Clothing, Amazon-Grocery Amazon-Sponte Sports, Amazon-Toyy Toys, and MmazonnOOfice Amazon-Office was exclusively set aside to test the model's adaptability to the new domain, while the remaining domains were used for training and in-domain evaluation. Similar to the setting of previous papers (Wei & He 2022 Wei et al., 2020), only users and items with minimum of five input interactions were considered for training. For each user, the most recent interaction was used as the test sample, the penultimate for validation, and all prior interactions for training. Additionally, we reserved 10% of users from the training domains to assess the model's generalization for new users during testing.
A.2 IMPLEMENTATION DETAILS
During model training, we train our model for 10 epochs with a learning rate searched from {2e- 2e-4 by default. We employ the AdamW optimizer, with 4.1e-4,le-5]. The learning rate is accumulation step of 2. For our visual encoder, we use the a warm-up ratio of 0.01 and a gradient
CLIP ViT-L model (Radford al. (02I). ). As for the pre-trained large language model, we employ the instruction- -uned 3B model (Together.ai) 2023), which has already undergone instruction tuning. We intend to explore different language model sizes in future research. The cooss-attention module integrated every 2nd language model layer. Our language model consists of 24 layers with a hidden dimension of 4096. Consistent with [Alayrac et al. (2022); Awadalla al. (2023), the pre-trained language and vision models remain frozen during our process. Only the cross-attention modules, input, and output layers are fine-tuned. Training is conducted with a batch size 24 on a node with 8 A100 40G GPUs. We employ the cosine learning scheduler for optimal convergence. For visual inputs, we utilize the original images, while textual data includes item category, title, price, ID, and brand. Each item in the corpus will be assigned unique new token in the token table. Evaluation based on a user's last five interactions as their history. For search and recommendation task, we use the beam search (Zhou Hansen 2005) of size 10 generate the products. The detailed task information will be introduced later.
We compare with the representative and state-of-the-art baselines of each task. Notably, P5 (Geng et al. 2022) and VIP5 (Geng et al., (023) are unified models that can work on partial of tasks, For recommendation, we split the baselines into four categories general recommenders: MF (Koren
ct al. 2009 Rendle et al.) 2012). MACR (Wei et al. 2021), LightGCN (He et al. 2020). Ultra-
GCN Mao al. 2021). Sequential recommenders: HGN (Ma et al. 2019). GRU4Rec (Jannach
& Ludewig 2017). SASRec (Kang & McAuley 2018) and S*-Rec (Zhou et al. 2020). Genera-
tive recommenders: BERT4Rec (Sun et al.) 2019). UniSRec (Hou et al., 2022). P5 (Geng et al.,
2022) and VIP5 /Gng et al. 2023). Multi-modal recommenders: VBPR (He & McAuley 2016b), CausalRec (Qiu et al. 2. MMGCL (Yi al. 2022) and MMSSL (Wei et 2023b). In addition the recommenders,
use MRG (Truong& & Lauw| 2019) baseline for explaination generation, SANCL Han et al. (2022) for preference prediction, S*eRec (Zhou et al. 2020)) and UniSRec (Hou
al. 2022) utilizing threshold-based method for multi-modal selection, BM25 (Robertson al.
15

---

Published as a conference paper at ICLR 2024
2009). HEM (Ai et al., 2017). ZAM (Ai et al., 2019), RTM (Bi et al.
2021 Note that general recommenders operate differently compared to other types and are included primarily to ensure a thorough and comprchensive comparison. It's important to note that the other categories (e.g., gen- erative recommenders) are our primary baselines for comparison. We leave the comparisons with multi-view learning methods (Fu et al. (2200) as future work.
We propose multiple personalized multi-modal tasks to evaluate the performance of our proposed UniMP. Details of the tasks are shown below.
Personalized recommendation. In this task, given the user's history information, we want the predict what product the user will prefer next. Combining with the user history, the task instruction is denoted as "Can you recommend the next item to the user?".
Personalized preference prediction. In this task, given the user's history information, we want the predict the exact preference (1-5) of the user to each specific product. Given an item and user history, the task instruction is represented as "Given ih, can you predict the preference level of the user to the product?" where in is the contextual information of item h including in = {[MGG, U1, v1, 42, U2,,.... ... kim Um: [EOC]]. The visual information In will also be processed and cross-attended by the model.
Personalized explanation generation. In this task, given the user's history information and an item, we want to explain why the user may want the product. Given an item h, the task instruc- tion is represented as "Given in, can you explain to the user why the product might be suitable or unsuitable?".
Personalized multi-modal selection. In this task, given the user's history information, we want the select the appropriate products for the user from the candidates { fuz: ..d J. The task instruction is represented as "Given bu dug ...,,, can you select all the appropriate items for the user?". In practice, to construct the candidates, we randomly select varying number (1-3) of positive items along with a diverse set of negative samples.
Personalized multi-modal search. In this task, using the user's historical data, we aim retrieve relevant products in response to the user's query. The task instruction is phrased as: "[Query] Can you retrieve the items that aligns with both the user's history and the given query?". As per Ai et al. (2177] 2019); Bi et al. (2021). we generate the query based on the item's 's category. It's essential to note that the same query can be associated with hundreds of distinct items. Hence, the user's personalized content is crucial for accurately retrieving the appropriate item.
Personalized user-guided image generation. In this task, based on the user's history information, we want the generate the product image for the user given the user's query. We first retrieve the relevant items based on history using the personalized multi-modal search instruction to filter out noise. Then we generate the product figure based on the retrieved products. Combined with the user history, the retrieval instruction is formulated as: "[Query] Can you retrieve the items that aligns with both the user's history and the given query?". With the retrieved images, the image generation instruction is represented as: "[Retrieved Images] [Query] Can you generate the product figure that the user may be interested in?". In this way, users can more easily find and discover items of interest, leading to improved user experience, which supplement users' diverse information needs. The number of retrieved images is set to 2 in the paper.
A.3 ROBUSTNESS OF PROPOSED METHON TO MISSING MODALITY (RQ5)
In Table 6 we present the robustness of UniMP in the face of noisy textual and visual inputs. Here, NT represents textual input in which 20% of the attribute information is absent, and NV denotes visual input where 20% of the images do not match their corresponding product. Both of these scenarios are reflective of real-world application conditions. It is evident from the table that UniMP is robust to such noisy input data, and it will effectively leverages the other modality to deliver accurate predictions with minimal performance loss. On the other hand, the performance of the baseline methods MMSSL and S S deteriorates considerably in these conditions.
16

---

Published as a conference paper at ICLR 2024
Table Robustness Textual and Visual Information. NT represents textual input in which partial attribute information is absent, and NV denotes visual input where partial images do not match their corresponding product.
HR@5
NDCG@5
MRR@5
S RRe with NT
0.0275
0.0158
0.0133
S' Ree
0.0311
0,0193
0.0156
MMSSL with NV
0.0253
0,0171
0,0144
MMSSL UniMP with NT
0.0281
0,0194
0.0164
0.0332
0.0228
0.0194
UniMP with NV
0.0328
0.0226
0.0192
UniMP (Ours)
0.0337
0.0231
0.0196
Table 7: Analysis of Learning Strategy.
Recommendation NDCG@5
Search NDCG@S
H
MRR @5
HR 5
MRR @5
Continue
0.0197
0,0162
0.0134
0.0876
0,0672
0,0583
Multi-task
0,0293
0.0220
0.0196
0.1632
0,1232
0.1105
Multi-task=Full
0.0325
0.0227
0.0189
0.1654
0.1254
0.1121
Multi-task+ Efffccinnn
0.0337
0.0231
0.0196
0.1769
0.1309
0.1158
A.4 ANALYSIS OF LEARNING STRATEGY
In Table examine various learning strategies, leading to several observations: Continuous learning tends to result the loss of previously acquired knowledge, leading to subpar performance. Multi-task learning surpasses the performance of continuous learning, indicating the acquisition of transferable task knowledge. However, multi-task learning might not achieve optimal results due the balancing between different tasks. When further fine-tuned the model for specific tasks (denoted as Multi-task++Full, we observed that adjusting only subset of parameters (specifically, the last layer, denoted as fine-tuning the entire model. This also motivates future work on designing more effective parameter- efficient fine-tuning strategies for the multi-modal user modeling method.
enhances overall performance more effectively than
A.5 RATIONALE OF DATA FUSION METHOD
Our fusion method's main innovation, compared to CLIP (Radford et al. 1021 and VIP5 (Geng ctt al.) 2023), lies in our unique approach to fusion strategy and fusion position. CLIP processes visual and textual data separately through modality-specific encoders and only combines them the final layer to compute similarity scores. VIP5, on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language mod- eling. We refer to these as "late fusion" and "carly fusion" strategies, respectively. These methods focus on processing single image-text pair.
In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data dur- ing fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337-40.2444). This fusion also performed at every layer, allowing for a more nuanced and de- tailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've we've included an experiment titled 'w/o Fine-grained" in Table 3 showing our method's performance without the cross-ate aateentioo design. Additionally, sion, Late Fusion, and without Exclusive Attention strategies respectively in Table g which further validate the effectiveness and rationale behind our proposed fusion method.
have conducted comparisons with Early Fu-
17

---

Published as a conference paper at ICLR 2024
Table 8: Ablation on Data Fusion Strategies.
HR@5
NDCG@5
MRR @
VIPS
0,0262
0.0163
0.0127
UniMP w/o Exclusive Attention
0.0244
0.0165
0.0132
UniMP w/o Cross-attention
0.0257
0.0192
0.0161
UniMP with Early Fusion
0.0271
0.0185
0.0135
UniMP with Late Fusion
0,0274
0,0193
0.0156
UniMP
0.0337
0.0231
0.0196
A.6 NECESSITY OF CONTEXT RECONSTRUCTION AND TOKEN-LEVEL REWEIGHTING
For the context reconstruction loss, this term involves predicting product attributes for each histori- cal product image in user's sequence. Unlike our primary task the paper (e.g., recommendation) which utilizes the entire user history encompassing multiple products, context reconstruction loss focuses on individual products. This approach aligns with the pre-training objective the model, such as generating detailed descriptions for each specific image. The integration of context construction loss serves as a bridge, harmonizing the optimization our model. Importantly, the inclusion of context reconstruction loss doesn't it incur extra computational costs, yet it substantially enhances training effectiveness by enriching the information learned.
As for token-level reweighting, recognize that reconstructing context is generally simpler than accurately predicting user preferences, which is key motivation for our approach. To address this, propose reweighting term designed to automatically differentiate between tokens that are easier or harder to predict.
A.7 EMPIRICAL PERFORMANCE ON MORE MULTI-MODAL DATASETS
We primarily selected the Amazon dataset as our experimental platform due to its extensive and varied content, including reviews, ratings, images, interactions, and product attributes. This com- prehensiveness makes it particularly suited for multi-modal, multi-task evaluations. Recognizing the importance of validating method to the Netflix and HM datasets. For Netflix, the images are obtained by crawling movie posters from the website. The results, detailed in Table affirm the effectiveness and adaptabil- ity of UniMP across these varied datasets.
findings across different datasets, we have also applied our UniMP
Table Recommendation Results on HM and Netflix Datasets.
НM
Neulli
HR@5
NDCG@5
MRR @5
HR @5
NDCG@5
MRR@5
0.0101
0.0063
0.0046
0.0742
0.0413
0.0316
VIPS
0.0122
0.0118
0.0093
0.0936
0.0589
0.0395
S3-Rec
0.0185
0.0121
0.0102
0.1155
0,0632
0.0511
UniSRee
0.0196
0.0139
0.0107
0.1324
0.0856
0.0644
UniMP (Ours)
0.0313
0.0206
0.0172
0.1723
0.1196
0.1024
While we are keen to broaden our multi-modal validation to encompass video and audio data, we currently face the limitation of having access to publicly available datasets containing raw video or audio content. Despite this, our method holds the capability for extension to various other data modalities. For instance, we could process image frames from video through our vision encoder and apply pooling techniques to derive representations for further analysis.
A.8 CHOICES OF LANGUAGE MODELS
In our paper, we utilized the 3B INCITE model, an instruction tuned large language model, for its ease of use and open-source availability. It is important to note that framework is adaptable to various large language models (LMs). To further explore this adaptability, we tested UniMP's performance with Mosaic ML's MPT-1B model, denoted UniMP-Small, The results of these
18

---

Published as a conference paper at ICLR 2024
Table 10: Recommendation Results with Different LMs.
HR@5
NDCG@5
MRR@5
UniMP-Small
0.0315
0,0216
0.0179
UniMP
0.0337
0.0231
0.0196
experiments, detailed T0 suggest that larger LMs can more effectively improve personalized user modeling.
A.9 FUTURE WORK
For the future work of our model, there's potential to seamlessly integrate active user feedback loops. This would facilitate real-time, accurate evaluations rooted in actual user interactions and preferences. Additionally, expanding the range of incorporated modalities, such as integrating audio data, promises more holistic user experience. As we strive for better product ID generation, delving into semantic ID strategies (Rajput al., Moreover, the emphasis on transferable learning highlights the importance of researching parameter- efficient transfer mechanisms (Hu et al. 2021 Wei et al. (023a) in multi-moda user modeling.
2023 Zeng et al. 2023 Jin et al. 2023) becomes crucial.
The development of adaptive algorithms (Ban et al. 2022 Qi et al. 2023 2024g and decentralized methods (Wu al., 2024 Bao et al.) [2024] (Sun al.] 2023) also crucial for the real-world applications of multi-modal personalized models. Lastly, for content generation tasks, leveraging larger and more diverse datasets for pre-training may further enhance model effectiveness.
19


Reviews:
[{"review_id": "ZfeGeRZNfH", "reviewer": "Reviewer_Epwj", "summary": "This paper introduces UniMP, a unified framework for multi-modal personalization that seeks to simplify the integration of various data modalities and tasks. It constructs a universal data format that facilitates the incorporation of diverse user historical data. It also presents a cross-attention mechanism that enables multi-modal user modeling. Furthermore, it combines several personalization tasks into a cohesive token generation framework and introduces context reconstruction and token-level reweighting for alignment. The experimental results show that UniMP can outperform competitive baselines on various benchmark tasks.", "questions": "The paper introduces a unified framework for multi-modal recommendation, which is commendably motivated. Nonetheless, I have concerns regarding the design of its approaches and the evaluation setups, as outlined below:\n- The authors introduce a cross-attention mechanism to merge visual and textual data. The rationale for this choice remains ambiguous. It is important to discuss how this approach compares to other vision-language fusion techniques such as CLIP[1] and other multi-modal recommendation systems like VIP5.\n- The necessity of the context reconstruction loss is still unclear to me. I wonder the detailed design of this loss item and why it can benefit alignment. And about the token-level reweighting, I think it is important to give more definitions/explanations of easy/hard tokens and how to distinguish.\n- In the evaluation part, it is crucial to offer clarity on the experimental settings. Some baseline models like MF and LightGCN operate differently from sequential models like S3Rec and BERT4Rec. Ensuring a clear distinction between these setups is essential to maintain fairness in comparisons.\n- The experiments utilize sub-datasets from the Amazon dataset, which might possess similar data distributions. This might limit the generalizability of the results. It would be informative to observe the recommendation model's performance, especially in scenarios of zero/few-shot recommendations, on diverse datasets with visual content.\n\n[1] Radford et al. Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation to establish a unified paradigm for multi-modal recommendation is good.\n- The structure is clear and it is well-written.\n- The evaluation is extensive and the experimental results look promising.", "weaknesses": "- The rational behind the design choice of its approach is not well-explained.\n- The experimental setting needs clarification.\n- Whether this method can perform well on other datasets except for Amazon datasets is unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pgnHWhIBzv", "reviewer": "Reviewer_Y26x", "summary": "This work proposes a unified personalization generative framework from multi-modal sources with the help of various LMs. Specifically, this work devises a generative personalization framework, UniMP, that can suit many downstream applications, including item recommendation, product search, preference prediction, explanation generation, etc. To achieve this, UniMP devises a universal data format, a user modeling architecture by combining a vision model and a language model, and a token generation framework by integrating multiple personalization learning tasks. Experiments on e-commerce datasets validate the effectiveness of the proposed framework.", "questions": "1. How do you incorporate the behavioral data, like click, browse, and scroll?\n\n2. How do you define and characterize the multi-modal information? The category, brand, description, and price are all textual inputs; why are they defined as multi-modalities in the paper?\n\n3. What are the choices of backbone LMs?\n\n4. How does a user handle the token limits of the LLMs when s/he aims to apply this framework?\n\n5. How to apply the proposed framework to other domains other than e-commerce?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea of proposing a unified personalization framework is intriguing and promising in light of the rapid development of various LMs. \n\n2. This work addresses several challenges encountered in multi-task learning when the different backbone pre-trained LMs are in different modalities.", "weaknesses": "1. Regarding the technical novelty in data fusion and user modeling, the contributions of this work are not impressive. \nIn particular, the strategies of data fusion and user modeling are kind of straightforward. The effectiveness is not validated. Please clarify or verify your choice. \n\n2. The presentation of this work should improved. Typos can be found without too much effort. \nFor instance,  \n\n- the abbreviation UniMP is not introduced when it first appears in the Introduction (see the 3rd paragraph on page 2); \n- in the 1st paragraph of Section 2.1, \"Based on s, ...\". \n\nBesides, the notations should be reorganized to make them more readable; e.g., $i_i$ (in Eq. (1)) is confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uihkLrMOSc", "reviewer": "Reviewer_s3tt", "summary": "This paper focuses on multi-modal personalization systems where the input consists not only text and ID's also images. The output is also multi-modal output generation. The authors take advantage of LLMS as multi-modal prompts.", "questions": "- What is x (the visual input of the item), how do you provide it in eq 1? Apart from the special token [IMG]?\n- If you are already providing the visual input in user's interaction history, you are using the same image in visual encoder again into cross attention module? (As far as l understand, no visual information is added to the user's history, the features come from visual encoder, if this is correct, l suggest that seperate image from the user's history in Fig 1.)\n- What is s in page 3 Section 2.1 line 5?\n\nTypos\n\nPage 2, traditional methods suffers -> suffer\npage 4, line 2,  infomration -> information\npage 4, Section 2.2, the visual input -> The visual input", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper has following strengths.\n  - Good experimental study and results\n  - Consider multi-modal multi-tasks", "weaknesses": "- This paper has a lot of things, but lacks of novelty.\n- All the components are already in the literature, bridging pre-trained vision and language models is not new, cross attention idea etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0mRFZiGtL", "reviewer": "Reviewer_4rAY", "summary": "This study proposes UniMP, a method for enhancing individual user\nexperiences by integrating multi-modal user information\neffectively. It introduces a flexible data format for combining\ndiverse user inputs, fine-grained user modeling, and a multi-task\noptimization approach. UniMP can outperform specialized methods in\nvarious personalization tasks and is particularly effective in\ntransfer learning scenarios with new users and domains. It also\ndemonstrates the ability to generate personalized content and handle\nnoisy multi-modal input. The study highlights the importance of\ncontext reconstruction and token-level re-weighting mechanisms in\nimproving training effectiveness. Overall, UniMP offers a versatile\napproach to multi-modal personalization.", "questions": "You claim it's \"multi-modal,\" but it only deals with language and\nimages. Since there's no validation with video or audio, the term\n\"universal\" might be overstated. Would it be advisable to revise the\ntitle or abstract to specify \"language and images\" to better reflect\nthe scope?\n\nThe dataset's limitation to Amazon data is acknowledged. To enhance\nthe study's scope, have you considered validating it with datasets\nlike Yelp, where there's a combination of image uploads, reviews, and\nratings? Additionally, could you extend the multi-modal validation to\ninclude video and audio data, such as YouTube viewing history?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. UniMP seamlessly integrates multi-modal user data, accommodating\nvarious input types, enhancing personalized recommendations.\n\n2. The model's fine-grained user modeling ensures accurate user\npreference predictions, improving overall performance.\n\n3. UniMP excels in transfer learning, adapting well to new users and\ndomains, providing robust personalized recommendations.", "weaknesses": "The study's weakness lies in its perceived lack of novelty in\ncombining image and language learning, which may not be considered\nhighly innovative.\n\nYou claim it's \"multi-modal,\" but it only deals with language and\nimages. Since there's no validation with video or audio, the term\n\"universal\" might be overstated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)