VisualLens: Personalization through Task-Agnostic Visual History
Authors: Wang Bill Zhu
Deqing Fu
Kai Sun
Yi Lu
Zhaojiang Lin
Seungwhan Moon
Kanika Narang
MUSTAFA CANIM
Yue Liu
Anuj Kumar
Xin Luna Dong
Venue: NeurIPS 2025 poster ()
Keywords: Recommendation
Personalization
LLM
Multimodal LLM
Primary area: applications
TL;DR: We propose VisualLens, which leverages task-agnostic user's visual record to enhance the understanding of individual preferences and deliver more personalized MLLM recommendations.

Abstract: Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals. 
However, item-based histories are not always accessible and generalizable for multimodal recommendation.
We hypothesize that a user's visual history --- comprising images from daily life --- can offer rich, task-agnostic insights into their interests and preferences, and thus be leveraged for effective personalization.
To this end, we propose VisualLens, a novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task-agnostic visual history.
VisualLens extracts, filters, and refines a spectrum user profile from the visual history to support personalized recommendation.
We created two new benchmarks, Google-Review-V and Yelp-V, with task-agnostic visual histories, and show that VisualLens improves over state-of-the-art item-based multimodal recommendations by 5-10\% on Hit@3, and outperforms GPT-4o by 2-5\%.
Further analysis shows that VisualLens is robust across varying history lengths and excels at adapting to both longer histories and unseen content categories.

VesualLens: Personalization through Task-Agnostic
Visual History
Wang Bill Zhu +
Deqing Fu o
Kai Sun
Yi Lu Mustafa Canim
Zhaojiang Lin"
Seungwhan Moon
Kanika Narang
Yue Liu
Anuj Kumar
Xin Luna Dong
Meta
USC
Abstract
Existing recommendation systems either rely on user interaction logs, such online shopping history for shopping recommendations, or focus on text signals. However, item-based histories not always accessible, and are not generalizable for multimodal recommendation. We hypothesize that a user's visual history comprising images from daily life- interests and preferences, and thus be leveraged for effective personalization. To this end, propose VISUALLEES. novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task- agnostic visual history. VISUALLENS extracts, filters, and refines spectrum from the visual history to support personalized recommendation. We created two henchmarks, Google Review-V and Yelp-V, with task-agnostic visual histories, and show that VisUAILLENS improves over state-of-the-art item-based multimodal recommendations by 5-10% on @i and outperforms GPT-4o by 2-5%. Further analysis shows that VISUALLENS is robust across varying history lengths and excels adapting both longer histories and unseen content categories.
offer rich, task-agnostic insights into their
profile
Introduction
Imagine personal assistant, similar to Vannevar Bush's MEMEX [4], observing what you do in your daily life. With her keen insight, she can make informed guesses about what you may enjoy or find intriguing. When you ask for recommendations on anything from restaurants and activities to movies, books, and products, based on her in-depth understanding of you she will provide suggestions tailored specifically to your tastes.
While the concept is intuitive, truly comprehensive personal assistant capable of making recommen- dations across all aspects of life has yet to be realized. Most existing multimodal recommendation systems remain domain- -specii- and rely heavily on item-based interaction histories [41, 34]. For example, an e-commerce platform may suggest products based on past purchases but ignore dining habits or interests outside shopping.
This work explores how can we leverage such a user's visual record to better understand individual preferences and enable more general, personalized recommendations. Achieving task-agnostic recommendations from visual history poses several challenges. First, visual histories are often diverse and noisy, containing images unrelated to any specific recommendation task, entities that fail to accurately represent user preference, or non-informative elements (e.g., background objects like
"Correspondence: wangzhu@usc.cr edu 'Wor done at Meta.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Recommendation Question Recommend museums for me in Los Angeles.
GoogleBeeiew-V
Yelp-V
Human Oracle
VisualLens
66.0 *
59.1
User Provided Photo History
Los Angeles County Museum of Art (Art Museum)
48.5
45.0 0k
E09TH
34.1
33.33
38,9
2. California Science
Center (Local Science
Museum)
3.
UniMP
GPT-4o
Figure 1: VISUALLENS leverages user's task-agnostic visual history to provide personalized recommendations. Our method outperforms GPT-4o by 1.6%.446%% on Hit@3.
trash cans). This creates a trade-off between preserving rich visual content and extracting clean, interpretable representations of user interests. Second, most current MLLMs can only process limited number of images, requiring selective retrieval of relevant user profile information for each query. Third, existing benchmarks are inadequate for evaluating task-agnostic visual recommendation systems, highlighting the need for new, purpose- built evaluation datasets.
We propose a novel framework VISUALLENS,
first step towards harnessing user's visual history for MLLM recommendation. VISUALLENS begins by extracting an offline spectrum user profile, compressing each image in the visual history into triplet: (raw image, caption, aspect words). This representation spans spectrum from rich but noisy content (raw image) to concise but clean semantic cues (aspect words). To improve the quality of aspect words, we employ an iterative refinement process that progressively enhances their alignment with user interests. Next, to efficiently incorporate multiple images during runtime recommendation, VISUALLENS retrieves the most relevant segments of the user profile based on the query context. These selected images are organized into d X d visual grid, accompanied by their corresponding captions and aspect words, enabling the model to jointly process and reason over a compact yet informative representation of the user's 's visual history. Finally, we train the system to perform both aspect refinement and recommendation question-answeriin (QA) in unified model. This design not only reduces parameter overhead but also strengthens the model's ability to interpret and utilize visual history for accurate, personalized recommendations.
To facilitate the evaluation, we created two new benchmarks, Google Review-V and Yelp-V, providing foundation for personalization assessments. We leverage user-taken photos address the challenge of history availability. Unlike extensive MEMEX video logs, these photos require less storage, often available in reviews and social media posts, and provide more insights into user's interests and preferences. Each benchmark includes a standard test set targeting generalization to new users, along with two additional test sets for transferring longer histories and wnseen categories.
Our experimental study shows promising recommendation quality of VISUALLENS. It achieved 82-91% % @it@ 10 the Google Review-V and Yelp-V benchmarks, outperforming state-of-the-art (UniMP [46]) by ~10%. Even comparing with GPT-40, our 8B model improves Hit@3 by 1.6% and 4.6% respectively on the two benchmarks. Further analysis reveals that VISUALLENS excels adapting to longer histories and unseen categories, while maintaining robustness with shorter histories.
2 Related Works
Recommendation system with large language models. Large Language Models (LLMs) have demonstrated strong potential in recommendation systems with their advanced language processing capabilities [41]. On item- based recommendation, studies such as LLM4RS [6], LLMRank [19], CLLM4Ree [62], P5 [12], and Sanner et al. [38] explored various LLM prompt and bootstrapping strategies, showing competitive performance, especially in cold-start scenarios. Generative rec- ommendation with open-domain prompts explored by GenRec [21], though fine-tuning remains
crucial. Fine-tuning approaches include personalized aspect extraction [26], and multitasking

---

Table I: Unlike representative related works, VISUALLENS is novel framework leveraging multimodal LLM for recommendation with task-agnostic visual content. CTR: click-through rate.
Multimodal
New Eval
User History
User Profile
Objective
LLM4RS |6]
x
X
Textual item features
Text sequence
Items
ReI.La 1271
X
X
Textual item features
Top-k hehaviors
CTR
ONCE [29]
X
X
Task-agnostic text
Content embedking
Items
PC¹L [52]
/
V
Multimodal item features
Selected images
Explanation
COURIER [54]
V
X
Multimodal item features Multimodal item features
Joint embedding
CTR
UniMP [46]
V
X
Joint embedding
Multi-task
LENU
Task-agnostic images
Spectrum
QA
LLaMa models [53]. Retrieval-enhanced models, such as ReLLa, improve recommendation by retrieving relevant behavior sequences [27]. Instruction-tunne -tning and graph augmentation approaches
are explored in InstructRec [57], LLMRee [48], and LKPNR [15]. Jang et al. [20] use RLHF methods to improve personalizations well. Recent advances in personalized conversation systems have utilized task-agnostic conversation logs to provide personalized answers [23, 16, 31, 29,51, However, these content-based recommendation approaches predominantly rely on textual data.
Multimodal recommendation systems. Multimodal recommendation systems [45, 54, 39] leverage multiple data types, such as text and images, to improve recommendation relevance and person- alization. Before the LLM era, Lee and Abu-EI-Haija [22] proposed systems for content-only video recommendation using similarity learning. PC [52] develop an LLM model that provides multimodal explanations for recommendations. On modalities beyond image and text, MMRF [59] built a joint recommendation system integrating comments with video items [7, 8, 11, 58]. Rec- Former [25] and UniSRec [18] convert images to short captions utilize text-only models. The current state-of-the-an image-text recommendation, UniMP [46], extended single-task multimodal personalization [17, 43, 49, 47] on multitask website-based shopping. However, most existing multimodal recommendation approaches rely on item-based user history, which is not always available. To address this limitation, we propose VISUALLENS, novel framework that leverages MLLMs to enable personalization using task- ausk-agnostic visual history.
We discuss more related works traditional recommendation and recommendation benchmarks in Appendix B.
3 Multimodal Task-Agnostic Recommendation
Consider a recommendation QA task, where the user asks a recommendation question and the recommender answers q with ranked list of candidate items. Good recommenders shall rank the items that the user is more likely to be interested in or willing to try early in the list.
In multimodal task-agnostic recommendation, the recommender is facilitated with task aask-agnottic visual history H., for each user u, which contains a series of photos, taken or posted by the user, not necessarily related to q. We state three assumptions allow generalization. First, the photos may not be directly relevant to the question. Second, an image may not be associated with any candidate item, and even so, the candidate ID not given. Third, photo does not necessarily imply strong preferences. Figure 1 shows an example question, visual history, and candidates.
To simplify the problem, assume candidate retriever exists to retrieve all candidates that satisfy the user's ' question. Each candidate s is represented with a (I..I.) pair, where I, is the name and text descriptions, and I, is an optional image set for the item.
Traditional recommendation setting considers two more types of signals. The first a task-specific set of items in the candidate set, which captures user interest or at least user history. The second is a set of user-specific attributes such as the user's age, gender, and interests. This paper focuses on task-agnostic visual history and leaves integration of these traditional signals for future extensions.
3

---

User history
History augmentation
Captions
F
Image Aarge, Image 2. the entronce to museum, with sgn "MUSEUM ENTRANCE Image J...
Fruzen
F
Caption generation
Tunable
(LLaVA-vo
Aspect words
Shared Weights
Aspect generation
Image museum arches, bench, dome, light, strortigght night halrary... Image museum ontrance, gloss, black. ion gate, reflection plants...
E
(MiniCPM-va..)
SUMT
History retrieval
Images Captions Aspect
Preference profiling
Rank candidates oulput probability
CIP 136pp)
(Gridiit Concateratioo)
Question
Relevant history
Prediction
What is the museum the user most most illey to go next?
Candidate matching
M
Wacan Park and Museum..
</1>
Candidates
Sloss Fumoces Vistor
(MiniCPM-v2.9)
</2
E
<13> Histrri Cahaba Pumpinng Storion..
VISUALLENS inference pipeline: the offline process augments images with captions and aspect words to generate a spectrum user profile; the runtime recommendation process retrieves relevant images, generate query-specific user profile accordingly. and then predict candidate preferences.
Figure 2:
VISUALLENS Framework
VISUALLENS framework contains two parts in inference (Figure 2): offline user profile generation, and runtime recommendation. Offline user profile generation ($4.1) augments each image in the visual history with captions and aspect words, and builds spectrum user profile. Runtime recommendation ($4.2) answers recommendation query q in three steps. First, the history retrieval step retrieves only images relevant to q, since the visual history can be diverse and not all photos are relevant to every query. Second, the preference profiling step uses the retrieved images and their augmented captions and aspects to generate query-specific profile of the user. Third, the candidate matching step matches the query-specific profile with each candidate, to generate the confidence score for each candidate for ranking. We discuss the joint training algorithm of VISUALLENS in $4.3.
4.1 Offline profile generation
To build user profile that retains rich visual content while offering clean, interpretable cues about user interests, we augment each image with a caption and a of aspect words. Each image is thus represented as spectrum triplet - (raw image, caption, aspect words) ordered by decreasing information richness and increasing semantic clarity. Ablation results (Table 4) confirm that these augmentations improve recommendation performance.
Image encoding. Each image is encoded using the CLIP ViT-L/14@336px model [36], producing embeddings used for history retrieval at query time.
Caption generation. Captions are generated using frozen LLaVA-v1.644 model [28], prompted to produce concise (=30 words) and grounded descriptions to minimize hallucinations.
Aspect word generation. Aspect words are concise descriptors of key image attributes (e.g., dome, balcony). We prompt the model list relevant terms without constraining the count, allowing flexibility based image complexity.
All modules are aan--aay and can be replaced with stronger alternatives. While LLaVA-v1.6 occasionally produces irrelevant or generic aspect words (e.g., blue, sky), we describe in Section 4.3.3 how joint finetuning improves their utility for recommendation tasks.
4.2 Runtime recommendation
History retrieval. Given a query y and user's visual history Hyp VISUAILLENS first retrieves images that are related to denoted by In.q. We choose up to images to cap the number of images

---

Table Dataset statistics of Google Review-V and Yelp-V.
Datasct
Train
Dev
Test
Categories
Avg. # of images
Avg. # of GT
Avg. # of candidates
GR-V
15.69M
2K
200K
66
157.0
2.7
43.1 66.7
Yclp-V
4.12M
2K
100K
35
263.6
8.2
process at rumtime, and ensure that only the most contextually relevant images retained for further processing, thereby reducing noise.
In general, we can use any image retrieval method such as DELG [5]. Here, we present a method for categorical recommendations such as restaurants and museums, popular in recommendation tasks. For each category we randomly select a set of candidate items in the category and average the visual embeddings of their images as the image embedding of the category, denoted by Ve. Specifically,
the category embedding is calculated as V. = E== (5) where is the number of candidates
()) indicates the visual embedding of the j-th item image in (see Appendix C for sensitivity), and ve
category c. The retrieval step measures the cosine similarity between the visual embedding v, of each image E H in the user's history and the image embedding V. of the relevant category c. We then select the top-w images based on the cosine similarity scores.
Preference profiling, Given set of retrieved images Iu.q, VISUALLENS then generates user's query-specific profile. A critical part of this step is image encoding. Even after retrieval, the number of images still large. Most MILMs allow context windows of limited sizes, constraining the number of images we can process. For example, for an input image of resolution 896 x 896, the PaliGemma model would generate embedding of up 4,096 tokens. A typical LLM with context window of 8, 199 tokens can take at most 2 images.
We propose to group relevant images In.q into d d grid, where d² = w, and treat all images the grid as a single image. If we have retrieved fewer than images, we pad with black background. Let be the maximum available resolution in a multimodal L.M.. The gridify process G takes the Additionally, we number each image from 1 to d to ensure the images are grounded to the corresponding caption and aspect words in the input to the candidate matching. We denote a user u's profile on question by (fu..g Fu.g). where i denotes the gridified image, and Zu,g denotes the concatenated captions and aspect words of relevant images.
d x d grid and generates an image of fixed size R hxxx3
Candidate matching. Finally, VISUALLENN takes the query-specific user profile (Nu.g: Fa.g) and set of candidates, each represented by II). and generates the matching score for each candidate, which will then be used for ranking. This achieved by prompting the multimodal candidate predictor, where we packed the user profile and candidates to the prompt through the image channel and the text channel separately (see the prompt template in Appendix D).
4.3 Iterative Refinement and Joint Training
VISUALLENS relies LLMs for image encoding, caption generation, aspect word extraction, and profile-cane candidate matching. Current the-the-shelf models perform suboptimally for these tasks, so we apply continued pretraining and task-specific fine-tuning to enhance performance.
4.3.1 Multi-image caption pretraining
To facilitate the model to ground to each grid faithfully, we perform LoRA continual pretraining on dense captions. We adopt the dense captioning dataset DOCCI dataset [32], which contains over 15,000 images and their corresponding dense captions. Each time, we randomly sample w images I {i,, ".., i and their corresponding caption C = {z,,... ... ...,,.",, and then we construct gridified input image G(I) and target output text description T(C) = "Image 1: 21, ... Image
w: xw Then we LoRA finetune the pretrained backbone model (e.g.. MiniCPM-V2.5) on all image-caption pairs (G(Z), }(C)} so that the model is able to process the gridified user history grid by grid, We then use the continual pretrained model as the starting point to apply joint training described in $..3.3.
5

---

4.3.2 Iterative aspect word refinement
Unlike image captioning, aspect word generation is not a standard multimodal task and lacks the extensive pretraining data. Hence, with zero-shot prompting, the generated aspect words have large quality gap across different images, and the extracted aspects may not indicate user preferences.
To finetune aspect word generation, first generate the training data. For each image i, start from an initial set of aspect words, denoted as w! which is generated by LLaVA-v1.6. In the th round, we prompt separate Llama-3.1 70B model with w candidates, and ground truths, and ask it to select useful aspect words that are helpful in ground truth prediction, which constitute w! This refinement process continues for several rounds, and the iterations allow for converging extracted aspect words toward more accurate and relevant subset. Empirically, we observe that the refinement
converges after approximately 4 rounds, and denote the 4th refined aspect word set w! as WA, which serves the training target.
The backbone model with parameter is finetuned to optimize the cross-entropy (CE) loss over all images I,
1 Σ
Cap
CE(W,,
Co(au, i)),
(1)
T
iEZ
where Tasp is the prompt for aspect words generation.
4.3.3 Joint training of aspect word generation and candidate matching
To take advantage of multitask training multimodal recommendation [46]. we jointly train the aspect word generator and the candidate predictor on the backbone model. This joint training strategy allows the model to simultaneously learn identify useful aspect words and make accurate predictions, thus improving overall performance.
The joint loss function balances aspect word generation and candidate matching with weighting factor X, where the candidate matching is optimized with binary cross-entrops (BCE) loss to handle multiple ground truth labels.
N >
Prod
BCE(S,, Pe ( Fuy.,y));
N
Jjoint
Lasp + >hprred
where Uj S, is the user, the question, and the ground truth set of candidates of the j-th example. The text prompt Zpred.j consists of the question 9j. the text query-specific profiles Fuzz,y and candidates. We LoRA finetune the model under the joint loss Cjoi--
Benchmarks and Experiments Setups
5.1 Benchmark creation
To the best of our knowledge, there no existing benchmark [16, 31, 50, 44, 37] to evaluate personalization with task-agnostic visual history. We created two benchmarks, Google Review-V and Yelp-V, leveraging publicly available data from Google Local Review [24] and Yelp [2].
For each user in the two datasets, we take the list of reviews in chronological order. Each review is associated with a business name, categories and description. In Google Review-V, each review is associated with few photos, used for image logs. Yelp-V does not associate review with photos, so we randomly subsample one-third of the store profile pictures, such that different reviews for the same business can be associated with different images.
User logs:
Questions and visual history: consider a special type of questions, category recommendattooss like "Recommend nearby museum". Such questions are both popular in real applications, and hard as there are many candidates satisfying the constraint. We remove small categories and most ambiguous categories, such as "place", and "spot".

---

Table 3: Hit rates and MRRs of VISUALLENS multiple baselines Google Review-V and Yelp-V. The result shows (a) VESUALLENS outperforms other baselines, though has gap with the human oracle; (b) model size greatly affects the performance; (c) simply rank by rating is worse design than the random baseline. Due to the large test set size (200K), an MRR difference greater than 0.4 yields p-value- 0.04.
Google Review-V
Yelp-V
Modality
Size
Hit@1
Hit@3
Hit@
MRR
Hit@
Hit@3
Hit@ 10
MRR
Naive baselines
Random
7.6
21.0
55.0
21.2
13.0
33.6
72.7
30.0
Rank by rating
3.9
15.8
55.5
17.7
8.7
28.0
72.3
25.9
Fine-tuned models
UniMP [46]
T+I
3B
13.8
34.1
73.0
30.5
22.4
48.5
85.0
38.3
Llama-3.1 Bomma-3.1-BB-iae 1-8-Innsttuct [30]
T
8B
15.8
36.3
77.2
32.9
24.1
52.2
88.5
39.6
PaliGemma [3]
T+1
3B
13.0
32.0
70.1
28.4
20.8
46.7
82.0
37.5
MiniCPM-V2.5 [56]
T+1
16.1
36.4
78.4
33.2
24.8
53.0
89.3
40.3
Direct inference
Jamma-3. 101--tt--ttttuct [30]
T
70B
16.2
35.9
75.7
33.1
25.22
53.2
88.5
40.6
GPT--4o [33]
T+1
17.1
37.3
80.1
34.3
26.1
54.5
90.5
41.7
Our method
VISUALLEENS (PaliGemmaa
T+1
16.7
36.3
77.1
33.5
27.8
58.8
90.4
44.3
VISUAL (MiniCPM-V2.55) T+1
18.5
38.9
82.3
35.4
28.3
59.1
91.0
44,9
Human annotations
22.0
45.0
36.0
66.0
For each review regarding business of category c, we create question asking recommend businesses in category c. We take all (and only) photos in the reviews before to generate the visual history. We consider the visual history task-agnostic since the categories are highly diverse (see Figure 4), and the photos are quite diverse too (e.g., park photo to illustrate happiness mentioned in the review). We filter cases where the history too short (<10) or does not contain the questioned category. By doing we ensure that the user history contains at least 10 relevant images (i.e., from the same category as the query) for each example in both Google Review-V and Yelp-V.
Candidates and ground truths: For a review we take all reviews starting from r to generate candidates and ground truths. To be realistic, consider only nearby businesses of the as the candidate set, and the number of candidates is a random number in [30, 100]. Candidates that also appear in the user's future reviews are considered as ground truths. To avoid falling into classification problem, we filter examples with only ground truth in Google Review-V and fewer than in Yelp-V.
category
Summary: Table gives the benchmark statistics. The ratio an average number candidates and that of ground truths is 16:1 for Google Review-V and 8:1 for Yelp-V. By default, the train, dev, and test data have disjoint users. We discuss other splitting Table and more details Appendix
5.2 Evaluation measures
We use two metrics evaluate recommendation quality.
Hit@k. Hit@k N ΣN [rannkrr;) k] checks if any relevant item within the top-k ranked results, where N is the number of examples, and rank(r;) ) is the rank of the first relevant item. We check Hit@3 (e.g., voice recommendations) and @it@ 10 ( e.g., on-screen recommendations).
Mean Reciprocal Rank (MRR). MRR N Σ rank(r;] measures the ranking quality by averaging
the reciprocal ranks of the first relevant item for each example. The MRR ranges from 1/S to where S is the number of candidates.
Additionally, we report wall-clock inference time in Appendix C to show the efficiency of VISUALLENS.

---

Table 4: Ablation study on PaliGemma. Different components of VISUALLENS model: joint training (Joint), iterative refinement (Iter), aspect words (Asp.), captions (Cap.), image embedding (Img.). and relevant image retrieval (Ret.). An MRR difference greater than 0.4 yields p-value < 0.04.
#
Representation
Ret.
Training
Google Review-V
Yelp-V
Asp.
Cap.
Img.
Joint
Hit@/
@it@ Hit@3
MRR
Hit@l
Hit@3
Hit @
MRR
1
/
16.7
36.3
77.1
33,5
27.8
58.8
90.4
44.3
2
/
/
16.1
35.8
76.2
33.0
27.2
57.9
88.9
43.3
3
V
/
15.7
35.2
75.4
32.5
26.9
57.5
88.2
42.9
4
15.2
34.7
74.2
31.9
25.8
55.3
86.1
41.2
5
14.8
33.9
73.0
31.2
25.0
53.9
84.9
40.4
6
13.5
32,5
71.9
29.6
22.0
48.2
83.6
38.8
7
V
11.5
27.9
67.3
25.9
20.1
45.7
81.7
36.8
5.3 Implementation and baselines
We ran VISUALLENS with two backbone models, a smaller 3B model PaliGemma and larger 8B model MiniCPM-V2.5. For optimal performance, we selected w 64 (112x112 each sub-image for PaliGemma, best in practice), corresponding to an 8 x 8 image grid, a candidate count of n = 10k, and loss weighting factor of À 2 (Eqn. 3). We compared LENU with solutions below.
Baselines: Randomly rank or select top-k ratings.
multimodal personalization model UniMP [46] (RedPajama 3B), with the adaptation to replace item images and attributes with image tokens in the visual history; Llama-3, 3, 3.1-8. 1-8B-Instrun [30] with text-only user preference profiles; PaliGemma 3B and MiniCPM-V2.55 8B [56] with image profiles.
Fine-tuned models: We fine-tuned three state-of-the-art solutions:
[30] with text-only preference profiles; GPT-4o [33] with multimodal profiles. To control the API cost, we subsample 1k instances from the test sets.
Direct inference: We compared with two out-of-box models: 1aama-..1-70B-11
Human annotation:
Finally, subsampled 50 examples from the test set for human annotation.
Results and Analysis
We conducted experiments to answer four questions:
Q1: Can we effectively leverage user's visual history improve personalization?
Q2: How does each element of VIsUALLenS contribute to the recommendation quality?
Q3: Can VISUALLENS transfer across users, unseen categories, longer history, and new benchmark?
Q4: What is the robustness VISUALLENS?
6.1 Recommendation effectiveness (Q1)
VISUALLEES significantly outperforms baselines. Our first observation from Table is that all recommendation models that leverage the visual history significantly outperform baseline solutions on all metrics. In particular, the best version VISUALLENS improves Hit 3 over Random by 18% Yelp-V, and even more over Rank by rating (apparently, overall ratings do not cater to specific users). There is still a gap between VISUALLENS and human annotations, but comparing Random, it fills ~75% of the gaps for hit@3 on both datasets.
on Google Review-V and by 26%
LENUALLENS outperforms state-of-the-art solutions. VISUALLEENS outperforms UniMP with the same number of trainable parameters. With the 8B MiniCPM-V2.5 backbone, VISUALLENS outperforms MiniCPM-V2.5 itself by 2.5% on Hit@3 on Google Review-V, and by 6% on Yelp-V, and we observe a similar trend for the 3B models. Even comparing with significantly larger models, including 1ama-3.1-70B, -----uuutttttt and GPT-4o without fine-tuning, VISUAL.LENS 7B improves by 1.6-5.69 @i.

---

(a)
(b)
Figure 3: (a) MRR distribution over number candidates, (b) MRR distribution over number of images. Both are on the User ID test set. We find (1) MRR converges when number of candidates exceeds 50; (2) MRR increases and flattens after reaching 100 images.
Table 5: Transferability: MRR of VISUALLENS models to different test setups. LongHis: train until certain timestamp and test afterwards. Category: held-out of categories for testing per user. Use ID: train and test set share common user ID.
Google Review-V
Yelp-V
LongHis
Category
User ID
LongHis
Category
User ID
VISUALLENS (PaliGemma)
35.9
34,9
33.5
46.6
45.2
44.3
VISUALLENS (MiniCPM-V2.55
38.0
37.1
35.4
47.2
45.5
44.9
6.2 Ablation studies (Q2)
We evaluate the usefulness of each component in VISUALLENS in Table 4, with PaliGemma as the backbone. We find history retrieval significantly improves the results, and can reduce Hit@3 by 7% on Google Review-V and by 12% on Yelp-V (#7 vs. #3). Besides, all three representations of the images (embedding, caption, aspects) play an important role. Removing captions and aspect words can reduce @it@3 @ by 3% Google Review-V and by 9% on Yelp-V, even without fine-tuning (#6 vs. #3). Between the two, aspect words play a more important role than captions (#5 vs. #4). Moreover, both iterative training and joint multi-task training improve the recommendation quality. Removing both of them lowers Hit @3 by 1%+ on both data sets (#3 vs #1).
6.3 Transferability (Q3)
We tested transfcrability over users (default setting), over categories, and over different (longer) history lengths. Table compares the MRR of VISUALLENS on both benchmarks, showing good transferability. MRR highest when applied to longer history, with the effectiveness of history retrieval. Transferability is higher across categories than across users, both demonstrating the promise of leveraging task-agnostic signals, and illustrating slightly more challenges to transfer between users of different interest patterns. We also present the generalization results across benchmarks in Appendix C.
6.4 Robustness and qualitative analysis (Q4)
We conducted several robustness tests as follows.
Candidate count: Figure 3a shows that the number of candidates grows, the recommendation becomes harder and MRR gets lower. However, when the number candidates exceeds 50, MRR converges.
Image count in user history: Figure 3b shows that as the history grows, MRR increases and flattens after reaching 100 images. This trend is related to our grid size 8 x 8, as smaller than 64 images will leverage spaces the grid, On the other hand, flat MRR after 100 images shows robustness against history noises with the retrieval step.

---

Google Review Catoger es
restaurant
Steak house
pub
reatuurant
sioskfoures
bar
station
area
golf course
Dart S rubs
yooun cream cuprasss
tourte attaation
hooel
hall
a8m
purger candaithes
fachecuu
part
brewery
Metorea landmart
neture preserum
Sroaclast Drunce
grocery
cafe
moreet
clu
church
coffee tea
grll
museum
brewpub
winery
aeatced
bubble
mail
dell
salon
wash
sica
bokcery
theater
beacch
camsground oundd
bakwout
garden
42-
10k
36
124
32-
10s
26
Caregories
100
Categuress
(a)
(b)
Figure 4: (a) MRR distribution over categories on Google Review-V, (b) MRR distribution over categories Yelp-V. We find (1) the performance per category is loosely correlated with number of training data; (2) when a category more general and less ambiguous, the performance on the category better.
Category distribution: Figure 4 plots MRR for different categories. There are few factors that can affect recommendation quality. First and foremost, ambiguous categories like "area", "station", and "market" get the lowest MRR in Google Review-V. Second, general categories (e.g., "museum", "hotel") with bigger category sizes obtain higher MRR than specific ones (e.g.. "historical landmark"). Third, transfer learning happens between neighbor categories; for example, "deli" and "takeout" achieve the top-2 and top-3 performance with less training data, since they are similar to the largest category "restaurant". We provide more qualitative analysis in Appendix C.
Conclusion and Discussion
In this paper, we proposed a novel approach VISUALLLENS for personalized recommendation using task-agnostic visual history. We advanced MLLMs to extract spectrum signals from images serve as user profile. We created two benchmarks, Google Review-V and Yelp-V, to evaluate VISUALLENS, affirming the efficacy and robustness of VISUALLENS.
VISUALLENS offers promising first step toward MILM recommendation systems that leverage task-agnostic visual history. Several future directions remain. First, we could integrate VISUALLEENS with additional data, such as image timestamps, locations, recognized fine-graine one-grained entities (e.g.. specific products), and user profiles. Second, we aim to extend the recommendation problems explored here to encompass broader QA tasks. Finally, we plan investigate privacy-preserving techniques, such as federated learning, during training.
Social Impact
The ability model user preferences from visual histories raises important considerations around privacy, consent, and data usage. While our work uses publicly available and anonymized images, real-world deployments would need carefully address how user data collected, stored, and interpreted, especially when recommendations extend beyond single domain. There is also potential for reinforcing biases or overfitting to superficial cues (e.g., location, aesthetics) that may not reflect deeper user intent. We encourage future research to incorporate fairness auditing, privacy-preserving training, and transparency mechanisms to ensure responsible use of multimodal personalization systems.
10

---

Acknowledgements
The rohot icon used in Figure 1 is from https://wwi flaticon com/ www/oom/eeeeeeeeeeeee com/free-icon, robot_2432846. The padlocks icons used Figure 2 are from https:// www.flaticon. com/ htt///freiiioonoeee iicon/ooc__99 29c3_59 and https://vee flatiioon..n.oo com/ The illustration images in Figures 1, 2 are from
wtteoo/ppalooc__3e2111e DOCCI [32].
References
[1] Anthropic. Claude 3.5 sonnet. https//www.atttroocc.cc///wwwwvw
3-5-sonnet, 2024, 17
[2] Nabiha Asghar. Yelp dataset challenge: Review rating prediction. 2016. 6,
[3] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi. Olivier Henaff, Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: versatile 3b vim for transfer, 2024. 7, 17
[4] Vannevar Bush. As we may think. The Atlantic Monthly,
1945. 1,
[5] Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings. Part XX 16, pages 726-743. Springer, 2020.
[6] Sunhao Dai, Ninglu Shao, Haiyuan Zhao. Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. Proceedings of the 17th ACM Conference on Recommender Systems, page 1126-1 1 1132. ACM, 2023. 3,
[7] James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. The youtube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems, page 193-296, New York, NY, USA, 2010. Association for Computing Machinery.
[8] Xingzhong Du, Hongzhi Yin, Ling Chen, Yang Wang, Yi Yang, and Xiaofang Zhou. Personalized vidco recommendation using rich contents from vidcos. IEEE Transactions on Knowledge and Data Engineering, 32(3):492-500 505, 2020. 3
[9] Deqing Fu. Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomorphic representations. First Conference on Language Modeling (COLM), 2024.
[10] Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. TLDR: Token-level detective reward model for large vision language models. In The Thirteenth International Conference on Learning Representations, 2025. 17
[11] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. A unified personalized video recommendation via dynamic recurrent neural networks. In Proceedings of the 25th ACM International Conference on Multimedia, page 127-135, New York, NY, USA, 2017. Association for Computing Machinery. 3
[12] Shijic Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rip): A unified pretrain, personalized prompt & predict paradigm (p5). Proceedings of the 16th ACM Conference Recommender Systems, 2022. 2
[13] Google. Gemini: A family of highly capable multimodal models, 2023. 17
[14] Yulong Gu, Zhuoye Ding. Shuaigiang Wang. Lixin Zou, Yiding Liu, and Dawei Yin. Deep multifaceted transformers for multi-objective ranking large-scale e-commerce recommender systems. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, page 2493-2500, New York, NY, USA, 2020. Association for Computing Machinery. 17
[15] Chen Hao, Xie Runfeng, Cui Xiangyang. Yan Zhou, Wang Xin, Xuan Zhanwei, and Zhang Kai. Lkpnr: Llm and kg for personalized news recommendation framework, 2023. 3
11

---

[16] F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context, ACM Transactions on Interactive Intelligent Systems (TiiS), 5(4). 19. 6, 17
[17] Ruining He and Julian McAuley. Vbpr: visual bayesian personalized ranking from implicit feedback. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, page 144-150. AAAI Press, 2016.
[18] Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In KDD, 2022. 3
[19] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shor shot rankers for recommender systems, 2024. 2
[20] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang. Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023. 3
[21] Jianchao Ji, Zelong Li, Shuyuan Xu, Wenyue Hua, Yinggiang Ge. Juntao Tan, and Yongfeng Zhang. Genrec: Large language model for generative recommendation, 2023. 2
[22] Joonseok Lee and Sami Abu-aa-Haija. Large-scale content-only video recommendation. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 987-995, 2017.
[23] Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. Teach lims to personalize an approach inspired by writing education, 2023. 3
[24] Jiacheng Li, Jingho Shang, and Julian McAuley. UCTopie: Unsupervised contrastive learning for phrase representations and topic mining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages $159-6169, Dublin, Ireland, 2022. Association for Computational Linguistics. 6, 17
[25] Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley. Text is all you need: Learning language representations for sequential recommendation. Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2023. 3
[26] Pan Li, Yuyan Wang, Ed H. Chi, and Minmin Chen. Prompt tuning large language models on personalized aspect extraction for recommendations, 2023.
[27] Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-cnhane large language models for lifelong sequential behavior comprehension in recommendation, 2024.
[28] Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 4,
[29] Qijiong Liu. Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. Once: Boosting content- recommendation with both open- and closed-source large language models, 2023. 3
[30] Meta. The Ilama 3 herd of models, 2024. 7, 8, 17
[31] Jianmo Ni. Jiacheng Li. and Julian McAuley. Justifying recommendations using distantly-labele reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Longuage Processing (EMNLPP-JCNIIP) pages 188-197, Hong Kong, China, 2019. Association for Computational Linguistics. 3, 6,
[32] Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge. Docci: Descriptions of connected and contrasting images, 2024. 5, 11
[33] OpenAL. Gpt-4o, 2024. 7, 8, 17
[34] Harris Papadakis, Antonis Papagrigoriou, Eleftherios Kosmas, Costas Panagiotakis, Smaragda Markaki, and Paraskevi Fragopoulou. Content-basse recommender systems taxonomy. Foundations of Computing and Decision Sciences, 48(2):21 (10))21-2200 1-241, 2023.
[35] Letitia Parcalabescu and Anette Frank. Do vision & language decoders use images and text equally? ? how self-consistent are their explanations? ArXiv, abs/2404. 18624, 2024. 17
12

---

[36] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger. and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 4
[37] Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. 2023. 6,
[38] Scott Sanner, Krisztian Balog, Filip Radlinski, Ben Wedin, and Lucas Dixon. Large language models are competitive near startstart recommenders for language- and item based preferences, 2023. 2
[39] Xiang-Rong Sheng, Feifan Yang, Litong Gong, Biao Wang. Zhangming Chan, Yujing Zhang, Yueyao Cheng, Yong-Nan Zhu, Tiezheng Ge, Han Zhu, Yuning Jiang. Jian Xu, and Bo Zheng. Enhancing taobao display advertising with multimodal representations: Challenges, approaches and insights. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, page 1958-4865, New York, NY, USA, 2024, Association for Computing Machinery. 3
[40] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference Information and Knowledge Management, page 1441-1450, New York,
NY, USA, 2019. Association for Computing Machinery. 17
[41] Zhaoxuan Tan and Meng Jiang. User modeling in the era of large language models: Current research and future directions, 2023. 1,2
[42] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, page 565-573, New York, NY, USA, 2018. Association for Computing Machinery. 17
[43] Jinhui Tang, Xiaoyu Du Xiangnan He, Fajie Yuan, Qi Tian, and Tat-Seng Chua. Adversarial training towards robust multimedia recommender system. IEEE Transactions on Knowledge and Data Engineering, 32(5):855 867, 2020. 3
[44] Mengting Wan and Julian J. McAuley, Item recommendation on monotonic behavior chains, In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, pages 86-94. ACM, 2018. 6, 17
[45] Jie Wang, Fajie Yuan, Mingyue Cheng, Joemon M. Jose, Chenyun Yu, Beibei Kong, Xiangnan He, Zhijin Wang, Bo Hu, and Zang Li. Transree: Learning transferable recommendation from cexture-of-modalit foedback, 2022. 3
[46] Tianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng. Zhengyang Wang, Jianhui Sun, Qingyu Yin, Hanqing Lu, Suhang Wang, Jingrui He, and Xianfeng Tang. Towards unified multi-modal personalization: Large vision-language models for generative recommendation and beyond, 2024. 2. 2, 7.8
[47] Wei Wei, Chao Huang, Lianhhao Xia, and Chuxu Zhang. Multi-modal self-supervised learning for recommendation. In Proceedings of the ACM Web Conference 2023, pages 790-800, 2023. 3
[48] Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Limrec: Large language models with graph augmentation for recommendation, 2024. 3
[49] Yinwei Wei, Xiang Wang, Liqiang Nic, Xiangnan Hc, Richang Hong, and Tat-Seng Chua. Mmgen: Multi-modal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1437-1445, 2019. 3
[50] Fangzhao Wa, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun I Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. MIND: A large-scalc dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3597-3606, Online, 2020. Association for Computational Linguistics. 17
[51] Wentao Xu, Qianqian Xie, Shuo Yang, Jiangxia Cao, and Shuchao Pang. Enhancing content-based recommendation via large language model. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, page 4153-4157, New York, NY, USA, 2024. Association for Computing Machinery.
[52] An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. Personalized showcases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1255-,255, New York. NY, USA, 2023, Association for Computing Machinery. 3, 17
13

---

[53] Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. Palr: Personalization aware Ilms for recommendation, 2023.
[54] Jia-Qi Yang, Chenglei Dai, Dan OU, Dongshuai Li, Ju Huang, De-Chuan Zhan, Xiaoyi Zeng, and Yang Yang. Courier: Contrastive user intention reconstruction for large-scale visual recommendation, 2024. 3
[55] Yuhao Yang, Chao Huang, Lianghao Xia, Yuxuan Liang, Yanwei Yu, and Chenliang Li. Multi-behavior hypergraph--nhhaneed transformer for sequential recommendation. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Diseovery and Data Mining, page 2263 2274, New York, NY, USA, Association for Computing Machinery. 17
2022.
[56] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm your phone. arXiv preprint 0408,01800, 2024. 8
[57] Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji- Ji-Rong Wen. Recommendation as instruction following: A large language model empowered recommendation approach, 2023. 3
[58] Bowen Zheng, Zihan Lin, Enze Liu, Chen Yang, Enyang Bai, Cheng Ling, Wayne Xin Zhao, and Ji Rong Wen, A large language model enhanced sequential recommender for joint video and comment recommendation, 2024. 3
[59] Peilun Zhou, Xiaoxiao Xu, Lantao Hu, Han Li, and Peng Jiang. A model-based multi-agent personalized short-video recommender system, 2024.
[60] Wang Zhu, Jesse Thomason, and Robin Jia. Generalization differences between end- end-to-end and neuro- symbolic vision-language reasoning systems. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4697-4711, Abu Dhabi, United Arah Emirates, 2022, Association for Computational Linguistics. 17
[61] Wang Zhu, Alekh Agarwal, Mandar Joshi, Robin Jia, Jesse Thomason, and Kristina Toutanova, Efficient end-to-end visual document understanding with rationale distillation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 8401-8424, Mexico City, Mexico, 2024. Association for Computational Linguistics. 17
[62] Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems, In Proceedings of the ACM Web Conference 2024, page 3162-3172. ACM, 2024.
14

---

Appendix
Benchmark Creation Details
15
A.1 Algorithm for candidate generation
15
A.2 Processing and filtering
16
A.3 Training data distribution
16
More Related Works
17
B.1 Traditional recommendation methods.
17
B.2 Multimodal large language models.
17
B.3 Datasets and evaluating multimodal recommendation.
17
More Analysis
17
C.1 Sensitivity of random sampling for candidate embedding
17
C.2 Generalization between Google Review-V and Yelp-V
17
C,3 Latency evaluation
18
C.4 Qualitative results
18
D Prompt Template
18
E Additional Implementation Details
18
F Licenses
18
G Limitation
21
A Benchmark Creation Details
A.1 Algorithm for candidate generation
We list the complete candidate and ground truth sets generation algorithm in Algorithm 1, where the function nearest(G, loc, m) returns the m nearest businesses of around certain location loc based on the graph G. The function (S removes the businesses in the set S with redundant names.
In both Google Review-V and Yelp-V, MCmin = 30, T' 100, fCmn 10.
massum
gastreuub
2919
watertsl
marro
petamame
300
charch
wineyy
q0
>losp mall
Tull
mean
km
textizus
baboty
Nestorical
campproand
entna
Jullon
plepmund
ssport
Tetal
beach
runey
trensry
gardan
mtduum
1zar
parking
Figure 5: The Google Review-Vision (Google Review-V) training data consists of 66 categories.
15

---

Algorithm Candidate and Ground Truth Sets Generation
Business geographics graph (V, E), Visit set B of user, Category e, Minimum random candidate count remin Maximum random candidate count rCmax: Minimum final candidate count MCin Minimum ground truth count gfCmin
Input:
Output: Candidate sets S Ground truth sets S1..n # Inilialization
do flag fo unsclected end for count +
for each business
Main algorithm
for each business b E B do
is in category c and f. = unselected then
counl. count
Candidate count m + random (rCmin TCmax)
Candidate set Sount nearest(G, broc, m)
CC
Candidate set Sounnt
cCD unique_ unique_namei name S
Ground truth set Scunt SCD n if |Scounel < sCmin or Scoune glCmin then
See
count count continue end if for each business Scount do flag fe selected end for end if end for return (SI..cuntt $1.
A.2 Processing and filtering
For Google Review-V and Yelp-V, the categories are selected as the last word of the annotated tags in the business. However, some category requires multiple words to express its meaning, such as "tourist attraction", "steak house", "historice landmark", "nature preserve", etc. We select and keep these multi-word categories.
In Google Review-V, we remove smaller categories with less than 10k occurrence in the dataset. Then, remove ambiguous non-difierentiable categories, including "shop", "store", "complex", "service". "company", "supplier", "caterer", "agency", "center", "organization", "attraction". "house". "mall". "landmark", "wash", "course". "preserve", "alley". "groomer", "field", "peak", "venue", "delivery", "dealer", "lounge", "office", "arcade". "court", "spot", "stop", "maintenance". "trainer", "wholesaler", "planner", "place". "facility". "school". "stand", "range", "consultant", "designer", "veterinarian" "ground", "contractor", "manufacturer"; "studio". "point", "lot".
In Yelp-V, as the category is very centralized, we remove smaller categories with less than 50k occurrence in the dataset. Then, we remove ambiguous non-differentiable categories, includ- ing "laannnni""" "nightlife", "services", "wings", "arts", "dogs", "tacos", "caribbean", "beer", "spirits", "wine", "venues", "fusion". "entertainment". "southern" "spaces", "lounges", "breweries", "shopping". "smoothies"; "flavor", "plates", "eastern", "tex-mex" "shop", "noodles", "markets", "market". "donuts", "gelato", "sum", "veggies", "fruits", "trucks", "bagels", "cheesesteaks", "clubs", "cuban", "ramen", "life", "roasteries", "stands", "brewpubs", "gluten-free free", "gardens", "travel".
A.3 Training data distribution
We list the training data distribution of Google Review-V in Figure 5,
16

---

B More Related Works
B.1 Traditional recommendation methods.
Before the LLM era, there are also line of work for recommendation with other networks, smaller language models, or ensembling methods. For example, Sun et al. [40] uses a bidirectional Transformer to model the item sequence. Tang and Wang [42] uses CNN to model user preference with convolutional filters. Gu et al. [14] and Yang et al. [55] uses ensembling methods for recommendation.
B.2 Multimodal large language models.
Multimodal LLMs are becoming increasingly powerful in processing images and videos and in
generating human-like text, exemplified by models like GPT-4 family [33], Claude 3.5 family [1], Gemini and PaliGemma [13, LLaVA model family [28], and Llama 3 Vision models [30]. However, they still suffer from strong language prior [9, 35]. generalization [60] and hallucinations [10], or
require heavy text transcription [61]. The extent to which they understand user history, in analogy to LLM recommendation systems, still unclear.
B.3 Datasets and evaluating multimodal recommendation.
The development of multimodal recommendation systems has been facilitated by the availability of diverse datasets that incorporate various types of data, including text, images, and user interaction histories. Notable datasets in this domain include MovieLens-IM [16], which is extensively used for movie recommendations, and the Amazon dataset [31], which includes user reviews and metadata for product recommendations. For news and books recommendations, the MIND [50] and Goodreads [44] datasets offer insights into user preferences in news and literature. Recently, LaMP [37] introduces 7 text classification and text generation tasks across long contexts to evaluate LLM's personalization capablity. In the context of outdoor activities, datasets such as Google Local Data [24, 52] and Yelp [2] are crucial. These datasets not only provide textual reviews but also include user ratings and geospatial data, which are essential for recommending local businesses and services. These benchmarks utilize the traditional recommendation systems that focus on ranking user preferences based on historical IDs. Instead, we introduce new visual history benchmarks Google Review-V and Yelp-V, using data from Google Local and Yelp, to evaluate the multimodal recommendation systems under more realistic visual history for task-agnostic setups.
C More Analysis
C.1 Sensitivity of random sampling for candidate embedding
We show that the recommendation results are not sensitive the random sampling of images in candidate embedding. Table 6 below shows that the robustness against sampling randomness: 1) the standard deviation <0.1 when we do 5 runs sampling: 2) as we reduce the size of the samples to 1K and 100, the MRR stays similar, with <0.5 difference.
Table MRR and its standard deviation (5 runs) of VISUALLENS (PaliGemma) on different sampling sizes.
Google Review-V
Yelp-V
10k
33.5 + 0.1
44.3 4 0.0
1k
33.4 0.1
44.3 0.1
100
33.0 + 0.2
44.1 0.1
C.2 Generalization between Google Review-V and Yelp-V
We report the generalization quality between Google Review-V and Yelp-V. Table 7 below shows that cross-source recommendation reduces MRR by 2-4% respectively vs. in-domain, an acceptable range, highlighting the transfcrability from the shared feature in the embedding space. Besides,
17

---

transferring from Google Review-V to Yelp-V is still better than the best baseline model in nn-domain (40.2), while the other side much worse (33.2). This as expected, since Google Review-V has much more categories and much lower photo quality.
Table MRR of VesuaLLens (PaliGemma) on in in-domain test and cross-doman transfer.
Train data
Test data
Google Review-V
Yelp-V
Google Review-V Yelp-V
33.5
41.9
29.4
44.3
C.3 Latency evaluation
To evaluate the latency of VISUALLENS, measured inference wall-clock time per example single NVIDIA L40S GPU. The results in Table 8 demonstrate that the joint image representation learned by VISUALLEES is more efficient and effective than the interleaved multimodal representation used UniMP, reducing latency by =0.5 sec.
Table 8: Average wall-clock inference time of VISUALLEnS compared with UniMP.
Model
Google Review-V
Yelp-V
UniMP (3B)
5.82s
4.92s
VISUALLENS (3B) VISUALLENS (8B)
5.29s
4.55s
7.31s
6.64s
C.4 Qualitative results
We show the input and output of positive example in Figure 6 and Figure 7. We show the input and output of a negative example in Figure and Figure 9. Please note that due to image licensing restrictions, we are only able provide the image names JPG format. For more details, you can download the image data from htttp::/bbusine yelp. com///atarressourcs// open- dataset/.
D Prompt Template
We list the prompt template for aspect word generation in Figure 10, and the prompt template for candidate matching in Figure 11. The model will fill in category information in the [[Category]] slot and predict the answer at the [[Answer]] slot.
Additional Implementation Details
We present the hyperparameters for training VISUALLENS in PaliGemma and MiniCPM-V2.55 in Table 9 and Table 10. Note that since there are at most 100 candidates, we add special tokens <I1>, <I100> identifier of the candidates in the vocab of the models. By doing so, we can predict the rank by taking the probability of only those special tokens in the first output token.
F
Licenses
We list the licenses involved in this work as follows.
Our usage of Google Local Data is under the license of Google Maps Platform Terms of Service.
Our usage of Yelp Data is under Yelp Dataset License.
18

---

Input
Question:
the Likely to go
Inages:
MgtnP 1ps. UmXX Moggsee
wFn4 www.srrwweellaa//ltt......... ahrataaaddsaatrraaas..... JPg, pkw1XI CwdNcua) 68.]V_A. J.
EX20P765 eecceiiii Vo-idvrá
Pwtrra---a311110 P03811991EVT onuatacuatrbbuu su200g.1p9.
hcstoobaeessrTThaYiiiggg 10 JPS.
meiggorandeccAoosttaaa JP9,
wyomfg3 www.1oonn22aaeeeee.e.....
8rxs 3MtatDFFUXdww0//p
7JT9006 MaaaceeeMyyRReRtta11
dwq77x- HAMevi Bec2 w.....-
h2vbwk1 Ry.. 199.
Mogyn 9ww.hhh............... 1pp.
ex0gsberssy@lephb páPA. JPS,
tfxri sertiwwgye-kkrbneee
Araskbenk 0 jp3.
yF
BUYSCRn4N
Téc8Sxiraeuyg J89.
8350K1 (3300A wiggoc/
-aatiaaa--EMMty2222eeooo 1P3.
quosh, eetngggrelouucci. Jpg,
mikoroaccdPppeebaaff SA.
y4radgsäfPorg SYATGHV..
WwTze:atALn10xEPP8y.... 1P3,
7EVeasaa Dmlwarg. 1P3,
ENEVsOvfinn eg$qs38902.jpp
heeodiyyrryySssehhiooo JPS:
pnt:Rrvoyekkmoot Yt
Nikepttscoos, www.eeegggg.....e
OcyPpSHInM
3Errod Xoyst
wrrascdliteegrrpeSSh..v jpg.
#ithaccRucc2kkwwWtBBB 1P3,
tayx0ol -et.ee-tnnn------ee BTMOI0. 1P4.
1p3,
exfsbeld10dttxxte
1pg,
1bp 20000pp 1P9,
R#061:690ReA57,
111357_55501110 15502mng0101 1P9.
Oke1582MBaYY
Q6ZNBL2V WkUcj ZAQS48C0, 1pg.
VaDyacEsaoC 2h1.
Jpg,
1p9.
grttwage Px2er0E011pY0.
159,
mkj pipppiaaaaaS RYOw. Jps,
TRsg5P)
(ASPP19922//1/00r2200 PXXVg - JP9,
28182 V5A.ovsstutttxeggg 1pg.
ULCkDeCBTa PeBre/vi 1F9,
[eRkkálzvpl 41 jps.
Bbepktzw/055pp3338yyy 199,
fojjNH-660CTL.NM JPB.
1Ipajc_ OF PeGuSbyUXitty
Candidates:
kiver Steakhoure and Grill
JVB Grill
<T3> Morton's a Steakhouse
<14> The Hamption Social Nashvi
<15> Char Restaurant
<I6> Batter'd ' & Fried Bostca Seafcod House
Captain
Willie
Kitchen Lounge
<19> Ruth'e Chris Steak house
<110> Bab's Steak & chop Reuse
Pish
Poke Bros
Rudie'e Seafood Sausage
<I14>
<I15> Damas' Beatauran:
<I16> Behrt Seafood Comрany
Firebirds Fired Grill
Nexican Grill Nariscon
<119>>
Crab
<I21> Sitar Indian Res:aurant
<I22> Carrabta's Italian Grill
<123>
Nashvill
Flening'a Prima SteaKhouse Mina
<I25> Amerigo I:alian Restaurant
<I26> The River House
<127> Rtch
<T28> Fannf e Mae's
<I29>
<I30>
<131> Pemrose
<T32> The Southern Steak $ 0yster
Nashville
Tapatio
<135> Red Perch
<136> March House
<I37> Suahi
<138> RuSan's Suahi and Seafood
Skull's
<I41> Little Octopus
<I42> The Cheesecake Factory Nashville
Juicy Seafood
of Cards
<I45> Joe Crab
Senrietta
<147> Urban Grub
<I48> Bombay Falace Restsurant
<I50>
<151> Redlande Grill Mashville
<I52> The Gzeen Pheasant
<I53> Nomailla! Suszi st datera
<I54> The Smiling Elephant
Sperry's Restaurant
<557> Rddia V'a Prine Seatood
<I58> The Optinist Mashville
doose
Cousins Naine lobster Nashville
<161: Jimny Kelly's Steakhouse
Alexander's Restaurant
<I63> 350 Nine Bar Bis:ro
<I64> The Palm Mashville
Materfrant Grille
Mina
<167> Boston Commone
<168> White Linozeen
c169> Saltine Fine Rood and Drink
<170> Seafood Sensation
<173> South Street
Figure 6: successful example of VISUALLENS Hit@ We list the input question, images and candidates.
PaliGemma models are under a custom license the Gemma Terms of Use (https: //ai. google dev/ hatvv/gema//// gemma/terms)
MiniCPM-V2.5 is under Apache License 2.0
Our usage of OpenAl's models for prompting is under the license OpenAl"s Terms of Service.
19

---

Output
Candidate Banking:
Bourbon
Mictael
Oak Steaxhouse Nashville
<I51> Jinmy Relly's Steakhouse
<I23> Jett Ruay's Steakbouse Machville
<I9> R Chria Steak House
<124; Planing's Prima SteaKhouse Kina Bar
<I10>
Steak chap Bouse
Steakbouse
<T64> The Palm Nashville
<172> The Cont fnental
<147> Ucban Grub
<197> addie V'a Prime Sesfood
<158> Optimias Nashville
<132> Southern Steak Oyster
<I17> Firebirde Wood Fired Grill
<I1> Steney River Steakboure and Grill
Redlande
Nsehville
<135> March Eouse
<189> Sal:ing Fine Foed and Drink
River
<144>
Carde Res
Alexander's Restaurant
<15>> Sperry's Restauran:
<12> JEB Wrill
Hampton Social Nashville
Fried Bosten Beatcod
<113> Rudie's Seafood Ssusage
<I16> Sehrt Seafood Company
<I5D> Cousins Maine Lobster Mashville
<I70> Beafood Sensation
<I43> Juicy Eeatood
<I45> Joe's Crab Stack
<149> The Bumbo Bros
<f11> Pish 0o
<138> RuBan's Suaht
Pever
<141> bittle Octopue
<165> Blue Moon Materfront Grille
Rtch
Skull'a Rainbow Room
<188> White Limozeen:
<136> Louse's wine Dive
<183> 360 Wine Bay Biatro
<130> Green If11a Grills
<131> Penrose
<112> Pake Aroa
Breen Fheasant
<159> silly Goose
<128> Pannie Mae
<18> Nillie B's Kitchen & Lounge
<137> Sushi 88
Regzsurant
<118> Reyes Mexican Grill Karisces
<I19> Eas: Side Zich
<I22> Carrabba's Italian Grill
<125> Amarigo Italian Restaurant
Tapatic
Cheesecake Factory
<I54> The Sniling Rleghant
<171> Sian Cafe
Mastville
Reataurant
<140>
<I48> Bombay Palace Beataurant
Ground Truth
<122>
Carrabba's 's Italian Grill
<I33> Oax Steakhouse Nashville
<I61> Jinny Kelly's Steakhouse
Bourbon. Steak Michael Mina
Figure successful example of VISUALLENN on Hit@1. We list the output candidate ranking and ground truth. The ranking follows the same left-right order the input candidates.
Table 9: Hyperparameters for training VISUALLEES with PaliGemma Backbone.
Hyperparameters for training PaliGemma
Parameter Size
3B
Image Resolution
896 896
Number of Image Tokens
4096
Hidden Dimension Size
2048
LoRA Rank
16
LoRA a
16
LoRA dropout
0.1
GPU
X NVIDIA H100
Batch Size
8
Gradient Accumulation Steps
8
Warmup Steps
200
Learning Rate
le-3
20

---

Input
Question: pizza
user most likely
go nexe?
Inages: aa8kgpdonggggnS5EE DYGRO. 1eg, pro2LDD23yysphhcoo0bbaaa 1pg,
RPxSLzegrM_
#4284447k0ggmmaaaaaaaataaa 199,
UPG0rAIm7ME118911
j- 389-
WwiorraVygnnnTDxoohssttiti JpS,
aCKGuasLLsarnstrrucaaaaa. JPG,
Qvdy
Sy97tswse 1P3.
#kveeaaweekkssssee0000000 Jpg.
ondeerrossarpgtuuuuuut.... jpg.
52gtz wtt.xelll....t...eee......
HI
BL3drxwbnP wwt.ggg/ccccc....pg_..
2ByRk B1EMpEt. -tt.aatttt---TT.....11
1pp.
xaaaudvwuuie WBeozz/U 3pg.
BBg59XRRbJDUnS kxjtaM.jp2
2367
JIMM815220MMK30 cA. 1p9.
jpg,
105xcTpP227AttDaaara.. 1 1pp..
79 Jps.
ligh 30zucHD--rr (rap-rraaaaaaa..ppp..
1878P00307AK2 VuG. 059.
BgtWea178y Bxcixaa; 9.G..
20e0p-E:00R 184 1006 (00000632000 2200 JP9.
SnkekBszM. mnkeensznnoQphaccjwec jpg.
VenxtrPggeTS100 d01x00
JF9.
ha4CYvrita. pM4.ga815..
(wwjjeeeokgggjigs--a.. ipg.
10000 1800 ipg.
awc74
VuLL:EEtvEE apt.aaakkjppppppppppp........ spQ.
JsNfuzopmopoi su
3pg,
bezt:hnaak1og@@:11xbbig. ipg
2aaaga00g02 1Pg.
VieiadCt 1 1-2213 bgnuCA.1p9
"p0FeR99suupshlaaeee ipp.
weElssbbnnscee2280 .pp.
3DaBniquMZe Mit.te---xxxaaaaaa 1F9,
g336Surs97JKK332]
p.BEB_ 1p9,
Oct6 OcFejxTNL 1Fg,
jpg.
GaysN htg.gg9999. JP9.
1pg.
MMOGRvvaaPOM--gfeve ips.
9n4a9an Bewaann-gPPPeeeeee waw.a/00..000s0000000..... Jpg
Candidates:
<I1> La Cucina Stalian Ratery
<12> Chuck E. Cheese
M3 Restaurant
<I3>
zupo
<Is> Brugos
<1'> kustemsn Brewing
<1d> Godfather's Pizza
Pizzava
bittle
<111> Electric Blue glephant
<112> Bclipee Pizza Co.
<I13> Knockouta Baz and Rockah Lounge
II14> Piretrail 2i2za
<I15>
Inn
d's
<I17> Smiling With Hope Pizza
<I15> Chicago's Zizza Mith A Twist Reno Virginia St. WV
<119> Monaciello
<120> Bick's pizsa, Reer, Hore.
<I21> Noble Pie Darlor Dovntom
<I22> Food Drink
<123> Korean Barbie
<124> Pizza Baron Reno
Pizzeria
Wild Garlic
Pub
<127> Casale's Hal.wwyy club
<128> The Pizza collective
<129> Pub X² Sub
<I30> Neble Pie Parler Kidrown Reno
waldorf
<133> Sizzle Fie
<134> Resz Street Market
Cafe Whitney
Piezzetta Zizza Kitchen
<I37> Pizanos Fizza
Pizza
<139> Raztlernake club
<I40> zpizza
Chicago
Playtield 7E
<T43> Bound Tabl's
<I44> Rlind Onion Pizsa & Pub
<I45> Opa Gournet Pizza Caisine
<I46> Pauzie's pizza
Blind Onion
Wings
Rack
<I49> Boulevard Pizza
<I50> MDD pizza
<151> Celafuria
<152> Atlantia café Alfresco
<I53> Domino's sizza
<I54> Peluso's Apizas
<15b> Chicago'a Pizza with A Twist seno, NY <1b6>
Liberty Food Mine Exchange
Cafe
<159> Californis Pizza Kitchen
<160> chicego Doge! Satery
<I61> Hidden Fizza
<I52> Pizza Factory
Figure 8: A failed example of VISUALLENS on Hit@3, We list the input question, images and candidates.
G
Limitation
While VISUAL.LENS demonstrates strong performance on task-agnostic multimodal recommendation, several limitations remain.
Modular design without optimal components. Our framework prioritizes modularity and extensi- bility, using a unified architecture across modules including image encoding, captioning, aspect word generation, and preference matching. However, we do not use the best-performing model for each subtask. Fach component can replaced with more advanced or domain-specific alternatives, which could potentially boost overall performance.
21

---

Output
machinery, Candidate Ranking:
<I4C> zpizza
<I42> Playfield 76
<ISC> MOD
<I57s Liberty Food
<131> Sizsle
<128> The Pizza Dollective
<126> Mild Garlic Pizza & 2ub
<I34> Wild Garlic
<I61> Bidden
Fiezzetta Pizza Xitchen
<13.> Smiling Mith Hopee
Pizseria Tupe
<121> Nobll Pie Parlor
DONICOWT
<130> Noble Pie Parlor Midtown Zeno
Sclipee pizza
<136> Feluso's Apizza
<134> Fizetrail Pizza
<IS2> Pizza Pactory
<137> Pizanos Pizza
<13> Pizzava
Semenza's Pizzeris
Faulic's
<144> The B1iad Onion Piaza
The
Pizza Kinga
<13> la Cucima Italian ian Ratery
<151> Calafuria
Monacisllo
Casale's Halfway Club
<I45> Opa Cournet Pizza
<I58> Opa
<159> California Pizza Kitchen
<120s Rick '8 Pisza, Rear, More
<I38> Chicago'e Pizza With Twist Reao Virginia
WV<155> Chicago's Fizza With Twiet
<I41> Tasze Chicago
<150> Chicago
Batery
<13: N3 Restaurant
<I22> Food Drink
Rattlesnake Club
Buntonan Brewing
<I11> Flectric Elephant
<134> Heat Street Market
<149> Boulevard Pizza
<136> J''' Fie
<129> Fub N Sub
<I31>
<I23> Krockouts Bar and Hookah Lounge
<I43> Round Table Pizza
Zizze.
Godfather's Fizza
<132> Mountain
Yumny's
<135> Cafe whitney
<115> Gold 'N silver Ian
Atlantic café Alfresco
Korean Barbie
<I1C> Litsle Caeaars
<I38>
<151> Domino'a Pizza
<12> Chock R. cheese
Ground Truth
<I57> Liberty Food Wine Exchange
Korean Barbie
<127>
Casale's Helfway club
<I17> Smiliag With Hope 212za
Nobla Parlor Midrow
A failed example of VISUALLENS on Hit@3. We list the output candidate ranking and ground truth. The ranking follows the same left-right order the input candidates.
Figure 9:
Aspect Word Generation
Here list aspect
given inage user
took at [[Category]]: industrial, pipea, abandoned, overgrown, bluesky.
What are useful sepects zo predict the [[ateegory]] the user will after?
Candidates of [[Category]]:
<Il>
Museun, symbelizing city'e industries sits atop sed Mountain, surrounded by park
<12> sloee Purnaces visicor Center, Blast furnace plant where from
with tours.
Bistoric Cababa Fumping Station,
Ground Fruth:
<I2> 8LosS Purnaces Vigizor Center
Provide useful aspecta help predict truth, separated by conna:
[[hnswer]]
Figure 10: The prompt template for aspect word generation.
Limited modality and domain coverage. Our experiments focus exclusively on static visual history (images) and do not cover more complex modalities such video, audio, or multimodal narratives over time. These richer formats are important for building truly comprehensive, lifelong user models, envisioned by early ideas like MEMEX [4], but remain out of scope for this work.
22

---

Candidate Matching
What is the [[Category]] the user nost likely to go to next?
Aspect worda:
industrial, metal, structure, machinezy,
Captions:
Image 1arge, rusted atructure, poasibly piece industrial machinery or equipment.
Image Induatrial landacapa with tall, metal structures under clear biue sky.
Candidates:
Museun, Statue symoolizing city'a industries sita atop Red Nountain, surrounded by pazk
Volcan
<12> 8'ose Purnaces Visitor Center, Blast furnace plant where education center
iron from 1882
with tours.
Ristoric Cababa Pumping Station.
Answer:
[(Answer))
Figure 11: The prompt template for candidate matching.
Hyperparameters for training on MiniCPM-V2.5
Parameter Size
8B
Image Resolution
980 x 980
Number of Image Tokens
Hidden Dimension Size
4096
LoRA Rank
LoRA
64
LoRA dropout
0.1
GPU
8 NVIDIA H100
Batch Size
Gradient Accumulation Steps Warmup Steps
8
200
Learning Rate
le-3
Table 10: Hyperparameters for training VISUALLEENS with MiniCPM-V2.5 Backbone.
Evaluation scope. We restrict the recommendation task to a QA format for consistency and interpretability. While this setup allows for controlled benchmarking and comparison, does not cover other common recommendation forms such as ranked lists, interactive dialogues, or implicit feedback modeling. Additionally, proposed benchmarks, Google Review-V and Yelp-V, capture a subset of the full task-agnostic recommendation landscape and may not reflect the full range of real-world personalization scenarios.
Future work could explore richer modalities, more diverse recommendation formats, and further improvements to the modular components of VISUALLENS.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Novelty in $2, method in $4, dataset creation in $5 and results in $6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
A limitation section is attached after Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should diseuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: In $5 and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer:
[No]
Justification: The validation and test sets of Google Review-V and Yelp-V is provided as supplementary, but the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/nnips.c.
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See NeurIPS code and data submission guidelines (https://nips http:/iisscccpppppppp wwtccc/pcccggupeueeeeeeeeeecooee Seaiidee//eeebbnne seimiiossiipoccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: In $5 and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We describe the p-value the captions of Table 3 and Table 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
In Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the ines?
NeurIPS Code of Ethics http://errrppssccccccc/////tteia
Answer: [Yes]
Justification: Reviewed NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consideration due to laws or regulations in their jurisdicioo))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Social impact section is after Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Not releasing pretrained language models, and image generators. Datasets are public before, we create benchmark based them.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Acknowledgements section is after Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See $5 and Appendix A.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The authors are human annotators for the human oracle, thus no compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Reviewed LLM policy.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
30


Reviews:
[{"review_id": "dGE3NnZDR5", "reviewer": "Reviewer_bz3j", "summary": "In this paper, the authors proposed VisualLens, a personalized‑recommendation framework that leverages a user’s task‑agnostic visual history. The method employs multimodal large language models to distill each image into a compact “spectrum” representation, forming a rich user profile. To benchmark the approach, the authors release two large‑scale datasets—Google Review‑V and Yelp‑V—and show that VisualLens delivers robust, state‑of‑the‑art performance across both.", "questions": "1. Could you specify which components of your pipeline are fundamentally new beyond CLIP4Clip (2023) and UniMP (2024), and provide an ablation showing the exact contribution (e.g., Hit@k delta) of each?\n2. Have you run any human double‐annotation or LLM‑as‑Judge audits on a stratified sample of Google Review‑V and Yelp‑V to estimate label noise or inter‑annotator agreement?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n\n1. The topic is both intriguing and holds potential for significant developments.\n2. The paper is well-organized and reader-friendly, making it easy to understand.\n3. The experiment results seem promising\n\nCons:\n\n1. The pipeline (CLIP retrieval → image‑grid prompt → LoRA‑tuned MLLM) replicates CLIP4Clip/UniMP, and the only new “aspect‑word” tweak lifts Hit@3 by < 2 pp, so novelty is incremental.\n2. Google Review‑V and Yelp‑V use raw user review tags without any human or LLM audit\n3. Results cover only oracle re‑ranking accuracy; the paper omits retrieval‑recall, latency, and etc", "was_revised": true, "final_justification": "Thanks for the thorough experiments. Many of my concerns are addressed.", "rebuttal": "We thank the reviewers for recognizing the potential impact of VisualLens in advancing personalized recommendation through task-agnostic visual histories. We appreciate the constructive feedback and, below, address the raised concerns and offer clarifications.\n\n**W1:** Which components of your pipeline are fundamentally new beyond CLIP4Clip (2023) and UniMP (2024), and provide an ablation?\n\n**A1:** Thank you for the thoughtful question. First, we would like to clarify that CLIP4Clip (2021) is a video-text retrieval model, while UniMP (2024) proposes a unified multimodal recommendation system trained via multi-task learning. In contrast, VisualLens tackles a fundamentally different problem: **task-agnostic visual recommendation**, which leverages a user’s general visual history (e.g., everyday photos) instead of task-specific interaction data. This setting presents new challenges, as conventional **weak image-text representations (e.g., used in UniMP)** fail to generalize effectively.\n\nVisualLens introduces several key innovations beyond CLIP4Clip and UniMP:\n\n**1. New problem setting:** We formulate the task of task-agnostic visual recommendation, where the user history is broad and not tied to a specific domain or task.\n\n**2. Spectrum image representation:** We enrich raw visual features with captions and aspect words, creating a spectrum of representations that capture varying levels of abstraction:\n+ Visual (dense but noisy),\n+ Captions (moderately abstract),\n+ Aspect words (sparse but precise).\n\nOur ablations in Table 4 demonstrate that each modality contributes meaningfully to performance.\n\n**3. Gridification of images:** To fit within the limited context window of multimodal LLMs, we propose a gridification strategy to encode multiple images efficiently. This is not addressed in CLIP4Clip or UniMP.\n\n**4. Joint fine-tuning:** We jointly train the aspect word generation and the preference prediction modules, improving both performance and efficiency, unlike prior works that rely on disjoint stages or fixed encoders.\n\n**Empirical support:** \n\nWe compare **VisualLens directly with UniMP** using the same retrieval setup in **Table 3, Row 3**. VisualLens-3B consistently outperforms UniMP across both benchmarks:\n+ MRR: +3.0% (Google Review-V), +5.0% (Yelp-V)\n+ Hit@3: +2.2% (Google Review-V), +10.3% (Yelp-V)\n\nFurther ablation studies in Table 4 show that removing captions and aspect words causes VisualLens to underperform UniMP. Notably, removing just the aspect words leads to a **statistically significant drop**:\n+ Google Review-V: -1.3% (Hit@3, MRR)\n+ Yelp-V: -3.6% (Hit@3), -2.5% (MRR), with p < 0.05\n\nThese results highlight the importance of each module in our system and the necessity of robust joint representations for task-agnostic recommendation.\n\nLastly, we are not aware of a 2023 version of CLIP4Clip. If the reviewer is referring to a newer paper, we would appreciate a reference and are happy to address it. Regardless, we will cite CLIP4Clip (2021) in our related work to properly acknowledge its relevance to retrieval-based multimodal systems.\n\n\n**W2:** Have you run any human double-annotation or LLM-as-Judge audits on a stratified sample of Google Review-V and Yelp-V to estimate label noise or inter-annotator agreement?\n\n**A2:** We did not conduct additional human annotations. However, we note that both Google Local Data and Yelp are widely used in the recommendation literature (e.g., [1], [2], [3]). Our work does not introduce new labels but restructures existing ones to support the task-agnostic visual recommendation benchmark. \n\nBesides, to reduce potential noise, we filter out businesses with fewer than 10 reviews, ensuring that each business label is supported by multiple human annotations. Additional details are provided in Appendix A.\n\nReferences:\n+ [1] Li et. al., UCTopic: Unsupervised contrastive learning for phrase representations and topic mining. ACL 2022.\n+ [2] Yan et al., Personalized showcases: Generating multi-modal explanations for recommendation. SIGIR 2023.\n+ [3] Mezentsev et al., Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs. RecSys 2024.\n\n**W3:** Results cover only oracle re‑ranking accuracy; the paper omits retrieval‑recall, latency.\n\n**A3:** For retrieval‑recall, ground truth for retrieval is not available, so we cannot report retrieval recall directly. In addition, recommendation tasks are fairly robust against small retrieval quality differences (as reported in Table 6). Table 4 ablation results suggest the retrieval stage is both necessary and effective, removing it leads to a 6.6% drop in MRR for Google Review-V and 6.1% drop for Yelp-V.\n\nRegarding latency, following reviewer’s suggestion, we measured inference wall-clock time per example on a single NVIDIA L40S GPU. The results demonstrate that the joint image representation learned by VisualLens is more **efficient and effective** than the interleaved multimodal representation used in UniMP, reducing latency by ~0.5 sec.\n\n| Model | Google-Review-V | Yelp-V |\n|------------------------|---------------------|----------------------|\n| **UniMP (3B)**   \t| 5.82s            \t| 4.92s            \t|\n| **Visuallens (3B)**   \t| 5.29s           \t| 4.55s            \t|\n| **Visuallens (8B)**  \t | 7.31s           \t| 6.64s            \t|\n\nWe will include these new latency results in the camera-ready version of the paper."}, {"review_id": "lzuR60MfTv", "reviewer": "Reviewer_bdom", "summary": "The authors present a system of multimodal language models for content recommendation. They intuitively propose a reasonable motivation and an application plan for multimodal large models in recommendation systems, aiming to improve the accuracy of the recommendation system by using users’ visual history as reference information. To this end, the authors introduce VisualLens, which functions like a data annotation pipeline. By leveraging the powerful understanding capabilities of multimodal large models, it re-encodes and compresses historical record information to assist in ranking the recommended content. The authors also propose a corresponding benchmark to validate the effectiveness of the method.", "questions": "- There are too many `Avg. # of images` in Table 2, is the model able to accept so much tokens?\n- It their any comparison to traditional recommandation system?", "limitations": "- It seems that the content is still image-level, while i consider the recommandation is more important for video recommandation.\n- Maybe a unified model can be implemented by getting embeddings from VLM, (like using VLM2Vec(ICLR2025)?)\nBecause the model you used whose ViT is not pre-trained with contrastive loss (for example: NTP loss), the matching may be misaligned.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- I agree with the motivation of this paper, which is to build an intelligent agent to understand user interests and habits for making recommendations. I think this promotes the application of multimodal large models in the next generation of recommendation systems. I believe this could help change the current state of the field, where contrastive learning is primarily used to apply multimodal image-text models in real-world applications.\n- Good performance and lower cost: their implementation achieved better performance while using a small MiniCPM.\n- Detailed description about the pipeline.\n\nWeaknesses:\n- It seems that the authors do not decide to open-source the data and implementation?\n- The 8B model is still big, a large amount of daily active user data on many platforms is difficult to process and infer quickly.", "was_revised": true, "final_justification": "I keep my rating and confidence. The authors resolve my confusion about model sizes, token number and comparison method. I consider this paper advance the MLLM application on recommandation system.", "rebuttal": "We thank the reviewer for the thoughtful and encouraging feedback. We are glad that the motivation behind VisualLens is well received. We appreciate the recognition of our system's practical relevance, the performance improvement, and the detailed design of our pipeline. Below, we respond to the specific questions and concerns raised.\n\n**W1:** Open-source.\n\n**A1:** We plan on open-sourcing the data and implementation (pending legal approval).\n\n**W2:** The 8B model is still big and hard to process data quickly.\n\n**A2:** We agree that 8B models can be resource-intensive. However, it already applies in chatbot use cases, which has tolerance to latency from large models.\nIn addition, VisualLens also performs well with a 3B model, and still **outperforms UniMP of the same size in both accuracy and latency** (results on a single NVIDIA L40S GPU below). \n\n| Model | Google-Review-V | Yelp-V |\n|------------------------|---------------------|----------------------|\n| **UniMP (3B)**   \t| 5.82s            \t| 4.92s            \t|\n| **Visuallens (3B)**   \t| 5.29s           \t| 4.55s            \t|\n| **Visuallens (8B)**  \t | 7.31s           \t| 6.64s            \t|\n\nAs the first study on task-agnostic visual recommendation, our goal is to establish a strong foundation. We anticipate future work exploring smaller models, efficient architectures, and even mobile-friendly deployments.\n\n\n**W3:** Avg. # of user history images is large, how to handle so many tokens?\n\n**A3:** To address VLM context length limitations, we apply image filtering and gridification on user history images, ensuring each example is encoded as a single gridified image with no more than 64 sub-images. This allows us to remain within context limits while retaining informative visual content.\n\n**W4:** Comparison to traditional recommandation system?\n\n**A4:** In Table 3, we compare VisualLens against one of the strongest traditional multimodal recommendation systems, UniMP. We also benchmark against various LLM-based recommendation methods, including both fine-tuned and prompting-based approaches.\n\n**W5:** Mostly image-level, no video-level.\n\n**A5:** We appreciate the reviewer pointing out this important direction for future research. While our current focus is on image-based recommendation, which has direct applications in e-commerce, travel planning, and social media, we agree that extending to video is a valuable next step. We have discussed  this in the Limitation section.  \n\nWe consider our work on static images to be a foundational step. Many state-of-the-art video analysis techniques rely on robust image-level understanding as a core component. One future study can be applying VisualLens to video keyframes and aggregating features over time.\n\n**W6:** Getting embeddings from VLM to implement a unified model?\n\n**A6:** Thank you for your suggestion! We find visual image representations alone are not sufficient for task-agnostic recommendation. This motivated our modular approach using joint image representations to capture more nuanced signals. We agree exploring the possibility of a unified system through better embedding is another promising direction. We will add this to the Limitations section and include related works such as VLM2Vec in our discussion."}, {"review_id": "HykrMvNPXR", "reviewer": "Reviewer_w61j", "summary": "This paper introduces VisualLens, a multimodal LLM-based framework for personalized recommendation using task-agnostic visual history—images from a user's daily life. It constructs a spectrum profile for each user, incorporating raw images, captions, and refined aspect words, and leverages a grid-based image aggregation for model input. The system is jointly trained for both aspect word refinement and recommendation. The authors also contribute two new benchmarks, Google Review-V and Yelp-V, and report improvements over existing SOTA models like GPT-4o and UniMP.", "questions": "- Would the model still work well on non-geolocated data (e.g., Pinterest photos, lifestyle vlogs)?\n\n- How well does the method generalize to non-category-based recommendation queries?\n\n- Is it possible to include metadata like timestamps or image locations, which are often present in photo histories?\n\nPlease also refer to the weakness part.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "For strengths,\n- The core idea of using casual, non-item-tied user images (e.g., photos from daily life) to personalize recommendations is both original and practically appealing.\n- The method consistently outperforms state-of-the-art fine-tuned models like UniMP, and even large proprietary models like GPT-4o, on both Google Review-V and Yelp-V benchmarks—showing the benefit of dedicated adaptation.\n- Google Review-V and Yelp-V are carefully curated datasets with realistic visual logs, candidate sets, and ground-truths, which will likely be valuable to the community.\n\nFor weaknesses,\n- The method is only tested on geographically grounded platforms (Google Reviews, Yelp), where photos often directly relate to physical entities (e.g., restaurants, parks). It remains unclear how this method would perform in abstract or sparse-visual platforms like Amazon, where visual signals are weaker or absent for many users.\n- Since the recommendation relies heavily on surface-level features like aspect words and image similarity, the method is potentially susceptible to irrelevant or adversarial content—e.g., a user uploading meme images or photos unrelated to preferences could degrade performance.\n- While VisualLens combines raw images with generated captions and aspect words, both of which are textual representations, the paper does not isolate the contribution of these text components. In other words, it's unclear whether the performance gain is due to the visual content itself or the associated textual annotations. An ablation using only the captions and aspect words—excluding the image—would help clarify whether visual signals contribute meaningfully beyond their textual projections.", "was_revised": true, "final_justification": "Thanks for the rebuttal. Many of my concerns are addressed. I'll keep my rating.", "rebuttal": "We thank the reviewer for the thoughtful feedback. We appreciate the recognition of our novelty, performance improvement against SOTA models, including GPT-4o and UniMP, as well as the value of our newly introduced Google Review-V and Yelp-V benchmarks. Below, we provide clarifications and address the questions and suggestions raised.\n\n**W1:** Would the model still work well on non-geolocated data?\n\n**A1:** We agree task-agnostic personalized QA on non-geolocated data, such as lifestyle vlogs, is an interesting future direction.Notably, Google Review-V already contains some lifestyle-oriented images (e.g., a cat in the garden), suggesting that VisualLens can generalize to such settings, provided that a sufficient number of relevant images are available (typically around 50, as shown in Figure 3). \n\nHowever, due to personal data privacy concerns, creating a public benchmark for this setting remains challenging. We have discussed this in our Limitation (L615).\n\n**W2:** The method is potentially susceptible to irrelevant or adversarial content.\n\n**A2:** VisualLens begins with an image retrieval step, which helps **filter out irrelevant content before personalization**, reducing the risk of noisy or adversarial inputs. As shown in our ablation studies in Table 4, this retrieval step plays a critical role in maintaining performance. Furthermore, Figure 3 illustrates that performance degrades as the proportion of relevant images decreases. However, **MRR stabilizes once the number of relevant images reaches approximately 50**, demonstrating a degree of robustness under typical usage conditions.\n\n**W3:** How well does the method generalize to non-category-based recommendation queries?\n\n**A3:** Most recommendation queries implicitly or explicitly involve a category, ranging from coarse-grained ones like \"restaurant\" to finer-grained ones like \"Italian restaurant\". In such cases, the category mainly defines the candidate set, and **VisualLens remains applicable regardless of granularity.** Our evaluation benchmarks include broad categories such as \"area\", which resemble open-ended queries like \"give me an idea on where to go?\" (see Figure 4 for the full category list). That said, extremely general prompts like \"recommend something\" fall outside the scope of this paper.\n\n**W4:** An ablation using only the captions and aspect words—excluding the image—would help clarify whether visual signals contribute meaningfully beyond their textual projections.\n\n**A4:** Thank you for the suggestion. We address this in **Table 3, Row 4**. Since Visuallens without image input cannot generate aspect words for joint fine-tuning, we instead evaluate a text-only variant of Visuallens-8B. In this variant, the same backbone LLM (Llama-3.1-8B-Instruct) is fine-tuned using only captions and aspect words. Compared to the full Visuallens-8B model, this leads to an MRR drop of **2.5 on Google-Review-V and 5.5 on Yelp-V**, highlighting the significant contribution of visual signals.\n\n**W5:** Is it possible to include metadata like timestamps or image locations, which are often present in photo histories?\n\n**A5:** Yes, incorporating metadata such as timestamps or location is possible in principle. However, due to the limited context length of our VLM backbone (MiniCPM), we prioritized optimizing image representations. Including metadata is a promising direction for future work."}, {"review_id": "jbb8MhgA89", "reviewer": "Reviewer_UTge", "summary": "This paper proposes VisualLens, a framework that leverages MLLMs to personalize with task-agnostic visual history, i.e., user-taken photos. The method constructs a spectrum user profile using raw images, captions, and aspect words, and employs a three-step runtime pipeline: history retrieval, preference profiling, and candidate matching. The authors also introduce two new benchmarks: Google Review-V and Yelp-V, and demonstrate that VisualLens outperforms existing baselines such as UniMP and GPT-4o by 2–10% in Hit@3 and MRR. Extensive experiments validate its robustness, transferability, and effectiveness.", "questions": "In Figure 2, the model for aspect generation is MiniCPM-V2.5, while in 4.3.2, the author mentioned that the initial set of aspect words is generated by LLaVA-v1.6. Is it a typo? \n\nIn Line 152, the author mentioned that they number each image to ensure the images are grounded to the corresponding caption and aspect words. How these numbers are used as inputs? By editing the images?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths: \n\n- The idea or using task-agnostic visual history is interesting and novel. \n- The overall framework design is clear and meet the needs for using task-agnostic visual history. \n- Strong empirical results with two new, large-scale and diverse benchmarks.\n- Robust and transferable performance across users, categories, and history lengths.\n\n\n\n### Weaknesses: \n\n- Privacy implications of using personal visual history are not discussed.\n- The framework requires multiple specialized components (captioning, retrieval, gridification), each with different baseline model, which may be non-trivial to deploy in real-world settings.", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal and would like to keep my current scores as they are a proper reflection of the quality of the paper in comparison with past NeurIPS accepted papers.", "rebuttal": "We thank the reviewer for the thoughtful feedback. We are glad that the novelty of using task-agnostic visual history for personalization, as well as the clarity and practicality of the VisualLens framework, were well appreciated. Below, we respond to the specific questions and concerns raised.\n\n**W1:** Privacy implications of using personal visual history are not discussed.\n\n**A1:** Thank you for the suggestion. We currently address privacy concerns in the **“Social Impact”** section following the Appendix. We acknowledge that real-world deployments must carefully handle user data collection, storage, and interpretation, especially when recommendations extend beyond a single domain. We will move this discussion to immediately after the main paper in the camera-ready version for greater visibility.\n\n**W2:** Modularized system is hard to deploy in real-world.\n\n**A2:** While the community increasingly favors unified systems as LLMs grow more powerful, modularized architectures still outperform in many real-world applications such as optical character recognition (OCR) and automatic speech recognition (ASR). Despite their structural complexity, modular pipelines are commonly implemented in industry, with well-established engineering practices supporting their development and maintenance.\n\nOn the other hand, the modular design of VisualLens offers greater efficiency than unified models, as demonstrated below. Since the caption and aspect-word generation modules operate offline, they incur no additional cost during inference, thereby reducing runtime latency.\n\n| Model | Google-Review-V | Yelp-V |\n|------------------------|---------------------|----------------------|\n| **UniMP (3B)**   \t| 5.82s            \t| 4.92s            \t|\n| **Visuallens (3B)**   \t| 5.29s           \t| 4.55s            \t|\n| **Visuallens (8B)**  \t | 7.31s           \t| 6.64s            \t|\n\n\n**W3:** The initial set of aspect words is generated by LLaVA-v1.6. Is it a typo?\n\n**A3:** No, this is correct. The initial aspect words are generated by LLaVA-v1.6. After finalizing these aspect words, we train MiniCPM to generate them for subsequent use. We will further clarify this in the camera-ready version of the paper.\n\n**W4:** How these image numbers are used as inputs? By editing the images?\n\n**A4:** Yes, each (sub-)image is annotated with a number in the upper-left corner, which is then used as part of the input to the model. This idea is inspired by [1], which used numbering to help the model better ground to parts of an image.\n\nReferences:\n+ [1] Yang et al., Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V."}]

Author rebuttal:

Meta-review: The reviews (5,4,4,4) for this paper have been collected and discussed. There is a general consensus among the reviewers that the paper, in its current form, is suitable for publication. After careful consideration of the feedback and the paper itself, the recommendation is to accept this submission.
Decision: Accept (poster)