Title: VisualLens: Personalization through Task-Agnostic Visual History
Venue: NeurIPS 2025
Accepted: yes
Keywords: Recommendation

Personalization

LLM

Multimodal LLM
Abstract: Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals. 
However, item-based histories are not always accessible and generalizable for multimodal recommendation.
We hypothesize that a user's visual history --- comprising images from daily life --- can offer rich, task-agnostic insights into their interests and preferences, and thus be leveraged for effective personalization.
To this end, we propose VisualLens, a novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task-agnostic visual history.
VisualLens extracts, filters, and refines a spectrum user profile from the visual history to support personalized recommendation.
We created two new benchmarks, Google-Review-V and Yelp-V, with task-agnostic visual histories, and show that VisualLens improves over state-of-the-art item-based multimodal recommendations by 5-10\% on Hit@3, and outperforms GPT-4o by 2-5\%.
Further analysis shows that VisualLens is robust across varying history lengths and excels at adapting to both longer histories and unseen content categories.

VisualLens: Personalization through Task-Agnostic Visual History

Abstract

Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals. However, item-based histories are not always accessible, and are not generalizable for multimodal recommendation . We hypothesize that a user's visual history comprising images from daily life - can offer rich, task-agnostic insights into their interests and preferences, and thus be leveraged for effective personalization. To this end, we propose VisualLens , a novel framework that leverages multimodal large language models (MLLMs) to enable personalization using task-agnostic visual history. VisualLens extracts, filters, and refines a spectrum user profile from the visual history to support personalized recommendation. We created two new benchmarks, Google Review-V and Yelp-V , with task-agnostic visual histories, and show that VisualLens improves over state-of-the-art item-based multimodal recommendations by 5-10% on Hit@3, and outperforms GPT-4o by 2-5%. Further analysis shows that VisualLens is robust across varying history lengths and excels at adapting to both longer histories and unseen content categories.

1 Introduction

Imagine a personal assistant, similar to Vannevar Bush's Memex [4], observing what you do in your daily life. With her keen insight, she can make informed guesses about what you may enjoy or find intriguing. When you ask for recommendations on anything from restaurants and activities to movies, books, and products, based on her in-depth understanding of you she will provide suggestions tailored specifically to your tastes.

While the concept is intuitive, a truly comprehensive personal assistant capable of making recommendations across all aspects of life has yet to be realized. Most existing multimodal recommendation systems remain domain-specific and rely heavily on item-based interaction histories [41, 34]. For example, an e-commerce platform may suggest products based on past purchases but ignore dining habits or interests outside shopping.

This work explores how can we leverage such a user's visual record to better understand individual preferences and enable more general, personalized recommendations. Achieving task-agnostic recommendations from visual history poses several challenges. First, visual histories are often diverse and noisy , containing images unrelated to any specific recommendation task, entities that fail to accurately represent user preference, or non-informative elements ( e.g ., background objects like

Recommend some museums for me

in Los Angeles.

User Provided Photo History

0

70

UniMP

GPT-4o

VisualLens

59.1

54.5

48.5

38.9

37.3

34.1

Los Angeles County Museum of Art (Art Museum)

California Science Center (Local Science Museum)

3. …

VisualLens

GoogleReview-V

Yelp-V

Hit@3

Human Oracle

45.0

66.0

trash cans). This creates a trade-off between preserving rich visual content and extracting clean, interpretable representations of user interests. Second, most current MLLMs can only process a limited number of images, requiring selective retrieval of relevant user profile information for each query. Third, existing benchmarks are inadequate for evaluating task-agnostic visual recommendation systems, highlighting the need for new, purpose-built evaluation datasets.

We propose a novel framework VisualLens, as a first step towards harnessing a user's visual history for MLLM recommendation. VisualLens begins by extracting an offline spectrum user profile, compressing each image in the visual history into a triplet: ( raw image, caption, aspect words ). This representation spans a spectrum from rich but noisy content ( raw image ) to concise but clean semantic cues ( aspect words ). To improve the quality of aspect words, we employ an iterative refinement process that progressively enhances their alignment with user interests. Next, to efficiently incorporate multiple images during runtime recommendation, VisualLens retrieves the most relevant segments of the user profile based on the query context. These selected images are organized into a d × d visual grid, accompanied by their corresponding captions and aspect words, enabling the model to jointly process and reason over a compact yet informative representation of the user's visual history. Finally, we train the system to perform both aspect refinement and recommendation question-answering (QA) in a unified model. This design not only reduces parameter overhead but also strengthens the model's ability to interpret and utilize visual history for accurate, personalized recommendations.

To facilitate the evaluation, we created two new benchmarks, Google Review-V and Yelp-V , providing a foundation for personalization assessments. We leverage user-taken photos to address the challenge of history availability. Unlike extensive Memex video logs, these photos require less storage, often available in reviews and social media posts, and provide more insights into a user's interests and preferences. Each benchmark includes a standard test set targeting generalization to new users , along with two additional test sets for transferring to longer histories and unseen categories .

Our experimental study shows promising recommendation quality of VisualLens. It achieved 82-91% Hit@10 on the Google Review-V and Yelp-V benchmarks, outperforming state-of-the-art (UniMP [46]) by ∼ 10% . Even comparing with GPT-4o, our 8B model improves Hit@3 by 1.6% and 4.6% respectively on the two benchmarks. Further analysis reveals that VisualLens excels in adapting to longer histories and unseen categories, while maintaining robustness with shorter histories.

2 Related Works

Recommendation system with large language models. Large Language Models (LLMs) have demonstrated strong potential in recommendation systems with their advanced language processing capabilities [41]. On item-based recommendation, studies such as LLM4RS [6], LLMRank [19], CLLM4Rec [62], P5 [12], and Sanner et al. [38] explored various LLM prompt and bootstrapping strategies, showing competitive performance, especially in cold-start scenarios. Generative recommendation with open-domain prompts is explored by GenRec [21], though fine-tuning remains crucial. Fine-tuning approaches include personalized aspect extraction [26], and multitasking on

LLaMa models [53]. Retrieval-enhanced models, such as ReLLa, improve recommendation by retrieving relevant behavior sequences [27]. Instruction-tuning and graph augmentation approaches are explored in InstructRec [57], LLMRec [48], and LKPNR [15]. Jang et al. [20] use RLHF methods to improve personalizations as well. Recent advances in personalized conversation systems have utilized task-agnostic conversation logs to provide personalized answers [23, 16, 31, 29, 51]. However, these content-based recommendation approaches predominantly rely on textual data.

Multimodal recommendation systems. Multimodal recommendation systems [45, 54, 39] leverage multiple data types, such as text and images, to improve recommendation relevance and personalization. Before the LLM era, Lee and Abu-El-Haija [22] proposed systems for content-only video recommendation using similarity learning. PC 2 L [52] develop an LLM model that provides multimodal explanations for recommendations. On modalities beyond image and text, MMRF [59] built a joint recommendation system integrating comments with video items [7, 8, 11, 58]. RecFormer [25] and UniSRec [18] convert images to short captions to utilize text-only models. The current state-of-the-art image-text recommendation, UniMP [46], extended single-task multimodal personalization [17, 43, 49, 47] on multitask website-based shopping. However, most existing multimodal recommendation approaches rely on item-based user history, which is not always available. To address this limitation, we propose VisualLens, a novel framework that leverages MLLMs to enable personalization using task-agnostic visual history .

We discuss more related works on traditional recommendation and recommendation benchmarks in Appendix B.

3 Multimodal Task-Agnostic Recommendation

Consider a recommendation QA task, where the user asks a recommendation question q , and the recommender answers q with a ranked list of candidate items . Good recommenders shall rank the items that the user is more likely to be interested in or willing to try early in the list.

In multimodal task-agnostic recommendation, the recommender is facilitated with a task-agnostic visual history H u for each user u , which contains a series of photos, taken or posted by the user, not necessarily related to q . We state three assumptions to allow generalization. First, the photos may not be directly relevant to the question. Second, an image may not be associated with any candidate item, and even so, the candidate ID is not given. Third, a photo does not necessarily imply strong preferences. Figure 1 shows an example question, visual history, and candidates.

To simplify the problem, we assume a candidate retriever exists to retrieve all candidates that satisfy the user's question. Each candidate s is represented with a ( x s , I s ) pair, where x s is the name and text descriptions, and I s is an optional image set for the item.

Traditional recommendation setting considers two more types of signals. The first is a task-specific set of items in the candidate set, which captures user interest or at least user history. The second is a set of user-specific attributes such as the user's age, gender, and interests. This paper focuses on task-agnostic visual history and leaves integration of these traditional signals for future extensions.

Candidate matching

(MiniCPM-v2.5)

User history

Question

What is the

museum

the user most likely to go to next?

Candidates

<I1> Vulcan Park and Museum...

<I2> Sloss Furnaces Visitor Center…

<I3> Historic Cahaba Pumping Station…

…

History augmentation

Caption generation

(LLaVA-v1.6)

Aspect generation

(MiniCPM-v2.5)

Relevant history

History retrieval

(CLIP ViT-L/14@336px)

Captions

Image 1. A large, white building three-story building with a clock tower…

Image 2. the entrance to a museum, with a sign 'MUSEUM ENTRANCE'…

Image 3. …

Aspect words

Image 1. museum, arches, bench, dome, light, streetlight, night, balcony, ...

Image 2. museum, entrance, glass, black, iron, gate, reflection, plants, …

Image 3. …

<I3>

<I1>

<I2>

…

Prediction

Rank candidates

with the output

probability

O

F

F

L

I

N

E

R

U

N

T

I

M

E

Frozen

Tunable

Shared

Weights

Preference profiling

(Gridify / Concatenation)

Images

Captions

Aspect words

4 VisualLens Framework

VisualLens framework contains two parts in inference (Figure 2): offline user profile generation , and runtime recommendation . Offline user profile generation (§4.1) augments each image in the visual history with captions and aspect words , and builds a spectrum user profile. Runtime recommendation (§4.2) answers a recommendation query q in three steps. First, the history retrieval step retrieves only images relevant to q , since the visual history can be diverse and not all photos are relevant to every query. Second, the preference profiling step uses the retrieved images and their augmented captions and aspects to generate a query-specific profile of the user. Third, the candidate matching step matches the query-specific profile with each candidate, to generate the confidence score for each candidate for ranking. We discuss the joint training algorithm of VisualLens in §4.3.

4.1 Offline user profile generation

To build a user profile that retains rich visual content while offering clean, interpretable cues about user interests, we augment each image with a caption and a set of aspect words. Each image is thus represented as a spectrum triplet - (raw image, caption, aspect words) - ordered by decreasing information richness and increasing semantic clarity. Ablation results (Table 4) confirm that these augmentations improve recommendation performance.

Image encoding. Each image is encoded using the CLIP ViT-L/14@336px model [36], producing embeddings used for history retrieval at query time.

Caption generation. Captions are generated using a frozen LLaVA-v1.6-8B model [28], prompted to produce concise ( ≤ 30 words) and grounded descriptions to minimize hallucinations.

Aspect word generation. Aspect words are concise descriptors of key image attributes ( e.g ., dome, balcony). We prompt the model to list relevant terms without constraining the count, allowing flexibility based on image complexity.

All modules are plug-and-play and can be replaced with stronger alternatives. While LLaVA-v1.6 occasionally produces irrelevant or generic aspect words ( e.g ., blue, sky), we describe in Section 4.3.3 how joint finetuning improves their utility for recommendation tasks.

4.2 Runtime recommendation

History retrieval. Given a query q and a user's visual history H u , VisualLens first retrieves images that are related to q , denoted by I u,q . We choose up to w images to cap the number of images we

process at runtime, and ensure that only the most contextually relevant images are retained for further processing, thereby reducing noise.

In general, we can use any image retrieval method such as DELG [5]. Here, we present a method for categorical recommendations such as restaurants and museums , popular in recommendation tasks. For each category c , we randomly select a set of candidate items in the category and average the visual embeddings of their images as the image embedding of the category , denoted by v . Specifically,

(see Appendix C for sensitivity), and v ( j ) c indicates the visual embedding of the j -th item image in category c . The retrieval step measures the cosine similarity between the visual embedding v i of each image i ∈ H in the user's history and the image embedding v c of the relevant category c . We then select the topw images based on the cosine similarity scores.

c the category embedding is calculated as v c = 1 n ∑ n j = 1 v ( j ) c , where n is the number of candidates

Preference profiling. Given a set of retrieved images I u,q , VisualLens then generates user's query-specific profile. A critical part of this step is image encoding. Even after retrieval, the number of images w is still large. Most MLLMs allow context windows of limited sizes, constraining the number of images we can process. For example, for an input image of resolution 896 × 896 , the PaliGemma model would generate an embedding of up to 4,096 tokens. A typical LLM with a context window of 8,192 tokens can take at most 2 images.

We propose to group relevant images I u,q into a d × d grid, where d 2 = w , and treat all images in the grid as a single image . If we have retrieved fewer than w images, we pad with a black background. Let h be the maximum available resolution in a multimodal LLM. The gridify process G takes the d × d grid and generates an image of fixed size R h × h × 3 . Additionally, we number each image from 1 to d 2 to ensure the images are grounded to the corresponding caption and aspect words in the input to the candidate matching. We denote a user u 's profile on question q by ( i u,q , x u,q ) , where i u,q denotes the gridified image, and x u,q denotes the concatenated captions and aspect words of relevant images.

Candidate matching. Finally, VisualLens takes the query-specific user profile ( i u,q , x u,q ) and a set of candidates, each represented by ( x s , I s ) , and generates the matching score for each candidate, which will then be used for ranking. This is achieved by prompting the multimodal candidate predictor, where we packed the user profile and candidates to the prompt through the image channel and the text channel separately (see the prompt template in Appendix D).

4.3 Iterative Refinement and Joint Training

VisualLens relies on LLMs for image encoding, caption generation, aspect word extraction, and profile-candidate matching. Current off-the-shelf models perform suboptimally for these tasks, so we apply continued pretraining and task-specific fine-tuning to enhance performance.

4.3.1 Multi-image caption pretraining

To facilitate the model to ground to each grid faithfully, we perform a LoRA continual pretraining on dense captions. We adopt the dense captioning dataset DOCCI dataset [32], which contains over 15,000 images and their corresponding dense captions. Each time, we randomly sample w images I = { i 1 , ⋯ , i w } and their corresponding caption C = { x 1 , ⋯ , x w } , and then we construct a gridified input image G ( I ) and a target output text description T ( C ) = ' Image 1: x 1 , ⋯ , Image w: x w '. Then we LoRA finetune the pretrained backbone model ( e.g ., MiniCPM-V2.5) on all image-caption pairs { G ( I ) , T ( C )} so that the model is able to process the gridified user history grid by grid. We then use the continual pretrained model as the starting point to apply joint training described in §4.3.3.

4.3.2 Iterative aspect word refinement

Unlike image captioning, aspect word generation is not a standard multimodal task and lacks the extensive pretraining data. Hence, with zero-shot prompting, the generated aspect words have a large quality gap across different images, and the extracted aspects may not indicate user preferences.

To finetune aspect word generation, we first generate the training data. For each image i , we start from an initial set of aspect words, denoted as W ( 0 ) i , which is generated by LLaVA-v1.6. In the j th round, we prompt a separate Llama-3.1 70B model with W ( j -1 ) i candidates, and ground truths, and ask it to select useful aspect words that are helpful in ground truth prediction, which constitute W ( j ) i . This refinement process continues for several rounds, and the iterations allow for converging extracted aspect words toward a more accurate and relevant subset. Empirically, we observe that the refinement converges after approximately 4 rounds, and denote the 4 th refined aspect word set W ( 4 ) i as W i , which serves as the training target.

The backbone model with parameter θ is finetuned to optimize the cross-entropy (CE) loss over all images I ,

where x asp is the prompt for aspect words generation.

4.3.3 Joint training of aspect word generation and candidate matching

To take advantage of multitask training in multimodal recommendation [46], we jointly train the aspect word generator and the candidate predictor on the backbone model. This joint training strategy allows the model to simultaneously learn to identify useful aspect words and make accurate predictions, thus improving overall performance.

The joint loss function balances aspect word generation and candidate matching with a weighting factor λ , where the candidate matching is optimized with binary cross-entropy (BCE) loss to handle multiple ground truth labels.

where u j , q j , S j is the user, the question, and the ground truth set of candidates of the j -th example. The text prompt x pred ,j consists of the question q j , the text query-specific profiles x u j ,q j , and candidates. We LoRA finetune the model under the joint loss L joint .

5 Benchmarks and Experiments Setups

5.1 Benchmark creation

To the best of our knowledge, there is no existing benchmark [16, 31, 50, 44, 37] to evaluate personalization with task-agnostic visual history. We created two benchmarks, Google Review-V and Yelp-V, leveraging publicly available data from Google Local Review [24] and Yelp [2].

User logs: For each user in the two datasets, we take the list of reviews in chronological order. Each review is associated with a business name, categories and description . In Google Review-V, each review is associated with a few photos, used for image logs. Yelp-V does not associate a review with photos, so we randomly subsample one-third of the store profile pictures, such that different reviews for the same business can be associated with different images.

Questions and visual history: We consider a special type of questions, category recommendations , like "Recommend a nearby museum" . Such questions are both popular in real applications, and hard as there are many candidates satisfying the constraint. We remove small categories and most ambiguous categories, such as 'place', and 'spot'.

For each review r regarding a business of category c , we create a question asking to recommend businesses in category c . We take all (and only) photos in the reviews before r to generate the visual history. We consider the visual history task-agnostic since the categories are highly diverse (see Figure 4), and the photos are quite diverse too ( e.g., a park photo to illustrate happiness mentioned in the review). We filter cases where the history is too short ( < 10 ) or does not contain the questioned category. By doing so, we ensure that the user history contains at least 10 relevant images ( i.e ., from the same category as the query) for each example in both Google Review-V and Yelp-V.

Candidates and ground truths: For a review r we take all reviews starting from r to generate candidates and ground truths. To be realistic, we consider only nearby businesses of the same category as the candidate set, and the number of candidates is a random number in [ 30 , 100 ] . Candidates that also appear in the user's future reviews are considered as ground truths. To avoid falling into a classification problem, we filter examples with only 1 ground truth in Google Review-V and fewer than 5 in Yelp-V.

Summary: Table 2 gives the benchmark statistics. The ratio of an average number of candidates and that of ground truths is 16:1 for Google Review-V and 8:1 for Yelp-V. By default, the train, dev, and test data have disjoint users. We discuss other splitting in Table 5 and more details in Appendix A.

5.2 Evaluation measures

We use two metrics to evaluate recommendation quality.

Hit@ k . Hit@ k = 1 N ∑ N i = 1 ✶ [ rank ( r i ) ≤ k ] checks if any relevant item is within the topk ranked results, where N is the number of examples, and rank ( r i ) is the rank of the first relevant item. We check Hit@3 ( e.g ., voice recommendations) and Hit@10 ( e.g ., on-screen recommendations).

Mean Reciprocal Rank (MRR). MRR = 1 N ∑ N i = 1 1 rank ( r i ) measures the ranking quality by averaging the reciprocal ranks of the first relevant item for each example. The MRR ranges from 1 / S to 1 , where S is the number of candidates.

Additionally, we report wall-clock inference time in Appendix C to show the efficiency of VisualLens.

5.3 Implementation and baselines

We ran VisualLens with two backbone models, a smaller 3B model PaliGemma and a larger 8B model MiniCPM-V2.5. For optimal performance, we selected w = 64 (112x112 each sub-image for PaliGemma, best in practice), corresponding to an 8 × 8 image grid, a candidate count of n = 10 k, and a loss weighting factor of λ = 2 (Eqn. 3). We compared VisualLens with solutions below.

Baselines : Randomly rank or select topk ratings.

Fine-tuned models : We fine-tuned three state-of-the-art solutions: multimodal personalization model UniMP [46] (RedPajama 3B), with the adaptation to replace item images and attributes with image tokens in the visual history; Llama-3.1-8B-Instruct [30] with text-only user preference profiles; PaliGemma 3B and MiniCPM-V2.5 8B [56] with image profiles.

Direct inference : We compared with two out-of-box models: Llama-3.1-70B-Instruct [30] with text-only preference profiles; GPT-4o [33] with multimodal profiles. To control the API cost, we subsample 1k instances from the test sets.

Human annotation : Finally, we subsampled 50 examples from the test set for human annotation.

6 Results and Analysis

We conducted experiments to answer four questions:

Q1 : Can we effectively leverage a user's visual history to improve personalization?

Q2 : How does each element of VisualLens contribute to the recommendation quality?

Q3 : Can VisualLens transfer across users, unseen categories, longer history, and new benchmark?

Q4 : What is the robustness of VisualLens?

6.1 Recommendation effectiveness (Q1)

VisualLens significantly outperforms baselines. Our first observation from Table 3 is that all recommendation models that leverage the visual history significantly outperform baseline solutions on all metrics. In particular, the best version of VisualLens improves Hit@3 over Random by 18% on Google Review-V and by 26% on Yelp-V, and even more over Rank by rating (apparently, overall ratings do not cater to specific users). There is still a gap between VisualLens and human annotations, but comparing w. Random , it fills ∼ 75% of the gaps for hit@3 on both datasets.

VisualLens outperforms state-of-the-art solutions. VisualLens outperforms UniMP with the same number of trainable parameters. With the 8B MiniCPM-V2.5 backbone, VisualLens outperforms MiniCPM-V2.5 itself by 2.5% on Hit@3 on Google Review-V, and by 6% on Yelp-V, and we observe a similar trend for the 3B models. Even comparing with significantly larger models, including Llama-3.1-70B-Instruct and GPT-4o without fine-tuning, VisualLens 7B improves by 1.6-5.6% on Hit@3.

10

20

30

40

50

60

70

80

90

100

# of Candidates

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MRR

Google Review-V

Yelp-V

(a)

10

20

30

40

50

100

150

200

250

300

400

500

600

700

800

900

# of Images

0.0

0.1

0.2

0.3

0.4

0.5

MRR

Google Review-V

Yelp-V

(b)

6.2 Ablation studies (Q2)

We evaluate the usefulness of each component in VisualLens in Table 4, with PaliGemma as the backbone. We find history retrieval significantly improves the results, and can reduce Hit@3 by 7% on Google Review-V and by 12% on Yelp-V (#7 vs. #3). Besides, all three representations of the images (embedding, caption, aspects) play an important role. Removing captions and aspect words can reduce Hit@3 by 3% on Google Review-V and by 9% on Yelp-V, even without fine-tuning (#6 vs. #3). Between the two, aspect words play a more important role than captions (#5 vs. #4). Moreover, both iterative training and joint multi-task training improve the recommendation quality. Removing both of them lowers Hit@3 by 1%+ on both data sets (#3 vs #1).

6.3 Transferability (Q3)

We tested transferability over users (default setting), over categories, and over different (longer) history lengths. Table 5 compares the MRR of VisualLens on both benchmarks, showing good transferability. MRR is highest when applied to longer history, with the effectiveness of history retrieval. Transferability is higher across categories than across users, both demonstrating the promise of leveraging task-agnostic signals, and illustrating slightly more challenges to transfer between users of different interest patterns. We also present the generalization results across benchmarks in Appendix C.

6.4 Robustness and qualitative analysis (Q4)

We conducted several robustness tests as follows.

Candidate count: Figure 3a shows that as the number of candidates grows, the recommendation becomes harder and MRR gets lower. However, when the number of candidates exceeds 50, MRR converges.

Image count in user history: Figure 3b shows that as the history grows, MRR increases and flattens after reaching ∼ 100 images. This trend is related to our grid size 8 × 8 , as smaller than 64 images will not leverage all spaces in the grid. On the other hand, flat MRR after 100 images shows robustness against history noises with the retrieval step.

Categories

26

28

30

32

34

36

38

40

42

44

MRR

Google Review-V Categories

restaurant

bar

tourist attraction

park

grill

shopping mall

bakery

supermarket

steak house

station

hotel

brewery

market

museum

deli

theater

takeout

pub

area

historical landmark

club

brewpub

salon

beach

garden

gastropub

golf course

atm

nature preserve

church

winery

car wash

campground

lodge

10 4

10 5

10 6

10 7

Number of training examples

(a)

Categories

38

40

42

44

46

48

50

MRR

Yelp-V Categories

restaurant

bars & pubs

burgers & sandwiches

breakfast & brunch

coffee & tea

seafood

pizza

desserts

salad

bakeries

steakhouses

yogurt & cream & cupcakes

barbeque

grocery

soup

bubble tea

buffets

chocolatiers

hotel

10 3

10 4

10 5

10 6

Number of training examples

(b)

Category distribution: Figure 4 plots MRR for different categories. There are a few factors that can affect recommendation quality. First and foremost, ambiguous categories like 'area', 'station', and 'market' get the lowest MRR in Google Review-V. Second, general categories ( e.g ., 'museum', 'hotel') with bigger category sizes obtain higher MRR than specific ones ( e.g ., 'historical landmark'). Third, transfer learning happens between neighbor categories; for example, 'deli' and 'takeout' achieve the top2 and top3 performance with less training data, since they are similar to the largest category 'restaurant'. We provide more qualitative analysis in Appendix C.

7 Conclusion and Discussion

In this paper, we proposed a novel approach VisualLens for personalized recommendation using task-agnostic visual history. We advanced MLLMs to extract spectrum signals from images to serve as user profile. We created two benchmarks, Google Review-V and Yelp-V, to evaluate VisualLens, affirming the efficacy and robustness of VisualLens.

VisualLens offers a promising first step toward MLLM recommendation systems that leverage task-agnostic visual history. Several future directions remain. First, we could integrate VisualLens with additional data, such as image timestamps, locations, recognized fine-grained entities ( e.g ., specific products), and user profiles. Second, we aim to extend the recommendation problems explored here to encompass broader QA tasks. Finally, we plan to investigate privacy-preserving techniques, such as federated learning, during training.

Social Impact

The ability to model user preferences from visual histories raises important considerations around privacy, consent, and data usage. While our work uses publicly available and anonymized images, real-world deployments would need to carefully address how user data is collected, stored, and interpreted, especially when recommendations extend beyond a single domain. There is also potential for reinforcing biases or overfitting to superficial cues ( e.g ., location, aesthetics) that may not reflect deeper user intent. We encourage future research to incorporate fairness auditing, privacy-preserving training, and transparency mechanisms to ensure responsible use of multimodal personalization systems.

References

Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet, 2024. 17

Nabiha Asghar. Yelp dataset challenge: Review rating prediction, 2016. 6, 17

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: A versatile 3b vlm for transfer, 2024. 7, 17

Vannevar Bush. As we may think. The Atlantic Monthly , 176(1):101-108, 1945. 1, 22

Bingyi Cao, Andre Araujo, and Jack Sim. Unifying deep local and global features for image search. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16 , pages 726-743. Springer, 2020. 5

Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In Proceedings of the 17th ACM Conference on Recommender Systems , page 1126-1132. ACM, 2023. 2, 3

James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. The youtube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems , page 293-296, New York, NY, USA, 2010. Association for Computing Machinery. 3

Xingzhong Du, Hongzhi Yin, Ling Chen, Yang Wang, Yi Yang, and Xiaofang Zhou. Personalized video recommendation using rich contents from videos. IEEE Transactions on Knowledge and Data Engineering , 32(3):492-505, 2020. 3

Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomorphic representations. In First Conference on Language Modeling (COLM) , 2024. 17

Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. TLDR: Token-level detective reward model for large vision language models. In The Thirteenth International Conference on Learning Representations , 2025. 17

Junyu Gao, Tianzhu Zhang, and Changsheng Xu. A unified personalized video recommendation via dynamic recurrent neural networks. In Proceedings of the 25th ACM International Conference on Multimedia , page 127-135, New York, NY, USA, 2017. Association for Computing Machinery. 3

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). Proceedings of the 16th ACM Conference on Recommender Systems , 2022. 2

Google. Gemini: A family of highly capable multimodal models, 2023. 17

Yulong Gu, Zhuoye Ding, Shuaiqiang Wang, Lixin Zou, Yiding Liu, and Dawei Yin. Deep multifaceted transformers for multi-objective ranking in large-scale e-commerce recommender systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , page 2493-2500, New York, NY, USA, 2020. Association for Computing Machinery. 17

Chen Hao, Xie Runfeng, Cui Xiangyang, Yan Zhou, Wang Xin, Xuan Zhanwei, and Zhang Kai. Lkpnr: Llm and kg for personalized news recommendation framework, 2023. 3

F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) , 5(4), 19. 3, 6, 17

Ruining He and Julian McAuley. Vbpr: visual bayesian personalized ranking from implicit feedback. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , page 144-150. AAAI Press, 2016. 3

Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In KDD , 2022. 3

Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems, 2024. 2

Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023. 3

Jianchao Ji, Zelong Li, Shuyuan Xu, Wenyue Hua, Yingqiang Ge, Juntao Tan, and Yongfeng Zhang. Genrec: Large language model for generative recommendation, 2023. 2

Joonseok Lee and Sami Abu-El-Haija. Large-scale content-only video recommendation. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) , pages 987-995, 2017. 3

Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, and Michael Bendersky. Teach llms to personalize - an approach inspired by writing education, 2023. 3

Jiacheng Li, Jingbo Shang, and Julian McAuley. UCTopic: Unsupervised contrastive learning for phrase representations and topic mining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6159-6169, Dublin, Ireland, 2022. Association for Computational Linguistics. 6, 17

Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley. Text is all you need: Learning language representations for sequential recommendation. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023. 3

Pan Li, Yuyan Wang, Ed H. Chi, and Minmin Chen. Prompt tuning large language models on personalized aspect extraction for recommendations, 2023. 2

Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation, 2024. 3

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 4, 17

Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. Once: Boosting content-based recommendation with both open- and closed-source large language models, 2023. 3

Meta. The llama 3 herd of models, 2024. 7, 8, 17

Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 188-197, Hong Kong, China, 2019. Association for Computational Linguistics. 3, 6, 17

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge. Docci: Descriptions of connected and contrasting images, 2024. 5, 11

OpenAI. Gpt-4o, 2024. 7, 8, 17

Harris Papadakis, Antonis Papagrigoriou, Eleftherios Kosmas, Costas Panagiotakis, Smaragda Markaki, and Paraskevi Fragopoulou. Content-based recommender systems taxonomy. Foundations of Computing and Decision Sciences , 48(2):211-241, 2023. 1

Letitia Parcalabescu and Anette Frank. Do vision & language decoders use images and text equally? how self-consistent are their explanations? ArXiv , abs/2404.18624, 2024. 17

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , 2021. 4

Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization, 2023. 6, 17

Scott Sanner, Krisztian Balog, Filip Radlinski, Ben Wedin, and Lucas Dixon. Large language models are competitive near cold-start recommenders for language- and item-based preferences, 2023. 2

Xiang-Rong Sheng, Feifan Yang, Litong Gong, Biao Wang, Zhangming Chan, Yujing Zhang, Yueyao Cheng, Yong-Nan Zhu, Tiezheng Ge, Han Zhu, Yuning Jiang, Jian Xu, and Bo Zheng. Enhancing taobao display advertising with multimodal representations: Challenges, approaches and insights. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , page 4858-4865, New York, NY, USA, 2024. Association for Computing Machinery. 3

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management , page 1441-1450, New York, NY, USA, 2019. Association for Computing Machinery. 17

Zhaoxuan Tan and Meng Jiang. User modeling in the era of large language models: Current research and future directions, 2023. 1, 2

Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining , page 565-573, New York, NY, USA, 2018. Association for Computing Machinery. 17

Jinhui Tang, Xiaoyu Du, Xiangnan He, Fajie Yuan, Qi Tian, and Tat-Seng Chua. Adversarial training towards robust multimedia recommender system. IEEE Transactions on Knowledge and Data Engineering , 32(5):855-867, 2020. 3

Mengting Wan and Julian J. McAuley. Item recommendation on monotonic behavior chains. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018 , pages 86-94. ACM, 2018. 6, 17

Jie Wang, Fajie Yuan, Mingyue Cheng, Joemon M. Jose, Chenyun Yu, Beibei Kong, Xiangnan He, Zhijin Wang, Bo Hu, and Zang Li. Transrec: Learning transferable recommendation from mixture-of-modality feedback, 2022. 3

Tianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng, Zhengyang Wang, Jianhui Sun, Qingyu Yin, Hanqing Lu, Suhang Wang, Jingrui He, and Xianfeng Tang. Towards unified multi-modal personalization: Large vision-language models for generative recommendation and beyond, 2024. 2, 3, 6, 7, 8

Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. Multi-modal self-supervised learning for recommendation. In Proceedings of the ACM Web Conference 2023 , pages 790-800, 2023. 3

Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Llmrec: Large language models with graph augmentation for recommendation, 2024. 3

Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro-video. In Proceedings of the 27th ACM International Conference on Multimedia , pages 1437-1445, 2019. 3

Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. MIND: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 3597-3606, Online, 2020. Association for Computational Linguistics. 6, 17

Wentao Xu, Qianqian Xie, Shuo Yang, Jiangxia Cao, and Shuchao Pang. Enhancing content-based recommendation via large language model. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , page 4153-4157, New York, NY, USA, 2024. Association for Computing Machinery. 3

An Yan, Zhankui He, Jiacheng Li, Tianyang Zhang, and Julian McAuley. Personalized showcases: Generating multi-modal explanations for recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , page 2251-2255, New York, NY, USA, 2023. Association for Computing Machinery. 3, 17

Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. Palr: Personalization aware llms for recommendation, 2023. 3

Jia-Qi Yang, Chenglei Dai, Dan OU, Dongshuai Li, Ju Huang, De-Chuan Zhan, Xiaoyi Zeng, and Yang Yang. Courier: Contrastive user intention reconstruction for large-scale visual recommendation, 2024. 3

Yuhao Yang, Chao Huang, Lianghao Xia, Yuxuan Liang, Yanwei Yu, and Chenliang Li. Multi-behavior hypergraph-enhanced transformer for sequential recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , page 2263-2274, New York, NY, USA, 2022. Association for Computing Machinery. 17

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint 2408.01800 , 2024. 7, 8

Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation as instruction following: A large language model empowered recommendation approach, 2023. 3

Bowen Zheng, Zihan Lin, Enze Liu, Chen Yang, Enyang Bai, Cheng Ling, Wayne Xin Zhao, and Ji-Rong Wen. A large language model enhanced sequential recommender for joint video and comment recommendation, 2024. 3

Peilun Zhou, Xiaoxiao Xu, Lantao Hu, Han Li, and Peng Jiang. A model-based multi-agent personalized short-video recommender system, 2024. 3

Wang Zhu, Jesse Thomason, and Robin Jia. Generalization differences between end-to-end and neurosymbolic vision-language reasoning systems. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 4697-4711, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. 17

Wang Zhu, Alekh Agarwal, Mandar Joshi, Robin Jia, Jesse Thomason, and Kristina Toutanova. Efficient end-to-end visual document understanding with rationale distillation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 8401-8424, Mexico City, Mexico, 2024. Association for Computational Linguistics. 17

Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. In Proceedings of the ACM Web Conference 2024 , page 3162-3172. ACM, 2024. 2

Appendix

A Benchmark Creation Details

A.1 Algorithm for candidate generation

We list the complete candidate and ground truth sets generation algorithm in Algorithm 1, where the function nearest( G , loc , m ) returns the m nearest businesses of around certain location loc based on the graph G . The function unique_name( S ) removes the businesses in the set S with redundant names.

In both Google Review-V and Yelp-V, rc min = 30 , rc max = 100 , fc min = 10 .

10 4

10 5

10 6

10 7

Number of examples

Data distribution

Categories

restaurant

bar

tourist attraction

park

grill

shopping mall

bakery

supermarket

steak house

station

hotel

brewery

market

museum

deli

theater

takeout

pub

area

historical landmark

club

brewpub

salon

beach

garden

gastropub

golf course

atm

nature preserve

church

winery

car wash

campground

lodge

casino

playground

gallery

stadium

bistro

marina

lake

gym

hospital

spa

farm

florist

bank

bowling alley

clinic

bridge

library

waterfall

pet groomer

mountain peak

zoo

pharmacy

roasters

boutique

arena

therapist

grocer

airport

nursery

parking lot

Categories

Algorithm 1 Candidate and Ground Truth Sets Generation

Input: Business geographics graph G = V , E , Visit set B of a user, Category c , Minimum random candidate count rc min , Maximum random candidate count rc max , Minimum final candidate count fc min , Minimum ground truth count gtc min .

A.2 Processing and filtering

For Google Review-V and Yelp-V, the categories are selected as the last word of the annotated tags in the business. However, some category requires multiple words to express its meaning, such as ' tourist attraction ', ' steak house ', ' historical landmark" , ' nature preserve ', etc. We select and keep these multi-word categories.

In Google Review-V, we remove smaller categories with less than 10k occurrence in the dataset. Then, we remove ambiguous or non-differentiable categories, including ' shop ', ' store ', ' complex ', ' service ', ' company ', ' supplier ', ' caterer ', ' agency ', ' center ', ' organization ', ' attraction ', ' house ', ' mall ', ' landmark ', ' wash ', ' course ', ' preserve ', ' alley ', ' groomer ', ' field ', ' peak ', ' venue ', ' delivery ', ' dealer ', ' lounge ', ' office ', ' arcade ', ' court ', ' spot ', ' stop ', ' maintenance ', ' trainer ', ' wholesaler ', ' planner ', ' place ', ' facility ', ' school ', ' stand ', ' range ', ' consultant ', ' designer ', ' veterinarian ', ' ground ', ' contractor ', ' manufacturer ', ' studio ', ' point ', ' lot '.

In Yelp-V, as the category is very centralized, we remove smaller categories with less than 50k occurrence in the dataset. Then, we remove ambiguous or non-differentiable categories, including' planning ', ' nightlife ', ' services ', ' wings ', ' arts ', ' dogs ', ' tacos ', ' caribbean ', ' beer ', ' spirits ', ' wine ', ' venues ', ' fusion ', ' entertainment ', ' southern ', ' spaces ', ' lounges ', ' breweries ', ' shopping ', ' smoothies ', ' flavor ', ' plates ', ' eastern ', ' tex-mex ', ' shop ', ' noodles ', ' markets ', ' market ', ' donuts ', ' gelato ', ' sum ', ' veggies ', ' fruits ', ' trucks ', ' bagels ', ' cheesesteaks ', ' clubs ', ' cuban ', ' ramen ', ' life ', ' roasteries ', ' stands ', ' brewpubs ', ' gluten-free ', ' gardens ', ' travel '.

A.3 Training data distribution

We list the training data distribution of Google Review-V in Figure 5.

B More Related Works

B.1 Traditional recommendation methods.

Before the LLM era, there are also a line of work for recommendation with other networks, smaller language models, or ensembling methods. For example, Sun et al. [40] uses a bidirectional Transformer to model the item sequence. Tang and Wang [42] uses CNN to model user preference with convolutional filters. Gu et al. [14] and Yang et al. [55] uses ensembling methods for recommendation.

B.2 Multimodal large language models.

Multimodal LLMs are becoming increasingly powerful in processing images and videos and in generating human-like text, exemplified by models like GPT-4 family [33], Claude 3.5 family [1], Gemini and PaliGemma [13, 3], LLaVA model family [28], and Llama 3 Vision models [30]. However, they still suffer from a strong language prior [9, 35], generalization [60] and hallucinations [10], or require heavy text transcription [61]. The extent to which they can understand user history, in analogy to LLM recommendation systems, is still unclear.

B.3 Datasets and evaluating multimodal recommendation.

The development of multimodal recommendation systems has been facilitated by the availability of diverse datasets that incorporate various types of data, including text, images, and user interaction histories. Notable datasets in this domain include MovieLens-1M [16], which is extensively used for movie recommendations, and the Amazon dataset [31], which includes user reviews and metadata for product recommendations. For news and books recommendations, the MIND [50] and Goodreads [44] datasets offer insights into user preferences in news and literature. Recently, LaMP [37] introduces 7 text classification and text generation tasks across long contexts to evaluate LLM's personalization capablity. In the context of outdoor activities, datasets such as Google Local Data [24, 52] and Yelp [2] are crucial. These datasets not only provide textual reviews but also include user ratings and geospatial data, which are essential for recommending local businesses and services. These benchmarks utilize the traditional recommendation systems that focus on ranking user preferences based on historical IDs. Instead, we introduce new visual history benchmarks Google Review-V and Yelp-V, using data from Google Local and Yelp, to evaluate the multimodal recommendation systems under a more realistic visual history for task-agnostic setups.

C More Analysis

C.1 Sensitivity of random sampling for candidate embedding

We show that the recommendation results are not sensitive to the random sampling of images in candidate embedding. Table 6 below shows that the robustness against sampling randomness: 1) the standard deviation is < 0.1 when we do 5 runs of sampling; 2) as we reduce the size of the samples to 1K and 100, the MRR stays similar, with < 0.5 difference.

C.2 Generalization between Google Review-V and Yelp-V

We report the generalization quality between Google Review-V and Yelp-V. Table 7 below shows that cross-source recommendation reduces MRR by 2-4% respectively vs . in-domain, in an acceptable range, highlighting the transferability from the shared feature in the embedding space. Besides,

transferring from Google Review-V to Yelp-V is still better than the best baseline model in-domain (40.2), while the other side is much worse (33.2). This is as expected, since Google Review-V has much more categories and much lower photo quality.

C.3 Latency evaluation

To evaluate the latency of VisualLens, we measured inference wall-clock time per example on a single NVIDIA L40S GPU. The results in Table 8 demonstrate that the joint image representation learned by VisualLens is more efficient and effective than the interleaved multimodal representation used in UniMP, reducing latency by ≈ 0.5 sec.

C.4 Qualitative results

We show the input and output of a positive example in Figure 6 and Figure 7. We show the input and output of a negative example in Figure 8 and Figure 9. Please note that due to image licensing restrictions, we are only able to provide the image names in JPG format. For more details, you can download the image data from https://business.yelp.com/data/resources/ open-dataset/ .

D Prompt Template

We list the prompt template for aspect word generation in Figure 10, and the prompt template for candidate matching in Figure 11. The model will fill in category information in the [[Category]] slot and predict the answer at the [[Answer]] slot.

E Additional Implementation Details

We present the hyperparameters for training VisualLens in PaliGemma and MiniCPM-V2.5 in Table 9 and Table 10. Note that since there are at most 100 candidates, we add special tokens <I1> , ..., <I100> as identifier of the candidates in the vocab of the models. By doing so, we can predict the rank by taking the probability of only those special tokens in the first output token.

F Licenses

We list the licenses involved in this work as follows.

Our usage of Google Local Data is under the license of Google Maps Platform Terms of Service.

Our usage of Yelp Data is under Yelp Dataset License.

Input

PaliGemma models are under a custom license the Gemma Terms of Use ( https://ai. google.dev/gemma/terms ).

MiniCPM-V2.5 is under Apache License 2.0

Our usage of OpenAI's models for prompting is under the license of OpenAI"s Terms of Service.

Output

<I12> Poke Bros

<I22> Carrabba's Italian Grill

<I33> Oak Steakhouse Nashville

<I57> Eddie V's Prime Seafood

<I68> White Limozeen

Ground Truth

Input

G Limitation

While VisualLens demonstrates strong performance on task-agnostic multimodal recommendation, several limitations remain.

Modular design without optimal components. Our framework prioritizes modularity and extensibility, using a unified architecture across modules including image encoding, captioning, aspect word generation, and preference matching. However, we do not use the best-performing model for each subtask. Each component can be replaced with more advanced or domain-specific alternatives, which could potentially boost overall performance.

Output

Limited modality and domain coverage. Our experiments focus exclusively on static visual history (images) and do not cover more complex modalities such as video, audio, or multimodal narratives over time. These richer formats are important for building truly comprehensive, lifelong user models, as envisioned by early ideas like Memex [4], but remain out of scope for this work.

What is the

[[Category]]

the user most likely to go to next?

Aspect words:

industrial, metal, structure, machinery, ...

Captions

:

Image 1: A large, rusted metal structure, possibly a piece of

industrial machinery or equipment.

Image 2: Industrial landscape with tall, rusted metal

structures under a clear blue sky.

...

Candidates:

<I1> Vulcan Park and Museum, Statue symbolizing city's

industries sits atop Red Mountain, surrounded by a park & a

museum.

<I2> Sloss Furnaces Visitor Center, Blast furnace plant where

iron was made from 1882-1971, now an arts & education center

with tours.

<I3> Historic Cahaba Pumping Station, ...

...

Answer: [[Answer]]

Candidate Matching

Evaluation scope. We restrict the recommendation task to a QA format for consistency and interpretability. While this setup allows for controlled benchmarking and comparison, it does not cover other common recommendation forms such as ranked lists, interactive dialogues, or implicit feedback modeling. Additionally, our proposed benchmarks, Google Review-V and Yelp-V, capture a subset of the full task-agnostic recommendation landscape and may not reflect the full range of real-world personalization scenarios.

Future work could explore richer modalities, more diverse recommendation formats, and further improvements to the modular components of VisualLens.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Novelty in §2, method in §4, dataset creation in §5 and results in §6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: A limitation section is attached after Appendix

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The validation and test sets of Google Review-V and Yelp-V is provided as supplementary, but not the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We describe the p -value in the captions of Table 3 and Table 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Reviewed NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Social impact section is after Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Not releasing pretrained language models, and image generators. Datasets are public before, we create a benchmark based on them.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Acknowledgements section is after Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See §5 and Appendix A.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The authors are human annotators for the human oracle, thus no compensation. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Reviewed LLM policy.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
comments: SUMMARY:
This paper proposes VisualLens, a framework that leverages MLLMs to personalize with task-agnostic visual history, i.e., user-taken photos. The method constructs a spectrum user profile using raw images, captions, and aspect words, and employs a three-step runtime pipeline: history retrieval, preference profiling, and candidate matching. The authors also introduce two new benchmarks: Google Review-V and Yelp-V, and demonstrate that VisualLens outperforms existing baselines such as UniMP and GPT-4o by 2–10% in Hit@3 and MRR. Extensive experiments validate its robustness, transferability, and effectiveness.

STRENGTHS AND WEAKNESSES:
### Strengths: 

- The idea or using task-agnostic visual history is interesting and novel. 
- The overall framework design is clear and meet the needs for using task-agnostic visual history. 
- Strong empirical results with two new, large-scale and diverse benchmarks.
- Robust and transferable performance across users, categories, and history lengths.



### Weaknesses: 

- Privacy implications of using personal visual history are not discussed.
- The framework requires multiple specialized components (captioning, retrieval, gridification), each with different baseline model, which may be non-trivial to deploy in real-world settings.

QUESTIONS:
In Figure 2, the model for aspect generation is MiniCPM-V2.5, while in 4.3.2, the author mentioned that the initial set of aspect words is generated by LLaVA-v1.6. Is it a typo? 

In Line 152, the author mentioned that they number each image to ensure the images are grounded to the corresponding caption and aspect words. How these numbers are used as inputs? By editing the images?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VisualLens, a multimodal LLM-based framework for personalized recommendation using task-agnostic visual history—images from a user's daily life. It constructs a spectrum profile for each user, incorporating raw images, captions, and refined aspect words, and leverages a grid-based image aggregation for model input. The system is jointly trained for both aspect word refinement and recommendation. The authors also contribute two new benchmarks, Google Review-V and Yelp-V, and report improvements over existing SOTA models like GPT-4o and UniMP.\n\nSTRENGTHS AND WEAKNESSES:\nFor strengths,\n- The core idea of using casual, non-item-tied user images (e.g., photos from daily life) to personalize recommendations is both original and practically appealing.\n- The method consistently outperforms state-of-the-art fine-tuned models like UniMP, and even large proprietary models like GPT-4o, on both Google Review-V and Yelp-V benchmarks—showing the benefit of dedicated adaptation.\n- Google Review-V and Yelp-V are carefully curated datasets with realistic visual logs, candidate sets, and ground-truths, which will likely be valuable to the community.\n\nFor weaknesses,\n- The method is only tested on geographically grounded platforms (Google Reviews, Yelp), where photos often directly relate to physical entities (e.g., restaurants, parks). It remains unclear how this method would perform in abstract or sparse-visual platforms like Amazon, where visual signals are weaker or absent for many users.\n- Since the recommendation relies heavily on surface-level features like aspect words and image similarity, the method is potentially susceptible to irrelevant or adversarial content—e.g., a user uploading meme images or photos unrelated to preferences could degrade performance.\n- While VisualLens combines raw images with generated captions and aspect words, both of which are textual representations, the paper does not isolate the contribution of these text components. In other words, it's unclear whether the performance gain is due to the visual content itself or the associated textual annotations. An ablation using only the captions and aspect words—excluding the image—would help clarify whether visual signals contribute meaningfully beyond their textual projections.\n\nQUESTIONS:\n- Would the model still work well on non-geolocated data (e.g., Pinterest photos, lifestyle vlogs)?\n\n- How well does the method generalize to non-category-based recommendation queries?\n\n- Is it possible to include metadata like timestamps or image locations, which are often present in photo histories?\n\nPlease also refer to the weakness part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a system of multimodal language models for content recommendation. They intuitively propose a reasonable motivation and an application plan for multimodal large models in recommendation systems, aiming to improve the accuracy of the recommendation system by using users’ visual history as reference information. To this end, the authors introduce VisualLens, which functions like a data annotation pipeline. By leveraging the powerful understanding capabilities of multimodal large models, it re-encodes and compresses historical record information to assist in ranking the recommended content. The authors also propose a corresponding benchmark to validate the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- I agree with the motivation of this paper, which is to build an intelligent agent to understand user interests and habits for making recommendations. I think this promotes the application of multimodal large models in the next generation of recommendation systems. I believe this could help change the current state of the field, where contrastive learning is primarily used to apply multimodal image-text models in real-world applications.\n- Good performance and lower cost: their implementation achieved better performance while using a small MiniCPM.\n- Detailed description about the pipeline.\n\nWeaknesses:\n- It seems that the authors do not decide to open-source the data and implementation?\n- The 8B model is still big, a large amount of daily active user data on many platforms is difficult to process and infer quickly.\n\nQUESTIONS:\n- There are too many `Avg. # of images` in Table 2, is the model able to accept so much tokens?\n- It their any comparison to traditional recommandation system?\n\nLIMITATIONS:\n- It seems that the content is still image-level, while i consider the recommandation is more important for video recommandation.\n- Maybe a unified model can be implemented by getting embeddings from VLM, (like using VLM2Vec(ICLR2025)?)\nBecause the model you used whose ViT is not pre-trained with contrastive loss (for example: NTP loss), the matching may be misaligned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
In this paper, the authors proposed VisualLens, a personalized‑recommendation framework that leverages a user’s task‑agnostic visual history. The method employs multimodal large language models to distill each image into a compact “spectrum” representation, forming a rich user profile. To benchmark the approach, the authors release two large‑scale datasets—Google Review‑V and Yelp‑V—and show that VisualLens delivers robust, state‑of‑the‑art performance across both.

STRENGTHS AND WEAKNESSES:
Pros:

1. The topic is both intriguing and holds potential for significant developments.
2. The paper is well-organized and reader-friendly, making it easy to understand.
3. The experiment results seem promising

Cons:

1. The pipeline (CLIP retrieval → image‑grid prompt → LoRA‑tuned MLLM) replicates CLIP4Clip/UniMP, and the only new “aspect‑word” tweak lifts Hit@3 by < 2 pp, so novelty is incremental.
2. Google Review‑V and Yelp‑V use raw user review tags without any human or LLM audit
3. Results cover only oracle re‑ranking accuracy; the paper omits retrieval‑recall, latency, and etc

QUESTIONS:
1. Could you specify which components of your pipeline are fundamentally new beyond CLIP4Clip (2023) and UniMP (2024), and provide an ablation showing the exact contribution (e.g., Hit@k delta) of each?
2. Have you run any human double‐annotation or LLM‑as‑Judge audits on a stratified sample of Google Review‑V and Yelp‑V to estimate label noise or inter‑annotator agreement?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-20