Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms
Authors: Miaosen Zhang
Yuxing Wei
Zhen Xing
Yifei Ma
Zuxuan Wu
Ji Li
Zheng Zhang
Qi Dai
Chong Luo
Xin Geng
Baining Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Image retrieval
Alignment
Aesthetics
Vision-Language Models
Primary area: machine_vision
TL;DR: Method for aligning visual retrieval model with human aesthetics

Abstract: Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics.  Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.

Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms
Miaosen Zhang
Yixuan Wei?
Zhen Xing
Yifei Ma*
Zuxuan Wu Ji Li
Zheng Zhang
Qi Dai Chong Luo
Xin Geng Baining Guo
Southeast University Tsinghua University {mi {miazhang, xgeng, 307 edu. en
Fudan University *Microsoft
qidžmi com
Abstract
Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibilityy In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in retrieval system. Advanced retrieval systems usually adopt a cascade aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a cefference-baseee reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Mcanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose novel dataset named HPIR to benchmark the alignment with human aesthetics. Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.
Introduction
Large-scale data pretrained models, e.g. CLIP [38], have been applied to broad range of fields, e.g. visual generation [40, 59, 58, 54, 56], understanding [33, 48], LMMs [26, 19]. They are trained on very large image-text pair datasets, e.g. LAION [43] and DataComp [8], rather than the traditional ImageNet [6]. These datasets contain noisy labels, and exhibit diverse data quality. As a result, though models trained on such datasets demonstrate strong capabilities on semantic matching the wild, they may prefer samples that violate the intents from users, as shown in Fig. 1. For example, using a langua-lagguage model an one stage retrieval system, with a huge amount of images in the database, the model may pick the images that exactly match the search query but with unappealing visual appearance. Moreover, may provides harmful results that disrupt the principle of responsible AI (RAI). Existing retrieval benchmarks [25, 63] also lack evaluation for aesthetics and RAI.
The work completed during internship at Microsoft Research Asia. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LLMs
Image Retrieval (Aesthetic)
alignment
alignment
alignment
w. elighment
Hoss to dextroy the world? ?
centroy the
Here are the chinge to presarte:
Query: lappy slovngwwth
Queey Happyying ball
(1) Large acale of trotagicaa Waappors, 121 The cuntrol of
sorry important aetinhhduutte planee
resources includi water and food anri
walli
Image Retrieval (RAI)
Image Retrieval (Visual Counting)
wio. atignment
alighment
wio. alignment
sliggment
Quary:
Quer:: tourappiee. ina
Absoyee of offeler delendinn
fourapptie ina
cnioe detending
Figure I: Alignment examples. W/o alignment, the models may prefer samples violating user intents.
These problems crucial in real retrieval engines, and lacking investigation in the research. Among the products of the industrial community (e.g., Google search, Bing search, etc.), such problems are mitigated by multi-stage approach, i.e., a cascade of semantic search and multiple quality filters or re-rankers. However, multi-stage approach may introduce extra latency and cascade of model biases, and requires more manpower and resources to maintain, debug and A/B test. Therefore, integrating human preferences into model features and simplifying retrieval into an end-to-end system shows great research value and engineering importance, especially in the scenarios where on-end devices and large-scale API services are arranged.
Luckily, in the field of natural language processing [69, 45, 2, 36), the problem of misalignment has been extensively studied. Supervised fine-tuning and reinforcement learning from human feedback (RLHF) [5] have been proven to be effective, which significantly improve the quality of model outputs. Similar method is also widely adopted in some vision-languaee tasks, primarily in image captioning [41], and has recently been extended to non-textual vision tasks [37]. Nevertheless, the utilization of RL. for subjective preferences in pure vision tasks has not yet been explored.
In this paper, target the realm of visual aesthetics representative of human preference and aim to align the pre- pre-trained vision models with human aesthetics. In Oxford dictionary [10], "Aesthetic" has two explanations:
(1) "Connected with beauty and art and the understanding of beautiful things." (2) "Made in an artistic way and beautiful to look at." We re-express this concept in Fig. 2. High level understanding of aesthetic may involve the cultural symbolic aspects that require reasoning related to the object. Low level part of aesthetic is related to image resolution, layout, saturation, etc. Particularly, this visual appeal (low level part) is considered as some statistical prior information. which can be learned by an end-to-end neural network to a great extent.
Based on the above understanding, strong reasoning ability of LLMS to extend the query with expectations of implicitly containing the understanding of beauty. We find that using this rephrased query in retrieval drastically boosts the aesthetic quality more than we ever expected. Then, we utilize public aesthetic models to re-rank the retrieved images, resulting in high quality image sequence that contains the inductive bias of both mechanisms and agrees with both aspects of aesthetic. Finally, a preference-based reinforcement learning method, adapted from DPO [39], is proposed to align the model with the sequence.
build our pipeline in Fig. 2, which first leverages the
Former well-known open-source aesthetic datasets (e.g., [15, 32, 30J) were mainly designed for image aesthetic assessment (IAA) task, which cannot be used for aesthetic retrieval evaluation without adaptations. Thus, we propose two methods to evaluate models. For system level evaluation, we use GPT-4V as a judge simulate to the subjective nature of aesthetic, we further construct a novel dataset (named HPIR) labeled by humans for model evaluation, and validating the reliability of the GPT-4V judge.
to choose a favored retrieval system within two candidates. Due
We make several contributions this work: (1) We benchmark the alignment of human aesthetics with two methods, using both novel dataset and using GPT-4V judge, which also investigates

---

:esthetic: Understanding of beauty
Image visual appeal
Understanding beauty
Image visual appeal
Composition Symbolism
Style
Cultural context
Resolution Saturation
Symmetry
Exposure
distill
Our approach
LLM rephrasing Aesthetic model
RL
LLM rephrasing
Aesthetic model
LLM extends conceptual aesthetic expectations
Re-rank with image visuol appeol
Query
LLM rephrasing
Retrieved result
Aesthetic model
Re-ronked resuit
RL finetune
The concept of aesthetic, which inspires our pipeline alignment. The specific and technical details are shown in Fig. 4.
Figure 2:
how to prompt those large mulli-modalle models toward better aesthetic judgement. We present that LLMs rephrasing of queries can significantly improve the aesthetic scores. (3) We propose preference- based reinforcement learning method align existing vision models with human aesthet- ics. Last but not least, aesthetics is one of the most subjective components of human preferences, and hence we believe our method can easily generalized to other aspects of human preference.
Method for Aesthetic Alignment
2.1 Model Pretraining
We use both self-pretrained model and open source models [38, 8] for alignment fine-tuning. We pretrain our vision-language model using the adapted CLIP contrastive loss [38], which can be formulated as follows:
N
exp(B1g/T)
Liext
log
(1)
Ek= exp($sa/TT)
i=
1=1
exp(sss/T)
Limage
Σ log
(2)
Ek=1 exp(sks/7))
j=1i=1
where Say = a? â, is the cosine similarity between the embeddings å, å, of the corresponding image and text. is the temperature parameter and = (1-e) -4 + c/N indicates a smoothed version of label 4 with factor of e. The final pretraining loss Cp is the sum of the text and image losses:
Lep
Liext + fimago-
The vision model and language model are initialized from the Swin- V2-L [28] and Roberta-L [27], respectively. We leverage several advanced techniques [57, 9, 55] and we leave our detailed insights into the pretraining process and data composition in Appendix. A.
2.2 Aesthetically Query Rephrasing with LLMs
According to the explanation of "aesthetic", a query with explicit understanding of aesthetic will potentially benefit the quality of retrieved images. While a typical user's query can be quite plain, we expect to leverage LLMs (e.g., GPT-3.5-turbo to enrich such concepts and contents. The participation LLMSs LMMs is crucial because high level of aesthetic understanding requires their strong reasoning ability. Existing aesthetic models do well to differentiate high and low quality images when they have great quality gap, when the gap is small, L.MS surpass all of them significantly via reasoning (Table 13 and 10 in Appendix). This indicates that to make further development of aesthetic understanding, reasoning is essential. While directly labeling a training dataset by LMMS not acceptable in both latency and cost, using LLMs to reason and extend the query becomes nice

---

w/o. LLM rephrasing "An IG style pic of sofa"
rephrasing
"Sleek modern sofa with clean lines, minimalist design, and neutral color palette, placed in a....."
Figure 3: Effect of LLM rephrasing. All images are retrieved from the same fixed engine. The advancement of LLM rephrasing has clearly enhanced the aesthetic quality of outputs, particularly in expressing abstract notions and stylistic elements.
substitute. In addition, LLMS can further refine the query with the following advantages: (1) Enrich queries with visual details. yielding more aesthetically appealing and expectation-aligned results. (2) Mitigate issues stemming from user search text style, misspellings, and incorrect emphasis.
The impact of the above enhancements will be quantified in Sec. 4.2. Our utilized prompt template can be found Appendix. F.I. in which a 'method' indicating the rules that query should obey is required. We evaluate four distinct method prompts in Sec. 4.2, and finally advocate the termed as <k list>:
Generate comma-separated list of succinet object descriptions, visual details, stylistic elements to extend the aesthetic understanding and expectations, ordered from the most to the least significant.
An example of query rephrasing is shown in Fig. 3. When we think about "Instagram style", we usually have an imagination of light scene with clean and a little minimalist design. LLM rephrasing adds these elements directly to the query, resulting in more satisfying retrieval results. In addition aligning with user's implicit imagination, when the standard of beauty associated with context of cultural or knowledge, LLM rephrasing can also significantly boost the results. More cases are presented in Appendix. C.
2.3 Aesthetic Alignment Fine-tuning
We aim to directly align the retrieval model with human aesthetics, eliminating the multi-stage retrieval system with re-ranker. To obtain the training data for fine-tuning, we leverage public aesthetic models to build a two-stage retrieval system, generating sorted high-quality image sequences. Particularly, given the images retrieved by pretrained model, we utilize well-known semantic (e.g., CLIP [38]) and aesthetic (e.g., CLIPIQA [50], IAP [42] and MANIQA [62]) models as the re-ranker. Although these models are not exactly the same category (e.g.. MANIQA is no-reference image quality assessment model instead of image aesthetic quality assessment model strictly speaking), we choose these models because during our engineering testing, they perform well as second stage re-rankers. Note that in real world engineering, we can adapt this pipeline to multi-stage system that may further leverage information like click rate, making the pipeline become RLHF.
Data preprocessing. We collect our training data using four-step process:
We analyze the topic distribution of user queries and employ GPT-3.5-turo to synthesize N = 80, 000 pseudo queries that mirrored the distribution of authentic user queries. This procedure protect user privacy well.
Each generated query subjected to rephrasing process as described Sec. 2.2. The modified outputs regarded rephrased queries.

---

For each query. we utilizee pretrained model along with Approximate Nearest Neighbor (ANN) scarch algorithm [4] to quickly retrieve the top K 400 images from a 75 million subset of DataComp, using the rephrased query.
We compute the score of re-ranker (semantic and aesthetic models) for each image in search results.
The final training dataset D is structured as follows:
D == [(a. 4: T:)li = I........ N;T; [xx xx
(
D.
(4)
is defined as a tuple that where qi and di are pseudo query and rephrased pscudo query, and each x; (1) contains image y(+) and the re-ranking scores:
x = (u S
(5)
Fine-tuning from AI feedback. We model the retrieval problem as reinforcement learning problem: for a given search query q and an image database y = {ymm J. we denote the retrieval system with learnable parameters as the policy model To (y g5 J)). For some of the retrieved images, e.g., y and Vý, we can establish a preference y/s > yy (or preferred retrieval response. Assume that these preferences are generated by some underlying reward model To (B, g), e.g., human/Al scorer, and aesthetic model. Reinforcement learning maximizes the expectation of rewards while using KL divergence for regularization to prevent training collapse and overfitting:
y/) to signify that image y/ (or yj) is
max Eq- D..tpt-ott))) (y 4)] x [Ne [x)] Ti.e [
(6)
Here, Tref is the reference model (i.e., the pretrained model). Following DPO [39], by choosing Bradley-Terry model [3] to formulate preference distribution, we can use the following policy objective loss to maximize the reward:
log a Blog To (ywlgy ))
To (unll::
Eap
E (a
Trey 3) Blog Tree (4;
(7)
where you is the preferred sample compared to yt- In retrieval scenario, given a user search query we build the partially ordered dataset Po for training by establishing those ordered pairs: Dro-
{gg yo, yy )ly < }- The probability that multimodal based policy model return response y: is given
by the normalized cosine similarity:
cos(S (os() (v:), SP (a))
To(usla;)) Σ Emey cos( (3N), (1N),, ff(()) (q)
(8)
Here, S and S represent the vision and language encoders of the multimodal model, respectively. is easy to observe that the denominator part of (b(9) Y)) will be cancelled out in Cdpo- thus in actual calculations, we only need to calculate the cosine similarity of the corresponding images and queries, which also makes Lápo independent of the image database y Compared to DPO [39], we utilize an ordered sequence to obtain samples for adapting the DPO loss, allowing producing (n²²) preference pairs from a sequence of length n. This approach significantly enhances the data utilization rate, making the modified DPO algorithm more scalable.
Following InstructGPT [36], we also integrate the pre-training loss Epe stabilize the training process and maintain retrieval capability. Consequently, the composite loss function formulated for fine-tuning expressed as:
L L + Wip Cpt-
Construction of Dpo We illustrate the construction of the partially ordered dataset Dpo in Fig. 4. For each query, images are intermittently selected from the retrieved results of the rephrased query at intervals defined as stride, aiming to amplify the quality discrepancy among the chosen images. Subsequently, these images are arranged into a matrix with u rows and v columns, following a row-major format. Samples within each row are sorted according to the score of the re-ranker. The re-ranker, which is finally designed by assembling open-source models, evaluated in Appendix. E. We then define the column dimension the aesthetic dimension, since the samples in each row are sorted aesthetically. The row dimension is defined as the semantic dimension because semantic relevance varies across different rows. We extract all rows and columns to obtain sequences with the form /1 > 1/2 > ... > yk, resulting in C2 pairs of (y/, Vj) in each sequence. To this end, total of uC + vC2 partial order pairs can be produced for each query. Note that numerous operations in this process can be tensorized and exccuted in parallel, thus the time cost is low.
ordered
5

---

query
Top images
LLM
Reshape
Aesthetic
Rerank
system
Semantic
Figure 4: An example illustration for the construction of partially ordered pair dataset.
Benchmarking Human Preference
Standard retrieval benchmarks, including MSCOCO [25] and Flickr30k [63]. lack the aesthetic evaluation. Aesthetic datasets, e.g. AVA [32], can only be used to evaluate the accuracy of an aesthetic prediction model, which needs additional efforts for retrieval system evaluation. Therefore, introduce the following two novel benchmarks assess whether retrieval model align with human aesthetics well: (1) testing model preference by human-labeled dataset, and (2) using GPT-4V [35] to determine the relative performance of two systems.
3.1 Human Preference of Image Retrieval (HPIR)
We introduce HPIR, test set of human preference alignment. leverages 150 pseudo queries for testing, which are generated using LLMS by requesting aligned distribution with the user's topics. For each query, we combine the results of multiple search engines and obtain 10 images, which divided into two groups (A and B). Human labelers asked evaluate the two groups of images, determining which group is more precise and visually appealing. To ensure robustness, each comparison annotated for 30 times. We also scrutinize the annotation time and order consistency (Sec. 3.2) to guarantee the quality. The label that predominated in the 30 annotations is designated the golden label. Let N Npos be denoted as the number of labelers that assign the golden label, and Nneg as the remaining number. We define the confidence score we (exemplified in Fig. 14 of Appendix) of this annotation as: Wc Npos + Nneg
2N
E [0, 1].
(10)
This confidence level has a similar effect variance, and the variance formula for human labelers in aesthetic annotations can be easily calculated as follows:
2N 2NN Noeg
ear (Npos Nneg)
(11)
To evaluate a model/search engine, we task it with discerning the better group between A and B for all queries, based designated criterion. Then the HPIR metric Masp (asp stands for either accuracy or aesthetics.) ) assessed by comparing the selections the model/engine to the human-annotated golden labels. Masp is formulated confidence weighted average over the queries:
Equery w. [choice golden label)
Masp
(12)
Equery We
where 1 {choice groddenlabel) is an indicator that equals 1 when the model's choice matches the golden label, and otherwise. This method can effectively assess the degree of alignment between a model and human preferences. For instance, to evaluate the CLIP [38], we simply compute the average CLIP similarities (to query) on group and B, choosing the group with the higher average as the model's choice. More details about data distribution, baseline results and aesthetic model evaluation can be found in Sec. and in Appendix. E.

---

3.2 GPT-4V Win Rate
LMMs have shown their strong abilities across numerous tasks. Thus, we directly compare two retrieval models/systems using GPT-4V [35]. Emulating the AlpacaEval [24] approach from LLMs, we first conduct image searches for a collection of queries using two retrieval systems, RI and R2. We then concatenate results in R1 and R2 into one large image and employ GPT-4V as the judge to assess which system performs better. We note that GPT-4V tends to prefer the first row when the results from both systems are comparable, a tendency that mirrors human behavior. To address the bias, we introduce an crdes-consistency (OC) strategy, where we initially place images from R1 on the first row and images from R2 the second for evaluation, then invert their positions for separate assessment. A visualization and more detailed description is provided in Appendix. D. If two assessments have conflicting conclusions, we say the two results are similar. Lets denote the
number of RI wins as Nw. loses as Ni and similar as Ng. System R2 serves as the baseline. We define the win rate of RI Ruins and win-and-simin rate Rwinksimstarr.
No. Rwin
(13)
No + N
N + N.
N
Ririnasimstar
Nw N. N =
(14)
N
Unlike HPIR, this approach lacks the supervision of human labelers, necessitating meticulous design and validation ensure its soundness. We thus leverage HPIR feedback to filter various prompts and evaluation methods, ultimately selecting prompt format referred to as <ranker> (see Appendix. G). Detailed experiments, shown in Appendix. D.2, yield that with this prompt and order-consistency. GPT-4V can present comparable aesthetic judgments to humans.
4
Experiments
In this section, we present our main experiments. More results, including ablations, benchmark evaluations (HPIR and GPT-4V judge), and LLM rephrasing, are in Appendix. B
4.1 Details and Evaluations of Alignment Fine-tuning
In the alignment fine-tuning loss, the Lyt component is configured identically to the pretraining phase described in Sec. 2.1, encompassing batch size, temperature, and data, with weight of Wpt = 1.0. For the remaining components, each batch comprises 128 queries. The overall learning rate is fixed to Ir = 5 x 10-5. The partially ordered set Dpo. as discussed in Sec. 2.3, is derived using u = 5, and a stride of 10.
We conduct the experiments with two other state-of-the-art models: CLIP [38] and DataComp [8]. The image encoders of CLIP and DataComp are ViT-L/14 models, trained on private 400M dataset and on the DataComp- IB dataset, respectively. We report their performance on classic retrieval benchmarks (ImageNetIK [6] zero-shot classification, MSCOCO [25] T2I retrieval Recall @ and Flickr30K [63] T2I retrieval Recall @ I) and on the proposed HPIR in Table 1. It is not surprising that our model performs worse DataComp on MSCOCO and Flickr30K, since our training budget is much smaller than DataComp. We can further observe that our alignment fine-tuning does not significantly impact the retrieval capability, but it greatly enhances the aesthetic scores of the retrieval results, surpassing both original CLIP and DataComp. While fine-tuming on the CLIP and DataComp. a similar increase is observed aesthetic ability, demonstrating the generalization of our method.
Table I: Performance traditional retrieval benehmarks and our proposed acsthetic alignment datasct. PT indicates pre-training, and RLFT indicates alignment fine-tuning.
Retrieval metrics (%)
HPIR (%)
Model
ImageNet1K 7S
MSCOCO
Flickr30K
Accuracy
Aesthetic
CLIP[38]
76.2
37.8
68.7
68.1
62.1
DataComp[8]
79.2
45.7
73.4
71.8
62.7
Ours-PT
40.2
66.5
CLIP RLFT
79.4
38.1
67.2
73.1
71.7
DatuComp RLFT
81.7
45.1
72.2
74.4
71.9
Ours: RLFT
82.2
40.2
66.4
71.7
67.6
In Table 2, we report the system-level comparison results with other models, approaches and two commercial search engines (Bing image search [31] and Getty search [13]). We report the win-
7

---

and-similar rate here because it is more in line with the user's thinking when choosing products (see other indices and details in Appendix. D.3). Experiments are conducted on a database of 15M images extracted from DataComp-1 IB and our internal database with 8M images. The experiments demonstrate the effectiveness of our alignment fine-tuning, the superiority of our final model, and the gap with commercial web-scale products. Our finetuned model shows comparable performance with 2-stage approach, yielding a possible latency and pipeline optimization to real world products (see win rate in Appendix. D.3). Systematically speaking, despite the vast difference in the size of the databases, our model still achieved a win-and-similar rate of 50%~~0% in comparisons with Bing image search and Getty search. Additional human user evaluations of selected experiments also validate the reliability the GPT-4V judge. These labeling processes can be seen as real world A/B test user studies.
Table 2: System level comparison results. Mark indicates using LLM rephrasing. DataComp- 15M a 15 million subset of DataComp- IB dataset [8].
System
System
similar (%)
ID
Name
Database
Stages
Name
Database
Stages
Accuracy
Aesthotic
1
Ours-FT
Datacompp 15M
Ours- PT
Datacomp- 15M
1
72.0
78.7
Datacomp-155
Datacomp-1555
3
Ours-FT
Datacomp-15M ISM
DataComp
Datacomp-1 15M
69.3
67.3
4
Ours-FT
intemal-8M
Ours-PT Rerank
internal-SM
68.0
70.0
5
Ours-F'I
intemal-SM
Bing search
web
>2
45.3
57.3
Ours: FT
internal
Gelly seurch
Gelly Images
>2
62.7
62.7
Human laheler judger (User study)
I'
Ours-FT
Datacomp-155
Ours-PT
Datacomp-15
65.3
74.7
4"
Ours-FTT
intemal-8M
Ours-PTI + Rerank
internal-8M -M
2
66.9
71.4
Ours-FT
internal
Bing search
6'
Ours-PT
intemal-RM
Getty search
Getty Images
>2
63.3
61.3
4.2
Effect of LLM Rephrasing
Here, we test different method prompts, which correspond to (method) as outlined in the template in Sec. 2.2, for search query rephrasing. We task GPT-3.5-turbo with the job of rephrasing queries to an approximate word count of 50. The method <k list>, as introduced Sec. 2.2, enumerates additional descriptions for the query. The <detail> method encourages the model to elaborate with more specifics, while <kw diet> instructs the LLM to enumerate keywords followed by detailed expansions for each. We employ two verification metrics described in this paper: HPIR and GPT-4V win rate, with the latter benchmarked against the original query's 's baseline results. The detailed prompts described in the Appendix. F.2.
Table reports the evaluation results, including the average scores from acsthetic models. The empirical evidence from all three metrics suggests that the utilization of LLMs for query rephrasing has enhanced the aesthetic appeal of the search results. We choose win rate to report because it brings clearer clues of which prompt is better (see other indices and details in Appendix. D.4).
Table 3: Evaluation of different method prompts for LLM rephrasing HPIR and GPT-4V win rate. Scores from aesthetic models are also provided.
Aesthetic Scores (5 Avg.)
HPIR (%)
GPT44V win rate (%)
prompt
CLIPIQA
IAP
MANIQA
Accuracy
Aesthetic
Accuracy
Aesthetic
original
0.8544
4,8047
0.4296
66.19
59.36
<detail>
0.8787
5.0698
0.4279
65.25
68.24
63.51
68.75
<k list>
0.8768
5.0772
0.4384
68.95
72.42
53.37
67.86
<kw diet>
0.8678
5.0140
0.4345
69.17
69.93
53.57
63.83
repeat 0.8554
4.9525
0.4395
67.96
65.67
62.00
63.16
<reorg> 0.8742
4.9925
0.4463
68.33
67.85
59.65
64.44
To investigate the mechanisms by which query rephrasing yields enhancements, we further evaluate two additional rephrasing methods for extending the length of the original query: the "repeat" method, which involves duplicating the original query n times, and the <reorg> method, which entails prompting the LLM to reformulate the query in diverse linguistic styles, then repeating it times

---

dashing Maine cat"
"Surrealist painting"
"grinning hoppy snowman" "
finetune
w/o finetune
Figure 5: Qualitative comparison of top-4 retrieval results between models with and without our proposed alignment fine-tuning.
without incorporating additional details. shown in Table 3, simply enlarging the length of the query, even in the absence of new details, can enhance the aesthetic performance. Leveraging L.MMs deepen the comprehension of the query and enrich the visual specifics allows for further aesthetic improvement in retrieval tasks. We further summarize possible reasons for this phenomenon in Appendix. F3.
5
Cases Study and Qualitative Comparison
Fig. shows the qualitative comparison between our fine-tuned model and pretrained model, where we retrieve top-4 images from the 75M subset of DataComp. It can be observed that the alignment fine-tuning endows the model with the capability to retrieve images with vivid background, rich texture details, and dynamic color contrast, leading to more aesthetically pleasing search engine.
More comparison result and analysis with and without LLM rephrasing using fine-tuned model can be found in Appendix. C. With LLM rephrasing, the retrieved images exhibit remarkable improvement visual coherence and enriched details. The styles of the images become more consistent with the search intent, capturing samples that align closely with human expectation.
6 Related Work
Vision Language Models. The availability of web-scale image-text pairs has sparked the research of
vision-language models [7, 21, 23, 29, 46, 51, 67]. The pioneering works, CLIP [38] and ALIGN [14], utilize contrastive loss during training to achieve remarkable generalization capabilities. Subsequent works [61, 65, 44] have expanded image-text contrastive
wider scope. BeiT3 [52], CoCa [64] and BLIP [20] further explore other pretraining methods. More recently, several large multi-modal models have emerged [26, 53, 12]. While most methods have shown strong retrieval capabilities, they often overlook aesthetics, frequently retrieving results with poor visual quality. Our work aims to fill this gap, focusing designing a vision- vision-language model with aligned aesthetics with humans.
Reinforcement Learning from Human Feedback (RLHF), RLHF has been widely adopted in LLMs [36, 45, 68, 49, 39]. Typically, reward model is trained as proxy for human preferences, providing feedback for model tuning. Recently, researchers focus and apply RLHF technique to computer vision [37, 16, 41]. In image generation, Lee el al. [18] and ImageR [60] utilize the reward modeling for text-to-image tasks. To the best of our knowledge, our work is the first focus on aligning human intents with the fundamental task of text-image retrieval.

---

Image Aesthetics Assessment. Numerous aesthetic evaluation protocols have been proposed [17. 47, 11, 22, 34, 42]. Kong et al. [17] propose relative ranking to model photo aesthetics. NIMA [47] assigns a score distribution to each photo, which captures the subjective variations in human aesthetic preferences. MANIQA [62] considers the no-reference image quality assessment task and employs a multi-scale attention mechanism to analyze images across various scales and regions. CLIPIQA [50] tries to understand image content and trains the model by comparing the quality of images. this work, we adopt a weighted combination of existing models [62, 50, 42, 38J to provide supervision for our training. In addition, several IAA datasets (AVA [32], PN [15], and CHUNK-PQ [30]) have been proposed for evaluating aesthetic models. However, in retrieval setting, we need to compare the aesthetic levels of two image groups corresponding to a shared query. Existing datasets cannot satisfy the needs in retrieval scenario.
7 Potential Social Impacts and Risks
We state that all labelers (mostly university students) were informed of all the uses of their labels, and they agreed on the use. All experiments were conducted with open-source datasets and proprietary data for which we own the full copyright.
Our work may cause potential social impacts and risks:
Diversity and quality trade-off:: The LLM rephrasing stage may increase the quality while decrease the diversity, which generally does more good than harm, but in cases where more diversity is required, several simple adjustments can be made to adapt. For example, repeating the rephrasing may benefit both direct inference and training models. In addition, increasing the pre- tre-training loss weight during fine-tuning may also be an option to balance the trade-off.
Potential biases: LLMS are trained with massive internet-scale data and may therefore contain potential biases. This may reinforce the biases of the retrieval system. For example, when searching for the word 'nurse', the vector search system using the CLIP feature returns approximately 90% female images and 10% male images. However, when this system uses LLM rephrasing as preprocessing, results nearly 100% female images. This because the rephrased sentence describes a female nurse. However, our approach also provides an opportunity to supervise and eliminate some biases. A vector search system using the CLIP feature is a black box, and we cannot predict or control when it may retrieve harmful content. By using LLM rephrasing, we can trace intermediate information and debug the system. We can further improve L.MSs or implement a text-based filter.
Specific RAI problems: As we mentioned, our algorithm can be adapted to other areas of RAI problems, such as nationality and race. While our approach may benefit many general cases, it may potentially ignore minorities, as aesthetics and ethics may differ across various cultures and regions. Specific adaptations should be made to both LLMs and vision models in accordance with the 'no free lunch' theorem.
8 Conclusion
In this paper, we attempted to align image retrieval models with human aesthetics. We presented preference-based reinforcement learning method to align retrieval models with human aesthetics by distilling knowledge from LLMs reasoning and aesthetic models. Extensive experiments demonstrated the effectiveness of our method, showing a possible alternative to the aulti-stage retrieval pipeline. Finally, we discussed the potential social impacts and risks. We believe our proposed approach can become a general practice for other misalignment problems in computer vision.
Acknowledgement This project was supported by the Fundamental Research Funds for the Central Universities (2242024430035).
References
[1] M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint 230.:2310.120366 2023.
10

---

[2] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint (exXix:224405880 2022.
[3] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, (3/4):324-34, 1952.
[4] Q. Chen, B. Zhao, H. Wang, M. Li, C. Liu, Z. Li, M. Yang, and Wang. Spann: Highly-efficient billion-scale approximate nearest neighborhood search. NeurIPS, 2021.
[5] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. NewrIPS, 2017.
[6] J. Deng, W. Dong, R. Socher, Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[7] A. Frome, G. S. Corrado, J. Shlens, S. Bengio. J. Dean, M. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. NeurIPS, 2013.
[8] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, Zhang, al. Datacomp: In search of the next generation of multimodal datasets. NeurlPS, 2023.
[9] K. He, Fan, Y. Wu, S. Xie, and Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[10] A. S. Hornby, J. Turnbull, D. Lea, D. Parkinson, P. Phillips, B. Francis, S. Webb, V. Bull, and
M. Ashby. Oxford advanced learner's dictionary. International Student's Edition, 2010.
[11] V. Hosu, B. Goldlucke, and D. Saupe. Effective aesthetics prediction with multi-level spatially pooled features. In CVPR, 2019.
[12] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al. Language is not all you need: Aligning perception with language models. NeurlPS, 2023.
[13] G. Images. Getty images. https: httpp://ww..etttnagggg.... www.geingeeeee..e..omm. com.
[14) C. Jia, Y. Yang, Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[15] D. Joshi, R. Datta, E. Fedorovskaya, Q--T. Luong, J. Z. Wang, J. Li, and Luo. Aesthetics and emotions in images. IEEE Signal Processing Magazine, 28(5):94 115, 2011.
[16] H. Kazemi, F. Taherkhani, and N. Nasrabadi. Preference-bassee image generation. In WACV, 2020.
[17] S. Kong, X. Shen. Lin. Mech, and C. Fowlkes, Photo aesthetics ranking network with attributes and content adaptation. In ECCV, 2016.
[18] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
[19] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[20] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[21] Li, R. Selvaraju, Gotmare, S. Joty, Xiong, and S. C. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NearIPS, 2021.

---

[22] Li, X. Huang, J. Lu, J. Feng, and J. Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In CVPR, 2021.
[23] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang. H. Hu, Dong, Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020.
[24] X. Li, I. Zhang, Y. Dubois, R. Taori, I. Guirajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloiing models. https: thub. com/ latts--labaappccaa _vall 2023.
[25] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[26] H. Liu, C. Li, Q. Wu, and Lee. Visual instruction tuning. NeurlPS, 2023.
[27] Y. Liu, M. Ott, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis. L. Zettlemoyer. and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXiv:190..11622 2019.
[28] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Zhang, L. Dong, al. Swin transformer v2: Scaling capacity and resolution. In CVPR, 2022.
[29] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-languaae tasks. NeurIPS, 2019.
[30] W. Luo, X. Wang, and Tang. Content-based photo quality assessment. In ICCV, 2011.
[31] Microsoft. Bing image search. https://www.bing comm/imggess/tttaala//00
[32] N. Murray, L. Marchesotti, and F. Perronnin. Ava: A large-scale database for aesthetic visual analysis. In CVPR, 2012.
[33] B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng. J. Fu, S. Xiang, and H. Ling. Expanding language-image pretrained models for general video recognition. In ECCV, 2022.
[34] D. V. Nieto, L. Celona, and C. Germandez-Labradon Understanding aesthetics with language: A photo critique dataset for aesthetic assessment. arXiv preprint arXiv:2206.086/ 2022.
[35] OpenAL. Gpt-4v. https: htttss:/openai....
2023.
[36] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
[37] A. S. Pinto, A. Kolesmikov, Y. Shi, L. Beyer, and X. Zhai Tuning computer vision models with task rewards. In ICML, 2023.
[38] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[39] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 2023.
[40] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[41] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In CVPR, 2017.
[42] C.
Schuhmann.
Laion-aestheties predictor v2.
https://github.com com/
prediticccpeedioooo. 2022.
[43] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes,
A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NewrIPS, 2022.
12

---

[44] J. Shao, S. Chen, Y. Li, K. Wang, Z. Yin, Y. He, J. Teng, Q. Sun, M. Gao. J. Liu, et al. Intern: A new learning paradigm towards general vision. arXiv preprint arXiv:2111.08687 2021.
[45] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning summarize with human feedback. NeurIPS, 2020.
[46] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai. VI-bert: Pre-training of generic visual-linguistic representations. ICLR, 2020.
[47] H. Talebi and P. Milanfar. Nima: Neural image assessment. 2018.
[48] S. Tu, Q. Dai, Z. Wu, Z.-Q. Cheng, H. Hu, and Y.-G. Jiang. Implicit temporal modeling with learnable alignment for video recognition. In ICCV, 2023.
[49] B. Wang, R. Zheng, L. Chen, Y. Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shi, et al. Secrets rlhf in large language models part ii: Reward modeling. arXiv preprint AXiv:2401.06000 2024.
[50] Wang, K. C. Chan, and C. Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
[51] L. Wang, Y. Li, and S. Lazebnik. Learning deep structure-preserving image-text embeddings. In CVPR, 2016.
[52] W. Wang, H. Bao, L. Dong, Bjorck, Z. Peng. Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[53] W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, Luo, T. Lu, J. Zhou, Y. Qiao, al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. NeurlPS, 2023.
[54] Y. Wang. J. Bao, W. Weng, R. Feng, D. Yin, T. Yang, J. Zhang, Q. Dai, Z. Zhao, C. Wang, et al. Microcinema: divide-andd-conuer approach for text-to-video generation. In CVPR, 2024.
[55] Y. Wei, Y. Cao, Z. Zhang, H. Peng, Z. Yao. Z. Xie, H. Hu, and B. Guo. iclip: Bridging image classification and contrastive language-image pre-training for visual recognition. In CVPR, 2023.
[56] W. Weng, R. Feng, Y. Wang, Q. Dai, C. Wang, D. Yin, Z. Zhao, K. Qiu, J. Bao, Y. Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In CVPRW, 2024.
[57] Z. Xie, Z. Zhang. Cao, Y. Lin, Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: simple framework for masked image modeling. In CVPR, 2022.
[58] Z. Xing. Q. Dai, H. Hu, Z. Wu, and Y.-G. Jiang. Simda: Simple diffusion adapter for efficient video generation. In CVPR, 2024.
[59] Z. Xing, Q Feng, H. Chen, Q Dai, H. Hu, H. Xu, Z. Wu, and Y.-G, Jiang, A survey on video diffiusion models. ACM Computing Surveys, 2024.
[60] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, Tang, and Y. Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurlPS, 2024.
[61] J. Yang, C. Li. P. Zhang, B. Xiao, C. Liu, L. Yuan, and J. Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022.
[62] S. Yang, T. Wu, S. Shi, S. Lao. Y. Gong, M. Cao, J. Wang, and Y. Yang. Maniqa: Multi- dimension attention network for no-reference image quality assessment. In CVPR, 2022.
[63] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denota- tions: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.
13

---

[64] J. Yu, Z. Wang, V. Vasudevan, L. Young, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
[65] L. Yuan, D. Chen, Y.-.. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv22111.11422 2021.
[66] Z. Yuan, H. Yuan, C. Tan, W. Wang. S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. NeurIPS, 2023.
[67] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, 2022.
[68] R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.00 04964, 2023.
[69] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv:199.008577 2019.
14

---

A Pre-training Details
We pretrain vision-language model, whose vision model (i.e., swin transformer) has been pretrained via SimMIM [57] and subsequently fine-tuned on ImageNet-22k [6] prior to the CLIP pretraining. To accommodate larger batch size and increased number of negative samples with limited resources, we implemented gradient accumulation and queue dictionary, similar to MoCo [9]. In line with iCLIP [55], we incorporated ImageNet-22k to bolster the model's s capability for short query and keyword searches. Thus, comparison of Table on ImageNet is not totally fair, we report only for representative keyword search capability.
Table lists the composition and amount of the data during model pretraining. Throughout the training process, employ fixed learning rate Ir x 10-4. Leveraging gradient accumulation and queue, we enable a batch size 32k for contrastive learning. Our training method, similar to label smoothing factor of 0.1. Difference
that of CLIP, incorporates the NCE Loss complemented from CLIP, we initialize the temperature parameter to 0.05 and treat it as a learnable parameter. The computational resources include 256 NVIDIA V100 GPUs.
Table 4: The composition and amount data during pretraining. SCVL indicates our self-collected image-text pair dataset.
Dataset
Samples (M)
Used Samples (M)
Used Epoches
DataComp [8]
1000
5000
5.0
SCVL
200
2000
ImageNet22K
14
140
10.0
Total
1214
Table 5 shows the specific prper-parameterss for pre-training, including the data augmentation and model settings. The loss curve and gradient norm curve during the training process are shown in Fig. 6, where the deep blue line is the curve after smoothing. Fig. demonstrates the curve of the retrieval evaluation metrics during the pre-training process, on three datasets (ImageNet1k--2 MSCOCO, Flickr30k).
Loss
Gradient
2.25
1.6
2.00
1.4
1.75
1.2
1.50
1.0
1.25
1.00
0.4
0.0
0.2
0.4
0.6
0.0
0.2
0.5
Steps (1e6)
Steps (1e6)
Figure Loss and gradient norm curves during pre- pre-training.
15

---

Table 5: Details of pre-training papee--aramtter...
Training
Loss
learning rate
5e-04
loss
NCE
total batch size
32768
label smoothing
0.1
batch per GPU
Data Augmentation
Ir dacay v
0.9
auto aug
rand-m9- rand-m9-mstd0.5-me
Ir dacay -1
0.9
color jitter
0.4
Ir scheduler
Fixed
cutmix
0.0
warm up steps
5000
mixup
0.0
weight decay
0.05
Model
optimizer
AdamW
tau init tau learnable
0.05
dropout drop path
0.0
True
0.0
project size
1024
grad clip
3.0
average pooling
Truc
amp opt level
True
vmodel
Swin-Large
O1
Imodel
RoBERTa-Large
ImageNeilK shot accop
MSCOCO T21 Recalig9
Flickr30K T21 Recalles
4025
821
40.00
82.0
81.-
818-
653
817-
39.25
65.0
81.6
39.00
815
38.75
814
Eporh
Spoch
Spoch
Figure 7: Evaluation curves on retrieval benchmarks during pre-training.
B Alignment Fine-tuning Details and Ablations
B.1
Fine-tuning Details and Curves
We show the details of fine-tuning jyper-parameterr Table 6. The whole training takes about 650 steps 4 NVIDIA A100 GPUs. According the ablation experiments in the following sections, found that using strong data augmentations have positive influence on HPIR-Accuracy and retrieval metrics (mostly focus semantic accuracy), but they may harm aesthetic performance. Thus do not add any data augmentation during alignment fine-tuning, Fig. 8 shows the loss curves and gradient norm curve during fine-tuning. Note that the total loss is the weighted sum of DPO loss and pre-training loss. Fig. shows the performance on retrieval benchmarks and HPIR during fine-tuning. can be observed that the retrieval performance stays stable. While some fluctuation happened on HPIR curves, we believe this is because of the unavoidable subjective nature of HPIR, as both options (groups) the dataset have close appearances.
Total
DPO
Gradient
8.94
0.72
2.5
0.99
0.88
0.68
2.0
1.
0.6
0.8
1.6
0.80
0.5
Figure 8: Total loss, DPO loss and gradient norm curves during fine-tuning.
16

---

Table 6: Details of fine-tuning hyper- parameters.
Training
Loss
learning rate
5e-5
pre-train loss
NCE
total batch size
128
label smoothing
0.1
batch size per GPU
32
Wpt
1.0
Ir dacay
0.9
finetune loss
Ranked DPO
dacay
0.9
B
0.05
Ir scheduler
Cosine
5
warm up steps
200
V
5
weight decay
0.0
stride
10
optimizer
Adam
Data augmentation
dropout drop path
0.0
auto augmentation
None
0.0
color jitter
0.0
grad clip
3.0
cutmix
0.0
amp opt level
True O1
hflip
0.0
mixup
0.0
ImagehetlK zoro shotocchtccoo accel
NSCOOD T2l Roca 191
Fick Reccall
66.9
64.0
40.1
400
26.0
ARI--AcoWrIcy weighted boc
APi-Aesthetic www.ghtdd acc
HRB-Overs weighted acc
55-
12-
47
40-
200
200
302
420
500
032
200
200
302
420
500
302
200
302
420
500
Strepe
SBepe
Figure 9: Evaluation curves benchmarks and HPIR during alignment fine-tuning.
B.2 Effect of 2-D sampling of Dpo
In order to perform a solid comparison, all experiments in Table are set to similar budgets and the same syper-parameters. The number of partial order pairs used for query, described in the main paper, is uC? + vC2. Thus we choose the u and v to have similar number of partial order pairs. [Dpol represent the total size of fine-tuning dataset.
We first ablate the impact of the two dimensional sampling strategy in the construction of Dpo In Table 7, the first line (u 15, v 1) represent that only sample from semantic dimension, and the last line of the first block (u = 1, = 15) shows the performance of sampling only from aesthetic dimension. As expected, the two dimensions bring different inductive biases derived from two aspects, and hence assembling both can benefit the alignment fine-tuning. The larger stride shows better result within certain range, but few influence when it is large enough.
We also have to re-emphasize that although we named u as the semantic dimension, however, because the ranked sequence is retrieved from rephrased query (instead of the original query), the sequence may contain aesthetie benefits from LLM rephrasing. Thus u dimension can also help aesthetic performance, As result, modifying the u, v may not cause remarkable change in results.
17

---

Table 7: Experiments the effect of the construction of partial order set po- Q is the number of queries.
stride
u (semantic)
v (aesthetic)
D Ppol
Accuracy
Aesthetic
15
1
105 x Q
72.1
66.5
8
108 Q
71.3
67.1
10
5
5
100 x Q
71.7
67.6
3
8
108 Q
70.3
65.4
1
15
105 x Q
68.1
67.5
5
68.7
64.4
10
5
100 x [
71.7
67.6
15
70.9
67.2
B.3
Loss Selection
In addition the ranked DPO loss as we introduced, we also adapt other losses our scenario. We introduce them in the follows:
B.3.1 RRHF Loss.
RRHF [66] is a more simple approach that do not need the reference model. In our scenario, is formulated as follow:
Lyrkf - (.. .1) DDo (No (z; Ta (3ulg:
(15)
C Cerhs Wythpt.
(16)
B.3.2 IPO Loss.
IPO [1] also general objective for RLHF, based on maximizing a non-linear function of preferences and keeps KL regularization. Similar to DPO, adapt IPO to ranking version, which formulated as follows:
To To
log To qi J) Toref (3 (7:
Lipo w)
log
(17)
Tree (#hr 14:
23
L Lipo + Wpt Ept.
(18)
For each loss, tuned their syper-parameters and reported their best results in Table 8. Firstly, all losses bring growth to the HPIR results. This proves again that the way we built the training set provides valid, learnable patterns. IPO and DPO perform similar results. RRHF loss, it does contain regularization terms, can not scale well. The peak result of RRHF loss is obtained at about 100 steps (of 650 steps in total). Then its retrieval ability (i.e., retrieval benchmark results) decreases rapidly to an unacceptable range.
Table 8: Results of alignment lne-tuning with different loss objective functions.
Retrieval benchmarks
HPIR
Fine-tune Loss
ImageNetl -ZS
MSCOCO
Flickr30K
Accuracy
Aesthetic
None (only pre-training)
82.1
40.2
66.5
66.2
RRHF
82.1
67.6
68.4
Ranked IPO
81.4
39.7
65.3
70.2
66.8
Ranked DPO
82.2
40.2
66.4
71.7
67.6
B.4 Impacts Data Augmentation
known to all, bring benefits to large range of computer vision tasks. Most of those tasks require model generalization capabilities on semantic dimension. Data augmentation greatly brings in important inductive biases, e.g., translation invariance, robustness of dealing occlusion, watermark, color change. However, we found the aesthetic related tasks, some data augmentations may harm the performances.
Data augmentation, as it
18

---

In Table 9, the best results are obtained by implementing the same image transform with the evaluation. The evaluation transform only contains an image size transform and color normalization. We separately add different data augmentations to the evaluation transform, and find that they affect aesthetic indicators to varying degrees, while the accuracy aspect of HPIR metric may gain benefits.
Table 9: Ablation of different data augmentations. Different strategies are added to the base transform separately.
Data augmentation
HPIR
type
values
Accuracy
Aesthetic
Eval-transform
71.7
67.6
auto-aug
rand-m9-mstd..55
74.4
66,9
random erase
0.25
72.5
65.2
color jitter
0.4
71.4
64.7
C More cases study of LLM rephrasing
"metallic mineral"
"Fluid mechanics sculpture" T
"Schematic diagram of neural network architecture"
"Super virus"
"The doomsday"
w/ LLM rephrasing
w/o LLM rephrasing
Figure 10: Qualitative comparison of top-4 images with and without LLM rephrasing. When user search query, they may implicitly have a expectation or imagination, LLM rephrasing help extent the imagined elements.
19

---

"Pretty girls of different colors"
"A room in the style of abstract paintings"
"Traditional Russian architecture" "
"Ancient Greek legends"
"Artsy outfit"
w/ LLM rephrasing
w/o LLM rephrasing
Figure 11: Qualitative comparison of top-4 images with and without LLM rephrasing. Some of the searching scenarios require cultural or knowledge context, LLM rephrasing helps extend the context.
The extension that LLM rephrasing provides can be divided as imagined elements and knowledge context, analyze the cases in the following:
Imagined elements. In Fig. 10, when user search "metallic mineral"or "Fluid mechanics sculpture", we may expect a shining and glaze surface. LLM rephrased results satisfy our expectation by adding our imagined styles into the query. In addition, our models may mix neural networks and website service networks, with adding the descriptions of the image, e.g. "with directed edges and nodes", the semantic accuracy can also be boosted. Another similar case is that models may mix virus and computer virus without extra details supplied.
Knowledge context. Some of the objects need cultural or knowledge context. For example, as shown Fig. 11. the understanding of abstract painting style and artsy style outfit. When searching cultural objects, LLM rephrased results seem more representative. The models are example,
also sometimes puzzled to some of the expression without further explanation. it may mix "girls of colors" and "girls paintings in different color", "a room in abstract paintings style" and "a room with abstract paintings" or "abstract painting of a room". "artsy outfit" and art work on the outfit".
20

---

D GPT-4V Judge Details
D.1
Order consistency and GPT-4V win rate
API Call
<query>
A.
<Result 1>
better
<Rasult 1> better
LMM
<Result 2>
<Result 1> win
API Call
<query>
is better
<Result 1> better
LMM
<Result
Figure 12: An example that result wins. If two calls insist that result 2 wins, then the number of result lose plus 1.
API Call
<query>
A better
<Result 1> better
B.
LMM
Similar
API Call
<query>
A.
Result 2>
A better
<Result 2> better
LMM
<Result 1>
Figure 13: An example that two calls hold different idea, indicating that the results have few difference in quality, and we say these two results are similar.
Order consistency (OC), As shown in Fig. 12 and 13, for each query and results from RI and R2, call GPT-4V twice. If and only if the results indicate same winner, commit that R1 wins or R2 wins (see Fig. 12). Otherwise we say they are similar and N,+ = 1. Specific input and output of one call can be found Appendix. G.I.
Win rate. When we are doing comparison experiments, we have to identify which one of the techniques is better. Thus using win rate is intuitional if the win rate is bigger than 50%, system A should perform better.
fiin-an-ssmilann rate. This metric is more in line with the user when they compare two products. For example, two products provide very similar results in most cases. this situation, using win rate will be high-error and unreliable. In fact, high similar rate should yield that the user may use both two products with high probability. Therefore, we introduce win-and-similar rate take similar cases into account, and we use this index when doing system level comparison.
21

---

D.2 Rationality of GPT-4V judge
While GPT-4V exhibits strong capabilities in various vision tasks, is still doubtful whether GPT-4V aligns well with human judgement such subjective tasks. Empirical validation of this methodology necessary. Therefore, we exploit our evaluation dataset, HPIR, to analyze the degree of consistency between GPT-4V's assessments and those of human labelers.
We directly compare the two groups (A and B) in HPIR using GPT-4V, and evaluate three different types of prompts, namely <ranker>, <scorer>, and <ep-scorer>. The <ranker> prompt involves merging ten images from group A and into single composite image (2 rows), upon which GPT-4V tasked with determining the better row. The <scorer> prompt entails providing GPT-4V with set of scoring guidelines to rate each group of images, where the group with the highest average score is deemed the winner. Lastly, the <ep-scorer> prompt requires merging ten images from group A and B into a single composite image too, then assigns scores to two groups concurrently. The winner is subsequently selected based the higher scores obtained. Additionally, order-consistency (OC) strategy discussed in Sec. 3. Given that the <scorer> approach evaluates system independently, this method free from order-consistency concerns. Furthermore, we engage five human experts who dedicate time to scoring for assessment purposes. The detailed descriptions of prompts are provided in supplementary materials.
also consider the
Table 10: Evaluation of GPT-4V with different prompts (<ranker>, <scorer>, and <cp-scorer>) HPIR. With order-consistency (OC), the <ranker> performs the best.
<ranker>
Kep-scorect
Human experts
HPIR Metric (%)
w/o OC
w/o OC
<scorer>
OC
OC
Accurncy
72.01
86.01
69.37
76.84
78.81
85.47
87.79
Acsthetic
86.70
60.94
80.79
77.83
82.52
85.00
The results are shown Table 10. We observe that the <ranker> prompt with consstconsistency performs the best among all the prompts, which is even comparable to human experts. This demon- strates the reliability of the GPT-4V judger. It is also evident that for pairs with minor aesthetic differences, GPT-4V's performance is considerably influenced by the order in which results presented, mirroring a similar characteristic in human evaluators. Utilizing the order-consistency approach, which computes the win rate exclusively is similar to that of humans. Given the subjective nature of aesthetic evaluation, the benchmarks set by humans HPIR represent the ceiling for this data, indicating substantial potential for advancements in multimodal models.
consistent data, GPT-4V's evaluative accuracy
D.3 Details of Table
Table 11: Comprehensive details Table
Accuracy
Aesthetic
win
similar
similar
similar
lose
simi lar
54
42
56.25%
72.00%
52
66
32
61.90%
78.67%
64.55%
74.00%
54
34
64.58%
77.33%
3
62
42
46
57.41%
69.33%
53
48
49
51.96%
67.33%
68
5
34
34
82
29.31%
45.33%
38
48
64
37.25%
57.33%
62.67%
60
56
37.78%
62.67%
Human labeler judge
1'
60
53.57%
65.33%
55
57
38
59.14%
74.67%
58
51.33%
63.33%
54
58
36.46%
61.33%
22

---

D.4 Details of Table 3
Table 12: Details of Table 3, origin query serves baseline method.
Accuracy
Aesibetic
prompt
win
similar
A losc
win
win & similar
A win
similar
lose
win
win & similar
original
<detail>>
47
76
27
63.51%
82.00%
44
86
20
68.75%
86.67%
list>
53.73%
<kw diet>
30
94
26
53.57%
82.67%
30
103
63.83%
88.67%
repeat
31
100
19
62.00%
87.33%
112
63.16%
90.67%
<reurg>
34
93
23
39.65%
84.67%
29
105
64.44%
89.33%
E Statistics of HPIR
In Table 13, we present HPIR results of our pretrained model, CLIP [38], and 3 aesthetic/quality models (CLIPIQA [50], MANIQA [62], IAP [42]). We further perform simple model ensemble to enhance the capability of aesthetic assessment simply by adding the clip score and the scaled scores of aesthetic models. Although still different from 2-stage approach, we use this index to select preferred aesthetic model as re-ranker in Sec. 2.3. This is because building a 2-stage retrieval system and searching scaling factors by calling LMMs to evaluate is unacceptable expansive. After grid search of models and scaling factors, we choose IAP [42] as the re-ranker.
Table 13: Performance of VL models and aesthetic models on HPIR benchmark.
VT.-model
Image AestheticQuality
Finsebbl
HIPRR-Metric.g (%)
Ours: PT
CLIP [38]
CLIPIQA [50]
IAP [42]
MANIQA [62]
CLIP + 1.25 IAP
Accuracy
66.19
68.11
58.22
61.22
55.46
72.63
Aesthetic
59.36
62.05
58.35
63.32
53.21
67.99
should be noted that unfeasible for the model attain perfect scores across both metrics concurrently. Additionally, aesthetic models only provide an aesthetic score. In our HPIR metric calculation, we use this score to evaluate both metrics. Therefore, the accuracy evaluations for aesthetic models have no reference value. Moreover, since HPIR cannot evaluate the retrieval prowess of the model, we can only compare the HPTR metrics of the two models when they have comparable retrieval capabilities.
Fig. 14 shows example query and its annotation file, each result is labeled with golden label and confidence. Fig. 15 displays the distribution of confidence scores with respect to two aspects (accuracy and aesthetic) in HPIR dataset. We observe majority of queries with confidence scores between and 0.5. Fig. 16 demonstrates the top-10 10 themes of the user queries with their frequencies. Because one query may have multiple themes, the sum of the count numbers can exceed 150. The most common themes are natural scene and human event, which follow the intuitive sense of human. Fig. 17 illustrates the interface of our labeling tool for labeling HPIR.
Query: vintage airplane collection
"eturacy's
Traan Jabelx": ("W", "0".-l. "2lder Label":
Group
"conFidence" 8.25
Thaao labelet: (1W". 197.-).
"aside 1ase1"
Group
Bisersity": {
19".-).
"poides lasel":
"Jabe"": '4",
"confidocce". 5.0925
Figure 14: An example of human labeling for the query "vintage airplane collection."
23

---

Accuracy confidence Distribution
Aesthetic confidence Distribution
25
20
Count
Count
0.0
0.2
0.4
0.6
0.0
0.2
Accuracy
Aesthetk
Figure 15: Distribution of confidence scores in HPIR dataset, with respect to three aspects (accuracy, aesthetic).
30
60-
30
scata
Tacheciates
Absured concests
rost
poot
Figure 16: Top 10 themes of user queries and their frequencies. Note that one query may have multiple themes.
User Query: rare wildflower species
Process: 17 / 300
Accurocy:
query
Aesthetic: Which
looks
beautiful
the
Diversity: which line is more diverse (in style and content)
Submi
Figure 17: Screen shot of labeling tool for human labelers to label the HPIR dataset.
24

---

LLM Rephrasing Prompts
F.1
Prompt Template
Our utilized prompt structure is as follows:
will given user query, and your task to generate concise image description in English that aligns with the user's intent. This description will facilitate the retrieval of images that are both accurate and aesthetically pleasing from the system. The description should be approximately (n_words) words length, constructed using the method outlined below:
(method)
user query: {query)
Here, {method} indicates the rules that query should obey.
F.2 Rephrasing Methods
We introduce the candidate methods of LLM rephrasing, the following items are possible substitutes for the method part of the template in the above template.
<dtaai>>:
Supply visual details of the objects given in the user's query.
<k list>:
Generate cemma-separated list of succinct object descriptions, visual details, or stylistic elements to extend the aesthetic understanding and expectations, ordered from the most to the least significant.
<kw diets:
Generate at least 2 key words for user query and then extend key word to description. Rank from most important key to least and follow this output format: <key>: <description>; <next :dda:cddeciiitionn
<reorg>:
Repeat the original query in different linguistic organization for several times, splited by "," and rank from the most accurate one.
We exemplify one query and various rephrasing results below, in Fig. 18-23.
Query: "Nice daily street"
Figure 18: Original search query and its retrieved results.
Query: "Nice daily street. Nice daily street. Nice daily street, Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street."
Figure 19: "repeat" method and its retrieved results.
25

---

Query: "Daily street scene with pleasant atmosphere, vibrant and inviting, A lovely urbon landscape capturing the beauty everyday life. Charming city street with relaxed and welcoming ambiance. Picturesque daily street scene with delightful inviting vibe. beautiful serene city street on peaceful day."
Figure 20: LLM rephrasing method: <reorg>, and its retrieved results.
Query: "A serene, tree-lined street on a clear day. The image coptures a peaceful urban scene with people strolling along the sidewalk, quaint shops and cafes lining the street, and colorful flower pots adding touch of charm. The warm sunlight bothes the scene, creating pleasant and inviting otmosphere."
Figure 21: LLM rephrasing method: <detail>, and its retrieved results.
Query: "People walking on a bustling city street, colorful storefronts with vibrant signs, urbon skyline in the background, street vendors selling fresh produce, pedestrians in stylish and casual attire, outdoor cafes with sidewalk seating, bicycles sharing the road, relaxed and inviting atmosphere."
Figure 22: LLM rephrasing method: <k list>, and its retrieved results.
Query: "Street: A bustling urbon street with pedestrians and vehicles, copturing the vibrant daily life in cit y; Daily: The routine activities of people going about their day, such commuting, shopping, and socializing."
STREET SPACE in BB PEDPLE
Figure 23: L.M rephrasing method: <kw dict>, and retrieved results.
F.3 Why Rephrasing Works
Our preliminary observation shows that there 2 main reasons for this phenomenon.
Sequence length is coupled with the image quality. As shown in Table 3 and Fig. 19, when we simply repeat the queries N times to extend the length the query, the aesthetic performance increases stably. This is possibly because most of the vision-language models, including ours, CLIP and DataComp, are trained on crawled images from websites, where images with high quality or meaningful content are more likely match longer caption.
Visual element extension continues to boost the aesthetic result. As shown Table using many of our rephrasing methods to extend the length of query can gain better result comparing to repeat method. This prove the assumption that LLM rephrasing bring deeper visual understanding to queries and further boost the result.
26

---

There are more possible aspects, for example, query style can influence image style, saturation and even image shape. We believe there are more interesting inductive biases hidden within the data, and hence we hope these interesting phenomenons can encourage further researches into the underlying reasons.
G GPT-4V Judger Prompts
We introduce the details GPT-4V judger in this section. We provide the system prompt (for GPT-4-vision model API) and example input and output for each.
G.1 Method <ranker>
System prompt:
judger inage retrieval systens that retrieve better images fron sbared data base. You will be given user query,
job identify syeten
image containing will he 2 rogs and each row will have 5 images, representing top retrieval images from 2 systens. Ne note that the first rou is the result from <aysten 1> is result from <aysten
snaller images.
provided inage,
and the second perspective: Accuracy, Aesthetic and Diversity. Requirenents:
Your judgenant will ba from 3
You NUST choose ene even if think two results are at same level*+ code that strictly follow fornat:
output
jaon
your analyze hou reaulta natch the user query accurately, shich row is better>, <int: only 2, row the second roe is better>,
"Accuracy analyza":
and
"Accuracy
"Aesthetic analyze": <atr: your
analyza tuo rou
is nore beautiful and elegant?,
"Aesthetic choice"; <int: only or 2, 1 for row 1 is better and for the row
"Diversity anslyze"; <str: your separate analyze of hov diveree of row 2 content style>,
"Diversity choice": <int: only or 2, 1 for row 1 is better and 2 for the second row ia batter?,
Example input and output:
Input
Output
[user query]: tender couple embraces [image]:
less intimane intersotio Raw2, howaver, corsistantty showa couples intimate brttece closely matching the euery:
has couple emoraceel induude
query
"Acnunery cholow" 3,
"Aertheiic wralyna"" Hlow 2 diappaps images with warmer tomes and mone coleodue visuo oppoat that ou eaest tendemeso, such os the stlvouttte and the imnggo with light. Row hos in var led seschedio quality, with some fet ages appio ling casuat less rocused the couple aspect). "Aeathetic choree": 2.
greater tery types imaggn settings
Rowr2, mal-te photogroshy incimate go... "Diveritty cat:
Figure 24: <ranker> (w/o OC) takes the input of two groups of images and prompts the model to choose the better one. In this example, line 2 wins the comparisons at accuracy and aesthetic aspects. In w/ OC setting, we take another call that exchanges line and 2 of the input images.
G.2 Method <scorer>
System prompt:
You judger inage retrieval systens whose is evaluate the systen by providing score for its retrieral results the given query. You sill be given user query, image containing inages, representing top
27

---

retrieval inages from systen. Your judgenent will be from perspective:
Accuracy, Aesthatic and Diversity. You should provide a 1-6 score for each of then.
your scoring
point: None of the result is eatistying,
totally wrong.
point: more than resulte are not acceptable, and the rest just nake sence.
point: all results
but only or satisfying.
point: nost of the results are satiafying, but or 2 of than are not so autstanding
just
point: all results are satistying.
Your output should be JSON code that strictly follov this fornat:
joon
"Accuracy analyza": <atr: your analyze of hou the resulta natch the user earp?,
query accurately,
"Accuracy score": <int: one integar number from I to 5, rapresenting the accuracy score?,
"Aesthetic analyze"; <etr: your sesthetic amalyze of hov beautiful and elegant the results are, then count the scores it can earn>, "Aasthatic
integar nunher from
representing the results
seathetic score?,
"Diversity analyze":
your analyze
content and style, then count the acorea it earn?, "Diversity score": <iat: one integer number from to 5, representing the diversity ...
Example input and output:
Input
Output
reolevol
floo"
"Accuray 1-4,
[user query]: tender couple embraces [image]]
"Aesthetic enetyze": The images pressried show nellenic to canton-tion rpporsntationss The first thee images, despin their swried art snylee, marrain a clear steualyy plessing dep ctonn of couphes emitrwcing. withs good use of color and comp oo tonn The fourth image, while siaually diecs, Bocs not the tonder oroth cup to subject motes The fitth imogee has on internating atyle may mature. Ovarat, the austhstic cuality ganeratty tight tormppovemeet.
artstic styles, from
unversslly corardered aesthetically plassing cuu
room
"Diorriity
Input
Output
"Acturayy The "Accurayy acone ". "Acsthetic andlyee" 2 "onsidoring sestheties, the imoges goneralty have soft. varm, and intimane hest which suts the fender sereet of the query. The lighting compostion the secor di and thrdd images are par cutarty stagent, entaroo the tender emotion the soones. The firs and fith images, while less compollie malrzain decent assthetic aigiment
[user query]: tender couple embraces [image]:
secthetic appeal. fourth image, however locks the subject being singl parson rotni: than couple. Thersforc, the acsthotic quality mostly high stagit
query'a
"Divoriit "The "Diversity soore':
Figure 25: <scorer> requires two calls for two groups separately. We only leave the analysis of aesthetic aspect for visualization. this example, first group wins accuracy and diversity aspects, and we will not take this sample into account when calculating aesthetic win-rate (as the two groups have equal scores).
G.3 Method <ep-scorer>
System prompt:
judger inage retrieval systens job by providing score for its retrieval results of the user's query. You vill be given user query, image, there will be rous and each rou will have inages, representing retrieval inages from 2 systens.
syeten
an image containing
images.
providad
note that the first rov is the result from <system and the second is the result from <aysten 2>. Your judgenent will be from perspactive: Accuracy, Aosthetic and Diversity. Your job is to give a systeme
ecore
aspects.
28

---

Hore is your scaring standard:
point: None of the
satistying, totally srong.
point: more than results are not accoptable, and the rest just nake sense.
point: reaulta
sense, only
satiafying-
point: nost of the resulta are satiafying, but or of then not
outstanding or just nake sense.
point: all results very satisfying.
output should sgaon
code that strictly follow
"Acouracy analyze': <stri analyze of how resulte of rov and rov match the user query accurately, then count the scores they can earn>, "Accuracy scores": [int]] list intogars fron 5, representing the accuracy acorea of row and rou 2>, "Aesthetic analyze"; <str: your
analyze
beautiful and elegant they than count the acores they can earn>, integers from to 5,
"Aesthetic acores"; <List [int]: list and row 2>, representing the accuracy
"Diversity analyze": <str: your analyze of hov diverse roe and row in content and style, "Diversity acores": <lit [int]: a list of 2 integers from 1 to 5, represeating accuracy
count: the
they
Example input and output:
Input
Output
"Accuracy anatyar:: Thow displas vanway of magas, howssees not all depect bender couple embrice us the user query. two images show indissclual and not couples, which not maton query, Row 2 more socurnety depions images of imagre bwing
[user query]: tender couple embraces [image]:
couples tender entrencse, matching: the user queey cleww rolovem::
"Accurayy scores"
onolyze"
images, popoliing acothodics comnibuting ansthetiaaly piwnsing
warm lightiig
collection?
"Acsthatlc secres": [3,
"Diversity analeec" "Row 1 showo divesiiyy tormo of ind vicuius and couple, but ludio diversity the cortect of the query, which serce for couples, Hano shows couples diftrant santings and embracea, providing prcvrtingeheerrtty white staging true to the query!
"Divenlly po 4]
Figure 26; <ep-scorer> (w/o OC) takes the input of two groups of images and prompts the model to score them separately. In this example, line 2 wins all comparisons. In w/ OC setting, we take another call that exchanges line and the input images and average the scores of two calls.
29

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We properly claim our contribution that we propose novel method to align image retrieval model with human aesthetics.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
30

---

Justification:
The performance of our method is limited to the public aesthetic models. We will discussion it in future.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided all the details in the paper.
Guidelines:
31

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: It is particularly costly to conduct a comprehensive code review. We plan to release the code in future. The training data won't ton't be released due to privacy reasons.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides getgggudd/ddduubbaeeeeee htte///oSSuumissioococo ceabbbiissiooooiccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / mnips.cc/publle htt//uuiiesssee (CodeSubmiii aasionoooicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
32

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Each experiment requires several days run, thus it is quite costly compute error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Appendix. A.
Guidelines:
The answer NA means that the paper does not include experiments.
33

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //apps:////uippe/// cwtpurips.ccppliiieeeeee http://errrpp...pppiiiiieeee wtwhieeiieellleeeeccccccc ines?
Answer:
[Yes]
Justification:
This work conforms every respect the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is on the image retrieval task, which does not involve societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
34

---

Justification:
Our retrieval model does not contain such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: have properly cited the assets under the license of CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [No]
Justification: Currently do not provide the code and data. When release them in future, we will provide document on them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
35

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "VgBvqYwhOh", "reviewer": "Reviewer_CcyJ", "summary": "This paper aligns the vision models with human values by leveraging LLM for query rephrasing and introducing preference-based reinforcement learning. The paper also presents a novel dataset named HPIR to benchmark the alignment with human aesthetics.", "questions": "Computational Cost: Can you elaborate on the computational costs and resource requirements of the reinforcement learning-based fine-tuning process? Are there any optimizations that could reduce the computational burden?\nUser Study: Have you conducted any user studies to validate the improvements in aesthetic alignment from a user perspective? If not, do you plan to include such studies in future work?", "limitations": "Exploring possible negative social impacts, such as implications for privacy or how the technology might be misused in unintended ways Research could benefit from deeper analysis of how biases in training data affect model outputs beyond just aesthetics, particularly with respect to cultural and demographic diversity.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper introduces a novel approach to align visual models with human aesthetics, combining LLM rewriting to enhance query understanding and using preference-based reinforcement learning to fine-tune the model. The paper is comprehensive in experiments and introduces the HPIR dataset for benchmarking. The paper is well-structured and the methods are clearly explained. Key concepts are well defined and the use of diagrams helps to effectively illustrate the results. And this paper improves the aesthetic quality of results in image retrieval by aligning visual models with human preferences. The proposed method and dataset provide valuable ideas for future research in this area.", "weaknesses": "[W1] This paper lacks a detailed user study to validate the actual effectiveness of the proposed method. Including a user study with different participants to evaluate the subjective improvement of aesthetic alignment could provide stronger evidence for the actual effectiveness of the method.\n[W2] Placing the related work in Section 6 makes it difficult for readers to have a clear understanding of the problem domain and existing research results before reading the specific methods and experiments, which is not conducive to the coherence of the paper structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1 & Q: Lack of user study\n**R:** In Table 2 on page 8 of the main paper, we have presented a user study (last two rows), where we let multiple human labelers judge the images retrieved from models w. and w/o. alignment (using the same queries). These labelers are expert search engine users. \nUsers judged which retrieved results from system A and B were better. \nThe first six rows of Table 2 are judged by GPT-4V.\n\nOur description of this part does not link it to the phrase \"user study\", causing confusion to the readers, and we will revise the description of this section. \n\nWe supplement more user studies over the experiments in Table 2 here:\n\n|   System A   |   System B   |   A to B win & similar rate by users   |  \n|:-----:|:-----:|:-----:|  \n| Ours-FT / Datacomp-15M | Ours-PT / Datacomp-15M | Acc: 65.3 / Aes: 74.7 |  \n| Ours-FT / Internal-8M | Ours-PT + Re-rank / Internal-8M | Acc: 66.9 / Aes: 71.4 |  \n| Ours-FT / Internal-8M | Bing Search / web | Acc: 49.1 / Aes: 56.6 |  \n| Ours-FT / Internal-8M | Getty Search / getty images | Acc: 63.3 / Aes: 61.3 |  \n\n\n\n## Q: Computational Cost\n**R:** The RL-finetuning process only used 4 NVIDIA A100 GPUs for fewer than 5 hours. If using smaller batch size or techniques like gradient accumulation, it can be trained within an acceptable time even using other consumer-grade GPUs with more than 4GB of memory. This is a small burden compared to most of today's work, including our pre-training phase.\n\n\n\n## W2: Related work\n**R:** Thank you for your comments, and we will make adjustments to the related work so that readers can better understand the article.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AKXp4WnbsR", "reviewer": "Reviewer_9cZ7", "summary": "This work aims to align vision models with human aesthetic standards in a retrieval system. To do this, the authors propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. The authors further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics.", "questions": "1.\tHave the authors considered human variance in aesthetics perception?\n2.\tAre the objective metrics enough for results evaluation? Have the authors considered using human studies to evaluate the results?", "limitations": "The authors did not indicate limitations in their paper, and mentioned that they will discuss it in future. I feel this paper has clear limitations such as the indication of human variance and the evaluation of the results.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of aligning vision models with human aesthetics in retrieval is interesting. This work has potential applications in various real-life applications. \n2.\tThe authors’ motivation of utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations is insightful. \n3.\tThe paper is well-written and informative.\n4.\tThe proposed dataset HPIR can be used by fellow researchers in the related fields.", "weaknesses": "I feel it can be further improved in the following ways. \n\n1.\tFor benchmarking human preferences, it might be better to record down the human variance in their annotations. I understand the authors used multiple annotations to ensure robustness, but since aesthetics is a subjective concept, human variance itself tells something.\n2.\tFollowing point 1, I feel the work can be made more solid if it includes some human evaluation studies on the experimental results. For example, in Fig. 5, it does not seem so obvious to me on the respective enhancement with finetuning. \n\nWithout the above two points, I feel the paper has somewhat overclaimed the \"alighing vision models with human aesthetics\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1 & Q1: Human variance\n**R:** Thank you for the suggestion. \nWe provide a metric 'confidence' for representing the robustness of the label.\nThe confidence score means the degree of agreement among all annotators, rather than a value provided by labeler. It is calculated through Equation 10.\nThis confidence score and variance have similar indicative meanings. For example, if a piece of data has the confidence score of 0.6, it means that 24 labelers hold the same choice and the other 6 labelers choose the opposite. Thus, $$Confidence=\\frac{2\\times24}{24+6}-1=0.6$$\nWe used this indicator and named it as confidence score because when both choices are supported by half of the labelers, $Confidence=0$, and when all labelers agree with a choice, $Confidence=1$. \nThe distributions of all confidence scores are shown in Fig. 15 of Appendix E.\n\nAs you mentioned, \"aesthetics is a subjective concept,\" we used the confidence as a weight (see Equation 11) to ensure that our accuracy was only calculated on less controversial comparisons. This ensures the robustness of the experiment and excludes the influence of subjectivity.\n\nPer your advice, we further calculate the variance for these labels. Let the positive choice be 1 and the negative choice be 0, and then the labeling results can be formulated as a sequence like [1,0,0,1,...]. It is easy to derive the following formula:\n$$\\text{variance} = \\frac{ 2N_{pos} N_{neg}}{( N_{pos}+ N_{neg})^2}.$$\nHere are the details of human variance of our labeled data, accompanied with confidence score:\n\n|  Confidence Score Range  |  Avg. Acc Confidence  |   Avg. Acc Variance  |  Avg. Aes Confidence  |  Avg. Aes Variance  |\n|:---:|:---:|:---:|:---:|:---:|\n|$0.0 \\leq c \\leq 0.2$| 0.108 | 0.492 | 0.106 | 0.492 |\n|$0.2 < c \\leq 0.4$| 0.324 | 0.446 | 0.343 | 0.440 |\n|$0.4 < c \\leq 0.6$| 0.490 | 0.379 | 0.506 | 0.371 |\n|$0.6 < c \\leq 0.8$| 0.676 | 0.269 | 0.676 | 0.269 |\n|$0.8 < c \\leq 1.0$| 0.950 | 0.047 | 0.966 | 0.032 |\n|All| 0.441 | 0.364 | 0.411 | 0.377 |\n\nLast but not least, perhaps our statement of confidence is bothering you because we didn't explain its similarity to variance. We'll modify the description to make it more explicit.\n\n## W2 & Q2: Lack of human evaluation\n**R:** Thank you for the suggestion. \nIn Table 2 on page 8 of the main paper, we have presented a user study (last two rows), where we let multiple human labelers judge the images retrieved from models w. and w/o. alignment (using the same queries). These labelers are expert search engine users. \nUsers judged which retrieved results from system A and B were better. \nThe first six rows of Table 2 are judged by GPT-4V.\n\nOur description of this part does not link it to the phrase \"user study\", causing confusion to the readers, and we will revise the description of this section. \n\nWe supplement more user studies over the experiments in Table 2 here:\n\n|   System A   |   System B   |   A to B win & similar rate by users   |  \n|:-----:|:-----:|:-----:|  \n| Ours-FT / Datacomp-15M | Ours-PT / Datacomp-15M | Acc: 65.3 / Aes: 74.7 |  \n| Ours-FT / Internal-8M | Ours-PT + Re-rank / Internal-8M | Acc: 66.9 / Aes: 71.4 |  \n| Ours-FT / Internal-8M | Bing Search / web | Acc: 49.1 / Aes: 56.6 |  \n| Ours-FT / Internal-8M | Getty Search / getty images | Acc: 63.3 / Aes: 61.3 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AmWSDQsDC0", "reviewer": "Reviewer_hvvN", "summary": "The paper looks into the alignment task for vision and language models within retrieval models where properties such as visual aesthetic comes to play. To achieve this, the paper collects some data to design a metric suitable for taking into account human aesthetic evaluation. And employs an RL-based technique to exploit the human opinion for better aligning the retrieved images with human aesthetic preferences.", "questions": "* The work has been focused on visual aesthetics, given the LLM rephrasing results, it may be beneficial to look into the sophistication of language parts and how that could correlate with the aesthetic of the image. Could that motivate higher level of language sophistication is also correlated with higher visual aesthetics?\n* Could you elaborate how the RL-based approach could scale?", "limitations": "The paper does not discuss any limitations with respect to the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It is well-written paper\n* The concept of aligning vision with aesthetic preferences is interesting and useful in some applications.\n* The experiments are well-designed and quite convincing. \n* It is interesting that LLM rephrasing could improve the quality of results", "weaknesses": "* The proposed metric could be elaborated better and maybe explained how the study ensured the metric is not designed under influence of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W: The design of the proposed metric\n\n**R:** Thanks.\nWe proposed two metrics in the paper: HPIR weighted accuracy and win rate. \n1. The construction of HPIR requires retrieving and filtering images according to the query, and then manually picking representative images to label. In order to exclude the influence of models during retrieving and filtering process, as depicted in Sec. 3.1, we leverage multiple retrieval systems (including Google, Getty, Bing etc.) and retrievers built upon various base models (CLIP, OpenCLIP, DataComp etc.) with different sizes and training data, to make it as comprehensive as possible. In addition, we merged results from these sources together and manually picked representative pairs. The performer does not know which model the data is coming from, and this process decoupled the data from the model's influence. \n2. We use win rate to quantify the comparison between different systems, which is a fair metric for different models. Additionally, the retrieval database we use is confidential, and we use human labeler to ensure that the results of GPT-4V are not seriously biased, and we use such indicators in the end-to-end evaluation at the system level, making it robust and reliable.\n\n## Q1: The correlation to the sophistication of language\n**R:** Yes, higher levels of language sophistication is also correlated with higher visual aesthetics. But, \"higher level of language sophistication\" is also coupled or correlated with \"aesthetic expectations and details\" in our paper. Like we found in Sec. 4.2, even using longer query brings increase to the results. Higher level of language sophistication is also correlated with higher visual aesthetics due to the inductive bias within pre-training data. \n\nHowever, a nice system should be user-friendly, without the need to write complex languages, and reduce the budget that comes with calling GPT. With our RL-finetuned model, users can achieve similar results with simple language as with complex language.\n\n## Q2: The scaling ability of RL-based approach\n**R:** Similar to RLHF in LLM, the more essential factor for whether an RL algorithm can scale mainly comes from the feedback. As an entry point to research, our feedback comes from aesthetic models, and it has a limited scaling upper-bound. In real application, we can obtain human feedback by, for example, user click-through rate. Feedback like this will exhibit excellent scaling properties and contribute to the continuous improvement of model capabilities, but due to privacy and policy reasons, it is out of the scope of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DezCTwzq1B", "reviewer": "Reviewer_amTt", "summary": "This paper studies the problem of aligning vision models with human aesthetic standards in a retrieval system. There are three key parts in the proposed model including LLM rephrasing, re-ranking, and RL fine-tuning. Two novel benchmarks are also introduced to integrate aesthetic quality into evaluation metrics. Experimental results demonstrate the effectiveness of the proposed method and the benchmarks.", "questions": "1. In line 62, it is said that the open-source IAA datasets cannot be used for aesthetic retrieval evaluation. Can you give a further explanation?\n2. In the first step of data preprocessing, the authors use the concept of \"topic\". Can you give a further explanation?", "limitations": "The paper does not discuss the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper addresses the aesthetic quality issue in image retrieval systems and introduces a reinforcement learning fine-tuning strategy that enables the retrieval model to directly retrieve images based on both semantics and aesthetic quality, eliminating the need for multi-stage filtering. This approach holds significant value.\n2. The paper introduces two evaluation benchmarks, addressing the limitation of current image retrieval benchmarks that fail to evaluate aesthetic quality.\n3. The experiments are comprehensive, validating the importance of each component in the proposed method.", "weaknesses": "1. The methodological process described in the article is somewhat cumbersome, with Figure 2 merely outlining key processes and concepts in a rudimentary manner, thereby increasing the difficulty for readers to comprehend.\n2. The authors appear to conflate \"no-reference image quality assessment\" with \"image aesthetic quality assessment.\" While these tasks are indeed closely related, they are distinct. MANIQA, for instance, should not be regarded as an aesthetic quality assessment model, and its paper does not evaluate the model's performance on aesthetic datasets.\n3. There remain some details in the article that are inadequately explained. It is peculiar that in Appendix Table 7, the same stride seemingly yields a different number of images.\n4. The manuscript contains typos. For example, the indicator function symbol in Equation 11 is clearly garbled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Cumbersome description of the method\n**R:** Thanks for your suggestion. We will find a way to further simplify the description of the method. The purpose of Fig. 2 is to illustrate the consistency of our approach and aesthetic concepts, and we will add more specific details to Fig. 3 that illustrates the specific steps. While at the same time, we will write a clearer index in the labels of Fig. 2 and Fig. 3 to avoid confusion for the reader.\n\n## W2: Conflate \"no-reference image quality assessment\" with \"image aesthetic quality assessment\"\n**R:** Thank you! We will modify the description to distinguish between MANIQA and the aesthetic quality assessment model. We use this model because we have experimented with more image quality-related models that include these three in the paper and found MANIQA performs well.\n\n## W3: Number of samples in Table 7\n**R:** Thanks! There may be some misunderstanding on the concept of stride, which refers to the interval between samples in our re-ordering sequence. The number of samples (as described in line 158-169, Sec. 2.3) is determined in terms of $u$ and $v$. The number of sampled images is $uv$ and the equation $|\\mathcal{D}_{po}| = uC_v^2+vC_u^2$ shows the number of comparison pairs.\n\nFor example, given a sequence of 100 images with $u$=$v$=5 and stride=2, we will sample $uv=25$ images with 2 as interval: [1,3,5,...,49]. If stride=3, it will be [1,4,7...,73]. The number of selected images is irrelevant to the stride.\n\n$|D_{po}|$ is the number of valid comparison pairs. When $u$=$v$=5 and stride=2 and we sampled [1,3,5...,49], we first split it with semantic dimension to [[1,3,5,7,9], [11,13,...],...,[...,47,49]]. There are 5 sequences of 5 images. For each sequence, such as [1,3,5,7,9], it contains $C_5^2$ valid comparison pairs: [(1,3), (1,5), ..., (7,9)]. Thus it will contain $5C_5^2$ pairs for all 5 sequences. Similarly, when we split with aesthetic dimension and result in [[1,11,21,31,41], [3,13,…]…, […,39,49]], it will also contain $5C_5^2$ pairs. Thus in this case $|\\mathcal{D}_{po}|=5C_5^2+5C_5^2=100$.\n\n## W4: Typos\n**R:** Thanks! We will fix the typos in the final version.\n\n\n## Q1: Explanation on why IAA dataset cannot be used for aesthetic retrieval evaluation\n**R:** A data item in IAA can be formulated as `(Img, Text, Score)`, for example, `(<cat img>, cat, 0.6)`; and `(<dog img>, dog, 0.73)`. To be fair, retrieval must be compared with the same query, therefore our required data should be in the following format: `(Text, [{img_1, score_1}, {img_2, score_2}...])`; for example, `(dog,[{<dog img1>, 0.76}, {<dog img2>, 0.54}…])`. An alternative way is to retrieve images based on the IAA's queries, but we still need human resources to label the comparison. In addition, the queries' distribution does not match to the distribution of retrieval system's users. Thus, we decided to directly label the testing set.\n\n## Q2: Explanation on \"topic\" in data preprocessing\n**R:** We expect that the distribution of the test set resembles the distribution of queries that are commonly searched by target users. User queries often involve user privacy that most companies will protect and cannot be directly obtained. A common alternative is to count the topic distribution, which are tags like \"financial scene\" or \"indoor decoration\" for queries. We generate queries based on the distribution of these topics to ensure that our query distribution is consistent with that of target users.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission proposes a method to improve language-based image retrieval in line with human aesthetic preferences. The method includes (i) rephrasing queries to elicit better retrieved results; and (ii) collecting preference data and using them for an RLHF-like LLM alignment.

Reviewers found the work:
1. Generally well-written, illustrated and structured
2. Well-motivated
3. Novel and with important contributions to the community, including the RLHF-like method, the prompt-rephrasing approach, and the proposed dataset.

There were concerns about lack of clarity in some areas which the reviewers felt were sufficiently addressed, and some misunderstandings about user study results which were resolved. In addition there were ethics concerns wrt consent and privacy of subjects and labellers in the collected dataset which were also satisfactorily addressed. Ultimately all 4 reviewers recommend acceptance and the AC sees no reason to override the unanimous recommendation.

The authors are asked to include the recommendations of the reviewers (including the ethics reviewer) in the camera ready version.
Decision: Accept (poster)