Title: Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image retrieval

Alignment

Aesthetics

Vision-Language Models
Abstract: Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics.  Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.

Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

Abstract

Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics. Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.

1 Introduction

Large-scale data pretrained models, e.g. CLIP [38], have been applied to a broad range of fields, e.g. visual generation [40, 59, 58, 54, 56], understanding [33, 48], LMMs [26, 19]. They are trained on very large image-text pair datasets, e.g. LAION [43] and DataComp [8], rather than the traditional ImageNet [6]. These datasets contain noisy labels, and exhibit diverse data quality. As a result, though models trained on such datasets demonstrate strong capabilities on semantic matching in the wild, they may prefer samples that violate the intents from users, as shown in Fig. 1. For example, using a vision-language model as an one stage retrieval system, with a huge amount of images in the database, the model may pick the images that exactly match the search query but with unappealing visual appearance. Moreover, it may provides harmful results that disrupt the principle of responsible AI (RAI). Existing retrieval benchmarks [25, 63] also lack evaluation for aesthetics and RAI.

LLMs

How to destroy the world ?

Here are the things to prepare:

(1) Large scale of biological

weapons. (2) The control of

resources including water and

food ……

User:

How to destroy the world ?

Model:

I'm sorry, but I cannot assist with

that request. It's important to

promote positive and constructive

actions that contribute to the well-

being of individuals and the planet.

w/o. alignment

Image Retrieval (Aesthetic)

Query:

Happy dog

playing with

a ball.

Query:

Happy dog

playing with

a ball.

Image Retrieval (RAI)

Query:

A lawyer

of color

defending

Image Retrieval (Visual Counting)

Query:

four apples

in a pic

Query:

four apples

in a pic

Query:

A lawyer

of color

defending

w/o. alignment

w/o. alignment

w/o. alignment

These problems are crucial in real retrieval engines, and are lacking investigation in the research. Among the products of the industrial community ( e.g. , Google search, Bing search, etc.), such problems are mitigated by a multi-stage approach, i.e. , a cascade of semantic search and multiple quality filters or re-rankers. However, multi-stage approach may introduce extra latency and a cascade of model biases, and it requires more manpower and resources to maintain, debug and A/B test. Therefore, integrating human preferences into model features and simplifying retrieval into an end-to-end system shows great research value and engineering importance, especially in the scenarios where on-end devices and large-scale API services are arranged.

Luckily, in the field of natural language processing [69, 45, 2, 36], the problem of misalignment has been extensively studied. Supervised fine-tuning and reinforcement learning from human feedback (RLHF) [5] have been proven to be effective, which significantly improve the quality of model outputs. Similar method is also widely adopted in some vision-language tasks, primarily in image captioning [41], and has recently been extended to non-textual vision tasks [37]. Nevertheless, the utilization of RL for subjective preferences in pure vision tasks has not yet been explored.

In this paper, we target the realm of visual aesthetics as a representative of human preference and aim to align the pre-trained vision models with human aesthetics. In Oxford dictionary [10], 'Aesthetic' has two explanations: (1) 'Connected with beauty and art and the understanding of beautiful things.' (2) 'Made in an artistic way and beautiful to look at.' We re-express this concept in Fig. 2. High level understanding of aesthetic may involve the cultural or symbolic aspects that require reasoning related to the object. Low level part of aesthetic is related to image resolution, layout, saturation, etc. Particularly, this visual appeal (low level part) is considered as some statistical prior information, which can be learned by an end-to-end neural network to a great extent.

Based on the above understanding, we build our pipeline as in Fig. 2, which first leverages the strong reasoning ability of LLMs to extend the query with expectations of implicitly containing the understanding of beauty. We find that using this rephrased query in retrieval drastically boosts the aesthetic quality more than we ever expected. Then, we utilize public aesthetic models to re-rank the retrieved images, resulting in a high quality image sequence that contains the inductive bias of both mechanisms and agrees with both aspects of aesthetic. Finally, a preference-based reinforcement learning method, adapted from DPO [39], is proposed to align the model with the sequence.

Former well-known open-source aesthetic datasets ( e.g. , [15, 32, 30]) were mainly designed for image aesthetic assessment (IAA) task, which cannot be used for aesthetic retrieval evaluation without adaptations. Thus, we propose two methods to evaluate models. For system level evaluation, we use GPT-4V as a judge to simulate users to choose a favored retrieval system within two candidates. Due to the subjective nature of aesthetic, we further construct a novel dataset (named HPIR) labeled by humans for model evaluation, and validating the reliability of the GPT-4V judge.

We make several contributions in this work: (1) We benchmark the alignment of human aesthetics with two methods, using both a novel dataset and using GPT-4V as a judge, which also investigates

Aesthetic :    Understanding of beauty  +  Image visual appeal

Understanding of beauty

Image visual appeal

Composition

Style

Symbolism

Cultural context

……

Resolution Saturation

Symmetry

Exposure

……

Our approach                      LLM rephrasing  +  Aesthetic model

distill

RL

Query

LLM rephrasing

Retrieved result

Aesthetic model

Re-rank with image visual appeal

LLM rephrasing

LLM extends conceptual aesthetic expectations

Aesthetic model

Re-ranked result

Vision

Language

>

RL finetune

how to prompt those large multi-modality models toward better aesthetic judgement. (2) We present that LLMs rephrasing of queries can significantly improve the aesthetic scores. (3) We propose a preference-based reinforcement learning method to align existing vision models with human aesthetics. Last but not least, aesthetics is one of the most subjective components of human preferences, and hence we believe our method can be easily generalized to other aspects of human preference.

2 Method for Aesthetic Alignment

2.1 Model Pretraining

We use both self-pretrained model and open source models [38, 8] for alignment fine-tuning. We pretrain our vision-language model using the adapted CLIP contrastive loss [38], which can be formulated as follows:

where s ij = ˆ a ⊤ i ˆ a j is the cosine similarity between the embeddings ˆ a i , ˆ a j of the corresponding image and text. τ is the temperature parameter and l ′ i = (1 -ϵ ) · l i + ϵ/N indicates a smoothed version of label l i with a factor of ϵ . The final pretraining loss L pt is the sum of the text and image losses:

The vision model and language model are initialized from the Swin-V2-L [28] and Roberta-L [27], respectively. We leverage several advanced techniques [57, 9, 55] and we leave our detailed insights into the pretraining process and data composition in Appendix. A.

2.2 Aesthetically Query Rephrasing with LLMs

According to the explanation of 'aesthetic', a query with explicit understanding of aesthetic will potentially benefit the quality of retrieved images. While a typical user's text query can be quite plain, we expect to leverage LLMs (e.g., GPT-3.5-turbo) to enrich such concepts and contents. The participation of LLMs or LMMs is crucial because high level of aesthetic understanding requires their strong reasoning ability. Existing aesthetic models do well to differentiate high and low quality images when they have a great quality gap, but when the gap is small, LMMs surpass all of them significantly via reasoning (Table 13 and 10 in Appendix). This indicates that to make further development of aesthetic understanding, reasoning is essential. While directly labeling a training dataset by LMMs is not acceptable in both latency and cost, using LLMs to reason and extend the query becomes a nice

w/o. LLM rephrasing

'An IG style pic of sofa'

w. LLM rephrasing

'Sleek modern sofa with clean lines, minimalist design, and neutral color palette, placed in a……'

substitute. In addition, LLMs can further refine the query with the following advantages: (1) Enrich queries with visual details, yielding more aesthetically appealing and expectation-aligned results. (2) Mitigate issues stemming from user search text style, misspellings, and incorrect emphasis.

The impact of the above enhancements will be quantified in Sec. 4.2. Our utilized prompt template can be found in Appendix. F.1, in which a 'method' indicating the rules that query should obey is required. We evaluate four distinct method prompts in Sec. 4.2, and finally advocate the one termed as <k list>:

Generate a comma-separated list of succinct object descriptions, visual details, or stylistic elements to extend the aesthetic understanding and expectations, ordered from the most to the least significant.

An example of query rephrasing is shown in Fig. 3. When we think about 'Instagram style', we usually have an imagination of light scene with clean and a little minimalist design. LLM rephrasing adds these elements directly to the query, resulting in a more satisfying retrieval results. In addition to aligning with user's implicit imagination, when the standard of beauty is associated with context of cultural or knowledge, LLM rephrasing can also significantly boost the results. More cases are presented in Appendix. C.

2.3 Aesthetic Alignment Fine-tuning

Weaimto directly align the retrieval model with human aesthetics, eliminating the multi-stage retrieval system with re-ranker. To obtain the training data for fine-tuning, we leverage public aesthetic models to build a two-stage retrieval system, generating sorted high-quality image sequences. Particularly, given the images retrieved by pretrained model, we utilize well-known semantic ( e.g. , CLIP [38]) and aesthetic ( e.g. , CLIPIQA [50], IAP [42] and MANIQA [62]) models as the re-ranker. Although these models are not exactly in the same category (e.g., MANIQA is a no-reference image quality assessment model instead of image aesthetic quality assessment model strictly speaking), we choose these models because during our engineering testing, they perform well as second stage re-rankers. Note that in real world engineering, we can adapt this pipeline to multi-stage system that may further leverage information like click rate, making the pipeline become RLHF.

Data preprocessing. We collect our training data using a four-step process:

We analyze the topic distribution of user queries and employ GPT-3.5-turbo to synthesize N = 80 , 000 pseudo queries that mirrored the distribution of authentic user queries. This procedure can protect user privacy well.

Each generated query is subjected to a rephrasing process as described in Sec. 2.2. The modified outputs are regarded as rephrased queries.

For each query, we utilize our pretrained model along with an Approximate Nearest Neighbor (ANN) search algorithm [4] to quickly retrieve the top K = 400 images from a 75 million subset of DataComp, using the rephrased query.

We compute the score of re-ranker (semantic and aesthetic models) for each image in search results.

The final training dataset D is structured as follows:

where q i and ˆ q i are pseudo query and rephrased pseudo query, and each x ( j ) i is defined as a tuple that contains image y ( j ) and the re-ranking scores:

Fine-tuning from AI feedback. We model the retrieval problem as a reinforcement learning problem: for a given search query q and an image database Y = { y n } , we denote the retrieval system with learnable parameters as the policy model π θ ( y | q ; Y ) . For some of the retrieved images, e.g., y i and y j , we can establish a preference y i > y j (or y i < y j ) to signify that image y i (or y j ) is a preferred retrieval response. Assume that these preferences are generated by some underlying reward model r ϕ ( y, q ) , e.g., human/AI scorer, and aesthetic model. Reinforcement learning maximizes the expectation of rewards while using KL divergence for regularization to prevent training collapse and overfitting:

Here, π ref is the reference model (i.e., the pretrained model). Following DPO [39], by choosing Bradley-Terry model [3] to formulate preference distribution, we can use the following policy objective loss to maximize the reward:

where y w is the preferred sample compared to y l . In retrieval scenario, given a user search query q , we build the partially ordered dataset D po for training by establishing those ordered pairs: D po = { ( q, y i , y j ) | y i < y j } . The probability that multimodal based policy model return response y i is given by the normalized cosine similarity:

Here, f θ v and f θ l represent the vision and language encoders of the multimodal model, respectively. It is easy to observe that the denominator part of π θ ( y | q ; Y ) will be cancelled out in L dpo , thus in actual calculations, we only need to calculate the cosine similarity of the corresponding images and queries, which also makes L dpo independent of the image database Y . Compared to DPO [39], we utilize an ordered sequence to obtain samples for adapting the DPO loss, allowing producing O ( n 2 ) preference pairs from a sequence of length n . This approach significantly enhances the data utilization rate, making the modified DPO algorithm more scalable.

Following InstructGPT [36], we also integrate the pre-training loss L pt to stabilize the training process and maintain retrieval capability. Consequently, the composite loss function formulated for fine-tuning is expressed as:

Construction of D po . We illustrate the construction of the partially ordered dataset D po in Fig. 4. For each query, images are intermittently selected from the retrieved results of the rephrased query at intervals defined as stride, aiming to amplify the quality discrepancy among the chosen images. Subsequently, these images are arranged into a matrix with u rows and v columns, following a row-major format. Samples within each row are sorted according to the score of the re-ranker. The re-ranker, which is finally designed by assembling open-source models, is evaluated in Appendix. E. We then define the column dimension as the aesthetic dimension, since the samples in each row are sorted aesthetically. The row dimension is defined as the semantic dimension because semantic relevance varies across different rows. We extract all rows and columns to obtain u + v ordered sequences with the form y 1 > y 2 > . . . > y k , resulting in C 2 k pairs of ( y i , y j ) in each sequence. To this end, a total of uC 2 v + vC 2 u partial order pairs can be produced for each query. Note that numerous operations in this process can be tensorized and executed in parallel, thus the time cost is low.

query

Top K images

>

>

>

>

>

>

……

……

Reshape

Rerank

Aesthetic

Semantic

LLM

IR system

3 Benchmarking Human Preference

Standard retrieval benchmarks, including MSCOCO [25] and Flickr30k [63], lack the aesthetic evaluation. Aesthetic datasets, e.g. AVA [32], can only be used to evaluate the accuracy of an aesthetic prediction model, which needs additional efforts for retrieval system evaluation. Therefore, we introduce the following two novel benchmarks to assess whether a retrieval model can align with human aesthetics well: (1) testing model preference by human-labeled dataset, and (2) using GPT-4V [35] to determine the relative performance of two systems.

3.1 Human Preference of Image Retrieval (HPIR)

We introduce HPIR, a test set of human preference alignment. It leverages 150 pseudo queries for testing, which are generated using LLMs by requesting an aligned distribution with the user's topics. For each query, we combine the results of multiple search engines and obtain 10 images, which are divided into two groups (A and B). Human labelers are asked to evaluate the two groups of images, determining which group is more precise and visually appealing. To ensure robustness, each comparison is annotated for 30 times. We also scrutinize the annotation time and order consistency (Sec. 3.2) to guarantee the quality. The label that predominated in the 30 annotations is designated as the golden label. Let N pos be denoted as the number of labelers that assign the golden label, and N neg as the remaining number. We define the confidence score w c (exemplified in Fig. 14 of Appendix) of this annotation as:

This confidence level has a similar effect to variance, and the variance formula for human labelers in aesthetic annotations can be easily calculated as follows:

To evaluate a model/search engine, we task it with discerning the better group between A and B for all queries, based on a designated criterion. Then the HPIR metric M asp ( asp stands for either accuracy or aesthetics.) is assessed by comparing the selections of the model/engine to the human-annotated golden labels. M asp is formulated as a confidence-weighted average over the queries:

where ✶ { choice = golden _ label } is an indicator that equals 1 when the model's choice matches the golden label, and 0 otherwise. This method can effectively assess the degree of alignment between a model and human preferences. For instance, to evaluate the CLIP [38], we simply compute the average CLIP similarities (to query) on group A and B, choosing the group with the higher average as the model's choice. More details about data distribution, baseline results and aesthetic model evaluation can be found in Sec. 4.1 and in Appendix. E.

3.2 GPT-4V Win Rate

LMMs have shown their strong abilities across numerous tasks. Thus, we directly compare two retrieval models/systems using GPT-4V [35]. Emulating the AlpacaEval [24] approach from LLMs, we first conduct image searches for a collection of queries using two retrieval systems, R1 and R2. We then concatenate results in R1 and R2 into one large image and employ GPT-4V as the judge to assess which system performs better. We note that GPT-4V tends to prefer the first row when the results from both systems are comparable, a tendency that mirrors human behavior. To address the bias, we introduce an order-consistency (OC) strategy, where we initially place images from R1 on the first row and images from R2 on the second for evaluation, then invert their positions for a separate assessment. A visualization and more detailed description is provided in Appendix. D.1. If two assessments have conflicting conclusions, we say the two results are similar. Lets denote the number of R1 wins as N w , loses as N l and similar as N s . System R2 serves as the baseline. We define the win rate of R1 as R win , and a win-and-similar rate as R win & similar :

Unlike HPIR, this approach lacks the supervision of human labelers, necessitating meticulous design and validation to ensure its soundness. We thus leverage HPIR feedback to filter various prompts and evaluation methods, ultimately selecting a prompt format referred to as <ranker> (see Appendix. G). Detailed experiments, shown in Appendix. D.2, yield that with this prompt and order-consistency, GPT-4V can present comparable aesthetic judgments to humans.

4 Experiments

In this section, we present our main experiments. More results, including ablations, benchmark evaluations (HPIR and GPT-4V judge), and LLM rephrasing, are in Appendix. B - F.

4.1 Details and Evaluations of Alignment Fine-tuning

In the alignment fine-tuning loss, the L pt component is configured identically to the pretraining phase described in Sec. 2.1, encompassing batch size, temperature, and data, with a weight of w pt = 1 . 0 . For the remaining components, each batch comprises 128 queries. The overall learning rate is fixed to lr = 5 × 10 -5 . The partially ordered set D po , as discussed in Sec. 2.3, is derived using u = v = 5 , and a stride of 10.

We conduct the experiments with two other state-of-the-art models: CLIP [38] and DataComp [8]. The image encoders of CLIP and DataComp are ViT-L/14 models, trained on a private 400M dataset and on the DataComp-1B dataset, respectively. We report their performance on classic retrieval benchmarks (ImageNet1K [6] zero-shot classification, MSCOCO [25] T2I retrieval Recall@1, and Flickr30K [63] T2I retrieval Recall@1) and on the proposed HPIR in Table 1. It is not surprising that our model performs worse to DataComp on MSCOCO and Flickr30K, since our training budget is much smaller than DataComp. We can further observe that our alignment fine-tuning does not significantly impact the retrieval capability, but it greatly enhances the aesthetic scores of the retrieval results, surpassing both original CLIP and DataComp. While fine-tuning on the CLIP and DataComp, a similar increase is observed in aesthetic ability, demonstrating the generalization of our method.

In Table 2, we report the system-level comparison results with other models, approaches and two commercial search engines (Bing image search [31] and Getty search [13]). We report the win-

and-similar rate here because it is more in line with the user's thinking when choosing products (see other indices and details in Appendix. D.3). Experiments are conducted on a database of 15M images extracted from DataComp-1B and our internal database with 8M images. The experiments demonstrate the effectiveness of our alignment fine-tuning, the superiority of our final model, and the gap with commercial web-scale products. Our finetuned model shows comparable performance with 2-stage approach, yielding a possible latency and pipeline optimization to real world products (see win rate in Appendix. D.3). Systematically speaking, despite the vast difference in the size of the databases, our model still achieved a win-and-similar rate of 50% ∼ 60% in comparisons with Bing image search and Getty search. Additional human user evaluations of selected experiments also validate the reliability of the GPT-4V judge. These labeling processes can be seen as real world A/B test user studies.

4.2 Effect of LLM Rephrasing

Here, we test different method prompts, which correspond to {method} as outlined in the template in Sec. 2.2, for search query rephrasing. We task GPT-3.5-turbo with the job of rephrasing queries to an approximate word count of 50. The method <k list>, as introduced in Sec. 2.2, enumerates additional descriptions for the query. The <detail> method encourages the model to elaborate with more specifics, while <kw dict> instructs the LLM to enumerate keywords followed by detailed expansions for each. We employ two verification metrics as described in this paper: HPIR and GPT-4V win rate, with the latter benchmarked against the original query's baseline results. The detailed prompts are described in the Appendix. F.2.

Table 3 reports the evaluation results, including the average scores from aesthetic models. The empirical evidence from all three metrics suggests that the utilization of LLMs for query rephrasing has enhanced the aesthetic appeal of the search results. We choose win rate to report because it brings clearer clues of which prompt is better (see other indices and details in Appendix. D.4).

To investigate the mechanisms by which query rephrasing yields enhancements, we further evaluate two additional rephrasing methods for extending the length of the original query: the 'repeat' method, which involves duplicating the original query n times, and the <reorg> method, which entails prompting the LLM to reformulate the query in diverse linguistic styles, then repeating it n times

'A dashing Maine cat'

'Surrealist art painting'

'grinning happy snowman'

w/ finetune

w/o finetune

一

一

A

without incorporating additional details. As shown in Table 3, simply enlarging the length of the query, even in the absence of new details, can enhance the aesthetic performance. Leveraging LLMs to deepen the comprehension of the query and enrich the visual specifics allows for further aesthetic improvement in retrieval tasks. We further summarize 2 possible reasons for this phenomenon in Appendix. F.3.

5 Cases Study and Qualitative Comparison

Fig. 5 shows the qualitative comparison between our fine-tuned model and pretrained model, where we retrieve top-4 images from the 75M subset of DataComp. It can be observed that the alignment fine-tuning endows the model with the capability to retrieve images with vivid background, rich texture details, and dynamic color contrast, leading to more aesthetically pleasing search engine.

More comparison result and analysis with and without LLM rephrasing using our fine-tuned model can be found in Appendix. C. With LLM rephrasing, the retrieved images exhibit remarkable improvement on visual coherence and enriched details. The styles of the images become more consistent with the search intent, capturing samples that align closely with human expectation.

6 Related Work

Vision Language Models. The availability of web-scale image-text pairs has sparked the research of vision-language models [7, 21, 23, 29, 46, 51, 67]. The pioneering works, CLIP [38] and ALIGN [14], utilize contrastive loss during training to achieve remarkable generalization capabilities. Subsequent works [61, 65, 44] have expanded image-text contrastive to a wider scope. BeiT3 [52], CoCa [64] and BLIP [20] further explore other pretraining methods. More recently, several large multi-modal models have emerged [26, 53, 12]. While most methods have shown strong retrieval capabilities, they often overlook aesthetics, frequently retrieving results with poor visual quality. Our work aims to fill this gap, focusing on designing a vision-language model with aligned aesthetics with humans.

Reinforcement Learning from Human Feedback (RLHF). RLHF has been widely adopted in LLMs [36, 45, 68, 49, 39]. Typically, a reward model is trained as a proxy for human preferences, providing feedback for model tuning. Recently, researchers focus and apply RLHF technique to computer vision [37, 16, 41]. In image generation, Lee et al. [18] and ImageReward [60] utilize the reward modeling for text-to-image tasks. To the best of our knowledge, our work is the first to focus on aligning human intents with the fundamental task of text-image retrieval.

Image Aesthetics Assessment. Numerous aesthetic evaluation protocols have been proposed [17, 47, 11, 22, 34, 42]. Kong et al. [17] propose relative ranking to model photo aesthetics. NIMA [47] assigns a score distribution to each photo, which captures the subjective variations in human aesthetic preferences. MANIQA [62] considers the no-reference image quality assessment task and employs a multi-scale attention mechanism to analyze images across various scales and regions. CLIPIQA [50] tries to understand image content and trains the model by comparing the quality of images. In this work, we adopt a weighted combination of existing models [62, 50, 42, 38] to provide supervision for our training. In addition, several IAA datasets (A V A [32], PN [15], and CHUNK-PQ [30]) have been proposed for evaluating aesthetic models. However, in retrieval setting, we need to compare the aesthetic levels of two image groups corresponding to a shared query. Existing datasets cannot satisfy the needs in retrieval scenario.

7 Potential Social Impacts and Risks

We state that all labelers (mostly university students) were informed of all the uses of their labels, and they agreed on the use. All experiments were conducted with open-source datasets and proprietary data for which we own the full copyright.

Our work may cause potential social impacts and risks:

Diversity and quality trade-off: The LLM rephrasing stage may increase the quality while decrease the diversity, which generally does more good than harm, but in cases where more diversity is required, several simple adjustments can be made to adapt. For example, repeating the rephrasing may benefit both direct inference and training models. In addition, increasing the pre-training loss weight during fine-tuning may also be an option to balance the trade-off.

Potential biases: LLMs are trained with massive internet-scale data and may therefore contain potential biases. This may reinforce the biases of the retrieval system. For example, when searching for the word 'nurse', the vector search system using the CLIP feature returns approximately 90% female images and 10% male images. However, when this system uses LLM rephrasing as preprocessing, it results in nearly 100% female images. This is because the rephrased sentence describes a female nurse. However, our approach also provides an opportunity to supervise and eliminate some biases. A vector search system using the CLIP feature is a black box, and we cannot predict or control when it may retrieve harmful content. By using LLM rephrasing, we can trace intermediate information and debug the system. We can further improve LLMs or implement a text-based filter.

Specific RAI problems: As we mentioned, our algorithm can be adapted to other areas of RAI problems, such as nationality and race. While our approach may benefit many general cases, it may potentially ignore minorities, as aesthetics and ethics may differ across various cultures and regions. Specific adaptations should be made to both LLMs and vision models in accordance with the 'no free lunch' theorem.

8 Conclusion

In this paper, we attempted to align image retrieval models with human aesthetics. We presented a preference-based reinforcement learning method to align retrieval models with human aesthetics by distilling knowledge from LLMs reasoning and aesthetic models. Extensive experiments demonstrated the effectiveness of our method, showing a possible alternative to the multi-stage retrieval pipeline. Finally, we discussed the potential social impacts and risks. We believe our proposed approach can become a general practice for other misalignment problems in computer vision.

Acknowledgement This project was supported by the Fundamental Research Funds for the Central Universities (2242024k30035).

References

M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310.12036 , 2023.

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , (3/4):324-345, 1952.

Q. Chen, B. Zhao, H. Wang, M. Li, C. Liu, Z. Li, M. Yang, and J. Wang. Spann: Highly-efficient billion-scale approximate nearest neighborhood search. NeurIPS , 2021.

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. NeurIPS , 2017.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. NeurIPS , 2013.

S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. NeurIPS , 2023.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

A. S. Hornby, J. Turnbull, D. Lea, D. Parkinson, P. Phillips, B. Francis, S. Webb, V. Bull, and M. Ashby. Oxford advanced learner's dictionary. International Student's Edition , 2010.

V. Hosu, B. Goldlucke, and D. Saupe. Effective aesthetics prediction with multi-level spatially pooled features. In CVPR , 2019.

S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, et al. Language is not all you need: Aligning perception with language models. NeurIPS , 2023.

G. Images. Getty images. https://www.gettyimages.com .

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

D. Joshi, R. Datta, E. Fedorovskaya, Q.-T. Luong, J. Z. Wang, J. Li, and J. Luo. Aesthetics and emotions in images. IEEE Signal Processing Magazine , 28(5):94-115, 2011.

H. Kazemi, F. Taherkhani, and N. Nasrabadi. Preference-based image generation. In WACV , 2020.

S. Kong, X. Shen, Z. Lin, R. Mech, and C. Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In ECCV , 2016.

K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS , 2021.

W. Li, X. Huang, J. Lu, J. Feng, and J. Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In CVPR , 2021.

X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV , 2020.

X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/ tatsu-lab/alpaca_eval , 2023.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. NeurIPS , 2023.

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR , 2022.

J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. NeurIPS , 2019.

W. Luo, X. Wang, and X. Tang. Content-based photo quality assessment. In ICCV , 2011.

Microsoft. Bing image search. https://www.bing.com/images/details/{0} .

N. Murray, L. Marchesotti, and F. Perronnin. Ava: A large-scale database for aesthetic visual analysis. In CVPR , 2012.

B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling. Expanding language-image pretrained models for general video recognition. In ECCV , 2022.

D. V. Nieto, L. Celona, and C. Fernandez-Labrador. Understanding aesthetics with language: A photo critique dataset for aesthetic assessment. arXiv preprint arXiv:2206.08614 , 2022.

OpenAI. Gpt-4v. https://openai.com/research/gpt-4v-system-card , 2023.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. NeurIPS , 2022.

A. S. Pinto, A. Kolesnikov, Y. Shi, L. Beyer, and X. Zhai. Tuning computer vision models with task rewards. In ICML , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS , 2023.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In ICML , 2021.

S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In CVPR , 2017.

C. Schuhmann. Laion-aesthetics predictor v2. https://github.com/ christophschuhmann/improved-aesthetic-predictor , 2022.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS , 2022.

J. Shao, S. Chen, Y. Li, K. Wang, Z. Yin, Y. He, J. Teng, Q. Sun, M. Gao, J. Liu, et al. Intern: A new learning paradigm towards general vision. arXiv preprint arXiv:2111.08687 , 2021.

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. NeurIPS , 2020.

W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai. Vl-bert: Pre-training of generic visual-linguistic representations. ICLR , 2020.

H. Talebi and P. Milanfar. Nima: Neural image assessment. TIP , 2018.

S. Tu, Q. Dai, Z. Wu, Z.-Q. Cheng, H. Hu, and Y.-G. Jiang. Implicit temporal modeling with learnable alignment for video recognition. In ICCV , 2023.

B. Wang, R. Zheng, L. Chen, Y. Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080 , 2024.

J. Wang, K. C. Chan, and C. C. Loy. Exploring clip for assessing the look and feel of images. In AAAI , 2023.

L. Wang, Y. Li, and S. Lazebnik. Learning deep structure-preserving image-text embeddings. In CVPR , 2016.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. NeurIPS , 2023.

Y. Wang, J. Bao, W. Weng, R. Feng, D. Yin, T. Yang, J. Zhang, Q. Dai, Z. Zhao, C. Wang, et al. Microcinema: A divide-and-conquer approach for text-to-video generation. In CVPR , 2024.

Y. Wei, Y. Cao, Z. Zhang, H. Peng, Z. Yao, Z. Xie, H. Hu, and B. Guo. iclip: Bridging image classification and contrastive language-image pre-training for visual recognition. In CVPR , 2023.

W. Weng, R. Feng, Y. Wang, Q. Dai, C. Wang, D. Yin, Z. Zhao, K. Qiu, J. Bao, Y. Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In CVPRW , 2024.

Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. In CVPR , 2022.

Z. Xing, Q. Dai, H. Hu, Z. Wu, and Y.-G. Jiang. Simda: Simple diffusion adapter for efficient video generation. In CVPR , 2024.

Z. Xing, Q. Feng, H. Chen, Q. Dai, H. Hu, H. Xu, Z. Wu, and Y.-G. Jiang. A survey on video diffusion models. ACM Computing Surveys , 2024.

J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS , 2024.

J. Yang, C. Li, P. Zhang, B. Xiao, C. Liu, L. Yuan, and J. Gao. Unified contrastive learning in image-text-label space. In CVPR , 2022.

S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, and Y. Yang. Maniqa: Multidimension attention network for no-reference image quality assessment. In CVPR , 2022.

P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2014.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022.

L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021.

Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. NeurIPS , 2023.

Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference , 2022.

R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964 , 2023.

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

A Pre-training Details

We pretrain our vision-language model, whose vision model (i.e., swin transformer) has been pretrained via SimMIM [57] and subsequently fine-tuned on ImageNet-22k [6] prior to the CLIP pretraining. To accommodate a larger batch size and an increased number of negative samples with limited resources, we implemented gradient accumulation and a queue dictionary, similar to MoCo [9]. In line with iCLIP [55], we incorporated ImageNet-22k to bolster the model's capability for short query and keyword searches. Thus, comparison of Table 1 on ImageNet is not totally fair, we report it only for a representative of keyword search capability.

Table 4 lists the composition and amount of the data during model pretraining. Throughout the training process, we employ a fixed learning rate lr = 5 × 10 -4 . Leveraging gradient accumulation and a queue, we enable a batch size of 32 k for contrastive learning. Our training method, similar to that of CLIP, incorporates the NCE Loss complemented by a label smoothing factor of 0 . 1 . Difference from CLIP, we initialize the temperature parameter to 0.05 and treat it as a learnable parameter. The computational resources include 256 NVIDIA V100 GPUs.

Table 5 shows the specific hyper-parameters for pre-training, including the data augmentation and model settings. The loss curve and gradient norm curve during the training process are shown in Fig. 6, where the deep blue line is the curve after smoothing. Fig. 7 demonstrates the curve of the retrieval evaluation metrics during the pre-training process, on three datasets (ImageNet1k-ZS, MSCOCO, Flickr30k).

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Steps (1e6)

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Loss

0.0

0.1

0.2

0.3

0.4

0.5

Steps (1e6)

0.50

0.75

1.00

1.25

1.50

1.75

2.00

2.25

2.50

Gradient norm

1

2

3

4

5

Epoch

81.4

81.5

81.6

81.7

81.8

81.9

82.0

82.1

ImageNet1K zero-shot acc@1

1

2

3

4

5

Epoch

38.50

38.75

39.00

39.25

39.50

39.75

40.00

40.25

MSCOCO T2I Recall@1

1

2

3

4

5

Epoch

64.5

65.0

65.5

66.0

66.5

Flickr30K T2I Recall@1

B Alignment Fine-tuning Details and Ablations

B.1 Fine-tuning Details and Curves

We show the details of fine-tuning hyper-parameters in Table 6. The whole training takes about 650 steps on 4 NVIDIA A100 GPUs. According to the ablation experiments in the following sections, we found that using strong data augmentations have positive influence on HPIR-Accuracy and retrieval metrics (mostly focus on semantic accuracy), but they may harm aesthetic performance. Thus we do not add any data augmentation during alignment fine-tuning. Fig. 8 shows the loss curves and gradient norm curve during fine-tuning. Note that the total loss is the weighted sum of DPO loss and pre-training loss. Fig. 9 shows the performance on retrieval benchmarks and HPIR during fine-tuning. It can be observed that the retrieval performance stays stable. While some fluctuation happened on HPIR curves, we believe this is because of the unavoidable subjective nature of HPIR, as both options (groups) of the dataset have close appearances.

0

100

200

300

400

500

600

Steps

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

Total loss

0

100

200

300

400

500

600

Steps

0.58

0.60

0.62

0.64

0.66

0.68

0.70

0.72

0.74

DPO Loss

0

100

200

300

400

500

600

Steps

0.5

1.0

1.5

2.0

2.5

3.0

Gradient norm

0

100

200

300

400

500

600

82.050

82.075

82.100

82.125

82.150

82.175

82.200

82.225

82.250

ImageNet1K zero-shot acc@1

0

100

200

300

400

500

600

39.9

40.0

40.1

40.2

40.3

40.4

40.5

MSCOCO T2I Recall@1

0

100

200

300

400

500

600

66.0

66.2

66.4

66.6

66.8

67.0

Flickr30K T2I Recall@1

0

100

200

300

400

500

600

Steps

66

67

68

69

70

71

HPIR-Accuracy weighted acc

0

100

200

300

400

500

600

Steps

60

62

64

66

68

HPIR-Aesthetic weighted acc

0

100

200

300

400

500

600

Steps

62

64

66

68

70

HPIR-Diversity weighted acc

B.2 Effect of 2-D sampling of D po

In order to perform a solid comparison, all experiments in Table 7 are set to similar budgets and the same hyper-parameters. The number of partial order pairs used for one query, as described in the main paper, is uC 2 v + vC 2 u . Thus we choose the u and v to have similar number of partial order pairs. |D po | represent the total size of fine-tuning dataset.

We first ablate the impact of the two dimensional sampling strategy in the construction of D po . In Table 7, the first line ( u = 15 , v = 1 ) represent that we only sample from semantic dimension, and the last line of the first block ( u = 1 , v = 15 ) shows the performance of sampling only from aesthetic dimension. As expected, the two dimensions bring different inductive biases derived from two aspects, and hence assembling both can benefit the alignment fine-tuning. The larger stride shows better result within a certain range, but few influence when it is large enough.

We also have to re-emphasize that although we named u as the semantic dimension, however, because the ranked sequence is retrieved from rephrased query (instead of the original query), the sequence may contain aesthetic benefits from LLM rephrasing. Thus u dimension can also help aesthetic performance. As a result, modifying the u, v may not cause a remarkable change in results.

B.3 Loss Selection

In addition to the ranked DPO loss as we introduced, we also adapt other losses to our scenario. We introduce them in the follows:

B.3.1 RRHF Loss.

RRHF [66] is a more simple approach that do not need the reference model. In our scenario, it is formulated as follow:

B.3.2 IPO Loss.

IPO [1] is also a general objective for RLHF. It is based on maximizing a non-linear function of preferences and keeps KL regularization. Similar to DPO, we adapt IPO to a ranking version, which is formulated as follows:

For each loss, we tuned their hyper-parameters and reported their best results in Table 8. Firstly, all losses bring growth to the HPIR results. This proves again that the way we built the training set provides valid, learnable patterns. IPO and DPO perform similar results. RRHF loss, as it does not contain regularization terms, can not scale well. The peak result of RRHF loss is obtained at about 100 steps (of 650 steps in total). Then its retrieval ability (i.e., retrieval benchmark results) decreases rapidly to an unacceptable range.

B.4 Impacts of Data Augmentation

Data augmentation, as it is known to all, bring benefits to a large range of computer vision tasks. Most of those tasks require model generalization capabilities on semantic dimension. Data augmentation greatly brings in important inductive biases, e.g., translation invariance, robustness of dealing occlusion, watermark, color change. However, we found in the aesthetic related tasks, some data augmentations may harm the performances.

In Table 9, the best results are obtained by implementing the same image transform with the evaluation. The evaluation transform only contains an image size transform and color normalization. We separately add different data augmentations to the evaluation transform, and find that they affect aesthetic indicators to varying degrees, while the accuracy aspect of HPIR metric may gain benefits.

C More cases study of LLM rephrasing

'metallic mineral'

'Fluid mechanics sculpture'

w/ LLM rephrasing

w/o LLM rephrasing

'Schematic diagram of neural network architecture'

'Super virus'

'The doomsday'

一

o

Hb S

Clienl

LAN NetworkDiagram

Cierl

VectorEPS10.

M

R

ALERT!

VIRUS

DOOMSOAY

ARY

人

DEAD

'Pretty girls of different colors'

w/ LLM rephrasing

w/o LLM rephrasing

'

A room in the style of abstract paintings

'

'

Traditional Russian architecture

'

'

Ancient Greek legends

'

'

Artsy outfit

'

000

一

一

The extension that LLM rephrasing provides can be divided as imagined elements and knowledge context, we analyze the cases in the following:

Imagined elements. In Fig. 10, when user search "metallic mineral"or "Fluid mechanics sculpture", we may expect a shining and glaze surface. LLM rephrased results satisfy our expectation by adding our imagined styles into the query. In addition, our models may mix neural networks and website service networks, with adding the descriptions of the image, e.g. "with directed edges and nodes", the semantic accuracy can also be boosted. Another similar case is that models may mix virus and computer virus without extra details supplied.

Knowledge context. Some of the objects need cultural or knowledge context. For example, as shown in Fig. 11, the understanding of abstract painting style and artsy style outfit. When searching cultural objects, LLM rephrased results seem more representative. The models are also sometimes puzzled to some of the expression without further explanation. For example, it may mix "girls of colors" and "girls paintings in different color", "a room in abstract paintings style" and "a room with abstract paintings" or "abstract painting of a room", "artsy outfit" and "art work on the outfit".

D GPT-4V Judge Details

D.1 Order consistency and GPT-4V win rate

<Result 1>

A.

B.

<query>

LMM

A is better

<Result 1> is better

API Call 1

<Result 1>

A.

B.

<query>

LMM

B is better

<Result 1> is better

API Call 2

<Result 1> win += 1

<Result 1>

A.

B.

<query>

LMM

A is better

<Result 1> is better

API Call 1

<Result 1>

A.

B.

<query>

LMM

A is better

API Call 2

Similar += 1

Order consistency (OC). As shown in Fig. 12 and 13, for each query and results from R1 and R2, we call GPT-4V twice. If and only if the two results indicate a same winner, we commit that R1 wins or R2 wins (see Fig. 12). Otherwise we say they are similar and N s + = 1 . Specific input and output of one call can be found at Appendix. G.1.

Win rate. When we are doing comparison experiments, we have to identify which one of the techniques is better. Thus using win rate is intuitional as if the win rate is bigger than 50%, system A should perform better.

Win-and-similar rate. This metric is more in line with the user when they compare two products. For example, If two products provide very similar results in most cases. In this situation, using win rate will be high-error and unreliable. In fact, high similar rate should yield that the user may use both two products with high probability. Therefore, we introduce win-and-similar rate to take similar cases into account, and we use this index when doing system level comparison.

D.2 Rationality of GPT-4V judge

While GPT-4V exhibits strong capabilities in various vision tasks, it is still doubtful whether GPT-4V aligns well with human judgement on such subjective tasks. Empirical validation of this methodology is necessary. Therefore, we exploit our evaluation dataset, HPIR, to analyze the degree of consistency between GPT-4V's assessments and those of human labelers.

We directly compare the two groups (A and B) in HPIR using GPT-4V, and evaluate three different types of prompts, namely <ranker>, <scorer>, and <cp-scorer>. The <ranker> prompt involves merging ten images from group A and B into a single composite image (2 rows), upon which GPT-4V is tasked with determining the better row. The <scorer> prompt entails providing GPT-4V with a set of scoring guidelines to rate each group of images, where the group with the highest average score is deemed the winner. Lastly, the <cp-scorer> prompt requires merging ten images from group A and B into a single composite image too, then assigns scores to two groups concurrently. The winner is subsequently selected based on the higher scores obtained. Additionally, we also consider the order-consistency (OC) strategy discussed in Sec. 3. Given that the <scorer> approach evaluates a system independently, this method is free from order-consistency concerns. Furthermore, we engage five human experts who dedicate time to scoring for assessment purposes. The detailed descriptions of prompts are provided in supplementary materials.

The results are shown in Table 10. We observe that the <ranker> prompt with order-consistency performs the best among all the prompts, which is even comparable to human experts. This demonstrates the reliability of the GPT-4V judger. It is also evident that for pairs with minor aesthetic differences, GPT-4V's performance is considerably influenced by the order in which results are presented, mirroring a similar characteristic in human evaluators. Utilizing the order-consistency approach, which computes the win rate exclusively on consistent data, GPT-4V's evaluative accuracy is similar to that of humans. Given the subjective nature of aesthetic evaluation, the benchmarks set by humans on HPIR represent the ceiling for this data, indicating substantial potential for advancements in multimodal models.

D.3 Details of Table 2

D.4 Details of Table 3

E Statistics of HPIR

In Table 13, we present HPIR results of our pretrained model, CLIP [38], and 3 aesthetic/quality models (CLIPIQA [50], MANIQA [62], IAP [42]). We further perform a simple model ensemble to enhance the capability of aesthetic assessment simply by adding the clip score and the scaled scores of aesthetic models. Although it is still different from 2-stage approach, we use this index to select a preferred aesthetic model as re-ranker in Sec. 2.3. This is because building a 2-stage retrieval system and searching scaling factors by calling LMMs to evaluate is unacceptable expansive. After grid search of models and scaling factors, we choose IAP [42] as the re-ranker.

It should be noted that it is unfeasible for the model to attain perfect scores across both metrics concurrently. Additionally, aesthetic models only provide an aesthetic score. In our HPIR metric calculation, we use this score to evaluate both metrics. Therefore, the accuracy evaluations for aesthetic models have no reference value. Moreover, since HPIR cannot evaluate the retrieval prowess of the model, we can only compare the HPIR metrics of the two models when they have comparable retrieval capabilities.

Fig. 14 shows an example query and its annotation file, each result is labeled with a golden label and a confidence. Fig. 15 displays the distribution of confidence scores with respect to two aspects (accuracy and aesthetic) in HPIR dataset. We observe a majority of queries with confidence scores between 0 and 0.5. Fig. 16 demonstrates the top-10 themes of the user queries with their frequencies. Because one query may have multiple themes, the sum of the count numbers can exceed 150. The most common themes are natural scene and human event, which follow the intuitive sense of human. Fig. 17 illustrates the interface of our labeling tool for labeling HPIR.

Group A

Group B

{

"Accuracy": {

"human labels": ["A", "B",…],

"golden label": {

"label": "B",

"confidence": 0.25

}

},

"Aesthetic": {

"human labels": ["B", "B",…],

"golden label": {

"label": "B",

"confidence": 0.4375

}

},

"Diversity": {

"human labels": ["A", "B",…],

"golden label": {

"label": "A",

"confidence": 0.0625

}

}

}

Query: vintage airplane collection

0.0

0.2

0.4

0.6

0.8

1.0

0

5

10

15

20

25

Count

Accuracy confidence Distribution

0.0

0.2

0.4

0.6

0.8

1.0

0

5

10

15

20

25

Count

Aesthetic confidence Distribution

Accuracy confidence

Aesthetic confidence

Natural scene

Human event

Arts

Daily object

Animal

Indoor scene

Tech-relative

Abstract concepts

Plant

Food

0

10

20

30

40

50

Frequency

Topic

Image Annotation

口

X

UserQuery:rarewildflowerspecies

Process: 17 / 300

line1

line2

Aesthetic:Whichlinelooksmorebeautiful

elegant?

line 1

line2

Diversity:whichline is more diverse (instyle and content)?

line 1

line2

Submit

Count

30

25

20

15

10

5

0

0.0

Diversity confidence Distri

0.2

0.4

0.6

Diversity confidence

F LLMRephrasing Prompts

F.1 Prompt Template

Our utilized prompt structure is as follows:

You will be given a user query, and your task is to generate a concise image description in English that aligns with the user's intent. This description will facilitate the retrieval of images that are both accurate and aesthetically pleasing from the system. The description should be approximately {n_words} words in length, constructed using the method outlined below:

Here, {method} indicates the rules that query should obey.

F.2 Rephrasing Methods

We introduce the candidate methods of LLM rephrasing, the following items are possible substitutes for the method part of the template in the above template.

<detail>:

Supply visual details of the objects given in the user's query.

<k list>:

Generate a comma-separated list of succinct object descriptions, visual details, or stylistic elements to extend the aesthetic understanding and expectations, ordered from the most to the least significant.

<kw dict>:

Generate at least 2 key words for user query and then extend key word to description. Rank from most important key to least and follow this output format: <key>: <description>; <next key>:<description>; ...

<reorg>:

Repeat the original query in different linguistic organization for several times, splited by "," and rank from the most accurate one.

We exemplify one query and various rephrasing results below, in Fig. 18-23.

Query: 'Nice daily street'

m

Query: 'Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street. Nice daily street.'

一一州

Query: 'Daily street scene with a pleasant atmosphere, vibrant and inviting. A lovely urban landscape capturing the beauty of everyday life. Charming city street with a relaxed and welcoming ambiance. Picturesque daily street scene with a delightful and inviting vibe. A beautiful and serene city street on a peaceful day.'

用

Query: 'A serene, tree -lined street on a clear day. The image captures a peaceful urban scene with people strolling along the sidewalk, quaint shops and cafes lining the street, and colorful flower pots adding a touch of charm. The warm sunlight bathes the scene, creating a pleasant and inviting atmosphere.'

Query: 'People walking on a bustling city street, colorful storefronts with vibrant signs, urban skyline in the background, street vendors selling fresh produce, pedestrians in stylish and casual attire, outdoor cafes with sidewalk seating, bicycles and cars sharing the road, a relaxed and inviting atmosphere. '

Query: 'Street: A bustling urban street with pedestrians and vehicles, capturing the vibrant daily life in a cit y; Daily: The routine activities of people going about their day, such as commuting, shopping, and socializing.'

G60

F.3 Why Rephrasing Works

Our preliminary observation shows that there are 2 main reasons for this phenomenon.

Sequence length is coupled with the image quality. As shown in Table 3 and Fig. 19, when we simply repeat the queries N times to extend the length of the query, the aesthetic performance increases stably. This is possibly because most of the vision-language models, including ours, CLIP and DataComp, are trained on crawled images from websites, where images with high quality or meaningful content are more likely to match a longer caption.

Visual element extension continues to boost the aesthetic result. As shown in Table 3, using many of our rephrasing methods to extend the length of query can gain a better result comparing to repeat method. This prove the assumption that LLM rephrasing bring deeper visual understanding to queries and further boost the result.

There are more possible aspects, for example, query style can influence image style, saturation and even image shape. We believe there are more interesting inductive biases hidden within the data, and hence we hope these interesting phenomenons can encourage further researches into the underlying reasons.

G GPT-4V Judger Prompts

We introduce the details of GPT-4V judger in this section. We provide the system prompt (for GPT-4-vision model API) and example input and output for each.

G.1 Method <ranker>

System prompt:

Example input and output:

[user query]: tender couple embraces [image]:

{

"Accuracy analyze": "Row 1 has images that are less relevant to the query 'tender

couple embraces' as they include non-couple images and less intimate interactions.

Row 2, however, consistently shows couples in intimate embrace, closely matching

the query.",

"Accuracy choice": 2,

"Aesthetic analyze": "Row 2 displays images with warmer tones and more cohesive

visual appeal that suggest tenderness, such as the silhouette and the image with

fairy lights. Row 1 has a more varied aesthetic quality, with some images appearing

more casual or less focused on the couple aspect.",

"Aesthetic choice": 2,

"Diversity analyze": "Row 1 shows a greater variety in types of images including

illustrations and photos, different settings, and a mix of close-up and distant shots.

Row 2, while more accurate, has less variety in terms of image type and setting,

focusing mainly on real-life photography in intimate settings.",

"Diversity choice": 1

}

Input

Output

Figure 24: <ranker> (w/o OC) takes the input of two groups of images and prompts the model to choose the better one. In this example, line 2 wins the comparisons at accuracy and aesthetic aspects. In w/ OC setting, we take another call that exchanges line 1 and 2 of the input images.

G.2 Method <scorer>

System prompt:

Example input and output:

Input

[user query]: tender couple embraces [image]:

[user query]: tender couple embraces [image]:

Input

{

"Accuracy analyze": "The image retrieval results …… is high but not perfect.", "Accuracy score": 4,

"Aesthetic analyze": "The images presented show a range of artistic styles, from realistic to cartoon-like representations. The first three images, despite their varied art styles, maintain a clear and visually pleasing depiction of couples embracing, with good use of color and composition. The fourth image, while visually clear, does not fit the tender aesthetic due to its subject matter. The fifth image has an interesting art style but may not be universally considered aesthetically pleasing due to its abstract nature. Overall, the aesthetic quality is generally high with some room for improvement.",

"Aesthetic score": 4,

"Diversity analyze": "The …… interpretations of the query.",

"Diversity score": 4

}

Output

{

"Accuracy analyze": "The user …… an average score.",

"Accuracy score": 3,

"Aesthetic analyze": "Considering the aesthetics, the images generally have a soft, warm, and intimate feel which suits the tender aspect of the query. The lighting and composition in the second and third images are particularly elegant, enhancing the tender emotion of the scenes. The first and fifth images, while less compelling, still maintain a decent level of aesthetic appeal. The fourth image, however, lacks the aesthetic alignment with the query's theme due to its subject being a single person rather than a couple. Therefore, the aesthetic quality is mostly high with a slight deduction for the less relevant image.",

"Aesthetic score": 4,

"Diversity analyze": "The …… improvement.",

"Diversity score": 3

}

Output

G.3 Method <cp-scorer>

System prompt:

You are a judger of image retrieval systems whose job is to evaluate the system by providing a score for its retrieval results of the user's query. You will be given a user query, and an image containing 10 smaller images. In the provided image, there will be 2 rows and each row will have 5 images, representing top 5 retrieval images from 2 systems. We note that the first row is the result from <system 1> and the second row is the result from <system 2>. Your judgement will be from 3 perspective: Accuracy, Aesthetic and Diversity. Your job is to give a score for both 2 systems from each of the above aspects.

Example input and output:

[user query]: tender couple embraces

[image]:

{

"Accuracy analyze": "Row 1 displays a variety of images, however, not all depict a

tender couple embrace as per the user query. Two images show individuals and not

couples, which do not match the query. Row 2 more accurately depicts images of

couples in tender embraces, matching the user query closely with all images being

relevant.",

"Accuracy scores": [3, 5],

"Aesthetic analyze": "Row 1 has a mix of images, some with appealing aesthetics

and others less so, particularly the images that do not match the couple embrace

theme. Row 2 images have a consistent theme of tender couple embraces, with

warm lighting and intimate settings contributing to a more aesthetically pleasing

collection.",

"Aesthetic scores": [3, 4],

"Diversity analyze": "Row 1 shows diversity in terms of individuals and a couple,

but lacks diversity in the context of the query, which asks for couples. Row 2 shows

couples in different settings and embraces, providing diversity while staying true to

the query.",

"Diversity scores": [2, 4]

}

Input

Output

Figure 26: <cp-scorer> (w/o OC) takes the input of two groups of images and prompts the model to score them separately. In this example, line 2 wins all comparisons. In w/ OC setting, we take another call that exchanges line 1 and 2 of the input images and average the scores of two calls.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We properly claim our contribution that we propose a novel method to align image retrieval model with human aesthetics.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: The performance of our method is limited to the public aesthetic models. We will discussion it in future.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical results in this work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided all the details in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: It is particularly costly to conduct a comprehensive code review. We plan to release the code in future. The training data won't be released due to privacy reasons.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See experiment part.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Each experiment requires several days to run, thus it is quite costly to compute error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix. A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms every respect of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is on the image retrieval task, which does not involve societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our retrieval model does not contain such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited all the assets under the license of CC-BY 4.0.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: Currently we do not provide the code and data. When we release them in future, we will provide document on them.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of aligning vision models with human aesthetic standards in a retrieval system. There are three key parts in the proposed model including LLM rephrasing, re-ranking, and RL fine-tuning. Two novel benchmarks are also introduced to integrate aesthetic quality into evaluation metrics. Experimental results demonstrate the effectiveness of the proposed method and the benchmarks.\n\nSTRENGTHS:\n1. This paper addresses the aesthetic quality issue in image retrieval systems and introduces a reinforcement learning fine-tuning strategy that enables the retrieval model to directly retrieve images based on both semantics and aesthetic quality, eliminating the need for multi-stage filtering. This approach holds significant value.\n2. The paper introduces two evaluation benchmarks, addressing the limitation of current image retrieval benchmarks that fail to evaluate aesthetic quality.\n3. The experiments are comprehensive, validating the importance of each component in the proposed method.\n\nWEAKNESSES:\n1. The methodological process described in the article is somewhat cumbersome, with Figure 2 merely outlining key processes and concepts in a rudimentary manner, thereby increasing the difficulty for readers to comprehend.\n2. The authors appear to conflate \"no-reference image quality assessment\" with \"image aesthetic quality assessment.\" While these tasks are indeed closely related, they are distinct. MANIQA, for instance, should not be regarded as an aesthetic quality assessment model, and its paper does not evaluate the model's performance on aesthetic datasets.\n3. There remain some details in the article that are inadequately explained. It is peculiar that in Appendix Table 7, the same stride seemingly yields a different number of images.\n4. The manuscript contains typos. For example, the indicator function symbol in Equation 11 is clearly garbled.\n\nQUESTIONS:\n1. In line 62, it is said that the open-source IAA datasets cannot be used for aesthetic retrieval evaluation. Can you give a further explanation?\n2. In the first step of data preprocessing, the authors use the concept of \"topic\". Can you give a further explanation?\n\nLIMITATIONS:\nThe paper does not discuss the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper looks into the alignment task for vision and language models within retrieval models where properties such as visual aesthetic comes to play. To achieve this, the paper collects some data to design a metric suitable for taking into account human aesthetic evaluation. And employs an RL-based technique to exploit the human opinion for better aligning the retrieved images with human aesthetic preferences.

STRENGTHS:
* It is well-written paper
* The concept of aligning vision with aesthetic preferences is interesting and useful in some applications.
* The experiments are well-designed and quite convincing. 
* It is interesting that LLM rephrasing could improve the quality of results

WEAKNESSES:
* The proposed metric could be elaborated better and maybe explained how the study ensured the metric is not designed under influence of the model.

QUESTIONS:
* The work has been focused on visual aesthetics, given the LLM rephrasing results, it may be beneficial to look into the sophistication of language parts and how that could correlate with the aesthetic of the image. Could that motivate higher level of language sophistication is also correlated with higher visual aesthetics?
* Could you elaborate how the RL-based approach could scale?

LIMITATIONS:
The paper does not discuss any limitations with respect to the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to align vision models with human aesthetic standards in a retrieval system. To do this, the authors propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. The authors further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics.\n\nSTRENGTHS:\n1.\tThe idea of aligning vision models with human aesthetics in retrieval is interesting. This work has potential applications in various real-life applications. \n2.\tThe authors’ motivation of utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations is insightful. \n3.\tThe paper is well-written and informative.\n4.\tThe proposed dataset HPIR can be used by fellow researchers in the related fields.\n\nWEAKNESSES:\nI feel it can be further improved in the following ways. \n\n1.\tFor benchmarking human preferences, it might be better to record down the human variance in their annotations. I understand the authors used multiple annotations to ensure robustness, but since aesthetics is a subjective concept, human variance itself tells something.\n2.\tFollowing point 1, I feel the work can be made more solid if it includes some human evaluation studies on the experimental results. For example, in Fig. 5, it does not seem so obvious to me on the respective enhancement with finetuning. \n\nWithout the above two points, I feel the paper has somewhat overclaimed the \"alighing vision models with human aesthetics\".\n\nQUESTIONS:\n1.\tHave the authors considered human variance in aesthetics perception?\n2.\tAre the objective metrics enough for results evaluation? Have the authors considered using human studies to evaluate the results?\n\nLIMITATIONS:\nThe authors did not indicate limitations in their paper, and mentioned that they will discuss it in future. I feel this paper has clear limitations such as the indication of human variance and the evaluation of the results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aligns the vision models with human values by leveraging LLM for query rephrasing and introducing preference-based reinforcement learning. The paper also presents a novel dataset named HPIR to benchmark the alignment with human aesthetics.\n\nSTRENGTHS:\nThis paper introduces a novel approach to align visual models with human aesthetics, combining LLM rewriting to enhance query understanding and using preference-based reinforcement learning to fine-tune the model. The paper is comprehensive in experiments and introduces the HPIR dataset for benchmarking. The paper is well-structured and the methods are clearly explained. Key concepts are well defined and the use of diagrams helps to effectively illustrate the results. And this paper improves the aesthetic quality of results in image retrieval by aligning visual models with human preferences. The proposed method and dataset provide valuable ideas for future research in this area.\n\nWEAKNESSES:\n[W1] This paper lacks a detailed user study to validate the actual effectiveness of the proposed method. Including a user study with different participants to evaluate the subjective improvement of aesthetic alignment could provide stronger evidence for the actual effectiveness of the method.\n[W2] Placing the related work in Section 6 makes it difficult for readers to have a clear understanding of the problem domain and existing research results before reading the specific methods and experiments, which is not conducive to the coherence of the paper structure.\n\nQUESTIONS:\nComputational Cost: Can you elaborate on the computational costs and resource requirements of the reinforcement learning-based fine-tuning process? Are there any optimizations that could reduce the computational burden?\nUser Study: Have you conducted any user studies to validate the improvements in aesthetic alignment from a user perspective? If not, do you plan to include such studies in future work?\n\nLIMITATIONS:\nExploring possible negative social impacts, such as implications for privacy or how the technology might be misused in unintended ways Research could benefit from deeper analysis of how biases in training data affect model outputs beyond just aesthetics, particularly with respect to cultural and demographic diversity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}