Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

Authors: miaosen  zhang
yixuan  wei
zhen  xing
yifei  ma
zuxuan  wu
ji  li
zheng  zhang
qi  dai
chong  luo
xin  geng
baining  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 07:24:18

Abstract: Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics.  Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.

Main paper: 
 introduction: 
 Large-scale data pretrained models, e.g. CLIP [38] , have been applied to a broad range of fields, e.g. visual generation [40, 59, 58, 54, 56] , understanding [33, 48] , LMMs [26, 19] . They are trained on very large image-text pair datasets, e.g. LAION [43] and DataComp [8] , rather than the traditional ImageNet [6] . These datasets contain noisy labels, and exhibit diverse data quality. As a result, though models trained on such datasets demonstrate strong capabilities on semantic matching in the wild, they may prefer samples that violate the intents from users, as shown in Fig. 1 . For example, using a vision-language model as an one stage retrieval system, with a huge amount of images in the database, the model may pick the images that exactly match the search query but with unappealing visual appearance. Moreover, it may provides harmful results that disrupt the principle of responsible AI (RAI). Existing retrieval benchmarks [25, 63] These problems are crucial in real retrieval engines, and are lacking investigation in the research. Among the products of the industrial community (e.g., Google search, Bing search, etc.), such problems are mitigated by a multi-stage approach, i.e., a cascade of semantic search and multiple quality filters or re-rankers. However, multi-stage approach may introduce extra latency and a cascade of model biases, and it requires more manpower and resources to maintain, debug and A/B test. Therefore, integrating human preferences into model features and simplifying retrieval into an end-to-end system shows great research value and engineering importance, especially in the scenarios where on-end devices and large-scale API services are arranged.Luckily, in the field of natural language processing [69, 45, 2, 36] , the problem of misalignment has been extensively studied. Supervised fine-tuning and reinforcement learning from human feedback (RLHF) [5] have been proven to be effective, which significantly improve the quality of model outputs. Similar method is also widely adopted in some vision-language tasks, primarily in image captioning [41] , and has recently been extended to non-textual vision tasks [37] . Nevertheless, the utilization of RL for subjective preferences in pure vision tasks has not yet been explored.In this paper, we target the realm of visual aesthetics as a representative of human preference and aim to align the pre-trained vision models with human aesthetics. In Oxford dictionary [10] , "Aesthetic" has two explanations: (1) "Connected with beauty and art and the understanding of beautiful things."(2) "Made in an artistic way and beautiful to look at." We re-express this concept in Fig. 2 . High level understanding of aesthetic may involve the cultural or symbolic aspects that require reasoning related to the object. Low level part of aesthetic is related to image resolution, layout, saturation, etc. Particularly, this visual appeal (low level part) is considered as some statistical prior information, which can be learned by an end-to-end neural network to a great extent.Based on the above understanding, we build our pipeline as in Fig. 2 , which first leverages the strong reasoning ability of LLMs to extend the query with expectations of implicitly containing the understanding of beauty. We find that using this rephrased query in retrieval drastically boosts the aesthetic quality more than we ever expected. Then, we utilize public aesthetic models to re-rank the retrieved images, resulting in a high quality image sequence that contains the inductive bias of both mechanisms and agrees with both aspects of aesthetic. Finally, a preference-based reinforcement learning method, adapted from DPO [39] , is proposed to align the model with the sequence.Former well-known open-source aesthetic datasets (e.g., [15, 32, 30] ) were mainly designed for image aesthetic assessment (IAA) task, which cannot be used for aesthetic retrieval evaluation without adaptations. Thus, we propose two methods to evaluate models. For system level evaluation, we use GPT-4V as a judge to simulate users to choose a favored retrieval system within two candidates. Due to the subjective nature of aesthetic, we further construct a novel dataset (named HPIR) labeled by humans for model evaluation, and validating the reliability of the GPT-4V judge.We make several contributions in this work: (1) We benchmark the alignment of human aesthetics with two methods, using both a novel dataset and using GPT-4V as a judge, which also investigates how to prompt those large multi-modality models toward better aesthetic judgement. (2) We present that LLMs rephrasing of queries can significantly improve the aesthetic scores.(3) We propose a preference-based reinforcement learning method to align existing vision models with human aesthetics. Last but not least, aesthetics is one of the most subjective components of human preferences, and hence we believe our method can be easily generalized to other aspects of human preference.2 Method for Aesthetic Alignment 
model pretraining: 
 We use both self-pretrained model and open source models [38, 8] for alignment fine-tuning. We pretrain our vision-language model using the adapted CLIP contrastive loss [38] , which can be formulated as follows:EQUATIONEQUATIONwhere s ij = â⊤ i âj is the cosine similarity between the embeddings âi , âj of the corresponding image and text. τ is the temperature parameter and l ′ i = (1 -ϵ) • l i + ϵ/N indicates a smoothed version of label l i with a factor of ϵ. The final pretraining loss L pt is the sum of the text and image losses:EQUATIONThe vision model and language model are initialized from the Swin-V2-L [28] and Roberta-L [27] , respectively. We leverage several advanced techniques [57, 9, 55] and we leave our detailed insights into the pretraining process and data composition in Appendix. A.According to the explanation of "aesthetic", a query with explicit understanding of aesthetic will potentially benefit the quality of retrieved images. While a typical user's text query can be quite plain, we expect to leverage LLMs (e.g., GPT-3.5-turbo) to enrich such concepts and contents. The participation of LLMs or LMMs is crucial because high level of aesthetic understanding requires their strong reasoning ability. Existing aesthetic models do well to differentiate high and low quality images when they have a great quality gap, but when the gap is small, LMMs surpass all of them significantly via reasoning ( substitute. In addition, LLMs can further refine the query with the following advantages: (1) Enrich queries with visual details, yielding more aesthetically appealing and expectation-aligned results. ( 2) Mitigate issues stemming from user search text style, misspellings, and incorrect emphasis.The impact of the above enhancements will be quantified in Sec. 4.2. Our utilized prompt template can be found in Appendix. F.1, in which a 'method' indicating the rules that query should obey is required. We evaluate four distinct method prompts in Sec. 4.2, and finally advocate the one termed as <k list>:Generate a comma-separated list of succinct object descriptions, visual details, or stylistic elements to extend the aesthetic understanding and expectations, ordered from the most to the least significant.An example of query rephrasing is shown in Fig. 3 . When we think about "Instagram style", we usually have an imagination of light scene with clean and a little minimalist design. LLM rephrasing adds these elements directly to the query, resulting in a more satisfying retrieval results. In addition to aligning with user's implicit imagination, when the standard of beauty is associated with context of cultural or knowledge, LLM rephrasing can also significantly boost the results. More cases are presented in Appendix. C.We aim to directly align the retrieval model with human aesthetics, eliminating the multi-stage retrieval system with re-ranker. To obtain the training data for fine-tuning, we leverage public aesthetic models to build a two-stage retrieval system, generating sorted high-quality image sequences. Particularly, given the images retrieved by pretrained model, we utilize well-known semantic (e.g., CLIP [38] ) and aesthetic (e.g., CLIPIQA [50] , IAP [42] and MANIQA [62] ) models as the re-ranker. Although these models are not exactly in the same category (e.g., MANIQA is a no-reference image quality assessment model instead of image aesthetic quality assessment model strictly speaking), we choose these models because during our engineering testing, they perform well as second stage re-rankers. Note that in real world engineering, we can adapt this pipeline to multi-stage system that may further leverage information like click rate, making the pipeline become RLHF.Data preprocessing. We collect our training data using a four-step process:• We analyze the topic distribution of user queries and employ GPT-3.5-turbo to synthesize N = 80, 000 pseudo queries that mirrored the distribution of authentic user queries. This procedure can protect user privacy well.• Each generated query is subjected to a rephrasing process as described in Sec. 2.2. The modified outputs are regarded as rephrased queries.• For each query, we utilize our pretrained model along with an Approximate Nearest Neighbor (ANN) search algorithm [4] to quickly retrieve the top K = 400 images from a 75 million subset of DataComp, using the rephrased query.• We compute the score of re-ranker (semantic and aesthetic models) for each image in search results.The final training dataset D is structured as follows:EQUATIONi , xi , . . . , xEQUATIONwhere q i and qi are pseudo query and rephrased pseudo query, and each x (j)i is defined as a tuple that contains image y (j) and the re-ranking scores:EQUATIONre-rank ).(EQUATIONFine-tuning from AI feedback. We model the retrieval problem as a reinforcement learning problem: for a given search query q and an image database Y = {y n }, we denote the retrieval system with learnable parameters as the policy model π θ (y|q; Y). For some of the retrieved images, e.g., y i and y j , we can establish a preference y i > y j (or y i < y j ) to signify that image y i (or y j ) is a preferred retrieval response. Assume that these preferences are generated by some underlying reward model r ϕ (y, q), e.g., human/AI scorer, and aesthetic model. Reinforcement learning maximizes the expectation of rewards while using KL divergence for regularization to prevent training collapse and overfitting:EQUATIONHere, π ref is the reference model (i.e., the pretrained model). Following DPO [39] , by choosing Bradley-Terry model [3] to formulate preference distribution, we can use the following policy objective loss to maximize the reward:EQUATIONwhere y w is the preferred sample compared to y l . In retrieval scenario, given a user search query q, we build the partially ordered dataset D po for training by establishing those ordered pairs: D po = {(q, y i , y j )|y i < y j }. The probability that multimodal based policy model return response y i is given by the normalized cosine similarity:EQUATIONHere, f θ v and f θ l represent the vision and language encoders of the multimodal model, respectively. It is easy to observe that the denominator part of π θ (y|q; Y) will be cancelled out in L dpo , thus in actual calculations, we only need to calculate the cosine similarity of the corresponding images and queries, which also makes L dpo independent of the image database Y. Compared to DPO [39] , we utilize an ordered sequence to obtain samples for adapting the DPO loss, allowing producing O(n 2 ) preference pairs from a sequence of length n. This approach significantly enhances the data utilization rate, making the modified DPO algorithm more scalable.Following InstructGPT [36] , we also integrate the pre-training loss L pt to stabilize the training process and maintain retrieval capability. Consequently, the composite loss function formulated for fine-tuning is expressed as:EQUATIONConstruction of D po . We illustrate the construction of the partially ordered dataset D po in Fig. 4 .For each query, images are intermittently selected from the retrieved results of the rephrased query at intervals defined as stride, aiming to amplify the quality discrepancy among the chosen images. Subsequently, these images are arranged into a matrix with u rows and v columns, following a row-major format. Samples within each row are sorted according to the score of the re-ranker. The re-ranker, which is finally designed by assembling open-source models, is evaluated in Appendix. E. We then define the column dimension as the aesthetic dimension, since the samples in each row are sorted aesthetically. The row dimension is defined as the semantic dimension because semantic relevance varies across different rows. We extract all rows and columns to obtain u + v ordered sequences with the form y 1 > y 2 > . . . > y k , resulting in C 2 k pairs of (y i , y j ) in each sequence. To this end, a total of uC 2 v + vC 2 u partial order pairs can be produced for each query. Note that numerous operations in this process can be tensorized and executed in parallel, thus the time cost is low.  
benchmarking human preference: 
 Standard retrieval benchmarks, including MSCOCO [25] and Flickr30k [63] , lack the aesthetic evaluation. Aesthetic datasets, e.g. AVA [32] , can only be used to evaluate the accuracy of an aesthetic prediction model, which needs additional efforts for retrieval system evaluation. Therefore, we introduce the following two novel benchmarks to assess whether a retrieval model can align with human aesthetics well: (1) testing model preference by human-labeled dataset, and (2) using GPT-4V [35] to determine the relative performance of two systems.We introduce HPIR, a test set of human preference alignment. It leverages 150 pseudo queries for testing, which are generated using LLMs by requesting an aligned distribution with the user's topics. For each query, we combine the results of multiple search engines and obtain 10 images, which are divided into two groups (A and B). Human labelers are asked to evaluate the two groups of images, determining which group is more precise and visually appealing. To ensure robustness, each comparison is annotated for 30 times. We also scrutinize the annotation time and order consistency (Sec. 3.2) to guarantee the quality. The label that predominated in the 30 annotations is designated as the golden label. Let N pos be denoted as the number of labelers that assign the golden label, and N neg as the remaining number. We define the confidence score w c (exemplified in Fig. 14 of Appendix) of this annotation as:EQUATIONThis confidence level has a similar effect to variance, and the variance formula for human labelers in aesthetic annotations can be easily calculated as follows:EQUATION)To evaluate a model/search engine, we task it with discerning the better group between A and B for all queries, based on a designated criterion. Then the HPIR metric M asp (asp stands for either accuracy or aesthetics.) is assessed by comparing the selections of the model/engine to the human-annotated golden labels. M asp is formulated as a confidence-weighted average over the queries:EQUATIONwhere 1{choice = golden_label} is an indicator that equals 1 when the model's choice matches the golden label, and 0 otherwise. This method can effectively assess the degree of alignment between a model and human preferences. For instance, to evaluate the CLIP [38] , we simply compute the average CLIP similarities (to query) on group A and B, choosing the group with the higher average as the model's choice. More details about data distribution, baseline results and aesthetic model evaluation can be found in Sec. 4.1 and in Appendix. E.LMMs have shown their strong abilities across numerous tasks. Thus, we directly compare two retrieval models/systems using GPT-4V [35] . Emulating the AlpacaEval [24] approach from LLMs, we first conduct image searches for a collection of queries using two retrieval systems, R1 and R2.We then concatenate results in R1 and R2 into one large image and employ GPT-4V as the judge to assess which system performs better. We note that GPT-4V tends to prefer the first row when the results from both systems are comparable, a tendency that mirrors human behavior. To address the bias, we introduce an order-consistency (OC) strategy, where we initially place images from R1 on the first row and images from R2 on the second for evaluation, then invert their positions for a separate assessment. A visualization and more detailed description is provided in Appendix. D.1.If two assessments have conflicting conclusions, we say the two results are similar. Lets denote the number of R1 wins as N w , loses as N l and similar as N s . System R2 serves as the baseline. We define the win rate of R1 as R win , and a win-and-similar rate as R win&similar :EQUATIONEQUATION)Unlike HPIR, this approach lacks the supervision of human labelers, necessitating meticulous design and validation to ensure its soundness. We thus leverage HPIR feedback to filter various prompts and evaluation methods, ultimately selecting a prompt format referred to as <ranker> (see Appendix. G). Detailed experiments, shown in Appendix. D.2, yield that with this prompt and order-consistency, GPT-4V can present comparable aesthetic judgments to humans. 
experiments: 
 In this section, we present our main experiments. More results, including ablations, benchmark evaluations (HPIR and GPT-4V judge), and LLM rephrasing, are in Appendix. B -F.In the alignment fine-tuning loss, the L pt component is configured identically to the pretraining phase described in Sec. 2.1, encompassing batch size, temperature, and data, with a weight of w pt = 1.0.For the remaining components, each batch comprises 128 queries. The overall learning rate is fixed to lr = 5 × 10 -5 . The partially ordered set D po , as discussed in Sec. 2.3, is derived using u = v = 5, and a stride of 10.We conduct the experiments with two other state-of-the-art models: CLIP [38] and DataComp [8] .The image encoders of CLIP and DataComp are ViT-L/14 models, trained on a private 400M dataset and on the DataComp-1B dataset, respectively. We report their performance on classic retrieval benchmarks (ImageNet1K [6] zero-shot classification, MSCOCO [25] T2I retrieval Recall@1, and Flickr30K [63] T2I retrieval Recall@1) and on the proposed HPIR in Table 1 . It is not surprising that our model performs worse to DataComp on MSCOCO and Flickr30K, since our training budget is much smaller than DataComp. We can further observe that our alignment fine-tuning does not significantly impact the retrieval capability, but it greatly enhances the aesthetic scores of the retrieval results, surpassing both original CLIP and DataComp. While fine-tuning on the CLIP and DataComp, a similar increase is observed in aesthetic ability, demonstrating the generalization of our method. In Table 2 , we report the system-level comparison results with other models, approaches and two commercial search engines (Bing image search [31] and Getty search [13] ). We report the win-and-similar rate here because it is more in line with the user's thinking when choosing products (see other indices and details in Appendix. D.3). Experiments are conducted on a database of 15M images extracted from DataComp-1B and our internal database with 8M images. The experiments demonstrate the effectiveness of our alignment fine-tuning, the superiority of our final model, and the gap with commercial web-scale products. Our finetuned model shows comparable performance with 2-stage approach, yielding a possible latency and pipeline optimization to real world products (see win rate in Appendix. D.3). Systematically speaking, despite the vast difference in the size of the databases, our model still achieved a win-and-similar rate of 50%∼60% in comparisons with Bing image search and Getty search. Additional human user evaluations of selected experiments also validate the reliability of the GPT-4V judge. These labeling processes can be seen as real world A/B test user studies. Here, we test different method prompts, which correspond to {method} as outlined in the template in Sec. 2.2, for search query rephrasing. We task GPT-3.5-turbo with the job of rephrasing queries to an approximate word count of 50. The method <k list>, as introduced in Sec. 2.2, enumerates additional descriptions for the query. The <detail> method encourages the model to elaborate with more specifics, while <kw dict> instructs the LLM to enumerate keywords followed by detailed expansions for each. We employ two verification metrics as described in this paper: HPIR and GPT-4V win rate, with the latter benchmarked against the original query's baseline results. The detailed prompts are described in the Appendix. F.2.Table 3 reports the evaluation results, including the average scores from aesthetic models. The empirical evidence from all three metrics suggests that the utilization of LLMs for query rephrasing has enhanced the aesthetic appeal of the search results. We choose win rate to report because it brings clearer clues of which prompt is better (see other indices and details in Appendix. D.4). To investigate the mechanisms by which query rephrasing yields enhancements, we further evaluate two additional rephrasing methods for extending the length of the original query: the "repeat" method, which involves duplicating the original query n times, and the <reorg> method, which entails prompting the LLM to reformulate the query in diverse linguistic styles, then repeating it n times without incorporating additional details. As shown in Table 3 , simply enlarging the length of the query, even in the absence of new details, can enhance the aesthetic performance. Leveraging LLMs to deepen the comprehension of the query and enrich the visual specifics allows for further aesthetic improvement in retrieval tasks. We further summarize 2 possible reasons for this phenomenon in Appendix. F.3. 
cases study and qualitative comparison: 
 Fig. 5 shows the qualitative comparison between our fine-tuned model and pretrained model, where we retrieve top-4 images from the 75M subset of DataComp. It can be observed that the alignment fine-tuning endows the model with the capability to retrieve images with vivid background, rich texture details, and dynamic color contrast, leading to more aesthetically pleasing search engine.More comparison result and analysis with and without LLM rephrasing using our fine-tuned model can be found in Appendix. C. With LLM rephrasing, the retrieved images exhibit remarkable improvement on visual coherence and enriched details. The styles of the images become more consistent with the search intent, capturing samples that align closely with human expectation.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: It is particularly costly to conduct a comprehensive code review. We plan to release the code in future. The training data won't be released due to privacy reasons.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Vision Language Models. The availability of web-scale image-text pairs has sparked the research of vision-language models [7, 21, 23, 29, 46, 51, 67] . The pioneering works, CLIP [38] and ALIGN [14] , utilize contrastive loss during training to achieve remarkable generalization capabilities. Subsequent works [61, 65, 44] have expanded image-text contrastive to a wider scope. BeiT3 [52] , CoCa [64] and BLIP [20] further explore other pretraining methods. More recently, several large multi-modal models have emerged [26, 53, 12] . While most methods have shown strong retrieval capabilities, they often overlook aesthetics, frequently retrieving results with poor visual quality. Our work aims to fill this gap, focusing on designing a vision-language model with aligned aesthetics with humans.Reinforcement Learning from Human Feedback (RLHF). RLHF has been widely adopted in LLMs [36, 45, 68, 49, 39] . Typically, a reward model is trained as a proxy for human preferences, providing feedback for model tuning. Recently, researchers focus and apply RLHF technique to computer vision [37, 16, 41] . In image generation, Lee et al. [18] and ImageReward [60] utilize the reward modeling for text-to-image tasks. To the best of our knowledge, our work is the first to focus on aligning human intents with the fundamental task of text-image retrieval.Image Aesthetics Assessment. Numerous aesthetic evaluation protocols have been proposed [17, 47, 11, 22, 34, 42] . Kong et al. [17] propose relative ranking to model photo aesthetics. NIMA [47] assigns a score distribution to each photo, which captures the subjective variations in human aesthetic preferences. MANIQA [62] considers the no-reference image quality assessment task and employs a multi-scale attention mechanism to analyze images across various scales and regions. CLIPIQA [50] tries to understand image content and trains the model by comparing the quality of images. In this work, we adopt a weighted combination of existing models [62, 50, 42, 38] to provide supervision for our training. In addition, several IAA datasets (AVA [32] , PN [15] , and CHUNK-PQ [30] ) have been proposed for evaluating aesthetic models. However, in retrieval setting, we need to compare the aesthetic levels of two image groups corresponding to a shared query. Existing datasets cannot satisfy the needs in retrieval scenario.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: See experiment part. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
potential social impacts and risks: 
 We state that all labelers (mostly university students) were informed of all the uses of their labels, and they agreed on the use. All experiments were conducted with open-source datasets and proprietary data for which we own the full copyright.Our work may cause potential social impacts and risks:• Diversity and quality trade-off: The LLM rephrasing stage may increase the quality while decrease the diversity, which generally does more good than harm, but in cases where more diversity is required, several simple adjustments can be made to adapt. For example, repeating the rephrasing may benefit both direct inference and training models. In addition, increasing the pre-training loss weight during fine-tuning may also be an option to balance the trade-off. • Potential biases: LLMs are trained with massive internet-scale data and may therefore contain potential biases. This may reinforce the biases of the retrieval system. For example, when searching for the word 'nurse', the vector search system using the CLIP feature returns approximately 90% female images and 10% male images. However, when this system uses LLM rephrasing as preprocessing, it results in nearly 100% female images. This is because the rephrased sentence describes a female nurse. However, our approach also provides an opportunity to supervise and eliminate some biases. A vector search system using the CLIP feature is a black box, and we cannot predict or control when it may retrieve harmful content. By using LLM rephrasing, we can trace intermediate information and debug the system. We can further improve LLMs or implement a text-based filter. • Specific RAI problems: As we mentioned, our algorithm can be adapted to other areas of RAI problems, such as nationality and race. While our approach may benefit many general cases, it may potentially ignore minorities, as aesthetics and ethics may differ across various cultures and regions. Specific adaptations should be made to both LLMs and vision models in accordance with the 'no free lunch' theorem.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Each experiment requires several days to run, thus it is quite costly to compute error bars. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 In this paper, we attempted to align image retrieval models with human aesthetics. We presented a preference-based reinforcement learning method to align retrieval models with human aesthetics by distilling knowledge from LLMs reasoning and aesthetic models. Extensive experiments demonstrated the effectiveness of our method, showing a possible alternative to the multi-stage retrieval pipeline.Finally, we discussed the potential social impacts and risks. We believe our proposed approach can become a general practice for other misalignment problems in computer vision.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: See Appendix. A. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: This work conforms every respect of the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work is on the image retrieval task, which does not involve societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
 
 Appendix: We pretrain our vision-language model, whose vision model (i.e., swin transformer) has been pretrained via SimMIM [57] and subsequently fine-tuned on ImageNet-22k [6] prior to the CLIP pretraining. To accommodate a larger batch size and an increased number of negative samples with limited resources, we implemented gradient accumulation and a queue dictionary, similar to MoCo [9] . In line with iCLIP [55] , we incorporated ImageNet-22k to bolster the model's capability for short query and keyword searches. Thus, comparison of Table 1 on ImageNet is not totally fair, we report it only for a representative of keyword search capability. Table 5 shows the specific hyper-parameters for pre-training, including the data augmentation and model settings. The loss curve and gradient norm curve during the training process are shown in Fig. 6 , where the deep blue line is the curve after smoothing. Fig. 7 demonstrates the curve of the retrieval evaluation metrics during the pre-training process, on three datasets (ImageNet1k-ZS, MSCOCO, Flickr30k). We show the details of fine-tuning hyper-parameters in Table 6 . The whole training takes about 650 steps on 4 NVIDIA A100 GPUs. According to the ablation experiments in the following sections, we found that using strong data augmentations have positive influence on HPIR-Accuracy and retrieval metrics (mostly focus on semantic accuracy), but they may harm aesthetic performance. Thus we do not add any data augmentation during alignment fine-tuning. Fig. 8 shows the loss curves and gradient norm curve during fine-tuning. Note that the total loss is the weighted sum of DPO loss and pre-training loss. Fig. 9 shows the performance on retrieval benchmarks and HPIR during fine-tuning. It can be observed that the retrieval performance stays stable. While some fluctuation happened on HPIR curves, we believe this is because of the unavoidable subjective nature of HPIR, as both options (groups) of the dataset have close appearances. In order to perform a solid comparison, all experiments in Table 7 are set to similar budgets and the same hyper-parameters. The number of partial order pairs used for one query, as described in the main paper, is uC 2 v + vC 2 u . Thus we choose the u and v to have similar number of partial order pairs. |D po | represent the total size of fine-tuning dataset.We first ablate the impact of the two dimensional sampling strategy in the construction of D po . In Table 7 , the first line (u = 15, v = 1) represent that we only sample from semantic dimension, and the last line of the first block (u = 1, v = 15) shows the performance of sampling only from aesthetic dimension. As expected, the two dimensions bring different inductive biases derived from two aspects, and hence assembling both can benefit the alignment fine-tuning. The larger stride shows better result within a certain range, but few influence when it is large enough.We also have to re-emphasize that although we named u as the semantic dimension, however, because the ranked sequence is retrieved from rephrased query (instead of the original query), the sequence may contain aesthetic benefits from LLM rephrasing. Thus u dimension can also help aesthetic performance. As a result, modifying the u, v may not cause a remarkable change in results. In addition to the ranked DPO loss as we introduced, we also adapt other losses to our scenario. We introduce them in the follows:B.3.1 RRHF Loss.RRHF [66] is a more simple approach that do not need the reference model. In our scenario, it is formulated as follow:EQUATIONEQUATIONIPO [1] is also a general objective for RLHF. It is based on maximizing a non-linear function of preferences and keeps KL regularization. Similar to DPO, we adapt IPO to a ranking version, which is formulated as follows:EQUATIONEQUATIONFor each loss, we tuned their hyper-parameters and reported their best results in Table 8 . Firstly, all losses bring growth to the HPIR results. This proves again that the way we built the training set provides valid, learnable patterns. IPO and DPO perform similar results. RRHF loss, as it does not contain regularization terms, can not scale well. The peak result of RRHF loss is obtained at about 100 steps (of 650 steps in total). Then its retrieval ability (i.e., retrieval benchmark results) decreases rapidly to an unacceptable range. Data augmentation, as it is known to all, bring benefits to a large range of computer vision tasks. Most of those tasks require model generalization capabilities on semantic dimension. Data augmentation greatly brings in important inductive biases, e.g., translation invariance, robustness of dealing occlusion, watermark, color change. However, we found in the aesthetic related tasks, some data augmentations may harm the performances.In Table 9 , the best results are obtained by implementing the same image transform with the evaluation. The evaluation transform only contains an image size transform and color normalization. We separately add different data augmentations to the evaluation transform, and find that they affect aesthetic indicators to varying degrees, while the accuracy aspect of HPIR metric may gain benefits. C More cases study of LLM rephrasing The extension that LLM rephrasing provides can be divided as imagined elements and knowledge context, we analyze the cases in the following:• Imagined elements. In Fig. 10 , when user search "metallic mineral"or "Fluid mechanics sculpture", we may expect a shining and glaze surface. LLM rephrased results satisfy our expectation by adding our imagined styles into the query. In addition, our models may mix neural networks and website service networks, with adding the descriptions of the image, e.g. "with directed edges and nodes", the semantic accuracy can also be boosted. Another similar case is that models may mix virus and computer virus without extra details supplied.• Knowledge context. Some of the objects need cultural or knowledge context. For example, as shown in Fig. 11 , the understanding of abstract painting style and artsy style outfit. When searching cultural objects, LLM rephrased results seem more representative. The models are also sometimes puzzled to some of the expression without further explanation. For example, it may mix "girls of colors" and "girls paintings in different color", "a room in abstract paintings style" and "a room with abstract paintings" or "abstract painting of a room", "artsy outfit" and "art work on the outfit". In Table 13 , we present HPIR results of our pretrained model, CLIP [38] , and 3 aesthetic/quality models (CLIPIQA [50] , MANIQA [62] , IAP [42] ). We further perform a simple model ensemble to enhance the capability of aesthetic assessment simply by adding the clip score and the scaled scores of aesthetic models. Although it is still different from 2-stage approach, we use this index to select a preferred aesthetic model as re-ranker in Sec. 2.3. This is because building a 2-stage retrieval system and searching scaling factors by calling LMMs to evaluate is unacceptable expansive. After grid search of models and scaling factors, we choose IAP [42] as the re-ranker. It should be noted that it is unfeasible for the model to attain perfect scores across both metrics concurrently. Additionally, aesthetic models only provide an aesthetic score. In our HPIR metric calculation, we use this score to evaluate both metrics. Therefore, the accuracy evaluations for aesthetic models have no reference value. Moreover, since HPIR cannot evaluate the retrieval prowess of the model, we can only compare the HPIR metrics of the two models when they have comparable retrieval capabilities.Fig. 14 shows an example query and its annotation file, each result is labeled with a golden label and a confidence. Fig. 15 displays the distribution of confidence scores with respect to two aspects (accuracy and aesthetic) in HPIR dataset. We observe a majority of queries with confidence scores between 0 and 0.5. Fig. 16 demonstrates the top-10 themes of the user queries with their frequencies.Because one query may have multiple themes, the sum of the count numbers can exceed 150. The most common themes are natural scene and human event, which follow the intuitive sense of human. Fig. 17 illustrates the interface of our labeling tool for labeling HPIR.{ "Accuracy": { "human labels": ["A", "B",…], "golden label": { "label": "B", "confidence": 0.25 } }, "Aesthetic": { "human labels": ["B", "B",…], "golden label": { "label": "B", "confidence": 0.4375 } }, "Diversity": { "human labels": ["A", "B",…], "golden label": { "label": "A", "confidence": 0.0625 } } } Our utilized prompt structure is as follows:You will be given a user query, and your task is to generate a concise image description in English that aligns with the user's intent. This description will facilitate the retrieval of images that are both accurate and aesthetically pleasing from the system. The description should be approximately {n_words} words in length, constructed using the method outlined below: {method} user query: {query}Here, {method} indicates the rules that query should obey.We introduce the candidate methods of LLM rephrasing, the following items are possible substitutes for the method part of the template in the above template.• <detail>:Supply visual details of the objects given in the user's query.Generate a comma-separated list of succinct object descriptions, visual details, or stylistic elements to extend the aesthetic understanding and expectations, ordered from the most to the least significant.Generate at least 2 key words for user query and then extend key word to description. Rank from most important key to least and follow this output format: <key>: <description>; <next key>:<description>; ...Repeat the original query in different linguistic organization for several times, splited by "," and rank from the most accurate one.We exemplify one query and various rephrasing results below, in Fig. 18 19 20 21 22 23 .Query: "Nice daily street" Figure 19 : "repeat" method and its retrieved results. Query: "People walking on a bustling city street, colorful storefronts with vibrant signs, urban skyline in the background, street vendors selling fresh produce, pedestrians in stylish and casual attire, outdoor cafes with sidewalk seating, bicycles and cars sharing the road, a relaxed and inviting atmosphere." Our preliminary observation shows that there are 2 main reasons for this phenomenon.• Sequence length is coupled with the image quality. As shown in Table 3 and Fig. 19 , when we simply repeat the queries N times to extend the length of the query, the aesthetic performance increases stably. This is possibly because most of the vision-language models, including ours, CLIP and DataComp, are trained on crawled images from websites, where images with high quality or meaningful content are more likely to match a longer caption.• Visual element extension continues to boost the aesthetic result. As shown in Table 3 , using many of our rephrasing methods to extend the length of query can gain a better result comparing to repeat method. This prove the assumption that LLM rephrasing bring deeper visual understanding to queries and further boost the result.There are more possible aspects, for example, query style can influence image style, saturation and even image shape. We believe there are more interesting inductive biases hidden within the data, and hence we hope these interesting phenomenons can encourage further researches into the underlying reasons.We introduce the details of GPT-4V judger in this section. We provide the system prompt (for GPT-4-vision model API) and example input and output for each.System prompt:You are a judger of image retrieval systems whose job is to identify the system that retrieve better images from a shared data base. You will be given a user query, and an image containing 10 smaller images. In the provided image, there will be 2 rows and each row will have 5 images, representing top 5 retrieval images from 2 systems. We note that the first row is the result from <system 1> and the second row is the result from <system 2>. Your judgement will be from In w/ OC setting, we take another call that exchanges line 1 and 2 of the input images.You are a judger of image retrieval systems whose job is to evaluate the system by providing a score for its retrieval results of the given query. You will be given a user query, and an image containing 5 smaller images, representing top 5 retrieval images from a system. Your judgement will be from 3 perspective: Accuracy, Aesthetic and Diversity. You should provide a 1-5 score for each of them.Here is your scoring standard: 1 point: None of the result is satisfying, or even totally wrong. 2 point: more than 1 results are not acceptable, and the rest just make sense.3 point: all results make sense, but only 1 or 2 are satisfying. 4 point: most of the results are satisfying, but 1 or 2 of them are not so outstanding or just make sense. { "Accuracy analyze": "The user …… an average score.", "Accuracy score": 3, "Aesthetic analyze": "Considering the aesthetics, the images generally have a soft, warm, and intimate feel which suits the tender aspect of the query. The lighting and composition in the second and third images are particularly elegant, enhancing the tender emotion of the scenes. The first and fifth images, while less compelling, still maintain a decent level of aesthetic appeal. The fourth image, however, lacks the aesthetic alignment with the query's theme due to its subject being a single person rather than a couple. Therefore, the aesthetic quality is mostly high with a slight deduction for the less relevant image.", "Aesthetic score": 4, "Diversity analyze": "The …… improvement.", "Diversity score": 3 }Figure 25 : <scorer> requires two calls for two groups separately. We only leave the analysis of aesthetic aspect for visualization. In this example, the first group wins at accuracy and diversity aspects, and we will not take this sample into account when calculating aesthetic win-rate (as the two groups have equal scores).System prompt:You are a judger of image retrieval systems whose job is to evaluate the system by providing a score for its retrieval results of the user's query. You will be given a user query, and an image containing 10 smaller images. In the provided image, there will be 2 rows and each row will have 5 images, representing top 5 retrieval images from 2 systems. We note that the first row is the result from <system 1> and the second row is the result from <system 2>. Your judgement will be from 3 perspective: Accuracy, Aesthetic and Diversity. Your job is to give a score for both 2 systems from each of the above aspects.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This submission proposes a method to improve language-based image retrieval in line with human aesthetic preferences. The method includes (i) rephrasing queries to elicit better retrieved results; and (ii) collecting preference data and using them for an RLHF-like LLM alignment.

Reviewers found the work:
1. Generally well-written, illustrated and structured
2. Well-motivated
3. Novel and with important contributions to the community, including the RLHF-like method, the prompt-rephrasing approach, and the proposed dataset.

There were concerns about lack of clarity in some areas which the reviewers felt were sufficiently addressed, and some misunderstandings about user study results which were resolved. In addition there were ethics concerns wrt consent and privacy of subjects and labellers in the collected dataset which were also satisfactorily addressed. Ultimately all 4 reviewers recommend acceptance and the AC sees no reason to override the unanimous recommendation.

The authors are asked to include the recommendations of the reviewers (including the ethics reviewer) in the camera ready version.

Reviews: review: paper_summary: paper_summary: This paper aligns the vision models with human values by leveraging LLM for query rephrasing and introducing preference-based reinforcement learning. The paper also presents a novel dataset named HPIR to benchmark the alignment with human aesthetics.
strength_weakness: strength_weakness: [W1] This paper lacks a detailed user study to validate the actual effectiveness of the proposed method. Including a user study with different participants to evaluate the subjective improvement of aesthetic alignment could provide stronger evidence for the actual effectiveness of the method.
[W2] Placing the related work in Section 6 makes it difficult for readers to have a clear understanding of the problem domain and existing research results before reading the specific methods and experiments, which is not conducive to the coherence of the paper structure.
questions: questions: Computational Cost: Can you elaborate on the computational costs and resource requirements of the reinforcement learning-based fine-tuning process? Are there any optimizations that could reduce the computational burden?
User Study: Have you conducted any user studies to validate the improvements in aesthetic alignment from a user perspective? If not, do you plan to include such studies in future work?
limitations: limitations: Exploring possible negative social impacts, such as implications for privacy or how the technology might be misused in unintended ways Research could benefit from deeper analysis of how biases in training data affect model outputs beyond just aesthetics, particularly with respect to cultural and demographic diversity.
paper_summary: paper_summary: This work aims to align vision models with human aesthetic standards in a retrieval system. To do this, the authors propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. The authors further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics.
strength_weakness: strength_weakness: I feel it can be further improved in the following ways. 

1.	For benchmarking human preferences, it might be better to record down the human variance in their annotations. I understand the authors used multiple annotations to ensure robustness, but since aesthetics is a subjective concept, human variance itself tells something.
2.	Following point 1, I feel the work can be made more solid if it includes some human evaluation studies on the experimental results. For example, in Fig. 5, it does not seem so obvious to me on the respective enhancement with finetuning. 

Without the above two points, I feel the paper has somewhat overclaimed the "alighing vision models with human aesthetics".
questions: questions: 1.	Have the authors considered human variance in aesthetics perception?
2.	Are the objective metrics enough for results evaluation? Have the authors considered using human studies to evaluate the results?
limitations: limitations: The authors did not indicate limitations in their paper, and mentioned that they will discuss it in future. I feel this paper has clear limitations such as the indication of human variance and the evaluation of the results.
paper_summary: paper_summary: The paper looks into the alignment task for vision and language models within retrieval models where properties such as visual aesthetic comes to play. To achieve this, the paper collects some data to design a metric suitable for taking into account human aesthetic evaluation. And employs an RL-based technique to exploit the human opinion for better aligning the retrieved images with human aesthetic preferences.
strength_weakness: strength_weakness: * The proposed metric could be elaborated better and maybe explained how the study ensured the metric is not designed under influence of the model.
questions: questions: * The work has been focused on visual aesthetics, given the LLM rephrasing results, it may be beneficial to look into the sophistication of language parts and how that could correlate with the aesthetic of the image. Could that motivate higher level of language sophistication is also correlated with higher visual aesthetics?
* Could you elaborate how the RL-based approach could scale?
limitations: limitations: The paper does not discuss any limitations with respect to the proposed method.
paper_summary: paper_summary: This paper studies the problem of aligning vision models with human aesthetic standards in a retrieval system. There are three key parts in the proposed model including LLM rephrasing, re-ranking, and RL fine-tuning. Two novel benchmarks are also introduced to integrate aesthetic quality into evaluation metrics. Experimental results demonstrate the effectiveness of the proposed method and the benchmarks.
strength_weakness: strength_weakness: 1. The methodological process described in the article is somewhat cumbersome, with Figure 2 merely outlining key processes and concepts in a rudimentary manner, thereby increasing the difficulty for readers to comprehend.
2. The authors appear to conflate "no-reference image quality assessment" with "image aesthetic quality assessment." While these tasks are indeed closely related, they are distinct. MANIQA, for instance, should not be regarded as an aesthetic quality assessment model, and its paper does not evaluate the model's performance on aesthetic datasets.
3. There remain some details in the article that are inadequately explained. It is peculiar that in Appendix Table 7, the same stride seemingly yields a different number of images.
4. The manuscript contains typos. For example, the indicator function symbol in Equation 11 is clearly garbled.
questions: questions: 1. In line 62, it is said that the open-source IAA datasets cannot be used for aesthetic retrieval evaluation. Can you give a further explanation?
2. In the first step of data preprocessing, the authors use the concept of "topic". Can you give a further explanation?
limitations: limitations: The paper does not discuss the limitations.

Comments: comment: Thank you for your valuable comments and your recognition of our work! We will add the explanation in the rebuttal to the revised version.
Thanks to the author's response. Most of my concerns are addressed. As my current rating is already the highest, l will maintain the original score.
We appreciate your recognition of our work and thank you for your valuable comments. We commit to add the human variance and additional human evaluation in the revised version.
Thanks authors for the responses. I think this paper has potential to be published given the minor revisions done. I would like to keep the current borderline accept rating.
Thanks for the comments and suggestions! It is certain that all labelers (mostly university students) were informed of all the uses of their labels, and they agreed on the use. All experiments were conducted with open-source datasets and proprietary data for which we own the full copyright. We will add more detailed statements in our final version. 

We will also start a new section in appendix to discuss potential negative social impacts and potential risks from the RAI perspective, including all the points mentioned by the ethics reviewer:
- Diversity and quality trade-off: The LLM rephrasing stage may increase the quality while decrease the diversity, which generally does more good than harm, but in cases where more diversity is required, several simple adjustments can be made to adapt. For example, repeating the rephrasing may benefit both direct inference and training models. In addition, increasing the pre-training loss weight during fine-tuning may also be an option to balance the trade-off.
- Potential biases: LLMs are trained with massive internet-scale data and may therefore contain potential biases. This may reinforce the biases of the retrieval system. For example, when searching for the word 'nurse', the vector search system using the CLIP feature returns approximately 90% female images and 10% male images. However, when this system uses LLM rephrasing as preprocessing, it results in nearly 100% female images. This is because the rephrased sentence describes a female nurse. However, our approach also provides an opportunity to supervise and eliminate some biases. A vector search system using the CLIP feature is a black box, and we cannot predict or control when it may retrieve harmful content. By using LLM rephrasing, we can trace intermediate information and debug the system. We can further improve LLMs or implement a text-based filter.
- Specific RAI problems: As we mentioned, our algorithm can be adapted to other areas of RAI problems, such as nationality and race. While our approach may benefit many general cases, it may potentially ignore minorities, as aesthetics and ethics may differ across various cultures and regions. Specific adaptations should be made to both LLMs and vision models in accordance with the 'no free lunch' theorem.