UniAR: A Unified model for predicting human Attention and Responses on visual content

Authors: peizhao  li
junfeng  he
gang  li
rachit  bhargava
shaolei  shen
nachiappan  valliappan
youwei  liang
hongxiang  gu
venky  ramachandran
golnaz  farhadi
yang  li
kai j kohlhoff
vidhya  navalpakkam
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:01:54

Abstract: Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.

Main paper: 
 introduction: 
 Implicit, early-stage perceptual behavior such as human attention is intricately linked with explicit, later-stage behavior such as subjective ratings/preferences. Yet prior research has often studied these in isolation. For example, there is a large body of work on predictive models of human attention that are known to be useful for various applications, ranging from basic attention/eye-movement research [32, 35] , to optimizing interaction designs [4, 65, 9] , enhancing webpage layouts [67, 85, 11] , improving user experience in immersive environments [5] and improving natural image and photo quality by reducing visual distraction [1] . Prior research has also explored predicting other kinds of implicit human behavior such as the sequence/order in which items are viewed (attention scanpath) in natural images or webpages [22, 19] , assessing visual importance in graphic designs [47, 63, 27] , and understanding visual clutter [52, 79, 71] .Separately from implicit, early-perceptual behavior, there has also been research in modeling explicit, later-stage decision-making behavior such as subjective preferences [20] and aesthetic quality [37, 21, 55, 30] . Prior research has been further fragmented due to dedicated models focusing on specific combinations of behavior tasks, input domain (e.g., natural images, designs, and webpages), and task scenarios (e.g., free viewing, object searching, and question answering).Figure 1 : Overview of our UniAR model. UniAR is a multimodal model that takes an image (could be a natural image, screenshot of a webpage, graphic design, or UI) along with a text prompt as input, and outputs heatmaps of human attention/interaction, scanpath or sequence of viewing/interaction, and subjective preference/likes. Example inputs and corresponding outputs for saliency, scanpath, and rating are shown on the left side, and the detailed model architecture is shown on the right side.To the best of our knowledge, a unified approach is still missing to modeling human visual behavior, ranging from implicit, early-perceptual behavior of what draws human attention, to explicit, later-stage decision-making on subjective preferences or likes.In this paper, we ask the following question: Can we build a unified model of human attention and preference behavior that reliably works across diverse types of visual content? If so, how does it compare with state-of-the-art (SOTA) models dedicated to specific domains and tasks? Such a unified model could enable a wide variety of applications. For instance, it could augment human decision making and accelerate evaluation of effective UIs by not only predicting preferences as rewards, but also providing additional insights in the form of predicted human attention behavior. UniAR. In this paper, we consider 11 public datasets consisting of different input domains/visual content (e.g., natural images, cartoons, art, graphic designs and webpages), behavior tasks (e.g., attention heatmaps, scanpath, likes/preference), and task-scenarios (e.g., free-viewing, object search, question answering). We introduce a new model, UniAR -A Unified model for predicting human Attention and Responses on visual content. UniAR is a multimodal transformer model that takes images and text prompts as input. The text prompt combines information about the input domain (e.g., natural image, graphics design or webpage), the desired behavior prediction task (e.g., attention heatmap or aesthetic score), and specifics of the task scenario when relevant (e.g., object name in an object search task). Our model generates predictions conditionally on these inputs. Experiments show that UniAR achieves SOTA performance across diverse datasets, spanning different input domains, behavior prediction tasks, and task scenarios.Main contributions of this work are summarized below:1. We proposed UniAR, a multimodal transformer model to predict different types of human behavior from attention to likes, across diverse types of visual content.2. We trained UniAR on 11 benchmark datasets with different input domains (natural images, webpages, and graphic designs) and output behavior types (attention/importance heatmaps, viewing sequence or scanpath, and aesthetics/quality scores), and showed that UniAR, which is a single unified model, can outperform or perform comparably to SOTA models trained on specific tasks and datasets. We further showed that UniAR generalizes well to tasks with unseen input and output combinations, under a zero-shot setting.We present various visualization results from UniAR on saliency/importance heatmap, scanpath, and ratings in Figure 2 , compared to ground truth. 
related work: 
 Saliency prediction. Saliency or attention heatmap prediction is a common implicit behavioral task aimed at predicting which areas within an image are more likely to draw human attention. Saliency models can be helpful for understanding human visual attention, and have been used for applications such as evaluating the quality of UIs, optimizing content placement in graphic designs, and improving perceptual quality of compressed images/videos (i.e., allocating more resources to visually important regions, and compressing the rest can preserve information while reducing bandwidth).Early work focused on the importance of low-level image features in saliency prediction [32, 35, 36, 28, 42] . Recent approaches for saliency modeling use Convolutional Neural Networks (CNNs), Transformers, or a mixture of models [40, 41, 33, 51, 23, 25] as the backbone architecture to extract deep representations and predict the probability distribution over human gaze or fixations (computed as a Gaussian-blurred 2D heatmap, which aggregates all fixations from multiple human observers) [33, 61, 12, 42, 18] . Customized modules, such as 1×1 read-out convolutions [33] and Attentive ConvLSTM [18] , have been introduced atop these CNNs to boost performance. Instead of the heatmap, regressing the Gaussian probability distribution has also been demonstrated as an alternative way for fixation predictions [61] . Chen et al. [12] propose to incorporate user profile information to personalize saliency predictions for each individual user.Scanpath prediction. Unlike saliency, which predicts a heatmap/probability distribution over attention/importance, the goal here is to predict the sequence of eye movements as humans engage with visual content, offering insights into how individuals observe and comprehend visual information.With graphic designs as an example, predicting scanpaths can help optimize content placement, ensuring priority content captures attention first.Prior work on human scanpath prediction has explored different task-scenarios, such as free-viewing, object searching, and visual question answering. Yang et al. [77] introduce a method utilizing inverse reinforcement learning for scanpath prediction during visual object searching. Continuing with this framework, Yang et al. [78] propose the concept of Foveated Feature Maps, enabling scanpath prediction when users search for an object that is not present in the initial image. To facilitate instruction following when performing a visual search over the image, Chen et al. [15] propose the use of a visual-question-answering model combined with a ConvLSTM module to predict the distribution of fixation positions and duration in response to a question regarding the associated natural image. In recent work, Mondal et al. [54] propose employing the Transformer model to regress coordinates for each fixation within a scanpath dedicated to object searching. This regression is conditioned on the embedding of the object's name from a pretrained language model. Subjective rating prediction. Predicting explicit human responses such as subjective preference/likes can help to better assess image quality and improve graphic designs. These responses can be continuous or discrete ratings, and may reflect both technical quality and aesthetic quality of an image. Explicit human feedback has been used in many applications. Statistic-based methods [53, 83] and Convolutional Neural Networks-based methods [84, 68] are proposed, and recently Vision Transformer has also been adopted for this task [37] .Limitations of prior work. While there has been significant progress in modeling behavioral tasks such as saliency, scanpaths and subjective preferences, a key limitation is that prior approaches often focused on a dedicated model for each specific task x input domain. As a result, there are saliency models for natural images, scanpath prediction models for graphic designs, or subjective ratings/likes on webpages, but there isn't a single unified model that generalizes across different tasks and domains. Instead of several dedicated per-task or per-domain models, our work seeks to build a single, unified model for these human-centered prediction tasks across diverse visual content.Multi-tasking unified model for language & vision. There have been significant recent advances in large language models for natural language processing and vision-language learning [14, 57, 17, 3, 69, 62] . The underlying modeling recipe involves fine-tuning large transformer models on datasets containing a variety of recognition and reasoning tasks such as text summarization, sentiment analysis, machine translation for language models, and image captioning, question-answering, detection, and segmentation for vision-language models. These fine-tuned large models show strong generalization capacity across various tasks and data domains. Inspired by these generalizable models for language and vision, we propose UniAR -a unified model for predicting different types of human visual behavior (from attention to likes) on a variety of visual content. 
unifying human attention and responses: 
 Our model architecture along with example inputs and corresponding outputs is shown in Figure 1 .Inspired by the recent progress in large vision-language models [14, 50, 45] , We adopt a multimodal encoder-decoder transformer model to unify the various human behavior modeling tasks. The model takes two types of inputs: an image and a text prompt. Its architecture comprises of the following components: a Vision Transformer model [24] for image encoding, a word embedding layer to embed text tokens, and a T5 [60] Transformer encoder to fuse image and text representations. Additionally, it has three separate predictors: a heatmap predictor for attention/saliency heatmaps or visual importance heatmaps, a scanpath predictor for the sequence/order of viewing, and a rating predictor for quality/aesthetic scores of images or webpages. These predictors are described in Sections 3.2 to 3.4. Besides the architecture, the text prompt is designed to encode relevant information about the input domain (e.g., natural image, graphic design, webpages), the expected prediction type of the model (e.g., interaction heatmaps, sequence-of-viewing, or aesthetic score), and other task-related information such as viewing scenarios (e.g., free-viewing or object-searching), target object names, or questions to be answered, as described in Section 3. 5 . More details about model architecture including # of layers and layer size can be found in Appendix A.To pretrain the model, we use both natural images from the WebLI dataset [14] and Web/mobile UI images [50] , to ensure that the model can generalize to multiple domains. Image captioning and captioning for a screen region are used as the pretraining tasks, as in the original papers. To support sequence tasks involving prediction of gaze/interaction coordinates, such as scanpath prediction, we also add a pretraining task to predict the coordinates of the bounding box of relevant items given a text snippet and the screenshot (for webpage and mobile interface data).Our model incorporates a heatmap head which is commonly used in attention/saliency research (i.e., predicting probability distribution of gaze over the input image). The heatmap prediction head takes the fused image tokens after the Transformer encoder, and processes the features via several read-out convolution layers, together with up-sampling so that the output will match the resolution of the input image. A sigmoid function is used at the end to ensure the generated values fall within the range [0, 1] for each pixel.In the experiments, we consider two different types of heatmaps, namely saliency and importance heatmap. A saliency heatmap is generated by aggregating human eye fixations from multiple participants viewing an image. On the other hand, importance heatmaps are obtained by participants highlighting or drawing bounding boxes to indicate the most critical design elements in a graphic design [27] . Each of these heatmaps reflects distinct aspects of human attention and preference.The text prompt specifies which heatmap-type to generate for a given input sample, thereby allowing our model to predict a variety of heatmap prediction tasks (e.g., attention, interaction, importance etc.) using a single heatmap prediction head. We adopt a pixel-wise ℓ 2 loss function for the heatmap predictor during training.The scanpath predictor takes both the fused image and text tokens after the Transformer encoder as input, and applies a Transformer decoder to generate the predicted scanpath.A scanpath is defined as a sequence of 2D locations (x 1 , y 1 ), (x 2 , y 2 ), . . . , (x N , y N ) with a total of N fixations, capturing the temporal aspect of attention and visual exploration. The subsequent fixations are conditional on all the previous fixations, thus fitting an autoregressive model with conditional generation. Inspired by previous literature, we use Transformer decoder for object detection and other localization tasks [13, 50] , and therefore generate the location end-to-end with Transformer and text generation. In general, we let the Transformer predict the sequence of coordinates as characters in a string one after one and readout the locations from the generated text subsequently.We spatially decompose the entire image into 1,000 × 1,000 bins with equal interval, and map each coordinate x n or y n to its nearest bin xn , ỹn ∈ Z in the range [0, 999].To formulate the target sequence for teacher-forcing training, we put a special token '<extra_id_01>' at the start of each target sequence, and attach another special token '<extra_id_02>' at the end, to indicate the entire scanpath sequence. We concatenate location coordinates with a separation word The training objective is to maximize the log-likelihood of tokens conditioned on the input image and all preceding tokens in ground-truth scanpath string, i.e., max 3N +1 j=1 w j log P (ỹ j |x, y 1:j-1 ),where x is the input image and text prompt, and y is the target sequence associated with x. w j is the weight for the j-th token.We use a unified weight for each token in experiments.Decoding during inference. During inference, it is not strictly guaranteed that the generated string will exactly follow the format of the target sequence, especially when the number of fixations is relatively large (e.g., N ≥ 30), resulting in an invalid sequence to readout. To deal with the possible invalid cases, in practice, we identify two special tokens '<extra_id_01>' and '<extra_id_02>' in the predicted string if available, and extract the context between these two special tokens. Then we split the extracted string with the separator word 'and'. For a pair of two tokens from the beginning, we check if they are both numerical. If so, we add one fixation with this coordinate after mapping them to the original resolution, then iteratively move to the following, and if not, we terminate the decoding process and keep the existing sequence. If there is no fixation available in the predicted string, we mark the scanpath as invalid. During training, we observe that the valid rate (#valid scanpaths / #scanpaths generated) of scanpath decoding quickly converges to 1, meaning every predicted scanpath will contain valid fixation(s).Compared to the scanpath predictors in GazeFormer [54] which predicts each point as 2D continuous numbers instead of two text tokens, our model seems less intuitive. However, as shown in Section 4.3, the proposed scanpath predictor works quite well. Moreover, one advantage of the current design of the scanpath predictor is that it can be easily extended to predict other types of human behavior sequences, e.g., text sequence, or 1-D number sequence, with minor modifications on the sequence output format. This flexibility is important for a unified model like ours.This prediction head takes image tokens after the Transformer encoder module, and processes the features via a few convolution and connected layers. An ℓ 2 loss is used for training the rating predictor with rating data.To [21] . We also provide model results from some other papers for comparison [25, 33] . The baseline results for CAT2000 are from CAT2000 Leaderboard5 .Evaluation metrics. Inheriting from the above benchmarks, we consider the following evaluation metrics. CC [44] : Pearson's Correlation Coefficient is used here to measure the linear relationship in all pixel values between the predicted and ground-truth saliency heatmaps; KLD [39] : the metric to use KL-Divergence between the predicted heatmap and ground-truth heatmap to measure the distribution discrepancy, with the prediction used as the target distribution. AUC-Judd [35] : Area under ROC curve (AUC) in the variant from Judd et al. [35] treating the heatmap prediction as binary classification with various thresholds. The specific calculations of true positive and false positive rates can be referred to [10] . sAUC [7] : the shuffled AUC metric samples negatives from other images for AUC calculation. NSS [59] : Normalized Scanpath Saliency is the average saliency strength (pixel values in the predicted heatmap) at all ground-truth fixation locations. SIM [64] : Similarity is computed as the sum of the minimum values among the normalized prediction and ground-truth heatmaps. RMSE: the root mean square error between the predicted and ground-truth heatmaps. R-Squared (R 2 ): the coefficient of determination applied to all values in the heatmap. SemSS [78] : Semantic Sequence Score converts each fixation to an ID decided by a semantic segmentation over the image, and compares two strings with a string-matching algorithm [56] . SemFED [78] : similar to SemSS, Semantic Fixation Edit Distance uses Levenshtein distance for string matching [48] .SequenceScore: similar to SemSS, but instead of using a semantic segmentation map, the Sequence Score uses the clustering results from a MeanShift clustering to segment the image and map the fixation to their ID. MultiMatch [22] : MultiMatch is the average of four metrics of scanpath, namely: Shape, Direction, Length, and Position, characterizing the similarity between the predicted scanpath and its ground-truth. SRCC and PLCC: refer to Spearman's rank correlation coefficient and Pearson's linear correlation coefficient, respectively, used to quantify the quality of predicted ratings.Experimental benchmarks for one task among different datasets may not have uniform evaluation metrics but most of their metrics are shared. For saliency and importance heatmap predictions, we resize the predicted heatmap back to its original image resolution for evaluation. 
model training: 
 We pretrain the model on a series of pre-training tasks, including Web/Mobile UI understanding [50] and natural image captioning [14] . Subsequently, we fine-tune the entire model using the Adafactor optimizer with a learning rate of 0. Datasets mixture. As we are combining a series of public datasets, in every iteration, for all training datasets in Table 1 , we employ a random sampling strategy that ensures an equal sampling rate across these datasets. This approach guarantees that each dataset has an equal probability of contributing a sample, irrespective of its sample volume.We present the results of UniAR for predicting heatmaps, scanpath-sequences as well as ratings across domains and datasets in Tables 2 to 4, in comparison with the baselines that are trained on a specific domain, task, or dataset.Heatmap prediction. Table 3 shows the performance of UniAR across 7 public benchmarks. A complete version of results including all baselines and metrics is presented in UniAR achieves the best result in 17 of them and the second best in 6 cases. In summary, UniAR experimentally demonstrates promising performance in saliency modeling in various fields.Scanpath prediction. In Table 4 , scanpath-sequence prediction results are shown for two datasets: COCO-Search18 [16] (scanpath in natural images for object searching) and WS-Scanpath [11] (scanpath on webpages under free viewing). On both the datasets, UniAR performs comparably to baselines, and further outperforms the baselines on all the metrics on WS-Scanpath. Among the 5 reported metrics in Table 4 , our model achieved the best result in 4 of them. A complete version of results including all baselines and metrics is presented in Table 7 in Appendix C.In Table 2 , we present rating prediction results on two datasets: KonIQ-10k [31] on natural images and Web Aesthetics [21] on webpages. UniAR achieves the best results for PLCC metrics on both datasets and the second best for SRCC on KonIQ-10k. Note that in Ke et al. [37] , a multi-scale version of MUSIQ performs slightly better than UniAR on SRCC (0.916 vs 0.905). However, since UniAR does not use multi-scale inputs, we did not include those results. 3) Employing both scanpath and saliency heatmap from natural image (COCO-Freeview), augmented with saliency heatmap from webpage (WS). Each scenario maintains some relevance to our test set by either sharing the same task or image domain, but never both.In Table 5 , we show experimental results for the above scenarios, and also attach the baseline results on this test set and the results from UniAR (full training data) as reference. As shown in Table 5 , our third training scenario: leveraging scanpath and saliency heatmaps from COCO-Freeview and saliency heatmap from WS, shows good results against previous SoTA [11] , despite the model not having seen webpage scanpath data during training. Prediction performance declines in scenarios 1 & 2, which take more limited datasets, but remain competitive.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: See experiments details.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
limitations and future work: 
 When modeling human preferences and behavior, it is important to carefully consider ethics and AI principles, and acknowledge dataset limitations.Ethics and AI principles. Modeling any aspect of human behavior should adhere to ethical guidelines on data collection and applications, and be conducted in a transparent way, including clarifying the limitations of the model when replicating human preferences. It should keep humans in the loop, as the model prediction is intended as a reference, not as a means to replace evolving human preferences with a synthetic guide. We take these ethical considerations and AI Principles into account, ensuring that the model usage remains socially beneficial and responsible.Aligning with human preference. While UniAR is trained to predict human preferences and behavior, we recognize that using it as a reward model may lead to reward hacking, which would make it less representative of genuine human preference. We suggest considering techniques [46, 26] to mitigate this.Representing diverse human preferences. Humans have diverse preferences for subjective notions like image attractiveness. Without personalization, the model converges towards a more uniform notion of preference -a common concern for ML models. To promote visual diversity when using UniAR, we propose two strategies: (1) using the model in a hybrid manner, providing insights to help humans make decisions in applications like web or visual content optimization, and (2) develop personalized models based on our initial unified model, which will help generate more diverse predictions based on user attributes. Adjusting to evolving human preferences. Human preferences evolve over time, and to remain accurate, the model has to adjust accordingly. Updates can be achieved by fine-tuning with more recent data. Future work will include updating the training data, and exploring concepts like continual learning techniques [72] , to keep the model up to date. Dataset limitations. Using diverse, representative datasets is important to minimize potential biases in the model. In this paper, we focus on a proof-of-concept for unified modeling of human attention/preference behavior, based on existing, publicly available datasets. Below is a listing of annotator demographics, as described in the original papers.1. WS-saliency [11] : "A total of 41 participants (19 females, 22 males; age range 17-23; with normal or corrected-to-normal vision) participated in our data collection." 2. Mobile UI [47] : "Thirty participants (12 male, 18 female). [...] The average age was 25.9 (SD=3.95). The participants had normal vision (8) or corrected-to-normal-vision (22) . Twenty of the 22 wore glasses and the remaining two wore contact lenses." 3. Imp1k [27] : "The data of 43 participants (29 male, most in their 20s and 30s) were used in the resulting analyses." 4. FiWI [67] : "11 students (4 males and 7 females) in the age range of 21 to 25 participated in data collection. All participants had normal vision or corrective visual apparatus."Despite some balance in male vs. female participants, the age distribution is skewed towards participants in their 20s. This is likely because most data were collected at universities. Crowdsourced datasets like Salicon and Koniq-10K are expected to cover a wider range of age and other attributes. Future work will focus on gathering a more diverse and representative dataset.Improve accessibility. UniAR predicts human attention, trained on datasets collected from humans not experiencing visual impairments beyond corrective lenses. UniAR cannot model behavior of, for example, blind and low-vision users directly, but it can still benefit them by acting as an accessibility tool for highlighting important areas of a webpage for a screen reader. One way to enhance the accessibility of the model is using multi-modal preference modeling, which incorporates not just visual cues, but also how users interact with content through screen readers, voice commands, and other assistive technologies. Collaborating with accessibility experts and organizations could help improve future iterations of our work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [NA] Justification: Not appliable for now.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We developed a multimodal, unified model UniAR to predict different types of implicit and explicit human behavior on visual content, from attention to subjective preferences/likes, using image and text prompts. This model, trained on diverse public datasets across natural images, graphic designs, webpages and UIs, effectively predicted human attention heatmaps, scanpath sequences, and aesthetic or quality scores. Our model achieved SOTA performance across multiple benchmarks and tasks. We plan to explore more behavior tasks and domains in future work. Table 6 : The full table for heatmap prediction results on 7 public datasets spanning digital design and natural images, benchmarking with eight metrics in total. The details on datasets and evaluation metrics can be found in Section 4.1. For Imp1k dataset we predict the importance heatmap, while for the rest of the four datasets, we predict attention/saliency heatmap. Our method is highlighted with green background . For each dataset and each metric, the best result in the current column is in bold, and the second best result is in blue. For our model, the relative performance change compared to the second best result (or the best result if we are not the best) in % is noted. We present full tables of UniAR's performance on heatmap and scanpath predictions in Tables 6 and 7 , with more baselines and a complete set of evaluation metrics. UniAR offers consistently good predictions on three tasks across multiple datasets, compared to the ground-truths.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 4.1Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow the existing references for experiment results on benchmark datasets, where only average metric number, but no error bar, are reported. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Section 4.1Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are all publicly available data with proper license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: No new human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Not applicable.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: • The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.There are two versions of Salicon data: Salicon 2015 and Salicon 2017. The results are on Salicon 2017.CAT2000 Leaderboard: https://saliency.tuebingen.ai/results_CAT2000.html 


Decision: The paper received 2 Borderline Accept, 1 Accept and 1 Reject ratings. While the authors could incorporate more experimental validations, the paper is well motivated and makes a solid contribution in unifying attention prediction. The AC reads the paper, the rebuttal, and the discussions and decides to accept it. Congratulations! It is encouraged that the authors incorporate reviewers' comments into their final version.

Reviews: review: paper_summary: paper_summary: In their paper "ALOHA: from Attention to Likes – a unified mOdel for understanding HumAn responses to diverse visual content" the authors describe a new unified model to predict human saliency(attention/importance), even more fine grained than that, scanpath, and ratings.

After nicely introduced the motivation and an extensive outlook on related work, the authors describe the multimodal VLM-based encoder-decoder transformer architecture in detail, especially the three predictor head. Exploting the power of instruction tuned LLM, the authors introduce additional tokens to  predict valid scanpath at inference time. 

The benefits of their model are evlauted based on many experiments, which are shortly discussed, followed by a short conclusion.
strength_weakness: strength_weakness: - The writing can be improved in several places, I'll note here a few:
1. Introduction: 
- In the introduction, it seems that the authors already have the conclusion at the end, switching from present to past tense for the main contributions of the paper. This makes it sounds as if the authors already published ALOHA previously. 
- In the same two points of the main contributions, the authors actually only have one contribution, their model ALOHA. The second "contribution" is only the evaluation of ALOHA.
- regarding the previous comment -- the second contribution, even though from my point of view it is no contribution is the evaluation of ALOHA and not the training of ALOHA.

3. Unifying Human Visual Behavior Modeling from Attention to Likes:
- The overall optimization criterion should be part of the main paper, not in the appendix.
- Model training (section 4) from my pov should be part of section 3, it is not part of the experiments since you do not report multiple training strategies.

Its unclear how hyperparameters were tuned.
Code will not be made public, which is a major concern in the current reproducibility crisis in ML. Additionally the question in the questionnaire is wrongfully answered as N/A; the paper DOES INCLUDE experiments requiring code. (GUIDELINE: The answer NA means that paper does not include experiments requiring code.)
questions: questions: You mention that you pad images to 512x512 -- what happens with images larger than 512x512?

How did you arrive at your optimization criterion, scaling for the learning rates and in general all hyperparameters? You did not report any hyperparameter tuning.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces ALOHA, a multimodal model that predicts human saliency, scanpath, and subjective rating of natural images, webpages, and graphic designs. ALOHA outperforms or performs similarly to baseline models across each of its prediction tasks while improving generalizability over task-specific models.
strength_weakness: strength_weakness: My biggest concern is the paper’s lack of engagement with the limitations and ethical considerations of modeling subjective human preferences. Currently there is a brief discussion of training data limitations in the supplement. I would like to see a thorough discussion of considerations included (or at least referenced) in the main text. Possible discussion points that come to mind include:

* What are the ethical considerations of replacing humans with models trained to replicate their preferences? Humans have diverse and often opposing preferences, particularly for subjective notions of aesthetics or attention. It is possible that models trained to replicate human preferences could learn a more uniform or singular notion of preference. Could using these models limit the visual diversity in webpages and content creation? Or could these models actually help us generate more visually diverse content?
* Whose preferences are included in the model and, more importantly, whose aren’t? Humans likely have diverse preferences when it comes to subjective notions like website attractiveness. I would suspect these preferences vary based on age, experience with technology, culture, etc. Building a model to replicate human perspectives may amplify a particular worldview that is not representative of all users. On the other hand, it may be easier to create a model trained on diverse human data than to get a diverse set of human feedback for every new webpage design.
* The paper suggests using the model as a reward function to optimize content creation. Could this result in negative consequences, such as content that is optimized for a model’s preference function and may not be value aligned with what humans want?
* How could human preference models, like ALOHA, be misused? For example, it seems like these models could be used to make ads more intrusive by optimizing their scanpath location or to make phising sites more convincing by optimizing the saliency of the credential input.
* How should human preference models take into account blind or low-vision users? Scanpaths and saliency likely differ between users who rely on screenreaders and those who don’t. Optimizing the web for sighted preferences could further exclude blind and low-vision from online spaces. Given the [emphasis](https://www.ada.gov/resources/web-guidance/) on web accessibility, how can we use human preference models to amplify (not weaken) online accessibility?
* How should models stay up to date with the changing human preferences? Human preferences, especially around aesthetics, are constantly changing. As a result, what are the implications for interpreting the evaluations used in this paper that span datasets from 2014–2023? 

The paper would also be strengthened by an audit of their model’s behavior beyond quantitative performance metrics, such as an analysis of the diversity of the humans included in the training dataset, a categorization of the types of mistakes the model makes, and a [Model Card](https://arxiv.org/abs/1810.03993) reflecting its design. 

I recognize that the paper is inheriting these ethical concerns from the preexisting datasets and tasks that it works on. I hope that by including a discussion of the models implications, the paper will progress research into these types of models and inspire work on new datasets, model analysis, and safe deployment techniques.

I might also suggest that the authors should select a new name for their model. I am not an expert in Hawaiian culture, but I know that the term ‘aloha’ is culturally significant, and there have been [Hawaiian movements to reclaim it](https://www.usatoday.com/story/life/health-wellness/2023/01/13/stop-saying-aloha-out-of-context/10990192002/), such as the [‘Aloha Not For Sale’ protests](https://kawaiola.news/cover/aloha-not-for-sale-cultural-in-appropriation/). Using it out of context as a model name, may minimize the rich cultural meaning behind the term, and given it is an imperfect acronym anyway, should be an easy change.
questions: questions: Please see the Weaknesses section above.
limitations: limitations: Please see the Weaknesses section above.
paper_summary: paper_summary: Noticing the existing issues in human behavior modeling, such as isolating the study of implicit, early-stage perceptual behavior (like human attention) from explicit, later-stage behavior (like subjective preferences), specified visual content type; in this manuscript, the author(s) aimed to build an integrated human attention and preference behavior model for addressing multiple visual content types. Through empirical experiments, the author(s) demonstrated the effectiveness of the proposed model.
strength_weakness: strength_weakness: In my opinion, the weaknesses of this manuscript are as follows:

1. This manuscript seems not to discuss the model's limitations in real-world situations, such as how its performance is in dynamic environments or its adaptability to changes in user behavior over time.

2. As the author(s) described, ALOHA has 848 million parameters. Training such a large model requires significant computational resources, which may not be accessible to most researchers or practitioners.
questions: questions: I read the manuscript, and I have the following questions/comments. Thanks.

1. Are there any potential applications of ALOHA in real-world scenarios or in dynamic environments? How can it be used to optimize elements, such as user interfaces, graphic designs, and content creation?

2. Are there any potential biases in the current model's predictions?

Tiny format issues in References:

(1) Sometimes, the journal/conference name used an abbreviation; sometimes, not, sometimes, both, such as Ref.[25], Ref.[67].

(2) Keep the format of the references consistent, such as Ref. [6] vs. Ref.[15].

Please check carefully and correct the issues.


Overall, ALOHA represents a significant advancement in modeling human visual behavior, offering a unified approach that spans from early-stage perceptual responses to later-stage decision-making. I think this is an interesting manuscript.

I look forward to hearing from the author(s). Thanks.
limitations: limitations: Yes.
paper_summary: paper_summary: In this paper, the authors introduce a novel text-image framework designed to integrate various human-response tasks and multiple image domains. These tasks include attention map generation, scanpath prediction, and subjective preference evaluation, applied to images such as webpages, natural scenes, graphic designs, and mobile user interfaces. The framework utilizes a transformer architecture, accepting both text and image inputs. Text inputs specify the tasks, while the model itself comprises a transformer encoder and three predictors (heatmap, rating, and scanpath) to generate 3 types of outputs. Initially pretrained on foundational text-image tasks, the model is fine-tuned for diverse human-response tasks
strength_weakness: strength_weakness: 1.	I have many concerns on the experimental results of this paper, where the authors try to demonstrate the superiority of their model performance by comparing with other methods. My concerns are as follows. 

a)    The metrics in Table 3 are not a standard set of metrics for saliency prediction. Why don’t the authors report the complete of evaluation metrics in Table 3? It seems strange to drop the popular metrics such as sAUC and SIM in Table 3. It is a standard practice to report all standard metrics, ie., those commonly used in saliency predictions, as in the online benchmark (https://saliency.tuebingen.ai/evaluation.html). It seems odd to me the authors only report selected metrics in Table 3 but report all in Table 6 (which is in the appendix). 

b)	Following a), why do the authors put both Table 3 and Table 6? Both of the tables occupy the same space. The authors can just put Table 6 in the main manuscript instead of Table 3. 

c)	Why do the comparison methods vary greatly across different testing datasets? It will be helpful if the authors stick to several methods and compare their performance across the same testing datasets. Deep Gaze II and IIE are recognized for their strong performance on images of general scenes. Why did the authors choose to test these models only on the CAT2000 dataset and not on other general scene datasets such as Salicon and OSIE?

d)	In the SalGAN paper [60], it reports the metrics on Salicon dataset, why the authors do not report this in Table 3 or Table 6 for SalGAN on Salicon dataset, but instead, only report its performance on the webpage dataset? If the authors have simply copied results from other works, incorporating these additional benchmarks would require minimal effort and should have been included. If the authors have tested the models, why not test them on the popular benchmarks such as Salicon and OSIE?

e) Can the authors explain why they don’t report SequenceScore for COCO-Search18? This column is denoted as "-" in Table 4. 

f) Furthermore, the reported results on COCO-Search18 of the comparison method FFM[79] in its own paper [79] are different from what what authors report in Table 4, both under target present conditions. The SemSS scores in [79] under target present conditions are consistenly above 0.53, and the method of Chen et al. [16] is 0.572. Can authors provide more details on how they obtained the current results in Table 4 for the comparison methods? 

g) Can authors explain why they didn’t compare scanpath prediction performance of ALOHA and baselines on COCO-FreeView in Table 4 even though they use COCO-FreeView in Sec. 4.4?

h)	It will be helpful if the authors can experiment with more popular benchmark datasets such as MIT300 and MIT1003, and compare with the performances published on the online benchmark (https://saliency.tuebingen.ai/results.html).

i)     Minor: some top performances are wrongly indicated, eg., on the webpage dataset (FiWI), the best performance on KLD should be Chen et al. [13].


In summary, excluding the SOTA models in common benchmark datasets significantly undermines the experimental evidence for ALOHA's competitiveness, especially considering that 7 out of the 11 datasets in the experimental section are for free-viewing saliency prediction (see Table 1). Based on all the above, I feel that current experiments cannot adequately supported the paper's claims.The paper needs more discussions and comparison results to support its claim that the model achieves SOTA performance. 

2.	Although the authors claim that the unified model could “act as a comprehensive reward model, predicting subjective ratings/preferences as rewards and providing additional insights from predicted human attention and behavior patterns,” the experiments do not clearly demonstrate the actual benefits of unifying the attention, rating, and scanpath tasks. The motiviation of this work is not well positionied. 

3.	Some technical details of the paper are not clear. E.g, a). the three predictors appear to operate independently, but it is unclear how they were chosen. b) Why text generation is chosen for scanpath generation. c) How is the model fine-tuned on various datasets.
questions: questions: 1. How are the three predictors chosen for each task? Are they fixed assigned based on different input texts?
2. Are there any differences between the salience map and the importance map in terms of architecture or loss function, or do the differences lie solely in the training data?
3. In the "decoding during inference" section, the paper mentions, “If there is no fixation available in the predicted string, we mark the scanpath as invalid.” This invalid case (“no fixation”) only covers situations where all token pairs are non-numerical. If some tokens in the output sequence are in an unrecognizable format, rendering part of the results invalid, how should this be evaluated and handled?
4. Why is text generation used as a scanpath predictor?
5. How is the model fine-tuned on various datasets, which parts are freezed?
6. All my above questions regarding the numerical results (Table 3 and Table 6) in the Weakness section.
limitations: limitations: The reported numerical results do not adequatly support the paper's claim.

More experiments to demonstrate the model's task transferring ability are needed. For example, transferring from saliency map to rating, or from scanpath to saliency map.

Comments: title: Thanks for the great feedback!
Response
Reply
comment: We would like to thank Reviewer Sv28 for the valuable feedback and suggestions. As mentioned in our response to Reviewer FgZM, we will significantly expand the relevant section to discuss the societal impact of this technique and propose strategies to mitigate any potential negative effects.

**Positive side**: In this paper, we proposed a unified model to predict both early-stage perceptual behaviors, such as human attention, and later-stage behaviors, such as subjective preferences and likes. This model offers several avenues for positive societal impact, including:
1. Ensuring visual content is better aligned with human perception and preferences, thereby enhancing user experience.
2. Training the model with a diverse set of human feedback, particularly from underrepresented groups, to ensure that visual content quality is optimized for individuals from various backgrounds. Achieving this goal is significantly more challenging without a unified model, as it would require training multiple individual models and evaluating them on new content for each group, which is not scalable.
3. Assisting individuals with blindness or low vision by enabling them to parse visual content more effectively. The model can highlight the most important areas of a webpage via heatmap predictions, making these areas more accessible through screen readers.

**Risk**: However, as with other research in the area of human behavior and preferences, we acknowledge that this technique could potentially have negative societal impacts, especially if misused, as highlighted by Reviewer FgZM. These potential risks include:
1. The possibility of making harmful websites more convincing.
2. Bias within the model could lead to unfair outcomes.
3. Over-reliance on the model, with the exclusion of human oversight, could result in issues such as reward hacking or other failures to align content with genuine human preferences.

**Our actions**: As mentioned in our response to Reviewer FgZM, we will incorporate a discussion on societal impact and other relevant topics (e.g., model card, limitations of current models) into the paper. Additionally, we plan to conduct follow-up research on issues such as model bias and fairness, and adhere to established guidelines on the societal impact of human behavior and preference models. We will also enforce both internal and external guidelines for the use of this technique, such as preventing unauthorized access and misuse, and ensuring human oversight rather than relying solely on the model. These measures will help our technique realize its potential for positive societal impact while mitigating potential negative effects.
Thank you for your prompt reply, we really appreciate it! We would love to have another round of response for some clarifications, and hope those help!

---

**Table 3 & Table 6**

We agree that there are some duplications between these two tables. Our intention in including Table 3 was primarily for presentational purposes.

We are happy to replace Table 3 with Table 6 in the main paper to include metrics sAUC and SIM, or the full set of eight metrics.

---

**Scanpath results for COCO-FreeView**

As one of the dataset we used for the experiment, we have the results from our model on COCO-FreeView. We would welcome the opportunity to compare our results with established baselines. If you could kindly direct us to a publicly available benchmark for scanpaths on the COCO-FreeView dataset, we would be glad to include a comparison with these baselines in our paper.

---

**SemSS score from FFM [79]**

Regarding the discrepancy in results, we have taken steps to clarify the issue.

We contacted the first author of FFM [79] directly to discuss the mismatch. The author confirmed that the results of this experiment in FFM [79] are indeed problematic due to incorrect labels. With the author's permission, we are willing to share their response for your reference, which validates our approach of referring to the updated results in Gazeformer [56].

> The SemSS of FFM in the two papers (GazeFormer and FFM) are different, because there was a mistake in generating the semantic labels in the FFM paper. Labels got mismatched in image size, which leads to systematic shift in SemSS for all methods (namely, all methods got higher SemSS scores, but the order maintains). We should refer to Table 6 (b) in GazeFormer as the more reliable SemSS scores for GazeFormer, FFM, and Chen et al.

Hope this clarifies the question.

---

**Unifying attention, rating, and scanpath**

1. Scanpath is naturally correlated with attention map: an attention map is aggregated from scanpaths from multiple users. In this way, rating, attention, and scanpath could be unified via a prediction model. We do have scanpath data, however, we will be cautious about releasing the scanpath data, due to privacy concerns.
2. We are happy that you agree the observed correlation between score and heatmap entropy is expected. This verified evidence further motivates the unified model, which is the foundation for studying the correlation between the heatmap and score for better performance. There are many other cases where subjective ratings/experiences and human attention (heatmap) are related, e.g., “Salient-Centeredness and Saliency Size in Computational Aesthetics”, ACM Transactions on Applied Perception 2023; “Aesthetic Attention”, https://philpapers.org/archive/NANAA-2.pdf. So a unified model to predict subjective ratings/experiences and human attention/scanpath will help advance the research in this direction.
3. We observed a p-value around 7e-7 for the correlation between easiness score and heatmap entropy, indicating the correlation between easiness score and gaze heatmap entropy is statistically significant.

---

We hope our clarification and the follow-up response will help us better understand our work. A reconsideration of our work is highly appreciated.
Thank you for helping us improve our paper and increasing your rating!

> discussing limitations in the demographic information that is not reported

That's a very good idea and thanks for your kind reminder! We will make sure to involve this part in our final discussion on demographic information. By quoting all the available information we have, we will also discuss the missing information in demographics, which could be useful for future reference.

> include all of the limitations you have mentioned in this rebuttal

For sure, we will add these thoughts in our rebuttal to the final version of the paper with a clear structure.

We appreciate your effort in reviewing this paper and we really enjoy talking to you.
Thank you to the authors for the rebuttal. However, I remain unconvinced by the explanations provided. As a researcher in the saliency field, I consider sAUC and SIM to be essential metrics for evaluating saliency prediction, yet they are notably absent from Table 3. This omission challenges the authors' assertion that Table 3 includes "the important metrics." 

Furthermore, in a scientific research paper, it is crucial to include key metrics and provide comprehensive evaluation, which should take precedence over concerns about font size. Therefore, I find the decision to place detailed information in Table 6 in the supplementary material, while presenting only partial and duplicate information in Table 3 in the main paper, to be unjustified, especially when both tables occupy the same length. The authors' reasoning to make Table 3 "more readable with a larger font size" seems insufficient and unconvincing.

Similarly, the explanation that the authors "did not find a good baseline" does not sufficiently justify the exclusion of scanpath results for COCO-FreeView, particularly given that COCO-FreeView is used in Section 4.4. Moreover, the authors did not adequately address the question regarding the SemSS scores in [79], where scores under target-present conditions are consistently above 0.53, and the method of Chen et al. [16] achieves 0.572.

The examples provided to demonstrate the benefits of unifying attention, rating, and scanpath tasks are rather weak and not sufficiently convincing. First, the examples involve only attention and rating, without scanpath, which undermines the claim of presenting a truly unifying example. Second, the observed correlation between score and heatmap entropy is expected, as it is natural for more challenging tasks to result in more complex browsing patterns. Additionally, the correlation observed is weak by statistical standards.

In view of the above, I would like to keep my rating as reject.
Thank you for taking the time to think deeply about the implications of this research! I appreciate your diligence in finding demographic information and doing error analysis, your thoughtfulness in responding to each of my concerns, and your commitment to include a model card and these considerations in the paper.

In addition to what you have listed in "New content to include", I would suggest:
* Beyond discussion limitations of the demographic information that exists, I suggest also discussing limitations in the demographic information that is not reported. From your demographic analysis, we only know the annotators age and gender. However, we do not know their location, race, country of origin, education level, etc. --- all of which contribute to someone's worldview and could change the way they interact with a webpage. 
* I would like to your discussion section include all of the limitations you have mentioned in this rebuttal --- a uniform model for diverse human preferences, reward hacking, safety measures for human preference models, making these models accessibility friendly, etc --- not just the demographic info, error analysis, and model card.

I appreciate your suggestions for new names. I think all the ones you suggested are great, so I will leave the name decision up to you.

Assuming these changes are reflected in the final manuscript, I am happy to accept the paper and have increased my score accordingly.