UniAR: A Unified model for predicting human Attention and Responses on visual content
Authors: Peizhao Li
Junfeng He
Gang Li
Rachit Bhargava
Shaolei Shen
NACHIAPPAN VALLIAPPAN
Youwei Liang
Hongxiang Gu
Venky Ramachandran
Golnaz farhadi
Yang Li
Kai J Kohlhoff
Vidhya Navalpakkam
Venue: NeurIPS 2024 poster (2024)
Keywords: attention
saliency
scanpath
aesthetics
Primary area: neuroscience_and_cognitive_science

Abstract: Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.

UniAR: A Unified model for predicting human Attention and Responses on visual content
Peizhao Li t1,, Junfeng He* 1 Gang Li Rachit Bhargava', Shaolei Shen', Nachiappan Valliappan', Youwei Liang f13 Hongxiang Gu', Venky Ramachandran' Golnaz Farhadi', Yang Li!,
Kai J Kohlhoff', and Vidhya Navalpakkam'
Google Research
Brandeis University
University of California San Diego
Abstract
Progress in human behavior modeling involves understanding both implicit, early- stage perceptual behavior, such as human attention, and explicit, later-stage behav- ior, such subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior isolation; and often limited to specific type of visual content. We propose UniAR unified model of human attention and preference behavior across diverse visual content. UniAR leverages multimodal transformer to predict subjective feedback, such satisfac- tion or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images. webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.
Introduction
Implicit, early-stage perceptual behavior such as human attention is intricately linked with explicit, later-stage behavior such as subjective ratinns/preferences. Yet prior research has often studied these in isolation. For example, there is a large body of work on predictive models of human attention that are known to be useful for various applications, ranging from basic attentionneye-movementt research 33 35,, to optimizing interaction designs 44 65 enhancing webpage layouts [67 85 [1)
improving user experience in immersive environments (5) and improving natural image and photo quality by reducing visual distraction
[ Prior research has also explored predicting other kinds of implicit human behavior such as the sequence/order in which items are viewed (attention scanpath) natural images or webpages [22 19], assessing visual importance in graphic designs [47 (2])
and understanding visual clutter [52 79
Separately from implicit, early- parry-lerceptul behavior, there has also been research in modeling explicit, later-stage decision- makinn behavior such as subjective preferences (20) and aesthetic quality [7
21 [55] 30]. Prior research has been further fragmented due to dedicated models focusing on specific combinations of behavior tasks, input domain (e.g., natural images, designs, and webpages), and task scenarios (e.g., free viewing, object searching, and question answering).
"Co-first authors, equal technical contribution
Work done during internship at Google Research I "oorespoondig authors, equal leading contribution: heae:eenghheeeeeerrra/dgooooooe
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"ON L QUINI IV' GXUTY
natural seathetios soore
Vision Transformor
Text Emberdinngg
Ml teney heareap
scnepeth
Image Tokens
Text Tokens
Trinsformet Encoder
UniAR
Pused Tokens
Fused Text Tokens
Saliency
Scanpath
Rating
Heedmap Predictor
Rali y Predisko
Scampplh Predicto
1.90
465
6.69
522 and
462 308 and
Figure I: Overview of our UniAR model. UniAR is multimodal model that takes an image (could be a natural image, screenshot of webpage, graphic design, or UT) along with a text prompt as input, and outputs heatmaps of human attention/interaction, scanpath or sequence of viewing/interaction and subjective preference/likes. Example inputs and corresponding outputs for saliency, scanpath, and rating are shown on the left side, and the detailed model architecture is shown the right side.
To the best of our knowledge, unified approach is still missing to modeling human visual behavior, ranging from implicit, early-perce parly-perceptual behavior of what draws human attention, to explicit, later-stage decision-makin
subjective preferences likes.
Can we build a unified model of human attention and preference behavior that reliably works across diverse types of visual content? If so, how does it compare with state-of-the-art (SOTA) models dedicated to specific domains and tasks? Such a unified model could enable wide variety of applications. For instance, it could augment human decision making and accelerate evaluation of effective UIs by not only predicting preferences as rewards, but also providing additional insights in the form of predicted human attention behavior.
In this paper, we ask the following question:
UniAR. this paper, we consider 11 public datasets consisting of different input domains/visual content (e.g., natural images, cartoons, art, graphic designs and webpages), behavior tasks (e.g., attention heatmaps, scanpath, likes/preference). and task-scenarios (c.g., free-viewing, object search, question answering). We introduce new model, UniAR A Unified model for predicting human Attention and Responses on visual content. UniAR is a multimodal transformer model that takes images and text prompts as input. The prompt combines information about the input domain (e.g., natural image, graphics design or webpage), the desired behavior prediction task (e.g.. attention heatmap or aesthetic score), and specifics of the task scenario when relevant (e.g., object name in an object search task). Our model generates predictions conditionally on these inputs. Experiments show that UniAR achieves SOTA performance across diverse datasets, spanning different input domains, behavior prediction tasks, and task scenarios.
Main contributions of this work are summarized below:
1. We proposed UniAR, multimodal transformer model to predict different types of human behavior from attention to likes, across diverse types of visual content.
2. We trained UniAR on 11 benchmark datasets with different input domains (natural images, webpages, and graphic designs) and output behavior types Cattention/imporancce heatmaps, viewing sequence or scanpath, and aesthetics/quality scores), and showed that UniAR, which is a single unified model, can outperform or perform comparably to SOTA models trained specific tasks and datasets. We further showed that UniAR generalizes well to tasks with unseen input and output combinations, under zero-shot setting.
We present various visualization results from UniAR on saliencylimportace heatmap, scanpath, and ratings in Figure 2 compared to ground truth.
2 Related Work
Saliency prediction. Saliency or attention heatmap prediction is common implicit behavioral task aimed at predicting which areas within an image are more likely to draw human attention. Saliency models can be helpful for understanding human visual attention, and have been used for applications such as evaluating the quality of UIs, optimizing content placement in graphic designs, and improving

---

SEST JUS
LUCY FELL
LUCY FELL
LUCY FELL
Free-viewing
Free-viewing
Search Bowl
Search Bowl
CT 0.89; 0.82
GT-0.41; Pred-0.48
0.30 0,32
Figure 2: Examples of UniAR's predictions across different tasks/domains. Images in green border are ground-truth, while images in orange border are UniAR's 's predictions. First row: attention/saliency heatmap prediction on natural images (Salicon) and webpages (WS-Saliency). Second row: im- portance heatmap graphic designs (Implk), and saliency heatmap on Mobile UI. Third row: scanpath-sequence during free-viewing of webpages (WS-Scanpath) and object-searching within images (COCO-Searchly) Fourth row: preference/rating prediction for natural images (Koniq-I0k) and webpages ( (Web Aesthetics).
perceptual quality of compressed images/videos (i.e., allocating more resources to visually important regions, and compressing the rest can preserve information while reducing bandwidth).
Early work focused on the importance of low-level image features saliency prediction
32 35
36 28 42
Recent approaches for saliency modeling use Convolutional Neural Networks (CNNs), Transformers, or a mixture of models [40]
25 as the backbone architecture to extract deep representations and predict the probability distribution over human gaze or fixations (computed as a Gaussian-blurred 2D heatmap, which aggregates all fixations from multiple human [8]. Customized modules, such as 1x1 read-out convolutions 33 and
41
33
51
23
observers) [33 61
42
Attentive ConVLSTM
18 have been introduced atop these CNNs to boost performance. Instead of the heatmap. regressing the Gaussian probability distribution has also been demonstrated as an alternative way for fixation predictions [6]]. Chen et al. [[2] propose to incorporate user profile information to personalize saliency predictions for each individual user.
Scanpath prediction. Unlike saliency, which predicts a heatmap/probability distribution over attention/importancee the goal here is to predict the sequence of eye movements as humans engage with visual content, offering insights into how individuals observe and comprehend visual information. With graphic designs as an example, predicting scanpaths can help optimize content placement, ensuring priority content captures attention first.
Prior work on human scanpath prediction has explored different task-scenarios, such as free-viewing object searching, and visual question answering. Yang et al. [77] introduce method utilizing inverse reinforcement learning for scanpath prediction during visual object searching. Continuing with this framework, Yang al. (78) propose the concept of Foveated Feature Maps, enabling scanpath prediction when users search for an object that is not present in the initial image. To facilitate instruction following when performing a visual search over the image, Chen et al. [55] propose the use of a www.--uuussionnnweeiiie model combined with CSTVLSTM module to predict the
3

---

distribution of fixation positions and duration in response to question regarding the associated natural image. recent work, Mondal et al. [54] propose employing the Transformer model to regress coordinates for each fixation within a scanpath dedicated to object searching. This regression conditioned on the embedding of the object's name from a pretrained language model.
Subjective rating prediction. Predicting explicit human responses such as subjective prefer- ence/likes can help to better assess image quality and improve graphic designs. These responses can be continuous discrete ratings, and may reflect both technical quality and aesthetic quality of an image. Explicit human feedback has been used in many applications. Statistic-based meth- ods (53 83 and Convolutional Neural Networks-
methods [84] 68 are proposed, and recently Vision Transformer has also been adopted for this task (37]
Limitations of prior work. While there has been significant progress in modeling behavioral tasks such as saliency, scanpaths and subjective preferences, key limitation is that prior approaches often focused on a dedicated model for each specific task x input domain. As a result, there are saliency models for natural images, scanpath prediction models for graphic designs, subjective ratings/likes on webpages, but there isn't It a single unified model that generalizes across different tasks and domains. Instead of several dedicated per-task or per-domain models, our work seeks to build a single, unified model for these human- cuntn-eentered prediction tasks across diverse visual content.
Multi-tasking unified model for language & vision. There have been significant recent advances
in large language models for natural language processing and vision-language learning 14 57 3
The underlying modeling recipe involves fine-tuning large transformer models datasets containing a variety of recognition and reasoning tasks such as text summarization, sentiment analysis, machine translation for language models, and image captioning, question answering, detection, and segmentation for vision-language models. These fine-tuned large models show strong generalization capacity across various tasks and data domains. Inspired by these generalizable models for language and vision, we propose UniAR unified model for predicting different types of human visual behavior (from attention to likes) on a variety of visual content.
69 62].
guestion-answering
3
Unifying Human Attention and Responses
Our model architecture along with example inputs and corresponding outputs is shown in Figure] 1
3.1 Model Architecture
We adopt multimodal encoder-decoder transformer model to unify the various human behavior modeling tasks. The model takes two types of inputs: an image and a text prompt. Its architecture comprises of the following components: a Vision Transformer model (24) for image encoding, a word embedding layer to embed text tokens, and a T5 [60] Transformer encoder to fuse image and text representations. Additionally, it has three separate predictors: a heatmap predictor for attention/salieney heatmaps or visual importance heatmaps, a scanpath predictor for the sequence/order of viewing, and a rating predictor for quality/aesthetic scores of images or webpages. These predictors are described in Sections 3.2 to 3.44 Besides the architecture, the text prompt is designed to encode relevant information about the input domain (e.g., natural image, graphic design, webpages), the expected prediction type of the or aesthetic score), and other task-related
Inspired by the recent progress in large vision-language models [14 50 45),
model (e.g., interaction heatmaps, sequence-of-viewing information such as viewing scenarios (e.g., free-viewing or object-searching), target object names, or questions to be answered, as described in Section 3.5 More details about model architecture including # of layers and layer size can be found in Appendix
To pretrain the model, we use both natural images from the WebLI dataset [4] and Web/mobile UI images [50], to ensure that the model can generalize to multiple domains. Image captioning and captioning for a screen region are used as the pretraining tasks, as in the original papers. To support sequence tasks involving prediction of gaze/interaction coordinates, such as scanpath prediction, we also add a pretraining task to predict the coordinates of the bounding box of relevant items given a text snippet and the screenshot (for webpage and mobile interface data).
3.2 Heatmap Predictor
Our model incorporates a heatmap head which is commonly used in attention/saliency research (i.e., predicting probability distribution of gaze over the input image). The heatmap prediction head takes

---

Table 1: List of all public datasets used to train our model. "# Image' denotes the number of unique
images in the entire dataset. Note that for annotation 'scanpath," there are multiple scanpaths recorded from group of users associated with one image, so "# Training Sample' is much larger than '# Image." During training, we randomly sample from all training datasets with an equal sampling rate.
Dataset
Domain
Annotation
Viewing style
Image
Resolution
Training
Salicon 34
Natural images
Attention heutmup
Free
15,000
640 480
10,000
OSIE
Natural images
Attention heatmap
Free-viewing
700
800 600
500
CA72000
Natural mnagss/catooonskat....
Atteation heatmap
Pres-viewing
4000
1920 1080
2000
WS-Sallenry
[
webpage
Attention heatmap
Free-viewing
1,280 x
392
Mobile 0/ +7
Mobile user interface
Attcation heatmap
Free-viowing
193
Varied
154
Imp/k 22
Graphic design
Importance heatmap
N/A
Varied
798
WS WS--Sampack
11
webpage
Scanpath (sequence)
Free. viewing
450
1,280 x 720
5,448
FiW/ [67]
webpuge
Attention heatmap
Free-viewing
159
1,360 768
121
COCO-Search/8
16
Natural images
Scanpath (sequeace)
Objectscarchhng
3.101
1,680 1,050
21,622
Konig-10k [3]
Natural images
Subjective rating
NIA
10,073
1,280 x 720
7,000
Wesheeess
21
webpage
Suhjective matnng
1,280 x 768
398
the fused image tokens after the Transformer encoder, and processes the features via several read-out convolution layers, together with up pp-sampling that the output will match the resolution of the input image. A sigmoid function is used at the end to ensure the generated values fall within the range [0, 1] for each pixel.
In the experiments, we consider two different types of heatmaps, namely saliency and importance heatmap. A saliency heatmap is generated by aggregating human eye fixations from multiple participants viewing an image. On the other hand, importance heatmaps are obtained by participants highlighting or drawing bounding boxes to indicate the most critical design elements in a graphic design 27 Each these heatmaps reflects distinct aspects of human attention and preference.
The text prompt specifies which heatmap-tyy type to generate for a given input sample, thereby allowing our model to predict a variety of heatmap prediction tasks (e.g., attention, interaction, importance etc.) using single heatmap prediction head. We adopt a pixel-wise & loss function for the heatmap predictor during training.
3.3 Scanpath (Sequence) Predictor
The scanpath predictor takes both the fused image and text tokens after the Transformer encoder as input, and applies a Transformer decoder to generate the predicted scanpath.
scanpath is defined as a sequence of 2D locations (I1, 31). (22, V2)..... (ZN, yNv with a total N fixations, capturing the temporal aspect of attention and visual exploration. The subsequent fixations are conditional on all the previous fixations, thus fitting an autoregressive model with conditional generation. Inspired by previous literature,
use Transformer decoder for object detection and other localization tasks [[3] (5). and therefore generate the location end-to-end with Transformer and text generation. In general, we let the Transformer predict the sequence of coordinates as characters in string one after one and readout the locations from the generated text subsequently.
We spatially decompose the entire image into 1,000 X 1,000 bins with equal interval, and map each coordinate n,, or yn to its nearest bin en Gn. E Z in the range [0, 999].
To formulate the target sequence for teacher- forcing training, we put special token "<extra_id__ _atra__id_>>>>>> at the start of each target sequence, and attach another special token '<extra_id" _aarrrrid_02____ at the end, to indicate the entire scanpath sequence. We concatenate location coordinates with a separation word 'and'. Let y indicate the target sequence with length 3N (corresponding to N fixations), we have the target sequence for teacher-forcing training as follows (-> indicates the line changing due to the paper format):
y <extra_id_o0 31 Ø1 and i22 J/2 and ... and nN UN mextra_id_02>..
The training objective is to maximize the log-likelihood of tokens conditioned on the input image and all preceding tokens in ground-truth scanpath string, i.c..
3N+1
max Σ
w log P(ū, x. V15-1),
where x is the input image and text prompt, and y is the target sequence associated with x. wy is the weight for the j-th token. We use unified weight for each token in experiments.
5

---

Decoding during inference. During inference. it is not strictly guaranteed that the generated string will exactly follow the format of the target sequence, especially when the number of fixations relatively large (e.g., N 2 30), resulting in an invalid sequence to readout. To deal with the possible invalid cases, in practice, we identify two special tokens '<extra_ id 01>' and "extra_id___ ____a_ii________ __02>" in the predicted string if available, and extract the context between these two special tokens. Then we split the extracted string with the separator word 'and'. For a pair of two tokens from the beginning. we check if they are both numerical. If so, add one fixation with this coordinate after mapping them to the original resolution, then iteratively move to the following, and if not, we terminate the decoding process and keep the existing sequence. If there is no fixation available in the predicted string, we mark the scanpath as invalid. During training, we observe that the valid rate (#valid scanpaths / #scanpaths generated) of scanpath decoding quickly converges to meaning every predicted scanpath will contain valid fixation(s).
Compared to the scanpath predictors in GazeFormer numbers instead of two text tokens, our model seems less intuitive. However, as shown in Section 4.3 the proposed scanpath predictor works quite well. Moreover, one advantage of the current design of the scanpath predictor is that it can be easily extended to predict other types of human behavior sequences, e.g., text sequence, or 1-D number sequence, with minor modifications on the sequence output format. This flexibility is important for a unified model like ours.
54 which predicts each point as 2D continuous
3.4 Rating Predictor
This prediction head takes image tokens after the Transformer encoder module, and processes the features via few convolution and connected layers. An lz loss used for training the rating predictor with rating data.
3.5
Text Prompt
To enhance the model's ability to generalize across a variety of visual content and task scenarios, we integrate specific task instructions into the model via text prompts. The prompts used in UniAR are structured as follows:
INPUT_TYPE TYPE: <input_ttppppe OUTPUT_TYPE coutpptt
QUERY: <query>>
We fill <input_type type> with string taken from [natural image webpage graphic design mobile user interface) and poutput_typed taken from (saliency heatmap importance heatmap aesthetics score scanpath). We append a query in string avery:<query> to the prompt if a task-specific query is available, for example, the object name to search, or the question to answer, depending on the use case. For example, an example full prompt "INPUT TYPE:: natural image OUTPUT_ "TTPUUTTTPEE scanpath QUERY:searching bowl", which guides the model to predict scanpath output on a natural image under the task of "searching bowl". The prompt we use modularized and can easily adapt to different types of datasets and scenarios.
4 Experiment
4.1
Protocol
Datasets.
Please refer to Table for all public datasets we consider in training and benchmarking. For more dataset processing details, please refer Appendix [
Benchmarks. We reuse benchmarks from recent literature for model comparison purposes. We adopt the benchmarks for WS- WS-Saliency and WS-Scanpath from Tables 3 and in Chakraborty et al.
11 respectively, Mobile UT from Table in Leiva et al. [47] Implk from Table 2 in Fosco et al. [27], OSIE from Table 4 in Chen al. [12). Salicon from Table 1 in Reddy al. [6], COCO-Search18 from Table in Mondal et al. (54] KontQ-10k from Table in Ke et al. (37. and Web Aesthetics
from Table 4 in Delitzas al. [21]. We also provide model results from some other papers for
comparison (25 33 The baseline results for CA72000 are from CAT2000 Leaderboard F
"There are two versions of Salicon data: Salicon 2015 and Salicon 2017. The results are on Salicon 2017. $CAT2000 Leaderboard: http:://alieenyttubbingnnaaaaae
CAT2000.htm html

---

Evaluation metrics. Inheriting from the above benchmarks, we consider the following evaluation metrics. CC 44]: Pearson's Correlation Coefficient is used here to measure the linear relationship in all pixel values between the predicted and ground-truth saliency heatmaps: KI.D to use Kl.-Divergence between the predicted heatmap and ground heatmap to measure the distribution discrepancy, with the prediction used as the target distribution. AUC-Judd [5] Area under ROC curve (AUC) in the variant from Judd et al. (35) treating the heatmap prediction as binary classification with various thresholds. The specific calculations of true positive and false positive rates can be referred to [O]. SAUC [7] the shuffled AUC metric samples negatives from other images for AUC calculation. NSS [59]: Normalized Scanpath Saliency is the average saliency strength (pixel values in the predicted heatmap) at all ground- truth fixation locations. SIM [64]: Similarity is computed the sum of the minimum values among the normalized prediction and ground-trut -uuuh heatmaps. RMSE: the root mean square error between the predicted and ground-truth heatmaps. R-Squared ( (: the coefficient of determination applied to all values in the heatmap. SemSS (78): Semantic Sequence Score converts each fixation to an II) decided by a semantic segmentation over the image, and compares two strings with string-matching algorithm [56]. SemFED [78): similar to SemSS, Semantic Fixation Edit Distance uses Levenshtein distance for string matching [48]. SequenceScore: similar SemSS, but instead of using a semantic segmentation map, the Sequence Score uses the clustering results from a MeanShift clustering to segment the image and map the fixation to their ID. MultiMatch [22]: MultiMatch the average of four metrics of scanpath, namely: Shape, Direction, Length, and Position, characterizing the similarity between the predicted scanpath and its ground-truth. SRCC and PLCC: refer Spearman's 's rank correlation coefficient and Pearson's linear correlation coefficient, respectively, used to quantify the quality of predicted ratings.
39 the metric
Experimental benchmarks for one task among different datasets may not have uniform evaluation metrics but most of their metrics are shared. For saliency and importance heatmap predictions, we resize the predicted heatmap back to its original image resolution for evaluation.
4.2 Model Training
We pretrain the model on a series of pre-training tasks, including Web/Mobile UI understanding [50] and natural image captioning [14]. Subsequently, we fine-tune the entire model using the Adafactor optimizer with a learning rate of 0.1, batch size of 128, and image resolution of 512x512. All images maintain their aspect ratio and padded fit the training resolution. The model uses VIT B16 as the vision encoder and T5 base as the Transformer encoder of the image and text tokens, resulting in total of 848 million parameters. The model is implemented in JAX. We use 64 Google Cloud TPU v3 to train UniAR for 20k steps in 12 hours.
Datasets mixture. As we are combining a series of public datasets, in every iteration, for all training datasets in Table we em- ploy a random sampling strategy that ensures an equal sampling rate across these datasets. This approach guarantees that each dataset has equal probability of contributing a sample, irrespective of its sample volume.
Table 2: Subjective rating prediction results on Natural image image dataset KonlQ- 1Ok and webpage dataset Web Aesthetics.
Dataset
Method
SRCC
PLCC
BRISQUE 33 122
0.665
0.681
ILNIQE
$15
HOSA 72
1
0.671
0.694
BIECON 16
0.618
0.651
WaDIQaM '7
0.797
0.805
Kon/Q- 10k 31 (Natural image)
PQR 31 17
0.880
0.884
SFA (4 18
0.856
DBCNN [54 18
0.875
0.884
MetaQA [66 20
0,850
0,887
BIQA (25 crops) 62 20
0.906
0.917
MUSIQ-iingl 37 21
0.905
0.919
UniAR
0.905
0.925 10.005
Rating-based Calista 21
0,770
4.3 Experiment Results
(webpage)
Web Aesthetics
211
Comparison- Calista
21
23
0.820
UniAR
0,811
0,839 /2319
We present the results of UniAR for predicting heatmaps,
well ratings across domains and datasets in Tables 4. in comparison with the basclines that are trained on a specific domain, task, or dataset.
Heatmap prediction. Table shows the performance of UniAR across 7 public benchmarks. complete version of results including all baselines and metrics is presented in Table in Appendix C Among these datasets, which vary in domains (natural images, webpages, and graphic designs) and tasks (heatmaps of attention, and importance), UniAR achieves SOTA performance compared to strong baselines, and outperforms previous SOTAs in many cases. On Mobile UI and Imp Ik datasets, UniAR outperforms previous SOTA across every metric. Out of the 27 metrics listed in Table 3
7

---

Table 3: Heatmap prediction results on 7 public datasets across natural images, art. cartoons, mobile Uls, and webpages (Please refer to Table in Appendix C for complete baselines & metries). For Imp/k we predict the importance heatmap, while for the remaining datasets, we predict the attention/saliency heatmap. For each dataset and metrie, the best result in bold, second best is in blue, and our method is highlighted in green. For our model, the relative performance change compared to the best result is noted. Note that the metric values for baseline models are obtained from existing references as described in the "Benchmarks" paragraph. "_" means the metrics are not reported in references. Also note that there are two versions of Salicon data, Salicon 2015 5 and
Salicon 2017. The results in this table are on Salicon 2017.
Dataset
Method
CC
KLD 1
AUC-Judd
NSS 1
SAM-ResNa [8] 18
0.596
1,506
0.795
1.284
EML-NET 33 20
0,565
2.110
0.790
1.277
WS-Soliency
UMSI 22 20
0.444
1.335
0.757
1.042
(Webpage)
Ni WS 19
0.798
0.690
0.852
1.777
AGD-F
AGD-F(Wo--) (Wo--L
22
0.815
0.637
0.858
L8002
UniAR
0.827
0.299
0.860
1,783
AGD-F 11 22
0.735
0.767
1.606
EML-NET 33 20
0.661
0.603
0,847
1.653
FIW/ 67
EMI-NET Salicon
68
20
0.689
0.567
0.848
1.722
(Webpage)
Chen et al.
23
0.699
0,564
0.851
1.752
UniAR
0.734 .0.14%
0.571 40.29%
0.859 40.94
1,838 +4.91%
ResNet-Sal
47
20
0.657
0.692
0.704
SAM-S2015
18
118
0.477
0.650
0.537
Mobile U/ 4v
SAM-S2017
18 '18
0.834
0.723
0.839
(Mobile interface)
SAM-mobile
20
0.621
0.666
0.655
UniAR
0.879
0.115
0.756 44.566
1.008
DeepGaze
17
0.795
0,382
0.864
1,962
CAT2000 (Natural images, cartoons, art....
UNISAL
0,740
0.470
1,936
DeepGaze IIR 21
0.819
0.345
0.869
2.112
SalFBNet 24
22
0,703
1,198
0.855
1,879
UniAR
0.870
-6.23%
0.613
477.68%
0.877 40.226
2.338
+10.709
SimpleNet ResNet-50 61 '0
0.895
0.211
0.868
1.881
SimpleNet w. PNASNet-5 20
0,907
0.193
0.871
1.926
Salicon 34
MDNSal 61
20
0.899
0.217
0.868
1.893
(Natural images)
UNISAL 25 '20
0.880
0,226
0.867
1,923
EML-NET '20
0.890
0,204
0.802
2.024
UniAR
0.77%
10088%
a.11%
1,946 3385%
SAM-ResNet 18 118
0.758
0.480
0.860
1.811
UMSI [27
0.746
0.513
1.788
OSIE 70
EML-NET
0.717
0.537
1.737
(Natural images)
Chen al. 12 23
0.761
0,506
0,860
1.840
UniAR
0.742
0.583
0,862 +0.2%
1.789 -277%
Dataset
Method
f
KLD 1
RMSE
R2 1
SAM [8 18
0.866
0.166
0.168
0. 108
UMSI-nc 27 20
0,802
0.177
0.152
0.095
Implk pr
UMSI-2stream |7
0.852
0.168
0.141
0.105
(Graphic design)
UMSI 20
0.875
0.164
0.134
0.115
UniAR
0.904 +3.31%
0.124
-25.006
0.079
-41.04%
0.823 4615.68
UniAR achieves the best result in 17 of them and the second best in 6 cases. In summary, UniAR experimentally demonstrates promising performance in saliency modeling in various fields.
Scanpath prediction. In Table 4 scanpath- prediction results are shown for two datasets:
1 (16] (scanpath natural images for object searching) and WS-Scanpath (II) (scanpath on webpages under free viewing). On both the datasets, UniAR performs comparably to baselines, and further outperforms the baselines on all the metrics on WS-Scampath. Among the
reported metrics in Table our model achieved the best result in 4 of them. A complete version of results including all baselines and metrics is presented in Table 7 in Appendix C
Score prediction. In Table 2 we present rating prediction results two datasets: KonIQ-10K [1] on natural images and Web Aesthetics [21 on webpages. UniAR achieves the best results for PLCC metrics both datasets and the second best for SRCC on KonlQ-10k. Note that Ke al. [37] a multi-scale version of MUSIQ performs slightly better than UniAR on SRCC (0.916 vs 0.905). However, since UniAR does not use multi-scale inputs, we did not include those results.

---

Table 4: Scanpath (sequence) prediction results on natural image and digital design datasets. Please refer Table in Appendix for complete baselines & metrics.
Dataset
Method
SemSS
SemFED J
Sequence Score
MultiMathh
IRL 777 20
0,481
2.259
0.833
COCO-Seareh18 [6
Chen ct al. 21
0.470
18998
0,820
(Natural images.
FFM 78
0.407
2.425
0,808
object searching)
Gazeformer
S 23
0.496
1.861
0.849
UniAR
0.521 4504
2.004 47.8%
0.874
SceneWalker
666 '20
0.194
0.716
WS Searpath I
AGD-F (w. layout) [ 22
0.203
0.719
(webpage.
AGD-S (w/o layout) [1 22
0.221
0,745
frce-viewing)
AGD-S (w. layout) 22
0.224
0.755
UniAR
0.267 +19.2066
0.887
Transferring knowledge be- tween tasks. We test UniAR's ability generalize and transfer to unseen tasks/domain combina- tions. Our model is trained on certain combinations of task and image domains, and tested on new, unseen combinations of be- havior tasks and image domains.
Table 5: Experiments on transferring knowledge from other do- main/task combinations to WS-Scanpath dataset for scanpath pre- dictions. CC = COCO-FreeViev dataset.
Training Set
Sequence Score
MultiMatch
WS- Scompath (previous SiTA
11
0.224
0.755
WS- WS--eanpath (ours)
0.261
0.894
UniAR full model
0.267
0.887
CC scanpath (ours)
0.196
0.836
CC scanpath + WS-Saliency (ours)
0.190
0.858
CC saliencylsanputth WS-Satiency (ours)
0.231
0.857
Our experiment uses WS (web- page) and COCO-Freeview (free-viewing on natural image) datasets, where WS-saliency, WS- scanpath, CC saliency and CC scanpath, are saliency and scanpath data for WS and COCO-Freeview data respectively. We test the model performance on scanpath prediction data (WS-Scampath dataset). We consider three different training scenarios: (1) Using scanpath data from natural image CCOCO-Freeviews (2) Combining scanpath from natural image (COCO-Freeviés with saliency heatmaps from webpage (WS); (3) Employing both scanpath and saliency heatmap from natural image (COCO-Freeview), augmented with saliency heatmap from webpage (WS). Each scenario maintains some relevance to our test set by either sharing the same task or image domain, but never both.
the webpage scanpath
In T we show experimental results for the above scenarios, and also attach the baseline results on this our third training scenario: leveraging scanpath and saliency heatmaps from COCO-Freeviee saliency heatmap from WS, shows good results against previous SoTA (TT, despite the model having seen webpage scanpath data during training. Prediction performance declines in scenarios & which take more limited datasets, but remain competitive.
and the results from UniAR (full training data) reference. As shown in Table 5
and
Limitations and Future Work
When modeling human preferences and behavior, it is important carefully consider ethics and AT principles, and acknowledge dataset limitations.
Ethics and AI principles. Modeling any aspect of human behavior should adhere to ethical guidelines data collection and applications, and be conducted in transparent way, including clarifying the limitations of the model when replicating human preferences. It should keep humans reference, not means replace evolving
in the loop, the model prediction is intended human preferences with a synthetic guide. We take these ethical considerations and AI Principles into account, ensuring that the model usage remains socially beneficial and responsible.
Aligning with human preference. While UniAR is trained to predict human preferences and behavior, we recognize that using as a reward model may lead to reward hacking, which would make it less representative of genuine human preference. to mitigate this.
We suggest considering techniques [46 26
Representing diverse human preferences. Humans have diverse preferences for subjective notions like image attractiveness. Without personalization, the model converges towards more uniform notion of preference common concern for MI models. To promote visual diversity when using UniAR, propose two strategies: (1) using the model in hybrid manner, providing insights to

---

help humans make decisions in applications like web or visual content optimization, and (2) develop personalized models based on our initial unified model, which will help generate more diverse predictions based on user attributes.
Adjusting to evolving human preferences. Human preferences evolve over time, and to remain accurate, the model has adjust accordingly. Updates can be achieved by fine-tuning with more recent data. Future work will include updating the training data, and exploring concepts like continual learning techniques [72]. keep the model up date.
Dataset limitations. Using diverse, representative datasets is important to minimize potential biases the model. In this paper, we focus on proof-of-concep for unified modeling of human attentionn/pefference behavior, based on existing, publicly available datasets. Below is a listing of annotator demographics, as described in the original papers.
"A total of 41 participants (19 females, 22 males; age range 17-23; with normal corrected-to-normat vision) participated in data collection."
WS-saliency [|:
2. Mobile U/ |77: "Thirty participants (12 male, 18 female). [...] The average age was 25.9 (SD=3.95), The participants had normal vision (8) or corrected tormraaallv (22).
Twenty of the 22 wore glasses and the remaining two wore contact lenses." "
3. Implk 277 "The data of 43 participants (29 male, most in their 20s and 30s) were used in the resulting analyses."
4. FIWI 67 "11 students (4 males and 7 females) in the age range of 21 to 25 participated in data collection. All participants had normal vision or corrective visual apparatus."
Despite some balance in male vs. female participants, the age distribution is skewed towards participants in their 20s. This is likely because most data were collected at universities. Crowdsourced datasets like Salicon and Koniq-1OK are expected to cover a wider range of age and other attributes. Future work will focus on gathering more diverse and representative dataset.
Improve accessibility. UniAR predicts human attention, trained on datasets collected from humans experiencing visual impairments beyond corrective lenses. UniAR cannot model behavior of, for example, blind and low-vision users directly, but it can still benefit them by acting as an accessibility tool for highlighting important areas of webpage for screen reader. One way to enhance the accessibility of the model is using multi-modal preference modeling, which incorporates not just visual cues, but also how users interact with content through screen readers, voice commands, and other assistive technologies. Collaborating with accessibility experts and organizations could help improve future iterations of our work.
Conclusion
We developed a multimodal, unified model UniAR to predict different types of implicit and explicit human behavior on visual content, from attention to subjective preferences/likes, using image and text prompts. This model, trained on diverse public datasets across natural images, graphic designs, webpages and Uls, effectively predicted human attention heatmaps, scanpath sequences, and aesthetic quality scores. Our model achieved SOTA performance across multiple benchmarks and tasks. We plan to explore more behavior tasks and domains in future work.
10

---

References
[1] Kfir Aberman, Junfeng He, Yossi Gandelsman, Inbar Mosseri, David E Jacobs, Kai Kohlhoff, Yael Pritch, and Michael Rubinstein. Deep saliency prior for reducing visual distraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[2] Hossein Adeli, Françoise Vitu, and Gregory J Zelinsky. model the superior colliculus predicts fixation locations during scene viewing and visual search. Journal of Neuroscience, 2017.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhni, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Saskia Bakker and Karin Niemantsverdriet. The intractioo-aatentien continuum: Considering various levels of human attention in interaction design. International Journal of Design, 2016.
[5] Leonardo Bonanni, Chia- Chia-Hsun Lee, and Ted Selker. Attention-base design of augmented reality interfaces. In CHI extended abstracts on Human factors in computing systems, 2005.
[6] Ali Borji and Laurent Itti. Cat2000: A large scale fixation dataset for boosting saliency research. arXiv preprint arXiv: (55.:/5555.38,,, 2015.
[7] Ali Borji, Hamed R Tavakoli, Dicky N Sihite, and Laurent Itti. Analysis of scores, datasets, and models in visual saliency prediction. In Proceedings of the IEEE International Conference on Computer Vision, 2013.
[8] Sebastian Bosse, Dominique Maniry, Klaus Roburt Müller, Thomas Wiegand, and Wojciech Samek. Deep neural networks for no-reference and full-referent image quality assessment. IEEE Transactions image processing, 2017.
[9] Zoya Bylinskii, Nam Wook Kim, Peter O'Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister. Fredo Durand, Bryan Russell, and Aaron Hertzmann. Learning visual importance for graphic designs and data visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 2017.
[10] Zoya Bylinskii, Tilke Judd. Aude Oliva, Antonio Torralba, and Frédo Durand. What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
[11] Souradeep Chakraborty, Zijun Wei, Conor Kelton, Scoyoung Ahn, Aruna Balasubramanian, Gregory J Zelinsky, and Dimitris Samaras. Predicting visual attention in graphic design documents. IEFE Transactions on Multimedia, 2022.
[12] Shi Chen, Nachiappan Valliappan, Shaolci Shen, Xinyu Ye, Kai Kohlhoff, and Junfeng He. Learning from unique perspectives: User-aware saliency modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023.
[13] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Yi Lin, David Fleet, and Geoffrey E. Hinton. A unified sequence interface for vision tasks. Advances Neural Information Processing Systems, 2022.
[14] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language- image model. arXiv preprint 22Xx:2209.00794 2022.
[15] Xianyu Chen, Ming Jiang, and Qi Zhao. Predicting human scanpaths in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[16] Yupei Chen, Zhibo Yang. Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. Coco- search 88 fixation dataset for predicting goal-directed attention control. Scientific Reports, 2021.
[17] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction finertundd- language models. arXiv preprint arXiv:2210. 10. /416, 2022.
[18] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Predicting Human Eye Fixations via LSTM-based Saliency Attentive Model. IEEE Transactions on Image Processing, 2018.
[19] Filipe Cristino, Sebastiaan Mathôt, Jan Theeuwes, and lain D Gilchrist. Scanmatch: A novel method for comparing fixation sequences. Behavior Research Methods, 2010.
11

---

[20] Benjamin de Haas, Alexios L. Iakovidis, D. Samuel Schwarzkopf. and Karl R. Gegenfurtner. Individual differences in visual salience vary along semantic dimensions. Proceedings the National Academy of Sciences, 2019.
[21] Alexandros Delitzas, Kyriakos C Chatzidimitriou, and Andreas I. Symeoniddi.. Calista: A deep learning- based system for understanding and evaluating website aestheties. International Journal of Human- Computer Studies, 2023.
[22] Richard Dewhurst, Marcus Nyström, Halszka Jarodzka. Tom Foulsham, Roger Johansson, and Kenneth Holmqvist. depends on how you look at it: Scanpath comparison in multiple dimensions with multimatch, vector-hased approach. Behavior Research Methods, 2012.
[23] Guanqun Ding, Nevrez Imamoğlu, Ali Caglayan, Masahiro Murakawa, and Ryosuke Nakamura. Salfbnet: Learning pseudo-salieng, distribution via feedback convolutional networks. Image and Vision Computing. 2022.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[25] Richard Droste, Jianbo Jiao, and J Alison Noble, Unified image and video saliency modeling. In The European Conference on Computer Vision, 2020.
[26] Luca Eyring. Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through soward-based noise optimization. arXiv preprint arXiv:2406.0431 2024.
[27] Camilo Fosco, Vincent Casser, Amish Kumar Bedi, Peter O'Donovan, Aaron Hertzmann, and Zoya Bylinskii. Predicting visual importance across graphic design types. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, 2020.
[28] Jonathan Harel, Christof Koch, and Pictro Perona. Graph based visual saliency. Advances in Neural Information Processing Systems, 2006.
[29] Sen He, Hamed Tavakoli, Ali Borji, Yang Mi, and Nicolas Pugeault. Understanding and visualizing deep visual saliency models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019.
[30] Weng Khuan Hoh, Fang-Iaue Zhang, and Neil A Dodgson. Salient-cetteredness and saliency size in computational aesthetics. ACM Transactions on Applied Perception, 2023.
[31] Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Koniq-106 10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 2020.
[32] Laurent Itti, Christof Koch, and Emst Niebur. A model of saliency-based -based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analyxis and Machine Intelligence, 1998.
[33] Sen Jia and Neil D.B. Bruce. Eml-net: An expandable multi-layer network for saliency prediction. Image and Vision Computing, 2020.
[34] Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2015.
[35] Tilke Judd, Krista Ehinger, Frédo Durand, and Antonio Torralba. Learning to predict where humans look. In IEEE International Conference Computer Vision, 2009.
[36] Tilke Judd, Frédo Durand, and Antonio Torralba. A benchmark of computational models of saliency to predict human fixations. MIT Technicul Report
2012.
[37] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEF/CVP International Conference on Computer Vision, 2021.
[38] Jongyoo Kim and Sanghoon Lee. Fully deep blind image quality predictor. IEEE Journal of selected topics in signal processing, 2016.
[39] Solomon Kullback. Information Theory and Statistics. 'Dover Publications, Inc.', 1959.
[40] Matthias Klimmerer, Lucas Theis, and Matthias Bethge. Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet. arXiv preprint arX1v:1411.1045,200 2014,
12

---

[41] Matthias Kümmerer. Thomas SA Wallis, and Matthias Bethge. Deepgaze ii: Reading fixations from deep features trained on object recognition. arXiv preprint arXi::1610.01533, 2016.
[42] Matthias Kummerer, Thomas SA Wallis, Leon A Gatys, and Matthias Bethge. Understanding low-and high-level contributions to fixation prediction. Proceedings of the IEEE International Conference on Computer Vision, 2017.
[43] Matthias Kümmerer, Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Frédo Durand, Aude Oliva, Antonio Torralba, and Matthias Bethge. Miv/tübingen saliency benchmark. htpps//aaleencyy.uebinnenn 2018.
[44] Olivier Le Meur, Patrick Le Callet, and Dominique Barha. Predicting visual fixations on video based on low-level visual features. Vision Research, 2007.
[45] Keaton Lee. Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos. Urvashi Khandelwal, Peter Shaw, Ming-Wos Wei Chang, and Kristina Toutanova. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In Proceedings of the 40th International Conference on Machine Learning. 2023.
[46] Seung Hyun Lec, Yinxiao Li, Junjie Ke, Innfarn Yoo, Han Zhang, Jiahui Yu, Qifei Wang, Fei Deng, Glenn Entis, Junfeng He, et al. Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to-mmage generation. In European Conference on Computer Vision, 2025.
[47] Luis A Leiva, Yunfei Xue, Avya Bansal, Hamed R Tavakoli, Tuôge Köroölu, Jingzhou Du, Niraj R Dayama, and Antti Oulasvirta. Understanding visual saliency in mobile user interfaces. In International Conference on Human- Computer Interaction with Mobile Devices and Services, 2020.
[48] Vladimir I Levenshtein al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet Physics Doklady. Soviet Union, 1966.
[49] Dingquan Li, Tingting Jiang, Weisi Lin, and Ming Jiang. Which has better visual quality: The clear blue sky or blurry animal? IEEE Transactions on Multimedia, 2018.
[50] Gang Li and Yang Li. Spotlight: Mobile UI understanding using rision-language models with a focus. In The Eleventh International Conference on Learning Representations, 2023.
[51] Akis Linardos, Matthias Kümmerer, Ori Press, and Matthias Bethge. Deepgaze iie: Calibrated prediction in and oft-oo-domain for state-of of the-ar saliency modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[52] Alaster Meehan and Joanne B Culpepper. Clutter estimation and perception. Optical Engineering, 2016.
[53] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on image processing. 2012.
[54] Sounak Mondal. Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. Gazeformer: Scalable, effective and fast prediction of goal-directed human attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[55] Bence Nanay. Aesthetic attention, Journal of Consciousness Studies, 20155
[56] Saul B Needleman and Christian D Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 1970.
[57] OpenAL. Gpt-4 technical report, 2023.
[58] Junting Pan, Cristian Canton Ferrer. Kevin McGuinness, Noel E O' O'Conoo, Jordi Torres, Elisa Sayrol, and Xavier Giro-i Nieto. Salgan: Visual saliency prediction with generative adversarial networks. arXiv preprint 1rXiv:1701.01000 2017.
[59] Robert J Peters, Asha Iyer, Laurent Itti, and Christof Kach. Components of bottom-up gaze allocation in natural images. Vision Research, 2005.
[60] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research. 2020.
[61] Navyasri Reddy, Samyak Jain, Pradeep Yarlagadda, and Vineet Gandhi. Tidying deep saliency prediction architectures. In JEEE/RSJ International Conference Intelligent Robots and Systems, 2020.
13

---

[62] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean-baptise Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser. al. Gemini 15; Unlocking multimodal understanding across millions tokens of context. arXiv preprint vrXiv:2403.05550 2024.
[63] Claudia Roda. Human attention and its implications for human computer interaction. Human attention in digital environments, 2011.
[64] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The carth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 2000.
[65] Eldon Schoop. Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. Predicting and explaining mobile ui tappability with vision modeling and saliency analysis. In Conference on Human Factors in Computing Systems, 2022.
[66] Lisa Schwetlick, Lars Oliver Martin Rothkegel, Hans Arne Trukenbrod, and Ralf Engbert. Modeling the effects perisaccadic attention on gaze statisties during seene viewing. Communications Biology, 2020.
[67] Chengyao Shen and Qi Zhao. Wehpage saliency. In Proceedings of the European Conference on Computer Vision, 2014.
[68] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang, Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[69] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrae. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
[70] Nachiappan Valliappan, Na Dai, Ethan Steinberg. Junfeng He, Kantwon Rogers, Venky Ramachandran, Pingmei Xu, Mina Shojaeizadeh, Li Guo, Kai Kohlhoff, et al. Accelerating eye movement research via accurate and affordable smartphone eye tracking. Nature Communications, 2020.
[71] Ronald Van den Berg, Frans W Cornelissen, and Jos BTM Roerdiak. A crowding model of visual clutter. Journal of Vision, 2009.
[72] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[73] Wenguan Wang and Jianbing Shen. Deep visual attention prediction. IEEE Transactions Image Processing, 2017.
[74] Jingtao Xu, Peng Ye, Qiaohong Li, Haiqing Du, Yong Liu, and David Doermann. Blind image quality assessment based on high order statistics aggregation. IEEE Transactions on Image Processing, 2016.
[75] Juan Xu, Ming Jiang, Shuo Wang. Mohan S Kankanhalli, and Qi Zhao. Predicting human gaze beyond pixels. Journal of Vision, 2014.
[76] Sheng Yang. Guosheng Lin, Qiuping Jiang, and Weisi Lin. A dilated inception network for visual saliency prediction. IEEE Transactions on Multimedia, 2019.
177] Zhibo Yang, Lihan Huang, Yupei Chen, Zijun Wei, Scoyoung Ahn, Gregory Zelinsky, Dimitris Samaras, and Minh Hoai. Predicting goal-directed human attention using inverse reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[78] Zhibo Yang. Sounak Mondal, Seoyoung Ahn, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras. Target-absent human attention. In Proceedings of the European Conference on Computer Vision, 2022.
[79] Chen-Ping Yu, Dimitris Samaras, and Gregory J Zelinsky. Modeling visual clutter perception using proto-object segmentation. Journal of vision, 2014.
[80] Dario Zanca, Stefano Melacci, and Marco Gori. Gravitational laws of focus of attention. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[81] Hui Zeng, Lei Zhang, and Alan C Bovik. A probabilistic quality representation approach to deep blind image quality prediction. arXiv preprini arXiv: 08090 2017.
[82] Jianming Zhang and Stan Sclaroff. Saliency detection: A boolean map approach. In Proceedings of the IEEE International Conference on Computer Vision, 2013.
14

---

[83] Lin Zhang, Lei Zhang, and Alan C Bovik. A feature-enriched completely blind image quality evaluator. IEEE Transactions Image Processing, 2015.
[84] Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using deep bilinear convolutional neural network. IEEE Transactions Circuits and Systems for Video Technology. 2018.
[85] Quanlong Zheng, Jianbo Jiao, Ying Cao, and Rynson WH Lau, Task-driven webpage saliency. In Proceedings of European Conference Computer Vision, 2018.
[86] Hancheng Zhu, Leida Li, Jinjian Wu, Weisheng Dong, and Guangming Shi. Metaiqa: Deep meta-learning no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
15

---

Enly Misel Whg mate poin tate so gool
Miaaia Whip
Euyy Wasste Whip make parn toste good
cite para
Smarrcn Microwave
Microwove
GT=0.77; Pred-0.73 0.73
GT-0.51; Pred-0.52
0.86
0:73
01.4. Pred-0.52
Figure 3: Another set of visualizations UniAR's predictions. Images in green border are ground- truth, while images in orange border are UniAR's predictions. First row: saliency heatmap on Salicon and WS-Saliency. Second row: importance heatmap on Implk, and saliency heatmap on Mobile UI. Third row: free-viewing scanpath on WS-Scanpath and object-searching scanpath on S0 Fourth row: rating prediction on Koniq-10k and Web Aestherics datasets.
A
Model Details
The main model components consist of VIT B16 encoder for image encoding, TS base encoder for mixing image and text tokens, and three predictors for rating, heatmap, and scanpath prediction, respectively.
Vision Transformer and T5 Encoder. The VIT B16 encoder uses 16 x 16 patch size, 12 layers with 12 heads, MLP dimension 3,072, and hidden dimension 768. The TS base encoder uses 12 layers with 12 heads and MLP dimension 2,048 and hidden dimension 768.
Score Predictor. The score predictor consists of four convolutional layers with Layer Normalization and ReLU activation. The filter size, kernel size, and strides are (768, 384, 128, 64), [2, 2, 2, 2), [1, 1, respec- tively. Three dense layers of size 2,048, 1,024 and are used generate scalar with ReI.U activations for the first two layers, and sigmoid for the last.
Heatmap Predictor. The heatmap predictor consists of two convolution layers with filter size, kernel size, and stride [768, 384], [3. 3]. [1, 1], respectively. then four de-convolution layers up-sample to the re- quired output resolution, with the filter size, kernel size, and stride as [768, 384, 384, 192]. [3, 3, 3,3,, 3), [2, 2, 2, 2]. respectively. Each de-convolution layer is with two read-out convolution layers of kernel size 3 and stride 1. Layer Normalization and ReLU are used for each layer. In the end, two read-out out convolution layers and a final sigmoid activation are used to generate the heatmap prediction.
Scanpath Predictor. The scanpath predictor is implemented using a T5 base decoder with 12 layers 12 heads and MLP dimension 2,048 and hidden dim 768. Output token length is 64.
We combine the losses from the three predictors, ie.. sequence cross-entropy loss, heatmap 1.2 loss, and score L2 loss using weights [1. 500, 50] empirically (to make them at the similar scale).
B Dataset Processing
In this section, we describe some of the key dataset processing details.
16

---

Table 6: The full table for heatmap prediction results on 7 public datasets spanning digital design and natural images, benchmarking with eight metrics in total. The details on datasets and evaluation metrics can be found in Section 4. 1 For Imp/k /k dataset we predict the importance heatmap. while for the rest of the four datasets, we predict sicention/aalincyy heatmap. Our method highlighted
with green background, For each dataset and each metric, the best result in the current column is in bold, and the second best result is in blue. For our model, the relative performance change compared to the second best result (or the best result if we are not the best) in % is noted.
Unranet
Method
AUC-Jodd
SALCT
NSS
RMSH
Ini et al 53 99
0,002
0.225
0.555
0.126
RMS 82
0.131
0.249
0.206
0.138
Modite Uf [X
(Mobile imnestuce)
GBVS
SAM-S3115 18
KesNesSoo
[+
04477
0.807
0.580
0.630
0.082
0.666
0.754
0.700
0.561
0.501
0,577
0.704
SAM
13
0.534
0.723
0.319
SAM
0,621
0.115
0,832
1,008
6.116
0,777
Ini [3] N6
0.57
0.540
(1710
0.661
1749
Dosp Gaaz 11 16
0.594
3.400
0.815
0.644
1.50
SaGAAN + W3 17
0.37
0.622
0.515
0.703
1 1.455
DVA F33 12
0.57
0.701
0,85
0.711
1 1.200
DAVIDISM (:30 10
0.510
0.588
0.739
0.668
1.135
RS Salienes 011
(wchyagge)
SAMt-Res-ee 18
13451 (2]
FMII-NIT
0.56
0.565
2.110
1.355
1.500
0.755
0.737
0.700
0.217
1.217
1.24
Freeklew
0.744
SAM--RoNN
0.718
D1 Nat WS [0
0.798
0,852
AGD-3- (W/o
22
0.15
0631
0,88
1,892
UniAK
0,27
0.299
0.800
0,779
0,757
1,785.1095
(LU8S
0.991
DeepGaze [1 To
0.685
0.797
0.625
1.239
SAM-RoNN 13 18
0.595
0.791
0.633
1.346
UMSI 221 20
0.47
0.755
0.675
0.938
AMV x7 (wobpegr)
AGIb-r [1] 21
6.738
0.967
0.748
1.000
FMIL-NH
0.661
8.00
0.547
0.675
EML NET Salicoe
0.699
0.567
0.445
UsiAR
0.749
1
0.082
0,544
SimpleNia ReaNe--00 AT
0.805
0:211
0.858
0.796
1.881
SimpleNia PNASN h1 300
0,907
0.193
0.871
0,797
1.926
Saticnn 3
MDNSal 61 20
0.559
0.217
0.86
0.797
1,855
(Naumal images)
UNIIAL
20
0.580
0.236
0.867
0.725
0.771
19923
EML-NET 20
0,500
0.204
0.802
0.778
0.785
2.024
UniAR
0900
0.214
11339
0.553
0.791 2710
1,946
3999
0.079
0,823
15
0.573
0.546
0.600
0.89)
oser
(77
(Nuural Imsges)
EMT
0.717
0.537
0,854
0.619
Chen
6.761
0:506
0,682
1,840
UsiAR
0 C742
0.581
0
0.745
0.640 18r
1789
EL163
0.558
I0# 43
0.780
0:445
0.856
0.619
0,670
1.99
DespChoo
+3
00205
0.362
0.864
0.650
0.687
19962
CAZ2000 35
UNISAL
30
0.740
0.570
0,800
0.688
0.063
19936
(Natural Image)
DespChoe
31
0.819
0.345
0.969
0L668
0.706
2.112
SuFFNNa
0.703
1.198
0,855
11009
0.752 JUS89
23388
0.91
Hylinskii atal.
17
0.758
0.301
01181
0,022
Bylinoki ctal.
0.732
0.368
0.205
0.061
(Grophicc design)
Is Fr
SAM [3
22 IMMSI-2arem
20 30
0.866
0.892
01902
0.166
0.177
0.168
0.168
0.152
0.141
0.108
13,095
0..05
UMSI
20
0,875
0.164
0.134
0.115
UnIAR
0,964 3335
0.124 22.0
0,856
0.079 20.055
0,533 4415355
Implk. In the lmp/k dataset, we observe some resolution mismatch between the image and its ground-truth importance map. To unify the image and the ground-truth into the same resolution, we find out the lower resolution (with a smaller area) between these two and downsample the larger one into the lower resolution.
WS-Scanpath. We asked Chakraborty al. [ for their code on evaluating the scanpath, and follow them to MeanShift clustering to generate spatial bins for the metric SequenceScore. We also follow their criteria to select the number of clusters in McanShift clustering which will be used in the evaluation later on.
We contacted the authors but the original dataset partition is missing. We randomly partitioned the dataset into a training and testing set with the number instances following Leiva et al. [47 We calculate the saliency results using images without padding.
Mobile UI.
COCO-Search
For the scanpath results, we follow the updated experimental results in the appendix of GazeFormer (54) The original results in the main paper are impacted by an image padding issue, as reported in their GitHub repo.
C Full Results and More Visualizations
We attach another set of visualizations of UniAR's predictions in Figure 3 inclluding heatmap, scanpath, and rating predictions.
17

---

Table 7: The full table for scanpath prediction results on natural image and digital design datasets, with searchin and free- viewing tasks.
Method
Length
[RI
22l 70
0.411
2.74
0.567
11:55
0.511
0.811
COCO Searc/8
16
Clan ct al [33
0,400
1,00
0,003
0,591
0.891
0.565
0.520
imaps.
FPM
0.007
0.513
object searchig))
0,46
enia 10
0.177
0.261
0.676
0.778
0.594
6.307
MASC
0.169
0.755
0,590
0.55
0.514
0.717
cwntpagg
WS Scarport
sieving)
11]
G Raroo
SerneWallen
lagua)
kgown
6a 30
22
0.218
0.194
03845
0320
0.616
0.573
01882
01816
0.560
0681
0.716
0.748
UnIAR
112667
6,907
IL836
0,908
0.798
01887
present full tables of UniAR's performance heatmap scanpath predictions Tables and y with more baselines and a complete set of evaluation metrics. UniAR offers consistently good predictions on three tasks across multiple datasets, compared to the ground-truths.
18

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Contributions and scope accurately reflected.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifienn specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: : No new theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
19

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See experiments details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [NA]
Justification:
Not appliable for now.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc.pspps.ccc/ppee cc/pubiic/ quides/CodeSubmn sww.ooooiiii for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nniss//.///////////e:::eee cc/public/ qu..dess/ooteSuunllesssooooo.... for more details.
20

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 41
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the existing references for experiment results on benchmark datasets, where only average metric number, but no error bar, are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform. every respect, with NeurlPS Code of Ethics http:////eerrpps.ccccubb///////a
Answer:
[Yes]
Justification:
We followed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
22

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
They are all publicly available data with proper license.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets.
Guidelines:
The NA means that the paper does release Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No new human subjects.
Guidelines:
answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Beard (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
23

---

Justification:
Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
24


Reviews:
[{"review_id": "Izlffq63Py", "reviewer": "Reviewer_RXVY", "summary": "In their paper \"ALOHA: from Attention to Likes – a unified mOdel for understanding HumAn responses to diverse visual content\" the authors describe a new unified model to predict human saliency(attention/importance), even more fine grained than that, scanpath, and ratings.\n\nAfter nicely introduced the motivation and an extensive outlook on related work, the authors describe the multimodal VLM-based encoder-decoder transformer architecture in detail, especially the three predictor head. Exploting the power of instruction tuned LLM, the authors introduce additional tokens to  predict valid scanpath at inference time. \n\nThe benefits of their model are evlauted based on many experiments, which are shortly discussed, followed by a short conclusion.", "questions": "You mention that you pad images to 512x512 -- what happens with images larger than 512x512?\n\nHow did you arrive at your optimization criterion, scaling for the learning rates and in general all hyperparameters? You did not report any hyperparameter tuning.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper are the experiments, which are extensive across several benchmarks and metrics.\nIntegrating a three headed VLM for the tasks is intuitive and elegant. \nTheir proposed ALOHA architecture is SOTA in sevaral settings (22/35)\nThe writing is mostly easy to follow and there is a clear picture the authors want to paint.", "weaknesses": "- The writing can be improved in several places, I'll note here a few:\n1. Introduction: \n- In the introduction, it seems that the authors already have the conclusion at the end, switching from present to past tense for the main contributions of the paper. This makes it sounds as if the authors already published ALOHA previously. \n- In the same two points of the main contributions, the authors actually only have one contribution, their model ALOHA. The second \"contribution\" is only the evaluation of ALOHA.\n- regarding the previous comment -- the second contribution, even though from my point of view it is no contribution is the evaluation of ALOHA and not the training of ALOHA.\n\n3. Unifying Human Visual Behavior Modeling from Attention to Likes:\n- The overall optimization criterion should be part of the main paper, not in the appendix.\n- Model training (section 4) from my pov should be part of section 3, it is not part of the experiments since you do not report multiple training strategies.\n\nIts unclear how hyperparameters were tuned.\nCode will not be made public, which is a major concern in the current reproducibility crisis in ML. Additionally the question in the questionnaire is wrongfully answered as N/A; the paper DOES INCLUDE experiments requiring code. (GUIDELINE: The answer NA means that paper does not include experiments requiring code.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and we address each point below.\n\n---\n\n**The writing can be improved in several places**\n\nA: We will edit the paper according to the suggestions.\n\n---\n\n**How hyperparameters are tuned**\n\nA: We did a training-validation split on our larger datasets and then tuned the hyperparameters (e.g., learning rate, batch size, dropout rate, loss weights of the 3 heads) on the validation set through grid search. The model was then trained with the full training data with the chosen hyperparameters.\n\n---\n\n**What happens with images larger than 512x512?**\n\nA: We resize the images to have a max height or width of 512 and center-pad the smaller dimension to 512.\n\n---\n\n**Code will not be made public, which is a major concern in the current reproducibility crisis in ML**\n\nA: Given the sensitivity of this topic, and our concern that the code may be misused if made public (see comments from Reviewer FgZM and our rebuttal on \"Properly using ALOHA model\"), our institution has a strict policy on opening source code in this area. But to help advance the research, we will provide enough details (including all the info in the rebuttal), all necessary communication and support for o researchers in this area to reproduce our work, who we believe will make use of this technique appropriately.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n31VtGnAO9", "reviewer": "Reviewer_FgZM", "summary": "This paper introduces ALOHA, a multimodal model that predicts human saliency, scanpath, and subjective rating of natural images, webpages, and graphic designs. ALOHA outperforms or performs similarly to baseline models across each of its prediction tasks while improving generalizability over task-specific models.", "questions": "Please see the Weaknesses section above.", "limitations": "Please see the Weaknesses section above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*Generalizable model of human visual behavior and preferences.* The ALOHA model predicts multiple forms of human preference --- the saliency, scanpath, and rating --- of various types of image inputs, including web pages, natural images, and cartoons. This improves the generalizability of ALHOA over prior models that focus on a single prediction task or data input modality.\n\n*In-depth analysis of model performance compared to prior benchmarks.* The paper provides a thorough evaluation of ALOHA's task-performance across 13 metrics, 11 datasets, and 4 tasks. ALOHA outperforms or performs on par with pre-existing models across all evaluations.\n\n*Easy to read paper.* The paper is well motivated and clearly explained. Figure 1 is a particularly helpful overview.", "weaknesses": "My biggest concern is the paper’s lack of engagement with the limitations and ethical considerations of modeling subjective human preferences. Currently there is a brief discussion of training data limitations in the supplement. I would like to see a thorough discussion of considerations included (or at least referenced) in the main text. Possible discussion points that come to mind include:\n\n* What are the ethical considerations of replacing humans with models trained to replicate their preferences? Humans have diverse and often opposing preferences, particularly for subjective notions of aesthetics or attention. It is possible that models trained to replicate human preferences could learn a more uniform or singular notion of preference. Could using these models limit the visual diversity in webpages and content creation? Or could these models actually help us generate more visually diverse content?\n* Whose preferences are included in the model and, more importantly, whose aren’t? Humans likely have diverse preferences when it comes to subjective notions like website attractiveness. I would suspect these preferences vary based on age, experience with technology, culture, etc. Building a model to replicate human perspectives may amplify a particular worldview that is not representative of all users. On the other hand, it may be easier to create a model trained on diverse human data than to get a diverse set of human feedback for every new webpage design.\n* The paper suggests using the model as a reward function to optimize content creation. Could this result in negative consequences, such as content that is optimized for a model’s preference function and may not be value aligned with what humans want?\n* How could human preference models, like ALOHA, be misused? For example, it seems like these models could be used to make ads more intrusive by optimizing their scanpath location or to make phising sites more convincing by optimizing the saliency of the credential input.\n* How should human preference models take into account blind or low-vision users? Scanpaths and saliency likely differ between users who rely on screenreaders and those who don’t. Optimizing the web for sighted preferences could further exclude blind and low-vision from online spaces. Given the [emphasis](https://www.ada.gov/resources/web-guidance/) on web accessibility, how can we use human preference models to amplify (not weaken) online accessibility?\n* How should models stay up to date with the changing human preferences? Human preferences, especially around aesthetics, are constantly changing. As a result, what are the implications for interpreting the evaluations used in this paper that span datasets from 2014–2023? \n\nThe paper would also be strengthened by an audit of their model’s behavior beyond quantitative performance metrics, such as an analysis of the diversity of the humans included in the training dataset, a categorization of the types of mistakes the model makes, and a [Model Card](https://arxiv.org/abs/1810.03993) reflecting its design. \n\nI recognize that the paper is inheriting these ethical concerns from the preexisting datasets and tasks that it works on. I hope that by including a discussion of the models implications, the paper will progress research into these types of models and inspire work on new datasets, model analysis, and safe deployment techniques.\n\nI might also suggest that the authors should select a new name for their model. I am not an expert in Hawaiian culture, but I know that the term ‘aloha’ is culturally significant, and there have been [Hawaiian movements to reclaim it](https://www.usatoday.com/story/life/health-wellness/2023/01/13/stop-saying-aloha-out-of-context/10990192002/), such as the [‘Aloha Not For Sale’ protests](https://kawaiola.news/cover/aloha-not-for-sale-cultural-in-appropriation/). Using it out of context as a model name, may minimize the rich cultural meaning behind the term, and given it is an imperfect acronym anyway, should be an easy change.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate you bringing the ethical concerns to our attention. We recognize that such concerns are common with machine learning models, particularly those involving user preference and behavior modeling. We are committed to addressing these issues to the best of our ability by expanding the ethics and limitations section.\n\n---\n\n**Uniform vs diverse human preferences**\n\nWe acknowledge that models carry the risk of converging towards a more uniform notion of preference, a concern shared by all machine learning models. To promote visual diversity in our model, we propose,\n1. Hybrid Approach: Initially, the model should be used in a hybrid manner, providing insights without replacing human decisions in web optimization, and\n2. Personalized Models: Develop personalized models based on our initial unified model. This approach will help generate more diverse predictions based on user attributes.\n\n---\n\n**Demographics of annotators**\n\nPlease see our response in the global comment.\n\n---\n\n**Aligning with human preference**\n\nWhile our model should be approximately aligned with human preferences, we recognize that using it as a reward model may lead to reward hacking. We plan to incorporate techniques [1, 2] for mitigating this when we use it as a reward mode.\n\n[1] Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation  \n[2] ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization\n\n---\n\n**Properly using ALOHA model**\n\nLike many technologies, human gaze prediction models could potentially be misused. Without regulations, such as unrestricted access to users without credentials and irresponsible data collection, the risk can be increased. To ensure the proper usage of our model, we propose,\n1. Placing restrictions on model access to prevent unauthorized applications or users;\n2. Adhering to strict ethical guidelines on data collection, and actively monitoring the collection process at scale; and\n3. Being transparent about our model's capabilities and limitations. We believe that keeping humans in the loop and only using the model prediction as a reference but not replacing humans is important.\n\nInternally, we strive to follow the ethical guidelines of our institution, ensuring that the model usage remains controlled and responsible. This has also caused us to be cautious about making our model publicly available.\n\n---\n\n**Blind and low-vision users**\n\nOur model can benefit blind and low-vision users for screen users by highlighting the most important areas of a webpage via heatmap predictions and making them more accessible via screen readers. We recognize that the current training data may have limitations in representing the full spectrum of user experiences. We plan to enhance the inclusiveness by,\n1. Multi-Modal Preference Modeling: We're developing our model to incorporate not just visual cues, but also how users interact with content through screen readers, voice commands, and other assistive technologies; \n2. Collaboration with Accessibility Experts: We plan to collaborate with accessibility experts and organizations representing blind and low-vision users for future iterations of our work.\n\n---\n\n**Stay up to date**\n\nStaying up to date is essential for models, and can be achieved by fine-tuning with more recent data. Continual learning techniques also support this idea [1]. However, updating the training data is out of the scope of our paper, as we want to focus on developing the first unified model on the diverse user modeling tasks.\n\n[1] A comprehensive survey of continual learning: theory, method and application\n\n---\n\n**Error Analysis**\n\nWe analyzed 50 error examples in the Koniq-10K dev set and found some interesting patterns, see examples in Figure 1 in our rebuttal PDF. The first example shows the most common error category when our model predicts a higher score for colorful but blurry images. The second example shows another common model error where a low-light image gets a higher prediction score. For the last black-and-white example, on the contrary, our model predicts a lower score than the groundtruth. These errors demonstrate that human preferences for image aesthetics can depend on diverse factors including clear focus, correct lighting, and artistic style. \n\nSimilarly, for heatmap prediction, we analyzed the 30 examples with the lowest NSS scores on the Salicon validation set. The example in Figure 2 in the rebuttal PDF demonstrates the most common error category where the groundtruth contains a more scattered gaze heatmap. In such cases, our model prediction might not focus on the right objects. \n\nWe will conduct more error analysis on other model predictions and include this info in the paper.\n\n---\n\n**New content to include**\n\nHappy to incorporate your recommendations and include a discussion on the following action items:\n1. The diversity / demographics of annotators\n2. Model failure cases and discussions\n3. A model card\n\n---\n\n**Model’s name**\n\nWe are happy to choose a new name from alternative names for our model, for example,\n1. GLAM: From Glances to Likes: A Unified Model for Understanding Human Visual Behavior\n2. TOTORO: from attention TO likes TO human RespOnses – a unified model of human visual behavior\n3. HABITAT: modeling Human Attention and Behavioral InTeractions Across diverse visual content using a multimodal Transformer", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3hh2n59mvN", "reviewer": "Reviewer_ew4J", "summary": "Noticing the existing issues in human behavior modeling, such as isolating the study of implicit, early-stage perceptual behavior (like human attention) from explicit, later-stage behavior (like subjective preferences), specified visual content type; in this manuscript, the author(s) aimed to build an integrated human attention and preference behavior model for addressing multiple visual content types. Through empirical experiments, the author(s) demonstrated the effectiveness of the proposed model.", "questions": "I read the manuscript, and I have the following questions/comments. Thanks.\n\n1. Are there any potential applications of ALOHA in real-world scenarios or in dynamic environments? How can it be used to optimize elements, such as user interfaces, graphic designs, and content creation?\n\n2. Are there any potential biases in the current model's predictions?\n\nTiny format issues in References:\n\n(1) Sometimes, the journal/conference name used an abbreviation; sometimes, not, sometimes, both, such as Ref.[25], Ref.[67].\n\n(2) Keep the format of the references consistent, such as Ref. [6] vs. Ref.[15].\n\nPlease check carefully and correct the issues.\n\n\nOverall, ALOHA represents a significant advancement in modeling human visual behavior, offering a unified approach that spans from early-stage perceptual responses to later-stage decision-making. I think this is an interesting manuscript.\n\nI look forward to hearing from the author(s). Thanks.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "In my opinion, the strengths of this manuscript are as follows:\n\n1. Designed a unified approach to model human visual behavior: image+text=>human perceptual behaviors.\n\n2. Extensive experiments are performed to validate the effectiveness of the proposed model.", "weaknesses": "In my opinion, the weaknesses of this manuscript are as follows:\n\n1. This manuscript seems not to discuss the model's limitations in real-world situations, such as how its performance is in dynamic environments or its adaptability to changes in user behavior over time.\n\n2. As the author(s) described, ALOHA has 848 million parameters. Training such a large model requires significant computational resources, which may not be accessible to most researchers or practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and we address each point below.\n\n---\n\n**Model's limitations in real-world situations, such as how its performance is in dynamic environments or its adaptability to changes in user behavior over time.**\n\nA: Current model does not consider dynamic environments or changes in user behavior over time, which is indeed a limitation. We will add this limitation to the discussion section. Please also see our response to the question of “Stay up to date” from reviewer FgZM.\n\n---\n\n**ALOHA has 848 million parameters. Training such a large model requires significant computational resources, which may not be accessible to most researchers or practitioners.**\n\nA: We acknowledge the resource requirement might be difficult for researchers without access to enough GPUs / TPUs. There are techniques to reduce model training cost, like LoRA, or Parameter-Efficient Fine-tuning (https://huggingface.co/blog/peft), which could be employed with much less resources. However, using such techniques would be beyond the scope of this work. We will include this info in the paper and develop models with lower resource requirements in our future work. \n\n---\n\n**How can it be used to optimize elements, such as user interfaces, graphic designs, and content creation?**\n\nA: ALOHA can help optimize content in several ways. For example, the predicted saliency heatmap can be used to remove the distracting areas in visual content, similar to “Deep Saliency Prior for Reducing Visual Distraction”:  https://arxiv.org/abs/2109.01980. If the content is generated by a generative model, ALOHA’s predicted score can be used as a reward score to improve the generative model, with learning from human feedback/preference method, such as DPOK (https://arxiv.org/pdf/2305.16381) and DRaFT (https://arxiv.org/abs/2309.17400).\n\n---\n\n**Are there any potential biases in the current model's predictions?**\n\nA: Please see the global rebuttal.\n\n---\n\n**Format issues**\n\nA: We will make the edits according to the feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIu3njZMjM", "reviewer": "Reviewer_Fbj6", "summary": "In this paper, the authors introduce a novel text-image framework designed to integrate various human-response tasks and multiple image domains. These tasks include attention map generation, scanpath prediction, and subjective preference evaluation, applied to images such as webpages, natural scenes, graphic designs, and mobile user interfaces. The framework utilizes a transformer architecture, accepting both text and image inputs. Text inputs specify the tasks, while the model itself comprises a transformer encoder and three predictors (heatmap, rating, and scanpath) to generate 3 types of outputs. Initially pretrained on foundational text-image tasks, the model is fine-tuned for diverse human-response tasks", "questions": "1. How are the three predictors chosen for each task? Are they fixed assigned based on different input texts?\n2. Are there any differences between the salience map and the importance map in terms of architecture or loss function, or do the differences lie solely in the training data?\n3. In the \"decoding during inference\" section, the paper mentions, “If there is no fixation available in the predicted string, we mark the scanpath as invalid.” This invalid case (“no fixation”) only covers situations where all token pairs are non-numerical. If some tokens in the output sequence are in an unrecognizable format, rendering part of the results invalid, how should this be evaluated and handled?\n4. Why is text generation used as a scanpath predictor?\n5. How is the model fine-tuned on various datasets, which parts are freezed?\n6. All my above questions regarding the numerical results (Table 3 and Table 6) in the Weakness section.", "limitations": "The reported numerical results do not adequatly support the paper's claim.\n\nMore experiments to demonstrate the model's task transferring ability are needed. For example, transferring from saliency map to rating, or from scanpath to saliency map.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe question of unifying human response-related vision models is novel and holds significant potential, particularly if the advantages of integrating tasks are thoroughly explored.\n2.\tThe paper is well-organized and easy to understand.", "weaknesses": "1.\tI have many concerns on the experimental results of this paper, where the authors try to demonstrate the superiority of their model performance by comparing with other methods. My concerns are as follows. \n\na)    The metrics in Table 3 are not a standard set of metrics for saliency prediction. Why don’t the authors report the complete of evaluation metrics in Table 3? It seems strange to drop the popular metrics such as sAUC and SIM in Table 3. It is a standard practice to report all standard metrics, ie., those commonly used in saliency predictions, as in the online benchmark (https://saliency.tuebingen.ai/evaluation.html). It seems odd to me the authors only report selected metrics in Table 3 but report all in Table 6 (which is in the appendix). \n\nb)\tFollowing a), why do the authors put both Table 3 and Table 6? Both of the tables occupy the same space. The authors can just put Table 6 in the main manuscript instead of Table 3. \n\nc)\tWhy do the comparison methods vary greatly across different testing datasets? It will be helpful if the authors stick to several methods and compare their performance across the same testing datasets. Deep Gaze II and IIE are recognized for their strong performance on images of general scenes. Why did the authors choose to test these models only on the CAT2000 dataset and not on other general scene datasets such as Salicon and OSIE?\n\nd)\tIn the SalGAN paper [60], it reports the metrics on Salicon dataset, why the authors do not report this in Table 3 or Table 6 for SalGAN on Salicon dataset, but instead, only report its performance on the webpage dataset? If the authors have simply copied results from other works, incorporating these additional benchmarks would require minimal effort and should have been included. If the authors have tested the models, why not test them on the popular benchmarks such as Salicon and OSIE?\n\ne) Can the authors explain why they don’t report SequenceScore for COCO-Search18? This column is denoted as \"-\" in Table 4. \n\nf) Furthermore, the reported results on COCO-Search18 of the comparison method FFM[79] in its own paper [79] are different from what what authors report in Table 4, both under target present conditions. The SemSS scores in [79] under target present conditions are consistenly above 0.53, and the method of Chen et al. [16] is 0.572. Can authors provide more details on how they obtained the current results in Table 4 for the comparison methods? \n\ng) Can authors explain why they didn’t compare scanpath prediction performance of ALOHA and baselines on COCO-FreeView in Table 4 even though they use COCO-FreeView in Sec. 4.4?\n\nh)\tIt will be helpful if the authors can experiment with more popular benchmark datasets such as MIT300 and MIT1003, and compare with the performances published on the online benchmark (https://saliency.tuebingen.ai/results.html).\n\ni)     Minor: some top performances are wrongly indicated, eg., on the webpage dataset (FiWI), the best performance on KLD should be Chen et al. [13].\n\n\nIn summary, excluding the SOTA models in common benchmark datasets significantly undermines the experimental evidence for ALOHA's competitiveness, especially considering that 7 out of the 11 datasets in the experimental section are for free-viewing saliency prediction (see Table 1). Based on all the above, I feel that current experiments cannot adequately supported the paper's claims.The paper needs more discussions and comparison results to support its claim that the model achieves SOTA performance. \n\n2.\tAlthough the authors claim that the unified model could “act as a comprehensive reward model, predicting subjective ratings/preferences as rewards and providing additional insights from predicted human attention and behavior patterns,” the experiments do not clearly demonstrate the actual benefits of unifying the attention, rating, and scanpath tasks. The motiviation of this work is not well positionied. \n\n3.\tSome technical details of the paper are not clear. E.g, a). the three predictors appear to operate independently, but it is unclear how they were chosen. b) Why text generation is chosen for scanpath generation. c) How is the model fine-tuned on various datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and we address each point below.\n\n---\n\n**Metrics in Table 3 and Table 6**\n\nA: We include the important metrics and baseline methods in Table 3 so that it is more readable with larger font size. We have included reference to Table 6 in Table 3 caption for completeness. Some metrics are not chosen in Table 3, because the baseline papers did not report them, and hence caused empty columns in Table 6.\n\n---\n\n**Different comparison methods across different testing datasets**\n\nA: For fair comparison, we refer to each baseline’s performance in their original papers and follow the same evaluation protocols / metrics, which can vary on different benchmarks.\n\n---\n\n**Why test Deep Gaze II / IIE only on CAT2000 and not on Salicon and OSIE?**\n\nA: Deep Gaze II / IIE papers reported results on CAT2000 but not on Salicon validation set / OSIE. We did not re-implement any baseline due to the large number of datasets / tasks we needed to compare. \n\n---\n\n**Why not include SalGAN on the Salicon dataset in Table 3 or Table 6?**\n\nA: There are two versions of Salicon data, Salicon 2017 (http://salicon.net/challenge-2017) and 2015 http://salicon.net/challenge-2015). All results in our table are on the newer Salicon 2017 dataset, in line with the baselines in Table 3, while SalGAN results were obtained on the Salicon 2015 dataset. The 2015 version differs in fixation ground truth, affecting metrics like NSS scores (higher for Salicon 2015 than 2017), so the results on Salicon 2015 are not comparable to Salicon 2017.\n\n---\n\n**Why no SequenceScore for COCO-Search18?**\n\nA: The computation of the SequenceScore requires a standardized superpixel segmentation, which is not available after we communicated with the authors. To avoid inaccurate / unfair comparison, we did not include this metric on this dataset.\n\n---\n\n**FFM [79] results on COCO-Search18 different from the ones in Table 4?**\n\nA: We refer to the results of FFM in Gazeformer [56] paper, which appeared in Table 6 in Appendix. There were some issues regarding the evaluation results on COCO-Search18 due to the misalignment of the ground-truth semantic segmentation map. The authors have updated their results in Appendix, which are the correct version that we cited in Table 4.\n\n---\n\n**Why not compare scanpath prediction performance of ALOHA and baselines on COCO-FreeView?**\n\nA: We have the scanpath results on COCO-FreeView, but as a relatively new dataset, we did not find a good baseline for scanpath prediction on this dataset, so we did not include this dataset in our table. One relevant work is [1]. However, it didn’t report scanpath prediction performance on COCO-FreeView.\n\n[1] Characterizing target-absent human attention. (CVPR'22)\n\n---\n\n**Additional experiments on MIT300 and MIT1003**\n\nA: We have a model that did not include MIT1003 as training and directly tested on MIT300, and the results are suboptimal. However, since many methods on MIT300 benchmark dashboard are trained on MIT1003, so for a fair comparison, we will train a model on MIT1003 and report the results on MIT300.\n\n---\n\n**The experiments do not clearly demonstrate the actual benefits of unifying the attention, rating, and scanpath tasks**\n\nA: The immediate benefit of an unified model is that only one model is needed instead of separate models for each task, leading to easier model serving with better performance. Furthermore, there are strong connections between attention and subjective experiences like aesthetic preference, subjective interpretation, emotional response, or ease of information finding (e.g., often focused attention means easy, while scattered attention means hard). To this end, we collect and will release a new dataset which contains both scores (easiness scores for question answering tasks) and gaze heatmaps (while performing question answering tasks) on digital images (designs, UIs, etc) , and show that the score and entropy of gaze heatmaps are correlated,  to better motivate the unified model. Please see more details in Figure 3 and 4 in the rebuttal PDF.\n\n---\n\n**How are the three predictors chosen for each task?**\n\nA: During training, we specify the task in the text prompt and compute gradients only using the loss of that task (e.g., scoring task if “OUTPUT: score” is in the prompt). During inference, the task info (i.e., “OUTPUT: score”) in the prompt will inform the model which task to perform and we retrieve results from the corresponding predictor.\n\n---\n\n**Differences between the salience map and the importance map?**\n\nA: The two tasks use the same heatmap L2 loss and only differ in data. Note that the two tasks will have different text prompts, namely “OUTPUT: saliency heatmap” for saliency, and “OUTPUT: importance heatmap” for importance.\n\n---\n\n**Invalid output format for scanpath prediction?**\n\nA: We first split the output sequence by the separator (“and” in our case). Then each component should contain two coordinates of x and y. If a <x, y> coordinate is invalid (e.g., only x but no y value), we skip that coordinate. All predicted scanpaths are valid after the models were finetuned.\n\n---\n\n**Why text generation as a scanpath predictor?**\n\nA: Text decoder can generate arbitrary length of tokens (up to max length) and is suitable for scanpath which also has variable length. As the vision-language model is pretrained on coordinate prediction task the decoder learned the notion of coordinate token and performed well for predicting scanpath, which mainly consists of coordinates. As a unified model, we hope the model can generalize to different kinds of sequences, which can be predicted by the text decoder.\n\n---\n\n**How is the model fine-tuned on various datasets, which parts are freezed?**\n\nA: All model weights are finetuned and no parts are frozen. We randomly sampled batches from all the datasets during training (more details in Section 4.2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thorough and constructive feedback. We have addressed each point in the individual responses. We have included some of the common points of discussion as below. Moreover, in the rebuttal pdf, we also included some figures to answer the questions of “Error Analysis” from reviewer FgZM, and “the actual benefits of unifying the attention, rating, and scanpath tasks” from reviewer Fbj6. We will also fix all the formatting issues pointed out by the feedback.

---

**Demographics of annotators [Reviewers FgZM, ew4]**

We check the training data collection processes and include the participant demographics that we have found.

WS-Saliency: “A total of 41 participants (19 females, 22 males; age range 17-23; with normal or corrected-to-normal vision) participated in our data collection.”

Mobile UI: “Thirty participants (12 male, 18 female). [...] The average age was 25.9 (SD=3.95). The participants had normal vision (8) or corrected-to-normal-vision (22). Twenty of the 22 wore glasses and the remaining two wore contact lenses.”

Imp1k: “The data of 43 participants (29 male, most in their 20s and 30s) were used in the resulting analyses.”

FiWI: “11 students (4 males and 7 females) in the age range of 21 to 25 participated in data collection. All participants had normal vision or corrective visual apparatus.”

Based on the available descriptions, we found that participants show good convergence on gender. However, the age distribution is somewhat skewed, likely because most collections were conducted at universities. Datasets like Salicon and Koniq-10K, which utilize crowdsourcing workers for annotations, are expected to have a better balance in terms of age and other attributes. We will add this information to the limitations discussion.

---

**Bias in the data sets or the model [Reviewer ew4]**

The model is trained with multiple public data sets, each of which might have some bias. So it is possible for our model to have also learned such bias from those datasets. Since this paper introduces the first model to unify all the human visual behavior tasks, we focus on getting a model to work for these tasks with good accuracy, and we plan to implement techniques to evaluate and mitigate bias in future iterations of our work. We will however add this discussion in our limitation section along with demographics information discussed above.

---

We look forward to the discussion phase and further improving the paper.

Meta-review: The paper received 2 Borderline Accept, 1 Accept and 1 Reject ratings. While the authors could incorporate more experimental validations, the paper is well motivated and makes a solid contribution in unifying attention prediction. The AC reads the paper, the rebuttal, and the discussions and decides to accept it. Congratulations! It is encouraged that the authors incorporate reviewers' comments into their final version.
Decision: Accept (poster)