The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric
Authors: Daniel Severo
Lucas Theis
Jona Ballé
Venue: ICLR 2024 poster (2024)
Keywords: perceptual metrics
lpips
perceptual embeddings
representation learning
self-supervised learning
FR-IQA
image quality assessment
compression
data-free
ssim
bapps
2afc
jnd
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We show how SOTA perceptual embeddings of the visual system can be constructed at inference-time with no training data or deep neural network features.

Abstract: We show how perceptual embeddings of the visual system can be constructed at inference-time with no training data or deep neural network features. Our perceptual embeddings are solutions to a weighted least squares (WLS) problem, defined at the pixel-level, and solved at inference-time, that can capture global and local image characteristics. The distance in embedding space is used to define a perceptual similary metric which we call \emph{LASI: Linear Autoregressive Similarity Index}. Experiments on full-reference image quality assessment datasets show LASI performs competitively with learned deep feature based methods like LPIPS \citep{zhang2018unreasonable} and PIM \citep{bhardwaj2020unsupervised}, at a similar computational cost to hand-crafted methods such as MS-SSIM \citep{wang2003multiscale}. We found that increasing the dimensionality of the embedding space consistently reduces the WLS loss while increasing performance on perceptual tasks, at the cost of increasing the computational complexity. LASI is fully differentiable, scales cubically with the number of embedding dimensions, and can be parallelized at the pixel-level. A Maximum Differentiation (MAD) competition \citep{wang2008maximum} between LASI and LPIPS shows that both methods are capable of finding failure points for the other, suggesting these metrics can be combined.

THE UNREASONABLE EFFECTIVENESS OF LINEAR PREDICTION AS A PERCEPTUAL METRIC
Daniel Severo" University of Toronto
Lucas Theis Google Deepmind
Johannes Ballé Google Research
Vector Institute for A.I. d. d.severo@mail utoronto.ca
theis@google.c com
jbalLe@google.com
Linear Autoregressive Similarity Index (LASI)
Deep featares (LPIPS)
Embewhlng
Embedding
Training
Kmbedding
Embedding
Solve WLS
Solve WLS
Comparison between method, Linear Autoregressive Similarity Index (LASI), and Learned Perceptual Image Patch Similarity (LPIPS) from Zhang et al. (2018). Our method solves a weighted least squares (WLS) problem to compute perceptual embeddings, inference time, with no prior training or neural networks. LPIPS uses pre-trained features from deep models as image embeddings and trains on human annotated data.
Figure 1:
ABSTRACT
We show how perceptual embeddings of the visual system can be constructed inference-time with no training data or deep neural network features. Our perceptual embeddings are solutions to weighted least squares (WLS) problem, defined at the pixel-level, and solved inference-time, that can capture global and local image characteristics. The distance in embedding space is used to define perceptual similarity metric which we call LASI: Linear Autoregressive Similarity Index. Experiments full-reference image quality assessment datasets show LASI performs competitively with learned deep feature based methods like LPIPS Zhang al., 2018) and PIM (Bhardwaj al.) (020). at similar computational cost to hand-crafted methods such as MS-SSIM (Wang et al. 2003). We found that increasing the dimensionality of the embedding space consistently reduces the WLS loss while increasing performance on perceptual tasks, at the cost of increasing the computational complexity. LASI is fully differentiable, scales cubically with the number of embedding dimensions, and can be parallelized at the pixel-level. A Maximum Differentiation (MAD) competition (Wang Simoncelli] (008) between LASI and LPIPS shows that both methods are capable of finding failure points for the other, suggesting these metrics can be combined. Code: /tttps://ithubb Index
com/ http///eeee////ooo/ Linear Autoregressive
INTRODUCTION
The applicability computer vision in real world applications hinges on how well the loss function aligns with the human visual system. Learning end-to-end solutions for applications such super- resolution and lossy compression (Ballé et al. 2016 2018) requires differentiable similarity metrics
Work done while Google Research

---

that correlate well with how humans perceive change in visual stimuli. Unfortunately, widely used metrics such as PSNR/MSE that measure change at the pixel-level, although differentiable, do not satisfy this criterion.
The failure of pixel-level metrics in capturing perception has prompted the design of similarity metrics at the patch level, inspired by subfield of human psychology known as psychophysics. The most successful one to date is the multi-scal structural similarity metric MS-SSIM (Wang et al.) 2003 2004) which models luminance and contrast perception. Despite these efforts, the complexity of the human visual system remains difficult to model by hand; evidenced by the failures of MS-SSIM in predicting human preferences in standardized image quality assessment (IQA) experiments (Zhang et al. 2018).
To move away from handcrafting similarity metrics the community has shifted towards using deep features from large pre-trained neural networks. For example, the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) metric assumes the L.2 distance between these deep features can capture human perception. In the same work, the authors introduce the Berkeley Adobe Perceptual Putch Similarity (BAPPS) dataset, which has become widely accepted as a benchmark for measuring perceptual alignment of similarity metrics. LPIPS uses deep features as inputs to a smaller neural network model that is trained on the human annotated data available in BAPPS which indicate human preference of certain images over others with respect to perceived similarity. Collecting this data is expensive as it requires large-scale human trials, and the generalization capabilities of metrics beyond this dataset are not well understood (Kumar et al. 2022).
To side-step expensive data collection procedures recent work has attempted to directly learn embed- dings inspired by well known phenomena of the human visual system. For example, the Perceptual Information Metric (PIM) (Bhardwaj et al., 2020) optimizes a mutual information (Cover 1999) based metric and does not use annotated labels. The deep features resulting from this procedure perform competitively with LPIPS on the BAPPS dataset as well as other settings (Bhardwaj et al. 2020). Other methods such as (Wei et al.) 2022) define self-supervised objective where the neural network must predict a label indicating which distortion type, from a predefined set, was used to corrupt the image.
In this work, we put into question the necessity of deep features to define similarity metries aligning with human preference. We take inspiration from recent work in the field of psychology which provide evidence that the visual working memory performs compression of visual stimuli (Bays)
ct al. 2022 Bates& & Jacobs, 2020b, Sims, 2018; Brady et al., 2009 Sims 2016; Bates & Jacobs 2020ap. We employ methods that compress a visual stimuli inference time with pre-training or prior knowledge of the data distribution. Applying this procedure at inference-time means we do not require any expensive labelling procedures, embeddings are learned at the pixel-level, but can capture patch-level semantics by solving a weighted least squares (WLS) problem from a neighborhood surrounding the pixel, a subcomponent of the lossless compression algorithm developed by Meyer & Tischer (00T).
unlabelled data, as in LPIPS PIM. Our
Our Linear Autoregressive Similarity Index (LASI) uses the 1.2 norm of the differences between embeddings of images, averaged over all pixels, to define a perceptual similarity metric. We find that inereasing the neighborhood size, which corresponds to the final embeddings dimensionality, consistently improves the WLS loss as well as performance on the tasks in BAPPS. This is in contrast to learned methods like LPIPS, where performance on perceptual tasks can correlate negatively with the classification performance from which the deep features are taken (Kumar et al.) 2022).
An overview of full-reference image quality assessment (FR-IQA) is provided in Section 2 while Section reviews representative sample of current state-of-the-art FR-IQA algorithms. Computing the embeddings as well as LASI is discussed, and an algorithm is given, in Section 4 LASI is benchmarked against LPIPS, PIM, and A-DISTS (Zhu et al., 2022) across 6 categories of image quality experiments in Section 5 In Section [.3] we employ the Maximum Differentiation (MAD)
competition (Wang& & Simoncelli 2008) to show that LPIPS and LASI can potentially be combined, as one can be used to find failure modes of the other (see the section for a formal definition).
2

---

2 FULL-REFERENCE IMAGE QUALITY ASSESSMENT (FR-IQA)
FR-IQA is an umbrella term for methods and datasets designed to evaluate the quality of a distorted image, relative to the uncorrupted reference, in way that correlates with human perception. Cor- relation is measured through benchmark datasets created by collecting data from psychophysical experiments such as two-alteratiie forced choice (2-AFC) human trials.
In 2-AFC image quality assessment experiments, subjects are forced to decide between two mutually exclusive alternatives relating the perceived quality of images. For example, in Zhang et al. (2018) subjects are shown 3 images, reference and 2 alternatives, and must indicate which of the 2 alternatives they perceive as being more similar to the reference. Just- must-noticeable differences (JND) is another type of 2-AFC experiment where two similar images are shown and subjects must generate a binary label indicating if they perceive the images as being the same or distinct.
The response of subjects in 2-AFC trials are taken to be the ground truth. For example, if images in a JND dataset have different pixel values, but are judged to be the same by all subjects, then a perfect FR-IQA algorithm must also decide they the same (Duanmu et al. (022I). In the case where there is disagreement between subjects on the same pair of images, then the uncertainty is considered inherent to human perception (i.e., aleatoric, not epistemic).
FR-IQA methods largely ignore perceptual uncertainty and instead attempt to learn a distance function between images that assigns small values to perceptually similar images. Algorithms can be categorized into data-free, unsupervised, and supervised, depending on what training data is needed to learn the distance function. Supervised methods require collecting annotated data from psychophysical experiments using human trials (e.g.. LPIPS from Zhang et al. (2018). Unsupervised methods can learn directly from unlabelled data (e.g.. PIM from Bhardwaj et (2020)). ) while data-free methods require no data or training at all (c.g.. MS- MSS-SSIM of Wang et al. (2003) and our method). These methods are trained and evaluated by performing train test splits on benchmark 2-AFC datasets such as the Berkeley Adobe Perceptual Patch Similarity (BAPPS) (Zhang et al. 2018).
3 RELATED WORK
In this section we review closely related literature for data-free and learned (both unsupervised and supervised) full-reference image quality assessment (FR-IQA). For an in-depth survey see Duanmu et al. ) Ding et al. (2021).
Data-free distortion metrics operating at the pixel level such as mean squared error (MSE) are commonly used in lossy compression applications (Cover 1999) but have long been known to correlate poorly with human perception (e.g.. Girod 1993). Patch-level metrics have been shown to correlate better with human judgement on psychophysical tasks. Most notably, the Structural Similarity Index (SSIM), as well as its multi-scale variate MS ASSM,,,, compare high level patch features such as luminance and contrast to define distance between images (Wang al. (004). SSIM is widely used in commercial television applications, and MS-SSIM is a standard metric for assessing performance many computer vision tasks. The method presented in this work is also data-free and outperforms MS-SSIM on benchmark datasets.
Many learned FR-IQA methods are designed mirroring the learned perceptual image patch similarity (LPIPS) method of Zhang et al. (0018), where neural network is trained on some auxiliary task and the intermediate layers are taken as perceptual representations of an input image. An unsupervised distance between images is defined as the L2 norm of the difference between their representations. A supervised distance uses the representations as inputs to a second model that is trained on human annotated data regarding the perceptual quality of the input images (e.g., labels of 2- 2-AFC datasets discussed in Section 2}. Taking representations from neural networks that perform well on their auxiliary task does not guarantee good performance perceptual tasks (Kumar al. 2022), making it difficult to decide which existing models will yield perceptually relevant distance functions. In contrast, for the same experimental setup, the performance of our method on perceptual tasks correlated well with performance on the auxiliary task (see Section 5.1). Self-supervision was used by Madhusudana et al. 2022bb and Wei al. (2022) for unsupervised and supervised IQA. Images are corrupted with pre-defined distortion functions and a neural network is trained with contrastive pairwise loss to predict the distortion type and degree. The
3

---

MSE
21
93
21
45
87
54
810
F11
112
151
1514
81
0.5
180
411
18
151
810
147
221
422
1834
11.5
23, 27.....
Original
N=
=
neighborhood size (N)
16
24
Figure 2: Left: Definition of previous pixels 11,) = (I1..... .II.).................... obeying raster-scan ordering,
and causal neighborhood n, (in yellow) of pixel H; for i = 18. The image has dimensions 5 x 5 1. n; defined as the N = 8 closest pixels to 418 in terms I.1 distance. For example, the distance from pixel 21 318 is while from 12 to 318 it is 2. Pixels in the red region not considered as they come after the 18-th pixel in the ordering. As the neighborhood size N increases, pixels with smaller LI distance are added to n; in the same order they appear in X(,))- Middle: An image of dimension 256 x 256 x 3 and the squared residuals of the prediction with Equation (T) Right: MSE of images reconstructed with Equation (T).
unsupervised distance is defined as discussed previously and ridge regression is used to learn supervised distance function. This method requires training data, while our method requires no training all.
4 METHOD
Here we present our data-free, self-supervised (at inference time) FR-IQA algorithm called Linear Autoregressive Similarity Index (LASI). We make of sub-component of the lossless compression algorithm available in Meyer & Tischer (200T) to define a distance between images, which is described in detail next.
4.1 CONSTRUCTING PERCEPTUAL EMBEDDINGS VIA WEIGHTED LEAST SQUARES
Our method relies on self-supervision (at inference-time) to learn a representation for each pixel that captures global perceptual semanties of the image. The underlying assumption that, given representation vector for some pixel, it must successfully predict the value of other pixels in the image order to capture useful semantics of the image's ' structure. Our method directly on images to compute distance d( ((x,)). y), similar to other data-free methods like 1.2 and MS-SSIM (Wang et al. 2003). We describe this formally next.
Weighted Least Squares Let [Z)....,24 Ix. E R represent a flattened image with height H. width W. number channels C, and & HWC pixels. Our method autoregressive and uses previous pixels X(1.)) = (II,... ... -z;-) 1) to predict the value of the i-th pixel. The number of previous pixels used will be equal to the dimensionality of the embeddings. Therefore, we restrict the algorithm to use a subset N i - 1 of pixels from ((,)). The subset made up of the elements in X(1.1) that are closest 0 in the coordinate space of the image the i-th pixel. We refer this as the causal neighborhood of pixel i, and represent it as a vector n, N See Figure 2 for an example.
For the i-th pixel of value 2;, we find a vector (x())).)) E N that minimizes the weighted least squares objective
(x( arg min
Σ Fi wo]===))"
WERN
where < w I 1 a hyperparameter and Es the Manhattan distance between coordinates of the i-th and j-th pixels in the image (see Figure 2 for example). Concatenating (x(X).)) column-wise yields the perceptual embedding matrix W W(x) R.N of image x.
In Manhattan distance.
4

---

Equation defines (weighted) least squares problem with data points (nn,, extracted from the previous pixels. The weights w decrease as the distance by between coordinates increases, biasing the objective to better predict closer pixels. The value of I; is not used to compute the representation (x.)) but is used in the computation of subsequent representations.
Distance Function The LASI distance between images is defined as the distance between their perceptual embeddings averaged over pixels d)x,y) = Σ [x,,()),s)) w.
Differentiiaility All operations. including solving Equation are diffcrentiable which allows us compute the gradients of d with respect both arguments. Section 5.3 we use differentiability to perform the Maximum Differentiation (MAD) Competition (Wang & Simoncelli 2008) between
our method and LPIPS (Zhang et al. 2018).
Predictions Meyer & Tischer (2001 solve Equation to generate prediction d; (x11)))) n. of the i-th pixel which is then used for lossless compression of the original im- age. Figure shows examples of the squared residual image made up of pixels 2; = (z; - 4;)² for varying sizes of neighborhood size N. In Section 5.1 Figure we show the prediction loss E of has strong correlation with performance on downstream 2-AFC tasks.
4.2 ALGORITHM
Here describe implementation which solves ① in 3 steps. The algorithm is differentiable and most operations can be run in parallel on a GPU. Compute time and memory can be traded-off by, for example, precomputing the rank-one matrices. It is also possible solve Equation ① thrice in parallel, once for each chanmel, and average the results at the expense of some performance on downstream perceptual tasks.
The steps of our method are:
1) Transform For the i-th pixel of value zi, compute rank-one matrix from the neighborhood, as well as another vector equal to the neighborhood scaled by the pixel itself:
A n.n. NNN,
b, r,n, R.N.
(2)
2) Weigh-and-Sum On second pass, for each pixel, compute weighted sum of the rank-one matrices of all previous pixels, weighted by why, where O < w < is a hyperparameter and lg the Manhattan distance between locations of pixels x; and Zy. Perform a similar procedure for vectors
b,:
A,
Σ j=1
60 Aj,
b = Σ by
3) Solve Finally, for each pixel, solve the least-squares problem with coefficients A, and target vector b; by computing the Moore-Penrose pseudo-inverse A of A,,
(x(,)))) = A!b..
(4)
The rank-one matrices A, have dimension x N. In our experiments, we found
12 was
sufficient to perform competitively with unsupervised methods on images of size 64 x 64 x 3. In this regime of small N computing the pseudo-inverse can be done directly using the singular value decomposition (SVD) (Klema & Laub 1980) (we use jax. numpy 1 ninal 1 pinv; Bradbury al. 2018).
Computational Complexity Solving Equation ( requires computing the SVD of A, which has worst case complexity of ((v3) (Klema & Laub 1980), where N is the neighborhood size and embedding dimensionality. This must be done for each pixel but can be parallelized at the expense of an increase in memory, with no loss in performance. Figure shows LASI is faster than PIM. which requires forward-passes in neural network, on the BAPPS dataset.
5

---

BAPPS-2AFC
BAPPS-IND
Metric
Trad.
CNN
SRes.
Deblur.
Color.
Interp.
Trad.
CNN
Majority
88.5
80.1
74.0
76.4
76.0
Ref.
Human
80.8
84.3
73.6
66.1
68.8
68.6
Supervised
LPIPS
79.3
83.7
71.5
61.2
65.6
63.3
51.9
67.9
LPIPS (fts. only)
73.3
83.1
71.7
60.7
65.0
62.7
46.9
67.9
Unsuperv.
A-DISTS
70.8
60.2
61.6
PIM
75.7
82.7
70.3
61.6
63.3
62.6
57.8
60.1
L.2 MS-SSIM
59.9
77.8
64.7
58.2
63.5
55.0
33.7
58.2
Data-free
61.2
79.3
64.6
58.9
57.3
57.3
36.2
63.8
LASI (Ours)
73.8
81,4
70.2
60.5
62.1
62.3
55.9
63.2
% Diff. best Unsuperv.
-2.5
-2.0
-2.1
-1.8
-4.5
-0.6
-3.3
-5.8
% Diff. to Supervised LPIPS
-6.9
-2.7
-1.9
-12
-5.3
-1.6
+9.3
-7.4
% Diff. to MS- MS-SSIM
+20.6
+2.6
+8.7
+2.7
+8.4
+8.7
+54.4
-0.9
Table 1: Results for just- just-notice differences (BAPPS-IND) and two-alternative forced choice (BAPPS-2AFC) experiments (higher better) of the BAPPS dataset (Zhang al. 2018). Numbers in gray are reference values. The first two entries under Ref. are theoretically calculated references. Majority is the highest score that can be achieved in each column while Human represents the average agreement between two randomly selected subjects. LPIPS (supervised) is trained human annotated labels. Unsupervised methods require unlabelled examples for training, while data-free methods do not require training data at all. "% Diff. to best Unsuperv." is the percentage difference between our method and the cherry-pickee Supervised LPIPS is trained the same distortion types present in "Trad." and "CNN" and therefore has an unfair advantage relative to other metrics in these categories (/hang et al. 2018). These values indicated in gray in the row labeled "% Diff. to Supervised LPIPS". See Section and Section for more details.
best performing unsupervised method shown in bold.
EXPERIMENTS
In this section we compare our method against state-of-the-art unsupervised FR-IQA algorithms. Our method is data-free but performs competitively with learned methods experiments of the Berkeley Adobe Perceptual Patch Similarity (BAPPS) dataset (Zhang et al. 2018) (the first row of Figure 4 shows examples from BAPPS), Experiments of Figure 3
indicate the prediction loss and performance on perceptual tasks correlate and improve as the neighborhood size increases.
In Table 1 results for PIM as well as LPIPS numbers for BAPPS-IND are taken from Table 1 of Bhardwaj et al. (2020). For LPIPS, we generously took the best models from Table 5 of Zhang al. (2018) for each 2-AFC category. For our method we used N 12 across all experiments and the decay parameter in Equation TT) was fixed to w 0.8, the same value used for the lossless compression algorithm of Meyer & Tischer (200T). Images in all categories have dimensions 64 x 64 x which is 1536 times larger than the neighborhood size.
The performance consistently improves with larger neighborhood sizes as can be seen from solid line in Figure 3 This suggests the parameter can be used to trade off computational complexity and performance and does not require tuning.
The last row of Table I shows the gap in performance between data-free method and the best performing unsupervised neural network approach. In the worst case, our method scores only 5.8% less while requiring no learning, data collection, expensive neural network models.
5.1 TWO-ALTERNATIVE FORCED CHOICE (BAPPS-2AFC)
This section discusses experiments on the BAPPS 2AFC dataset of Zhang et al. (2018). BAPPS- 2AFC was constructed by collecting data from a 2-AFC psychophysical experiment where subjects are shown reference images (() as well as alternatives x x and must decide which of the two alternatives they perceive as more similar to the reference. The share of subjects that select image
(e) (e)
x over x(( is available in the dataset as (t))

---

18
All
CNN
co
Debtur
Wall time (ms))
LASI (ours)
13
PIM
Prediction Loss Perceptual
Interp:
SRes.
Tred.
1
8
12
16
5
6
7
8
Neighborhood size
neighberhoad
Figure 3: Left: Total time to compute distances for 1000 examples from BAPPS NVIDIA V100. Results are averaged over 100 runs. Right: Correlation of performances on the self-supervised prediction task and perceptual 2-AFC task. Both curves normalized to lie between 0 and 1 for exposition. The prediction loss (dashed line) is calculated by computing the MSE between the original and reconstructed reference image. As the causal neighborhood size N increases, the prediction loss decreases (lower is better) as the performance on perceptual tasks increases (higher is better). The curves on the left plot are computed using all examples in the BAPPS-2AFC dataset (Zhang et al.) 2018), while the right plots are broken down by sub-categories of the same dataset.
The performance of FR-IQA algorithm, defined by a distance function d between images, 2-AFC dataset is measured by
Σ
p (p())a0
a0 (1-2(0) 1-000
ma
where (*) [(xx (0) d(x; (()))
Equality is achieved when the algorithm agrees
with the majority of subjects, i.c a = (p(0)
3l for all examples in the dataset. Human-level
performance is defined as
+ (000)++(1--0)))) (1-D(0)
(6)
which corresponds to the average probability of two randomly chosen subjects agreeing. Majority and human-level performance scores for our 2-AFC experiments are shown in the first rows of Table
Across all categories method scored competitively with the best performing unsupervised method, with the largest gap being 4.5% in the "Colorization" category. Our method outperforms MS-SSIM (Wang al. (003) on all 2-AFC categories, most notably in "Traditional" where the improvement is 20.6%, and provides perceptual embeddings (Equation I for use in downstream computer vision tasks. We highlight the overall improvements with respect MS-SSIM, in the last row of Table
Correlation with Self-Supervised Task In an empirical study, Kumar et al. (202)) investigate deep features from better performing classifiers achieve better perceptual scores on the BAPPS dataset. Surprisingly, their results suggest the correlation can be negative: more accurate models produce embeddings that capture less meaningful perceptual semantics, performing poorly on 2-AFC and JND tasks when used together with LPIPS. We perform similar study with our method on the prediction task defined in Meyer & Tischer (2001) as a function of neighborhood size. For each reference image ((),, we compute the prediction p(O), composed of pixels p((() w.(r).))) n;, and
report the residual 1 Ei= (r( +f(0)² alongside the score on the 2-AFC task. Results are shown in Figure 3 The performance on the prediction and perceptual tasks show strong correlation, and improve consistently, across all 2-AFC categories.
5.2 JUST-NOTICFABLE DIFFERENCES (BAPPS-JND)
In Table I we compare our data-free method to recent unsupervised methods on the JND subset the BAPPS dataset. The BAPPS-IND dataset ((x(), (() (p))) was created by showing two images
[[] = 1 iff statement A is truc.
[ rounds the nearest integer
7

---

LASI loss fixed
LPIPS loes fixed
between LASI (ours) and LPIPS (Zhang al. 2018). The first 10 images from the "Interp." category of BAPPS-JND are shown in the first row labelled Gaussian noise was added to to create images i shown in the second row. Images in the same column of the middle rows in "LASI loss fixed" are equidistant to their corresponding references
Figure 4: Results from the MAD competition (Wang & Simoncelli 2008
measured by LASL. The same is true for the bottom rows under LPIPS. These examples constitute failure points of LASI and LPIPS, as images the bottom rows of each box (x())) are perceptually closer to their references r, yet have the same distance the distorted images the top rows (X
x())))()))))))) to group of subjects where the latter is the former but corrupted by one of two different distortion types (identified by the last 2 columns Table Subjects must indicate if they perceive the images as being the same or not. The share of subjects that judged the images as being the same,
p()), is available in the dataset but not the individual responses. See (Zhang et al. 2018 for more details regarding the images as well distortion types.
As described in Section 2 an FR-IQA algorithm in the context of a JND task will attempt to output binary response that correlates with ( This defines binary classification task where the distance defined the FR-IQA algorithm must be thresholded to yield decision and precision/recall can be traded-off by varying the threshold value (Bishop & Nasrabadi] (0066). We evaluate the JND experiment with area-under-the-cuuve score known mean average precision (mAP), following Zhang et al. 20188) Bhardwaj et al. 2020)).
Results are shown in the last columns Table For CNN-based distortions our method scores similarly to MS-SSIM (63.8% vs 63.2%). PIM loses to our method (60.1% vs 63 63.2%) while LPIPS outperforms it only by 5.8%.
Similar to the 2-AFC experiments, the subcategory with the largest gap between data-free and unsupervised methods is "Traditional". The gap drastically reduced by our method by raising the highest score from 36.2% (achieved by MS-SSIM) to 55.9%, which is within 3.3% of the best scoring unsupervised method (PIM). this same subcategory our method significantly outperforms LPIPS (55.9% vs 46.9%).
5.3
MAXIMUM DIFFERENTIATION (MAD) COMPETITION
MAD competition (Wang & Simoncelli 2008) is a technique to discover failure modes in the perceptual space of differentiable similarity metrics. Failure modes image pairs (x1, X2) where
8

---

one image is clearly closer to reference upon inspection, but the metric assigns similar distances d(r, x1) ss d(r, x2).
We now describe the MAD competition outlined in Wang & Simoncelli (2008) that uses K steps of projected gradient descent to find a failure mode (Xmax x(X) in d. First, the reference r corrupted with noise yielding F which acts the starting point (Xmxx: X
(0), (F, for optimization. At the
i-th step, the image Xmax is updated using a gradient step in the direction that maximizes it's 1.2 distance to the uncorrupted reference r. However, before updating, the gradient is projected into the space orthogonal to V
d(r, (1) This projection step guarantees that the distance to the reference Xusa
does not change significantly, i.e.. d(r, Xmmak) d(r, Xmmxx)), The same procedure is done for X (K) but the gradient step is taken the opposite direction (minimizing). In practice an extra correction step is required as the projected gradient will be tangent to the set of equidistant images (i.e., the level set). It is common to replace L2 distance with another similarity metric as a way to contrast failure modes and possibly discover ways to combine models (Wang & Simoncelli 2008).
We performed the MAD competition between LPIPS and LASI. Qualitative results are shown in Figure 4 'he neighborhood size of LASI was held fixed at N 16 while deep features from VGG (Simonyan & Zisserman 2014) were used for LPIPS as in Zhang et al. (2018). Images are parameterized by an unconstrained tensor which is then passed through sigmoid function yield an image in the RGB space. Gaussian noise was added in the parameter space (i.e., before the sigmoid applied) to generate the corrupted reference to guarantee images are valid RGB images.
Results indicate LASI can find failure points for LPIPS and vice-versa. Each metric fails in different ways. Image (K) shows artifacts resembling the structure of the causal neighborhood for LASI while artifacts for LPIPS are smoother.
The difference in failure modes suggests LASI and LPIPS be combined. Merging these models is non-trivial as the embedding dimensions differ in size making difficult to perform simple aggregation techniques at the embedding level. We leave this as future work.
CONCLUSION
In this work show how perceptual embeddings can be constructed at inference time with training data or deep neural network features. Our Linear Autorcgressive Similarity Index (LASI) metric performs competitively with learned methods such LPIPS (Zhang al. (018) and PIM (Bhardwaj et al. 2020) on benchmark psychophysical datasets, and outperforms other untrained methods like MS-SSIN (Wang al. 2003
LASI solves a weighted least squares problem at inference time to create embeddings that capture meaningful semanties of the human visual system. Evidence shows increasing the embedding dimensionality improves the overall downstream performance on the tasks present in the BAPPS dataset (Zhang et al. 2018). while improving the WLS loss. This is in strong contrast learned methods like LPIPS, where the classification performance of deep networks can correlate negatively with perception (Kumar et al. 2022).
There are many candidate hypotheses for the unreasonable effectiveness of LASI in FR-IQA, of which we discuss two. First, it unclear how the performance of an algorithm on BAPPS generalizes to other tasks and datasets; warranting a discussion if BAPPS is indeed valid benchmark for FR-IQA beyond small image patches. Alternatively, it is possible the performance of LPIPS and LASI are due to different reasons. While LPIPS embeddings are constructed by indirectly compressing data samples during training, LASI embeddings are tasked with compressing a specific image.
We conclude with myriad of open directions explore. One such direction is investigate if LASI embeddings, L... the solutions to the WLS problem, have useful semantics in computer vision beyond perceptual tasks. The choice of using WLS was inspired by Meyer & Tischer (2001) who use it to perform lossless compression of grayscale images, but there are other small-scale regression tasks that could be used.

---

REFERENCES
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. arXiv preprint arXiv:1611.0100 2016.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint arXiv: arXiv:/802.0m 1802.01436, 2018.
Christopher Bates and Robert Jacobs. Optimal attentional allocation in the presence of capacity constraints visual search. 2020a.
Christopher Bates and Robert A Jacobs. Efficient data compression in perception and perceptual memory. Psychological review, 127(5):891, 2020b.
Paul Bays, Sebastian Schneegans, Wei Ji Ma, and Timothy Brady. Representation and computation in working memory. 2022.
Sangnie Bhardwaj. Ian Fischer, Johannes Ballé, and Troy Chinen. An unsupervised information- theoretic perceptual quality metric. Advances in Neural Information Processing Systems, 33:13-24, 2020.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milng and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http://gitttin ccm./ooglee/aa
Timothy F Brady, Talia Konkle, and George Alvarez. Compression in visual working memory: using statistical regularities to form more efficient memory representations. Journal of Experimental Psychology: General, 138(4):487, 2009.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Keyan Ding, Kede Ma. Shiqi Wang, and Ecro P Simoncelli. Comparison of full-reference image quality models for optimization of image processing systems. International Journal of Computer Vision, 12981228- 2021.
Zhengfang Duanmu, Wentao Liu, Zhongling Wang, and Zhou Wang. Quantifying visual image quality: A bayesian view. Annual Review of Vision Science, 7:437-464, 2021.
Bernd Girod, What's wrong with mean squared error?, pp. 207-220. MIT Press, 1993. ISBN 0-262-23171-9.
Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications. IEEE Transactions on automatic control, 25(2):164-176, 1980.
Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin Dogus Cubuk. Do better imagenet classifiers assess perceptual similarity better? Transactions of Machine Learning Research, 2022.
Pavan C Madhusudana, Neil Birkbeck, Yilin Wang, Balu Adsumilli, and Alan C Bovik. Conviqt: Contrastive video quality estimator. arXiv preprint arXiv:2206. 14713, 2022a.
Pavan C Madhusudana, Neil Birkbeck, Yilin Wang, Balu Adsumilli, and Alan C Bovik. Image quality assessment using contrastive learning. IEEE Transactions on Image Processing. 31:4149-41 4161, 2022b.
Bernd Meyer and Peter E Tischer. Glicbawls-grey level image compression by adaptive weighted least squares. In Data Compression Conference, volume 503, 2001.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 140.::1409.55.0 1556, 2014.
Chris R Sims. Rate-distortion theory and human perception. Cognition, 152:181-198, 2016.
10

---

Chris R Sims. Efficient coding explains the universal law of generalization in human perception. Science, 360(6389):652-655) 656, 2018.
Zhou Wang and Bero P Simoncelli. Maximum differentiation (mad) competition: A methodology for comparing computational models of perceptual quantities. Journal of Vision, 8(12):8-8, 2008.
Zhou Wang, Eero Simoncelli, and Alan C Bovik, Multiscale structural similarity for image quality assessment. In The S'hretySeventt Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402, Ieee, 2003.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
Xuekai Wei, Jing Li, Mingliang Zhou, and Xianmin Wang. Contrastive distortion-level learning- based no-reference image-quality assessment. International Journal of Intelligent Systems, 37(11): 8730-8746, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Hanwei Zhu, Baoliang Chen, Lingyu Zhu, and Shiqi Wang. From distance to dependency: A paradigm shift of full-reference image quality assessment. arXiv preprint arXiv:2211 1.04927, 2022.


Reviews:
[{"review_id": "a66I2On7yD", "reviewer": "Reviewer_WqhQ", "summary": "The paper focuses on generating perceptual embedding of an image, without using any training data. The paper also introduces and proposes a distance metric called LASI. The author goes on to compare the proposed method's performance with existing methods such as LPIPS and others.", "questions": "1. In the results presented in Table:1, why does MS-SSIM outdo the author's proposed method for the BAPPS-JND task.. whereas it outperforms it for the BAPPS-2AFC task?\n2. As mentioned in the weakness section, it is imperative that authors present more details qualitative and quantitative about the failure cases seen using the LASI method proposed in the paper.\n3. In Section 5.3: \"....Results indicate LASI can find failure points for LPIPS and vice-versa......\" Can authors elaborate on this point ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper looks to solve an important problem in the domain of computer vision which is to qualify the quality of embedding generated which matches with its perceptual quality, without using any training dataset. \n- Conducts evaluation on the BAPPS dataset with other metrics such as LPIPS, PIM, and MS-SSIM.\n- Achieves comparative and better results in some cases compared to current state-of-art methods.\n- A good amount of side experiment details are shown to better verify the claims presented in the paper.\n-The paper for the most part of it well organized without any obvious typos and a writing structure that is easy to follow.", "weaknesses": "- There is minimal discussion about the failure cases using the proposed method. Would be great to have some qualitative results and the probable reason we are seeing the results as we see it.\n- Authors fails to discuss adequately why in some cases other metrics (such as PIM) excel compared to the LASI metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2yPbZ9RNYT", "reviewer": "Reviewer_8dYU", "summary": "The paper introduces a new perceptual metric that requires no training data nor DNN features. In particular, taking inspiration from psychology finding that visual working memory compresses visual stimuli, the proposed method, named Linear Autoregressive Similarity Index (LASI), compresses a visual stimuli during inference.", "questions": "Written in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a new perceptual metric that requires no training data or DNN features.\n\n- The paper is clearly written and easy to follow.", "weaknesses": "- The paper claims to have competitive performance, compared with LPIPS method. But, 11% difference in SR in Table 1 seems rather high.\n\n- The paper claims that the advantage of the proposed method is that it requires no training data or DNN features. But, the method requires computations at inference time. Considering that once training is done, DNN-feature based methods do not require much of extra computations (hence the cost is amortized) while the proposed method requires extra computations, is requiring training data or DNN features really a bad thing?\n\n- All experiments are performed with 64x64 resolution, which seem rather small. Is the proposed method effective for larger images, compared to other works? Is the choice of $N$ the size of neighborhood robust against the image size? It seems $N$ needs to be tuned for each image resolution, which can be critical, since images can come in at various sizes.\n\n- The paper claims that the method can be combined with LPIPS but I cannot find the experimental results on this. Does the combination actually bring improvements?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "topUyGblnL", "reviewer": "Reviewer_QSZk", "summary": "This work focuses on solving the weighted least squares problem for image quality distance in full-reference image quality assessment. Inspired by lossless compression, this work proposes the Autoregressive Similarity Index (LASI), which obtains perceptual embeddings by calculating a weighted sum of the causal neighborhood subset of pixel values to predict the current pixel value. The performance of the data-free LASI is comparable to that of supervised methods such as LPIPS and unsupervised methods like PIM.", "questions": "1. The sentence in the section 4.1 ‘’Our method relies on self-supervision (at inference-time) to learn a representation for each pixel that captures global perceptual semantics of the image. The underlying assumption is that, given a representation vector for some pixel, it must successfully predict the value of other pixels in the image in order to capture useful semantics of the image’s structure.” But the relationship of the perceptual embedding in FR-IQA and the semantic extraction is confused in this work. Because many previous methods also use semantic features extracted by pre-trained models on high-level classification tasks to calculate perceptual distance. Is there any difference between this semantic feature and the embedding derived by Eq 1 in this work?\n2. Since the LASI is designed on the semantic extraction in the images’ structure, so the correlation of the prediction task and perceptual task is good, which is a little obvious.  \n4. Perhaps LASI can only measure perceptual distance at patch level, and will it be useful for high-resolution images? \n5. An ablation study is missing for the causal neighborhood and non-causal neighborhood, as semantic understanding relies on contextual relationships around pixels or regions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe LASI metric is designed in a simple yet solid manner. Full-reference Image Quality Assessment (FR IQA) can be approached from the perspective of lossless compression and the semantic information within the neighborhood of pixels.\n2.\tThe experiments of JND，2AFC and MAD are very detailed and the explanations are very clear.", "weaknesses": "1. The experimental dataset consists only BAPPS. \n2. An ablation study is missing for the causal neighborhood and non-causal neighborhood, as semantic understanding relies on contextual relationships around pixels or regions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)