Evaluating Perceptual Distances Models by Fitting Binomial Distributions to Two-Alternative Forced Choice Data

## Abstract

Abstract The abstract should summarize the contents of the paper.
LNCS guidelines indicate it should be at least 70 and at most 150 words.
Please include keywords as in the example below.
This is required for papers in LNCS proceedings.

###### Keywords:

## 1 Introduction

This document serves as an example submission to ECCV \ECCVyear.
It illustrates the format authors must follow when submitting a paper.
At the same time, it gives details on various aspects of paper submission, including preservation of anonymity and how to deal with dual submissions.
We advise authors to read this document carefully.

The document is based on Springer LNCS instructions as well as on ECCV policies, as established over the years.

## 2 Initial Submission

### 2.1 Language

All manuscripts must be in English.

### 2.2 Template

Papers must be prepared with the official LNCS style from Springer.
This applies to both review and camera-ready versions.
Springer requires manuscripts to be prepared in LaTeX (strongly encouraged) or Microsoft Word.

Authors preparing their paper with LaTeX must use the template provided by ECCV, which is based on the corresponding Springer class file llncs.cls but includes line numbers for review (\crefsec:line-numbering) and properly anonymizes the paper for review (as in this example document).
Authors who – for whatever reason – cannot use LaTeX can alternatively use the official LNCS Word template from Springer.
However, it is the authors’ responsibility to ensure that the resulting PDF file is consistent with this example paper and follows it as closely as possible (\ie, includes line numbers, is properly anonymized, \etc).

We would like to stress that the class/style files and the template must not be manipulated and that the guidelines regarding font sizes and format must be adhered to.
For example, please refrain from using any LaTeX or TeX command that modifies the layout settings of the template (\eg, \textheight, \vspace, \baselinestretch, \etc).
Such manual layout adjustments should be limited to very exceptional cases.
This is to ensure that the end product is as homogeneous as possible.

Papers that differ significantly from the required style may be rejected without review.

#### 2.2.1 Fonts.

Springer’s templates for LaTeX are based on CMR, and the XML templates for Word are based on Times.
We ask you to use the font according to the template used for your papers.
Specifically, please refrain from using Times when preparing your paper with LaTeX.
Using a different font can be interpreted as purposely circumventing the length limitations and may lead to rejection without review.

### 2.3 Paper Length

Papers submitted for review must be complete.
The length should match that intended for final publication.
Papers accepted for the conference will be allocated 14 pages (plus additional pages for references) in the proceedings.
Note that the allocated 14 pages do not include the references.
The reason for this policy is that we do not want authors to omit references for sake of space limitations.

Papers with more than 14 pages (excluding references) will be rejected without review.
This includes papers where the margins and formatting including the font are deemed to have been significantly altered from those laid down by this style guide.

The reason such papers will not be reviewed is that there is no provision for supervised revisions of manuscripts.
The reviewing process cannot determine the suitability of the paper for presentation in 14 pages if it is reviewed in 16.

### 2.4 Paper ID

It is imperative that the paper ID is mentioned on each page of the manuscript of the review version.
Enter your paper ID in the appropriate place in the LaTeX template (see %TODO REVIEW).
The paper ID is a number automatically assigned to your submission when registering your paper submission on the submission site.

### 2.5 Line Numbering

All lines should be numbered in the initial submission, as in this example document.
This makes reviewing more efficient, because reviewers can refer to a line on a page.
Line numbering is removed in the camera-ready version.

## 3 Policies

The policies governing the review process of ECCV \ECCVyear are detailed on the conference webpage (see [https://eccv2024.ecva.net/](https://eccv2024.ecva.net/)), such as regarding confidentiality, dual submissions, double-blind reviewing, plagiarism, and more.
By submitting a paper to ECCV, the authors acknowledge that they have read the submission policies and that the submission follows the rules set forth therein.

Accepted papers will be published in LNCS proceedings with Springer.
To that end, authors must follow the Springer Nature Code of Conduct for Authors (see [https://www.springernature.com/gp/authors/book-authors-code-of-conduct](https://www.springernature.com/gp/authors/book-authors-code-of-conduct)).
We would like to draw particular attention to the policies regarding figures and illustrations, as well as ethical approval and informed consent, which are also reproduced on the ECCV website.

## 4 Preserving Anonymity

ECCV reviewing is double blind, in that authors do not know the names of the area chair/reviewers of their papers, and the area chairs/reviewers cannot, beyond reasonable doubt, infer the names of the authors from the submission and the additional material.
You must not identify the authors nor provide links to websites that identify the authors, neither in the paper nor in the supplemental material.
If you need to cite a different paper of yours that is being submitted concurrently to ECCV, the authors should *(1)* cite these papers anonymously, *(2)* argue in the body of your paper why your ECCV paper is non trivially different from these concurrent submissions, and *(3)* include anonymized versions of those papers in the supplemental material.
Violation of any of these guidelines may lead to rejection without review.

Many authors misunderstand the concept of anonymizing for blind review.
Blind review does not mean that one must remove citations to one’s own work—in fact it is often impossible to review a paper unless the previous citations are known and available.

Blind review means that you do not use the words “my” or “our” when citing previous work.
That is all.
(But see below for tech reports.)

Saying “this builds on the work of Lucy Smith [1]” does not say that you are Lucy Smith;
it says that you are building on her work.
If you are Smith and Jones, do not say “as we show in [7]”, say “as Smith and Jones show in [7]” and at the end of the paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:

> An analysis of the frobnicatable foo filter. In this paper we present a performance analysis of our previous paper [1], and show it to be inferior to all previously known methods. Why the previous paper was accepted without this analysis is beyond me. [1] Removed for blind review

An example of an acceptable paper:

> An analysis of the frobnicatable foo filter. In this paper we present a performance analysis of the paper of Smith \etal[1], and show it to be inferior to all previously known methods. Why the previous paper was accepted without this analysis is beyond me. [1] Smith, L and Jones, C. “The frobnicatable foo filter, a fundamental contribution to human knowledge”. Nature 381(12), 1-213.

If you are making a submission to another conference at the same time, which covers similar or overlapping material, you may need to refer to that submission in order to explain the differences, just as you would if you had previously published related work.
In such cases, include the anonymized parallel submission [1] as supplemental material and cite it as

> [1] Authors. “The frobnicatable foo filter”, ECCV \ECCVyearSubmission ID 00324, Supplied as supplemental material 00324.pdf.

Finally, you may feel you need to tell the reader that more details can be found elsewhere, and refer them to a technical report.
For conference submissions, the paper must stand on its own, and not *require* the reviewer to go to a tech report for further details.
Thus, you may say in the body of the paper “further details may be found in [Authors14b]”.
Then submit the tech report as supplemental material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem, which you tested using a tool that is widely known to be restricted to a single institution.
For example, let’s say it’s 1969, you have solved a key problem on the Apollo lander, and you believe that the ECCV audience would like to hear about your
solution.
The work is a development of your celebrated 1968 paper entitled “Zero-g frobnication: How being the only people in the world with access to the Apollo lander source code makes us a wow at parties”, by Zeus \etal.

You can handle this paper like any other.
Do not write “We show how to improve our previous work [Anonymous, 1968].
This time we tested the algorithm on a lunar lander [name of lander removed for blind review]”.
That would be silly, and would immediately identify the authors.
Instead write the following:

> We describe a system for zero-g frobnication. This system is new because it handles the following cases: A, B. Previous systems [Zeus et al. 1968] did not handle case B properly. Ours handles it by including a foo term in the bar integral. … The proposed system was integrated with the Apollo lunar lander, and went all the way to the moon, don’t you know. It displayed the following behaviours, which show how well we solved cases A and B: …

As you can see, the above text follows standard scientific convention, reads better than the first version, and does not explicitly name you as the authors.
A reviewer might think it likely that the new paper was written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been contracted to solve problem B.

For sake of anonymity, authors must omit acknowledgements in the review copy.
They can be added later when you prepare the final copy.

## 5 Formatting Guidelines

### 5.1 Headings

Headings should be capitalized (\ie, nouns, verbs, and all other words except articles, prepositions, and conjunctions should be set with an initial capital) and should, with the exception of the title, be aligned to the left.
Only the first two levels of section headings should be numbered, as shown in \creftab:headings.
The respective font sizes are also given in \creftab:headings.
Kindly refrain from using “0” when numbering your section headings.
Words joined by a hyphen are subject to a special rule.
If the first word can stand alone, the second word should be capitalized.

**Table 1: Font sizes of headings. Table captions should always be positioned *above* the tables.**
| Heading level | Example | Font size and style |
| --- | --- | --- |
| Title (centered) | Lecture Notes … | 14 point, bold |
| 1st-level heading | 1 Introduction | 12 point, bold |
| 2nd-level heading | 2.1 Printing Area | 10 point, bold |
| 3rd-level heading | Headings. Text follows … | 10 point, bold |
| 4th-level heading | Remark. Text follows … | 10 point, italic |

Here are some examples of headings:
“Criteria to Disprove Context-Freeness of Collage Languages”, “On Correcting the Intrusion of Tracing Non-deterministic Programs by Software”, “A User-Friendly and Extendable Data Distribution System”, “Multi-flip Networks: Parallelizing GenSAT”, “Self-determinations of Man”.

### 5.2 Figures

For LaTeX users, we recommend integrating figures in your paper using the package graphicx.

It is essential that all illustrations are clear and legible.
Vector graphics (rather than rasterized images) should be used for diagrams and schemas whenever possible.
Please check that the lines in line drawings are not interrupted and have a constant width.
Line drawings are to have a resolution of at least 800 dpi (preferably 1200 dpi).
Grids and details within figures must be clearly legible and may not be written one on top of the other.
The lettering in figures should not use font sizes smaller than 6 pt ($\sim$2 mm character height).

Figures should be numbered and should have a caption, which should always be positioned *under* the figures, in contrast to the caption belonging to a table, which should always appear *above* the table.
Figures and Tables should be cross-referred in the text.

If they are short, they are centered between the margins (\cf\creffig:short).
Longer captions, covering more than one line, are justified (\creffig:example shows an example).
Captions that do not constitute a full sentence, do not have a period.

Figure: Figure 1: One kernel at $x_{s}$ (*dotted kernel*) or two kernels at $x_{i}$ and $x_{j}$ (*left and right*) lead to the same summed estimate at $x_{s}$. This shows a figure consisting of different types of lines. Elements of the figure described in the caption should be set in italics, in parentheses, as shown in this sample caption. The last sentence of a figure caption should generally end with a full stop, except when the caption is not a full sentence.
Refer to caption: x1.png

Figure: Figure 2: An example of a subfigure

If possible (\eg, if you use LaTeX) please define figures as floating objects.
LaTeX users, please avoid using the location parameter “h” for “here”.
If you have to insert a pagebreak before a figure, please ensure that the previous page is completely filled.

### 5.3 Formulas

Displayed equations or formulas are centered and set on a separate line (with an extra line or half line space above and below).
Equations should be numbered for reference.
The numbers should be consecutive within the contribution, with numbers enclosed in parentheses and set on the right margin.
For example,

$$ $\displaystyle\psi(u)$ $\displaystyle=\int_{0}^{T}\left[\frac{1}{2}\left(\Lambda_{0}^{-1}u,u\right)+N^ {\ast}(-u)\right]\text{d}t\;$ (1) $\displaystyle=0$ (2) $$

and

$$ $E=m\cdot c^{2}.$ (3) $$

Please do not include section counters in the numbering.

Numbering equations makes reviewing more efficient, because reviewers can refer to a line on a page.
It is important for readers to be able to refer to any particular equation.
Just because you did not refer to it in the text does not mean some future reader might not need to refer to it.
It is cumbersome to have to use circumlocutions like “the equation second from the top of page 3”.
(Note that the ruler will not be present in the final copy, so is not an alternative to equation numbers).
All authors will benefit from reading Mermin’s description of how to write mathematics:
[https://doi.org/10.1063/1.2811173](https://doi.org/10.1063/1.2811173).

Equations should never be in color and should be punctuated in the same way as ordinary text.
They should not be pasted in as figures.

#### 5.3.1 Lemmas, Propositions, and Theorems.

The numbers accorded to lemmas, propositions, and theorems, \etcshould appear in consecutive order, starting with Lemma 1.
Please do not include section counters in the numbering like “Theorem 1.1”.

### 5.4 Footnotes.

The superscript numeral used to refer to a footnote appears in the text either directly after the word to be discussed or – in relation to a phrase or a sentence – following the punctuation mark (comma, semicolon, or period).(^1^11The footnote numeral is set flush left and the text follows with the usual word spacing. Second and subsequent lines are indented.)
For remarks pertaining to the title or the authors’ names, in the header of a paper, symbols should be used instead of a number.
Please note that no footnotes may be included in the abstract.

### 5.5 Cross References

For the benefit of author(s) and readers, please use the

command for cross-referencing to figures, tables, equations, or sections.
This will automatically insert the appropriate label alongside the cross reference as in this example:

> To see how our method outperforms previous work, please see \creffig:example and \creftab:headings. It is also possible to refer to multiple targets as once, \eg to \creffig:example,fig:short-a. You may also return to \crefsec:intro or look at \crefeq:important.

If you do not wish to abbreviate the label, for example at the beginning of the sentence, you can use the

command. Here is an example:

> \Cref fig:example is also quite important.

### 5.6 Program Code

Program listings or program commands in the text are normally set in typewriter font (\eg, printf("Hello world!\n");).

### 5.7 Citations

Arabic numbers are used for citation, which is sequential either by order of citation or by alphabetical order of the references, depending on which sequence is used in the list of references.
The reference numbers are given in brackets and are not superscript.
Please observe the following guidelines:

- •
Single citation: [Authors14]
- •
Multiple citation: [Alpher02, Alpher03, Alpher05, Authors14b, Authors14].
The numbers should be listed in numerical order.
If you use the template as advised, this will be taken care of automatically.
- •
If an author’s name is used in the text: Alpher [Alpher02] was the first …

Please write all references using the Latin alphabet. If the title of the book you are referring to is, \eg, in Russian or Chinese, then please write (in Russian) or (in Chinese) at the end of the transcript or translation of the title.
All references cited in the text should be in the list of references and vice versa.

References should be formatted with the official LNCS reference style.
The LaTeX template already takes care of that through the use of the splncs04.bst BibTeX style file.
Springer strongly encourages you to include DOIs (Digital Object Identifiers) in your references (\cf[ECCV2022]).
The DOI is a unique code allotted by the publisher to each online paper or journal article.
It provides a stable way of finding published papers and their metadata.
The insertion of DOIs increases the overall length of the references section, but this should not concern you as the reference section is not counted toward the page limit.

### 5.8 Miscellaneous

Compare the following:

| $conf_a$ | $\qquad conf_{a}$ |
| --- | --- |
| $\mathit{conf}_a$ | $\qquad\mathit{conf}_{a}$ |

See The TeXbook, p. 165.

The space after \eg, meaning “for example”, should not be a sentence-ending space.
So \egis correct, *e.g.* is not.
The provided \eg macro takes care of this.

When citing a multi-author paper, you may save space by using “et alia”,
shortened to “\etal” (not “et. al.” as “et ” is a complete word).
If you use the \etal macro provided, then you need not worry about double periods when used at the end of a sentence as in Alpher \etal.
However, use it only when there are three or more authors.
Thus, the following is correct:
“Frobnication has been trendy lately.
It was introduced by Alpher [Alpher02], and subsequently developed by
Alpher and Fotheringham-Smythe [Alpher03], and Alpher \etal [Alpher04].”

This is incorrect: “… subsequently developed by Alpher \etal [Alpher03] …” because reference [Alpher03] has just two authors.

## 6 Camera-Ready Manuscript Preparation

This information will follow after paper decisions have been announced.

## 7 Conclusion

The paper ends with a conclusion.

Page 0 of the manuscript.
Page 0 of the manuscript.
Page 0 of the manuscript.
Page 0 of the manuscript.
Page 0 of the manuscript. This is the last page.

Now we have reached the maximum length of an ECCV \ECCVyear submission (excluding references).
References should start immediately after the main text, but can continue past p. 14 if needed.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: Please see the weaknesses.
soundness: 1
strengths: The issue of modeling 2AFC experimental data is an interesting topic.

The proposed method enjoys simplicity.
summary: The paper proposes a probabilistic approach to model the subjective judgment obtained from 2AFC experiments about visual quality using a binomial distribution. The goal is to evaluate perceptual distance models predicting visual quality with respect to the 2AFC experimental data. Three metrics are derived for evaluation and the results on BAPPS are shown.
weaknesses: 1. The paper is not easy to follow, mainly due to lack of definition, usage inconsistency, and unusual usage of notations and symbols. For instance, $t$ in $x_0(t)$ is not defined, $T$ is not defined, $d_v$ is not defined, $N$ is not defined, $P$ and $p$ are used without clear distinction, $j$ in Fig. 1 is not defined, the arguments are not consistent in $P$ (sometimes two arguments, sometimes three arguments, sometimes no argument, arguments do not match), {$(d_0, d_1), n=j$} doesn't make much sense, etc. In addition, some sentences are not easily understood, e.g., first sentence of Section 2.1, last sentence in page 6, what does 'preferred smoothness' mean and how can it be determined, etc.

2. Although the proposed method provides (maybe) new ways to evaluate the predicted quality scores by perceptual distance models, it is not experimentally shown whether the proposed method has any advantage over existing approaches including checking whether the image preferred by a majority of human raters is predicted to have a higher quality score by a model, comparing the predicted scores to the MOS estimated using the Thurstone or Bradley-Terry model on the subjective data, etc. In fact, in Table 2, the proposed method shows very similar results to the distance-only case, so the proposed method doesn't seem advantageous. 

3. Section 4 presents experimental results, but they do not draw much meaningful conclusions. Above all, they do not prove that the proposed method is correct. Table 1 only shows that PIM, LPIPS, and DISTS are better than Euclidean, NLPD, and SSIM, which would be obvious without relying on the proposed method. Fig. 3 shows some difference among distance models, but not much more than that; it is said that humans perceive sharper differences when images are far from their reference, which I wouldn't agree. Fig. 4 is an interesting result, but it is not clear whether it can be useful practically. Fig. 5 shows the robustness of the proposed method against the choice of hyperparameters, but it does not prove that the results are correct.

4. It is said that the proposed method is free from assumptions unlike existing methods. However, the proposed method also imposes certain assumptions. For instance, it is assumed that the subjective preference strictly follows the binomial distribution without any noise, the probability in the binomial distribution depends only on the quality distances and nothing else, etc. Justification on these issues would be needed.

5. Even though the assumptions made in the proposed method are correct, the proposed model is fitted through maximum likelihood estimation, where fitting error wouldn't be zero. How accurate is the fitting in the experiments? The fitting error could be from from the inadequacy of the model, the multimodality of the likelihood, the violation of convexity of the optimization problem, or something else. Analysis on this would be needed to justify the adequacy of the proposed model.

6. Sampling for $\hat{n}$ is performed to obtain an 'upper bound' of AJ(?) and 'minimum possible' NLL. However, in Table 1, AJ with $\hat{n}$ doesn't act as an upper bound and NLL with $\hat{n}$ is not a lower bound. The meaning of the results of $\hat{n}$ should be interpreted appropriately. The same applies to Table 3.

(minor) Line 105, based off -> based on

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The authors should compare your method with state-of-the-art perceptual distance models.
2. It would be beneficial to have a more detailed and step-by-step explanation of the algorithm.
soundness: 2
strengths: 1. The paper introduces a novel method for evaluating perceptual distance models using binomial distributions to model decision-making in two-alternative forced choice (2AFC) experiments, which is an innovative attempt in the field of image quality assessment.
2. The paper demonstrates the robustness of the model to changes in hyperparameters and offers insights into the interpretability of the results.
summary: The paper presents an approach to evaluate perceptual distance models using two-alternative forced choice (2AFC) experiments, especially with crowd-sourced data where traditional ranking methods are not applicable. The authors use a binomial distribution to model the decision-making process, estimate parameters with maximum likelihood, and propose metrics that surpass simple accuracy measures. This method offers a more robust evaluation of perceptual models compared to classical and neural network approaches.
weaknesses: 1. The paper compares the proposed method with classical and neural network methods but does not extensively compare it with the latest state-of-the-art approaches in perceptual distance modeling.
2. The method is evaluated across different datasets, but there is insufficient evidence to demonstrate its generalizability across different types of images and distortions, particularly on traditional perceptual datasets like TID.
3. The experimental section may lack comparative experiments to show the specific improvements of the proposed method over existing ones.
4. The paper may suffer from a lack of clarity in certain sections, Improving the organization of the content can help address this issue.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weakness.
soundness: 2
strengths: 1. Rigorous Statistical Framework: By employing binomial distribution and maximum likelihood, the approach brings statistical rigor to perceptual distance model evaluation.
2. Improved Insight into Judgments: The additional metrics offer nuanced insights into how perceptual models align with human judgments, capturing nuances beyond binary agreement.
summary: The paper presents a statistical model to improve the evaluation of perceptual distance models based on crowd-sourced 2AFC (two-alternative forced choice) data. The topic is interesting but there are some serious issues with the applicability and experiment.
weaknesses: **1. Applicability and Generalizability of the Proposed Metrics**

Could the evaluation metrics proposed in this study be extended to no-reference image quality assessment (NR-IQA) settings? In NR-IQA, pairwise comparisons, where participants judge the relative quality of two images, are common. This scenario bears similarities to 2AFC binary decision tasks and could benefit from the uncertainty modeling approach presented here. I recommend that the authors consider and discuss the metrics’ applicability in no-reference scenarios, potentially running experiments on pairwise NR-IQA datasets to further validate the transferability and generalizability of the metrics. This could significantly enhance the utility and impact of the proposed metrics in image quality assessment.

**2. Superiority Over Traditional Evaluation Metrics**

The study does not provide a clear demonstration of how the proposed metrics outperform traditional evaluation methods such as SRCC (Spearman’s Rank Correlation Coefficient), PLCC (Pearson Linear Correlation Coefficient), or KRCC (Kendall Rank Correlation Coefficient). While the authors show that deep learning models perform well using the new metrics, it is unclear whether this suffices to validate the effectiveness of the new metrics. To address this, I recommend a direct experimental comparison with these established metrics, which would highlight whether the new metrics provide additional information or improved accuracy. Moreover, it would be beneficial to discuss the potential of the new metrics to capture aspects like human judgment uncertainty and finer model differences, which traditional metrics may overlook, to enhance the credibility of this research.

**3. Dataset Selection and Applicability of Use Cases**

The experiments primarily rely on two datasets: BAPPS, which is diverse in distortion types, and CLIC, which is compression-specific. This dataset selection might limit the study’s ability to generalize the metrics’ effectiveness across different use cases. For instance, if the goal is to develop a general FR-IQA (full-reference image quality assessment) method, testing on a broader range of distortions—such as noise, blur, and color distortions—would be beneficial, possibly including datasets like TID2013 or LIVE. Conversely, if the focus is specifically on compression distortions, the paper would benefit from more explicitly discussing this application focus and how the proposed metrics could, for example, aid in choosing compression algorithms based on perceptual quality. Clearer examples or case studies illustrating how the metrics would assist in practical applications would improve the paper’s practical relevance.

**4. Relationship Between Uncertainty Modeling and MOS Distribution**

There seems to be a potential relationship between the proposed uncertainty modeling approach and MOS (Mean Opinion Score) distributions. MOS scores typically provide a more fine-grained rating (e.g., scores from 1 to 5), which offers a probabilistic view of perceived quality rather than binary decisions. I encourage the authors to explore how MOS distributions might contribute to refining the uncertainty modeling in this study, leveraging the probability distribution across the MOS scale to capture subtle perceptual differences more effectively. Discussing or experimenting with this connection could improve the interpretability and granularity of the proposed approach.

Meta-review: The paper was reviewed by three experts and there is an unanimity on the negative side with respect to the acceptance of this paper.

The responses provided by the authors did not convince the reviewers to improve their ratings.

The AC agrees with the reviewers that the paper is below the acceptance bar and invites the authors to benefit from the received feedback and further improve their work.

Decision: Reject