Perceptual Fairness in Image Restoration
Authors: Guy Ohayon
Michael Elad
Tomer Michaeli
Venue: NeurIPS 2024 poster (2024)
Keywords: fairness
bias
inverse problems
image restoration
image processing
machine learning
computer vision
responsible AI
super-resolution
deblurring
denoising
Primary area: fairness
TL;DR: We propose a new definition of fairness for image restoration algorithms, draw its connection to previous ones, study its theoretical properties, and demonstrate its practical utility.

Abstract: Fairness in image restoration tasks is the desire to treat different sub-groups of images equally well. Existing definitions of fairness in image restoration are highly restrictive. They consider a reconstruction to be a correct outcome for a group (e.g., women) *only* if it falls within the group's set of ground truth images (e.g., natural images of women); otherwise, it is considered *entirely* incorrect. Consequently, such definitions are prone to controversy, as errors in image restoration can manifest in various ways. In this work we offer an alternative approach towards fairness in image restoration, by considering the *Group Perceptual Index* (GPI), which we define as the statistical distance between the distribution of the group's ground truth images and the distribution of their reconstructions. We assess the fairness of an algorithm by comparing the GPI of different groups, and say that it achieves perfect *Perceptual Fairness* (PF) if the GPIs of all groups are identical. We motivate and theoretically study our new notion of fairness, draw its connection to previous ones, and demonstrate its utility on state-of-the-art face image restoration algorithms.

Perceptual Fairness in Image Restoration
Guy Ohayon
Faculty Computer Science Technion- Institute of Technology ohayonguy@cs. technion. wec.iiioonsstttionn....mom.mmmo. ac
Michael Elad
Tomer Michaeli
Faculty of Computer Science
Faculty of Electrical and Computer Engineering Technion-Israid Institute of Technology tomer m@ee. technion...... ac.
Technion Israel Institute of Technology
elad@cs technion.ac...
Abstract
Fairness in image restoration tasks is the desire to treat different sub-groups of images equally well. Existing definitions of fairness image restoration are highly restrictive. They consider reconstruction to be correct outcome for group (e.g.. women) only if falls within the group's 's of ground truth images (e.g., natural images of women); otherwise, it is considered entirely incorrect. Consequently, such definitions are prone to controversy, as errors image restoration can manifest in various ways. In this work we offer an alternative approach towards fairness in image restoration, by considering the Group Perceptual Index (GPI), which we define as the statistical distance between the distribution of the group's s ground truth images and the distribution of their reconstructions. We assess the fairness an algorithm by comparing the GPI of different groups, and say that it achieves perfect Perceptual Fairness (PF) if the GPIs all groups are identical. We motivate and theoretically study our new notion of fairness, draw its connection to previous ones, and demonstrate its utility on state-of-the-att face image restoration algorithms.
Introduction
Tremendous efforts have been dedicated understanding, formalizing. and mitigating fairness issues in various tasks, including classification [17
22
29
81
94
95), regression [2] 7
g
12
43
655)
clustering 4-6
13 70
73), recommendation 25
26
46
52
92, and generative modeling
[15
224] [44 66
74
75
96 Fairness definitions remain largely controversial, yet broadly speaking,
they typically advocate for independence (or conditional independence) between sensitive attributes (ethnicity, gender, etc.) and the predictions of algorithm. In classification tasks, for instance, the input data carries sensitive attributes. which are often required to statistically independent of the predictions (e.g., deciding whether to grant a loan should not be influenced by the applicant's 's gender). Similarly, in coxt-to-image generation, fairness often advocates for statistical independence between the sensitive attributes of the generated images and the text instruction used [24). For instance, the prompt "An image of Firefighters should result in images featuring people of various genders, ethnicities, etc.
While fairness is commonly associated with the desire to eliminate the dependencies between sensitive attributes and the predictions, fairness in image restoration tasks (e.g.. denoising, super-resolution) has fundamentally different meaning. In image restoration, both the input and the output carry sensitive attributes, and the goal is to preserve the attributes of different groups equally well [34]. But what exactly constitutes such a preservation of sensitive attributes? Let us denote by x, y, and the unobserved source image, its degraded version (e.g., noisy, blurry), and the reconstruction of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

X
py
P
Px]a(-la)
GPI > GPT of of a=l
of GPI of
PYJa(1a)
of of = GPI of
Pxia((a)
Good
Figure 1: Illustrative example of the proposed notion of Perceptual Fairness (PF). This figure presents four possible restoration algorithms exhibiting different hehaviors and fairness performance. In this example, the sensitive attribute A takes the values 0 with probabilities P(A= 0) < = The distributions px and py correspond to the ground truth signals (e.g., natural images) and their degraded measurements (e.g.. noisy images). respectively. The distribution Px (-Ia) corresponds to the ground truth signals associated with the attribute value a, and pYd a-Ila) J is the distribution of their degraded measurements. The distribution of all reconstructions is denoted by Px. and a (-la) the distribution of the reconstructions associated with attribute value a. The Group Perceptual Index (GPI) of the group associated with is defined as the statistical distance between (- and px A (-la), ) and good PF is achieved when the GPIs all groups are (roughly) similar. For example, X1 achieves good PF since the GPIs of both and
roughly equal, while Xs achieves poor PF since the GPI of a = is worse (larger) than that of 1. See Section 2 for more details.
from y. respectively. Additionally, let X, denote the of images x carrying the sensitive attributes Jalal et al. [34] deem the reconstruction of any x E X₂ as correct only if å E X. This allows practitioners evaluate fairness in intuitive way, by classifying the reconstructed images produced for different groups. For instance, regarding x. and & as realizations of random vectors X, Y, and X, respectively, Representation Demographic Parity (RDP) states that P(X X. X E Xx) should be the same for all and Proportional Representation (PR) states that P(X E Xa) = P(X E Xa) should hold for every a. However, the idea that a reconstructed image ê can either be an entirely correct output (f E X₂) or an entirely incorrect output (& & X,)) is highly limiting, as errors in image restoration can manifest many different ways. Indeed, what if one algorithm always produces blank images given inputs from specific group, and another algorithm produces images that are "almost" in X, for such inputs (e.g., each output only close to some image in Xx)? Should both algorithms be considered equally (and completely) erroneous for that group? Furthermore, quantities of the form P E (-)-) completely neglect the distribution the images within X. For example, assuming the groups are women and non-women, an algorithm that always outputs the same image of woman when the source image is woman, but produces diverse non-women images when the source is not woman, still satisfies RDP. Does this algorithm truly treat women fairly?
To address these controversies, we propose to examine how the restoration method affects the distribu- tion of each group of interest (e.g., the distribution of images of women or non-women), Specifically, we define the Group Perceptual Index (GPI) to be the statistical distance (e.g., Wasserstein) between the distribution of the group's s ground truth images and the distribution of their reconstructions. We then associate Perceptual Fairness (PF) with the degree which the GPIs of the different groups are close to one another. In other words, the PF of an algorithm corresponds to the parity among the GPIs of the groups interest (see Figure for intuition). The rationale behind using such an index two-fold. First, it solves the aforementioned controversies. For example, algorithm that always outputs the same image of a woman when the source image is a woman, and diverse non-women images otherwise, would achieve poor GPI for women and good GPI for womew, thus resulting in poor PF. Second, the GPI reflects the ability of humans to distinguish between samples of group's s ground truth images and samples of the reconstructions obtained from the degraded images of that group [0]. Thus, achieving good PF (i.e., parity in the GPIs) suggests that this ability is the same for all groups
This paper is structured as follows. Section we formulate the image restoration task and present the mathematical notations necessary for this paper. This includes a review of prior fairness definitions image restoration, alongside our proposed definition. We also discuss why PF can be considered as generalization of RDP. In Section 3 we present our theoretical findings. For instance, prove that

---

(GPPIII (P(()
0.3
DPS
PIGDM
DDNM
DDRM
10
DPS
PiGDM
DDNM
DDRM
Young & Asian
Young & non-Asian
Old & Asian
Old & non-Asian
DPS PIGDM DDNM DDRM
Figure 2: Examining fairness in face image super-resolution techniques through the lens of RDP 34] PF (our proposed notion fairness). Both RDP and PF assess how well an algorithm treats different fairness groups Specifically, RDP evaluates the parity in the GP of different groups (higher GP is better), and PF evaluates the parity in the GPI of different groups (lower GPI is better). The results show that the groups old& Asian and old&non-Asian attain similar treatment according RDP (similar GP scores that roughly zero), while the latter group attains better treatment according PF. Section Appendix G.7 PP the desired one.
we show why this outcome
achieving perfect GPI for all groups simultaneously is not feasible when the degradation sufficiently severe. We also establish an interesting (and perhaps counter-intuitive) relationship between the GPI of different groups for algorithms attaining perfect Perceptual Index (PI) [10], and show that PF and the PI are often at odds with each other. In Section 4 we demonstrate the practical advantages of PF over RDP. In particular, we show that PF detects bias in cases where RDP fails to do so. Lastly, in Section 5 we discuss the limitations of this work and propose ideas for the future.
2 Problem formulation and preliminaries
We adopt the Bayesian perspective of inverse problems, where an image a is regarded as realization of random vector X with probability density function Px. Consequently, an input is realization of random vector Y (e.g., noisy version of X). which related to X via the conditional probability density function PyJx- The task of an estimator (in this paper, an image restoration algorithm) is estimate only from Y. such that statistically independent given Y). Given an input y, the estimator X generates outputs according to the conditional density P
Y > X is a Markov chain (X and X

---

2.1 Perceptual index
common way to evaluate the quality of images produced by an image restoration algorithm is to assess the ability of humans to distinguish between samples of ground truth images and samples of the algorithm's 's outputs. This is typically done by conducting experiments where human observers
vote on whether the generated images are real or fake [88 20 28
32
33 72
[OI 102]. Importantly,
this ability can be quantified by the Perceptual Index [10], which the statistical distance between
the distribution of the source images and the distribution of the reconstructed ones,
PI == d(px.PX)
where .) is some divergence between distributions (Kullback-Love divergence, total variation distance, Wassersterin distance, etc.).
2.2 Fairness
2.2.1 Previous notions of fairness
Jalal et al. 34 introduced three pioneering notions of fairness for image restoration algorithms:
Representation Demographie Parity (RDP), Proportional Representation (PR), and Conditional Proportional Representation (CPR). Formally, given a collection of sets images { } where a, is vector of sensitive attributes and each Xx, represents the group carrying the sensitive attributes as these notions are defined by
RDP: P(X E Xa, E E x..) = P(X E Na, XX E Xa,) for every i,
(2)
PR: P(X E xx,)) = P(X E Xa.) for every i;
(3)
CPR: P( E
y) y) P(X X2, = y) for every i,
While such definitions are intuitive and practically appealing, they have several limitations. First, any reconstruction that falls even "slightly off" the set Xa; is considered entirely wrong outcome for its corresponding group. In other words, reconstructions with minor errors are treated the same as completely wrong ones. Second, these definitions neglect the distribution of the groups' images. Consequently, an algorithm can satisfy RDP, PR, CPR, etc., while treating some groups much worse than others in terms of the statistics of the reconstructed images. For instance, consider dogs and cats as the two fairness groups. Let Adogs and Xeats be the sets of images of dogs and cats, respectively, dog. Furthermore, suppose that the species can be
and let Toog E Xaugs be a particular image perfectly identified from any degraded measurement, i.e.,
PPX E Kdogs Y = y) = 1 or P(X E Meaal = y)=1
(5)
for every y. Now, suppose that X always produces the image Tdog from any degraded dog image, while generating diverse, high-quality cat images from any degraded cat image. Namely, for every y. we have
P(X Felog X E Kdogs) P(X E Kaogs X Mangs) = P( E Mewss Neas);
(6)
P(X Edog Y = y) = IP( E Adogs Y 1) = IP = -Nooss Y = =y, P(X Xans = y) IP(X Xears y).
(7)
Although this algorithm satisfies RDP (Equation (6) and CPR (Equations 7 and (8) which entails PR [34], it is clearly useless for dogs. Should such an algorithm really be deemed as fair, then?
To address such controversies, we propose to represent each group by the distribution of their images, and measure the representation error of group by the extent which an algorithm "preserves" such a distribution. This requires more general formulation of fairness groups, which is provided next.
2.2.2 Rethinking fairness groups
We denote by (a random vector) the sensitive attributes of the degraded measurement Y, so that (-1a( is the distribution of degraded images associated with the attributes A = (e.g., the distribution of noisy women images). Consequently, the distribution of the ground truth images that possess the sensitive attributes is given by (-la), and the distribution of their reconstructions is

---

given by & Moreover, we assume that A + Y - X forms a Markov chain, implying that knowing A does not affect the reconstructions when Y is given. This assumption is not limiting. since image restoration algorithms are mostly designed to estimate X solely from Y, without taking the sensitive attributes as an additional input. See Figure for an illustrative example of the proposed formulation.
Note that such formulation is quite general, as it does not make any assumptions regarding the nature of the image distributions, whether they have overlapping supports or not, etc. Our formulation also generalizes the previous notion of fairness groups, which considers only the support of px ( (-la) for every Indeed, one can think of Xa = P (-la) as the set of images corresponding to some group, and of |o 2a as the collection of all sets. Furthermore, notice that A can also (-la) is the posterior
be the degraded measurement itself. i.e. A = Y. In this case, Pxia(-la) distribution of ground truth images given the measurement a, and (*)a(-1a) = P Pxjy (-la) is the distribution of the reconstructions of the measurement a. Namely, our mathematical formulation is adaptive to the granularity of fairness groups considered.
2.2.3 Perceptual fairness
We define the fairness of an image restoration algorithm as its ability to equally preserve the distribu- tion Pxja(-la) across all possible values of a. Formally, we measure the extent to which an algorithm X preserves this distribution by the Group Perceptual Index, defined as
GPLa(a) == (-lp)a((-ll))2))) PX a-lla)),
where d(..-) is some divergence between distributions. Then, we say that X achieves perfect Perceptual Fairness with respect to d, or perfect PF& in short, if
GPIa(a,) GPL
(10)
for every d1, 02 E supp (see Figure to gain intuition). In practice, algorithms may rarely achieve exactly perfect PFa, while the GPI, of different groups may still be roughly equal. In such cases, we say that X achieves good PFa. In contrast, if there exists at least one group that attains far worse GPI, than some other group, we say that X achieves poor/bad PFa. Importantly, note that achieving good PF does not necessarily indicate good PI, and/or good GPI, values.
2.2.4 Group Precision, Group Recall, and connection to RDP
In addition to the PLa defined in I the performance of image restoration algorithms is often measured via the following complementary measures [455 M (1) Precision, which is the probability that a sample from Px falls within the support of px. TP X E supppx). and (2) Recall, which is the probability that a sample from px falls within the support of px- PP( E su Achieving low precision implies that the reconstructed images may not always appear as valid samples from Px- Thus, precision reflects the perceptual quality of the reconstructed images. Achieving low recall implics that some portions of the support of px may never be generated as outputs by X. Hence, recall reflects the perceptual variation of the reconstructed images.
Since here we are interested in the perceptual quality and the perceptual variation of group's reconstructions, let us define the Group Precision and the Group Recall by
GP(a) == P(X E Xo|A= 11A00 a), a), GR = P(X 8))A=o),
(11)
(12)
where X, supp (xppax))()1a) and &. Pu (-la). Hence, when adopting our formulation of fairness groups, satisfying RDP simply means that the GP values of all groups are the same. However, as hinted in previous sections, two groups with similar GP values may still differ signif- icantly in their GR. From the following theorem, we conclude that attaining perfect Pr where dTv(p.g) g) J [p(x) <(x)
((z))d |dr is the total variation distance between distributions, guarantees that both the GP and the GR of all groups have a common lower bound. This implies that PFay can be
considered generalization of RDP.
Theorem 1. The Group Precision and Group Recall of any restoration method satisfy
GP(a) > 1 GPIan
(13)
(R)a) 2 GPIan,
(14)
for all a E suppPA-
5

---

Although using drv(ss) provides straightforward relationship between PF and RDP, other types of divergences may not necessarily indicate GP and GR so explicitly. The perceptual quality & variation of group's reconstructions may be defined in many different ways 71,. and the GPI implicitly entangles these two desired properties.
The mathematical notations and fairness definitions are summarized in Appendix A To further develop our understanding of PF, the next section presents several introductory theorems.
3 Theoretical results
Image restoration algorithms can generally be categorized into three groups: (1) Algorithms targeting the best possible average distortion (e.g., good PSNR) 3
47 48 83
85
900
(2) algorithms
that strive to achieve good average distortion but prioritize attaining best PI [
19 27
47
61
80 83
85
88 89, 93 100, [04]. and (3) algorithms attempting to sample from the posterior distribution
Pxly of the given task at hand [16 40 42
51
58
76 86 9T]. In Appendix B we demonstrate
simple toy example that all these types of algorithms may achieve poor PF, implying that perfect PF is not property that can be obtained trivially. Namely, even when using common reconstruction algorithms such as the Minimum Brean-Squared-mree one group may attain far worse GPI than another group. It is therefore tempting ask in which scenarios there exists an algorithm capable of achieving perfect GPI for all groups simultancously: As stated in the following theorem, this desired property unattainable when the degradation is sufficiently severe.
(MMSE) estimator or the posterior sampler,
Theorem 2. Suppose that 3a1, 02 E supp PA such that
P(X E Na; n Xag A = 0;) < P(Y E Va, nV = az);
(15)
for both 1,2, where Xx, supp supppxia and Vo. = supp (y)ala)))).... Then, GPIa(a1) and GPIa(a2) cannot both be equal to zero.
In words. Theorem states that when the degraded images of different groups are "more overlapping" than their ground truth images, at least one group must have sub-optimal GPI. Importantly, note that perfect GPI can always be achieved for some group corresponding to A = a individually. by ignoring the input and sampling from (-aa. Hence, Theorem 2 implies that, for sufficiently severe degradations, one may attempt to approach zero GPI for all groups simultaneously, until the GPI of one group hinders that of another one. But what about algorithms that just attain perfect overall PI? Can such algorithms also attain perfect PF? As stated in the following theorem, it turns out that these two desired properties (perfect PI and perfect PF) are often incompatible.
Theorem 3. Suppose that A takes discrete values, X attains perfect Pla (Px = Px), and 3a, Om E supp PA such that GPIa(a) > O and IP(A- = am) 0.5. Then, X cannot achieve perfect PF
In words, when there exists majority group in the data distribution, Theorem states that an algorithm with perfect PI, whose GPI is not perfect even for only one group, cannot achieve perfect PF.... This intriguing outcome results from the following convenient relationship between the GPIs of different groups for algorithms with perfect PI.
Theorem 4. Suppose that A takes discrete values and X attains perfect Pla (P& = Px). Then,
GP) < P(A= a)
Σ F(A=='')o d')GPla, a'ya
(16)
for every a with P(A = a) >
This theorem is, perhaps, counter-intuitive. Indeed, for algorithms with perfect PI, improving the GPLd of one group can only improve the GPI of other groups, and this is true even if the groups do not o While this may seem contradictory to Theorem 2 note that such relationship holds until the algorithm can no longer attain perfect Pl. The example in Appendix B demonstrates this theorem.
'wwo groups with attributes are overlapping if P(X E Xx, ( > 0. where Xo. supp (-)aaa))

---

4 Experiments
We demonstrate the superiority of PF over RDP in detecting fairness bias in face image super- resolution. Our analysis considers various aspects, including different types of degradations, and faireess evaluations across four groups categorized by ethnicity and age. First, we show that RDP incorrectly attributes faimess in simple scenario where faimess is clearly violated. In contrast, PF successfully detects the bias. Second, we showcase a scenario where PF uncovers potential malicious intent. Specifically, it can detect bias injected into the system via adversarial attacks, situation again missed by RDP.
4.1 Synthetic data sets
In the following sections we assess the fairness of leading face image restoration methods through the lens of PF and RDP. Such methods are often trained and evaluated on high-quality, aligned face image datasets like CelebA-HQ [36] and FFHQ [37], which lack ground truth labels for sensitive attributes such ethnicity. Moreover, these datasets are prone to inherent biases, e.g., they contain very few images for certain demographic groups [31 35 69), and it is unclear whether images from different groups have similar levels of image quality and variation (prior work suggests that they might not [T].. To address these limitations, we leverage an image-to-image translation model that takes text instruction as additional input. This model allows to generate four synthetic fairness groups with high-quality, aligned face images. Specifically, we translate each image x from the Ce [36] test partition into four different images representing Asian/non-Asian and young/old individual We use a unique text instruction for each translation. For example, the text instruction "120 years old human, Asian, natural image, sharp, DSLR" translates xr into an image of an old&Asial individual. Finally, we include each resulting image in its corresponding group data only if all translations are successful according to the FairFace combined age & ethnicity classifier (35). This involves classifying the ethnicity and age of the translated images and ensuring that old individuals are categorized as 70+ years old, young individuals are categorized as any other age group, Asian individuals are classified as either Southeast or East Asian, and non-Asian individuals are classified as belonging to any other ethnicity group. See Appendix [..] for more details and for the visualization of the results.
2
Disclaimer. Importantly, we note that the generated synthetic data sets may impose offensive biases and stereotypes. We use such data sets solely to investigate the fairness of image restoration methods and verify the practical utility of our work. We do not intend to discriminate against any identity group or cultures in any way.
4.2 Perceptual Fairness vs. Representation Demographic Parity
We consider several image super-resolution tasks using the average-pooling down-sampling operator with scale factors 8 E [4, 8, 32). and statistically independent additive white Gaussian noise of standard deviation nN {0, 0.1, 0.25). ). In Appendix [ we also conduct experiments image denoising and deblurring. The algorithms DDNM* [86). DDRM [42), DPS [16). and PiGDM [76] evaluated on all scale factors, and GFPGAN 884, VQFR 27
GPEN 933. DifIBIR 449. Code- 88 are evaluated only on the x4 and x8
Former 104), RestoreFormer++ [89, and RestoreFormer
scale factors (these algorithms produce completely wrong outputs for the other scale factors). To assess the PF of each algorithm, we compute the GPIKID of each group using the Kernel Inception Distance (KID) [9] and the features extracted from the last pooling layer of the FairFace combined age & ethnicity classifier (35). In Appendix G.4 we utilize the Fréchet Inception Distance (FID) (30) instead of KID, and in Appendix G.5 we assess other types of group metrics such as PSNR. Addi- tionally, we provide in Appendix G.6 an ablation study of alternative feature extractors. To assess RDP. we use the same FairFace classifier to compute the GP of each group. As done in [34], we approximate the GP of each group by the classification hit rate, which is the ratio between the number of the group's reconstructions that are classified as belonging to the group and the total number of the group's inputs. Qualitative and quantitative results for 32, N 0.0 are presented in Figure 2
We choose to consider these fairness groups since image restoration algorithms are likely biased towards young and white demographics, given the overrepresentation of such groups in common training datasets (e.g.. FFHQ, CelebA). Namely, groups of Asian and/or old individuals typically underrepresented in such datasets.
7

---

4, O 0.0
== 8, ON 0.0
16, ON = 0.0
32, C.v 0.0
Young
Non-Asian
Old
Asian
Old
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
GP(c)
1 CP(a)
t GP())
GP))
Young
Asian
Young
Non-Asian
Old
Asian
OMd
Non
10
10
GPigmie)
GP
GPPgin(a)
GPlinn(e)
DPS PiGDM
DDNM
ReetoreFormer
DirBIR
GPEN
VQFR
DDRM
Former
CFPGAN
Reatore
Figure 3: Comparison of the GP and the GPlıD of different fairness groups, using various state-of sfate-of-the-at the-th--rt face image super-resolution methods. In most experiments, GPlиn suggests fairness discrepancy between the groups old&non-Asian and AldaAsian, while the GP of these groups is roughly equal.
Quantitative results for values of and aN 0.0 are shown in Figure Complementary details and results are provided in Appendix G
receives the best overall treatment in terms of both GP and GPIKID. This result is not surprising, since the training data sets of the evaluated algorithms (e.g., FFHQ)
Figure shows that the group young&non-
known be biased towards young and white demographics [50] 63]. However,
while most algorithms appear to treat the groups oldaAssian and ald&non-Asiaa quite similarly in terms of GP, the GPIxID indicates clear disadvantage for the former group. Indeed, by examining ethnicity and age separately using the FairFace classifier, we show in Appendix G.7 that, according to RDP, the group old&non-Asian exhibits better preservation of the ethnicity attribute compared the group old& while the age attribute remains equally preserved for both groups. This highlights that RDP is strongly dependent on the granularity of the fairness groups (as suggested in (34] ), since slightly altering the groups' partitioning may completely obscure the fact that an algorithm treats certain attributes more favorably than others. However, as our results show, this issue is alleviated when adopting GPIRID instead of GP. Namely, the ethnicity bias is still detected by comparing the GPIKID of different groups, even though the fairness groups are partitioned based on age and ethnicity combined.
4.3 Adversarial bias detection
In Section 1 we discussed the limitations of fairness definitions such as RDP. For instance, an algorithm might satisfy RDP by always generating the same output for degraded images of a particular group, even if it produces perfect results for another. However, such an extreme scenario is not common practice. Indeed, real-world imaging systems often involve degradations that are too severe, and well-trained algorithms perform impressively well when applied to different groups (see, e.g., Figure So what practical advantage does PF have over RDP in such cireumstances? Here, we demonstrate that a malicious user can manipulate the facial features (e.g., wrinkles) of group's reconstructions without violating fairness according to RDP, but violating fairness according to PF In particular, we consider only the ethnicity sensitive attribute by taking the young& Asian group as Asian, and the young&non-Asin group as non-Asian. Then, we use RestoreFormer++ method, which roughly satisfies RDP with respect to these groups (see Figure 4a where GP is evaluated by

---

CD (((pon-Arian) Asian)
Asian
Asiansian
0.00
0.25
0.50
0.75
1.00
GPIxID (mon Asian)
(P(Asii
9
VAdv
y
BAdv
WAdY
y
MAdv
CPIgID (Asian)
No attacks
Apde
Adv. attacks Asian
Adv. attacks non-Astan
(a) RestoreFormer++ achieves roughly similar GP for both groups (i.e., roughly satisfies RDP). The adversarial attacks on the inputs from the non- Asian group remain undetected by GP, while they highly affect the GPI.
(b) Visual results. y and x are the original input and the source image, respectively. yads and a are the adversarial input and its corresponding out- put, respectively. Each yaáv successfully alters the output facial features. Indeed, &adv clearly con- tains a face with more wrinkles than x.
Using adversarial attacks to inject bias into the outputs of RestoreFormer++, in setting where it (roughly) satisfies RDP, Such attacks are detected by PF but not by RDP.
Figure 4:
classifying ethnicity alone), and perform adversarial attacks on the inputs of each group to manipulate the outputs such that they are classified belonging to the 70+ age category. The fact that the GP of each group is quite large implies that the malicious user can classify ethnicity quite accurately from the degraded images, and then manipulate the inputs only for the group it wishes harm (we skip such a classification step and simply attack all of the group's inputs). Such attacks are anticipated to Complementary details of this experiment
succeed due to the arrceptionrrobustess tradeoff 59 60
are provided in Appendix H
In Figure 4 we present both quantitative and qualitative results demonstrating that the attacks the non-Asian group are not detected by RDP. However, we clearly observe that these attacks are successfully identified by the GPIKID of each group. This again highlights that PF less sensitive the choice (partitioning) of fairness groups compared to RDP. Specifically, age must be considered as a sensitive attribute to detect such bias via RDP. Yet, even then, the malicious user may still inject other types of biases. Conversely, PF does not suffer from this limitation, as any attempt to manipulate the distribution of a group's 's reconstructions would be reflected in the group's ss GPI.
Discussion
Different demographic groups can utilize an image restoration algorithm, and fairness in this context asserts whether the algorithm "treats" all groups equally well. In this paper, we introduce the notion of Perceptual Fairness (PF) to assess whether such a desired property is upheld. We delve into the theoretical foundation of PF, demonstrate its practical utility, and discuss its superiority over existing fairness definitions. Still, our work is not without limitations. First, while PF alleviates the strong dependence of RDP on the choice of fairness groups 34 (as demonstrated in Section 4 still cannot guarantee fairness for any arbitrary group partitioning simultaneously (a property referred to as obliviousness in (34)). Second, our current theorems are preliminary, requiring further research to fully understand the nature of PF. For example, the severity of the tradeoff between the GPI scores of different groups (Theorem 2 and that the tradeoff between PF and (Theorem 3 remain unclear. Third, we do not address the nature of optimal estimators that achieve good or perfect PF. What is their best possible distortion (e.g., MSE) and best possible Fourth, on the practical side, we show in Appendix G.6 that effectively evaluating PF using metrics such KID necessitates utilizing image

---

features extracted from a classifier dedicated to handling the considered sensitive attributes (e.g.. an age and ethnicity classifier). However, this is not a disadvantage compared to previous fairness notions (RDP, CPR and PR). which also require such a classifier. Lastly, while the proposed GPI may be suitable for evaluating fairness in general-content natural images, we considered only human face images due to their societal implications, namely since fairesss issues are particularly critical when dealing with such images. For example, if a general-content image restoration algorithm performs better on images with complex structures than images of clear skies, this discrepancy is unlikely to be problematic for practitioners, as long as the algorithm attains good performance overall. Moreover, previous works [3] evaluated fairesss with respect non-human subjects (e.g.. dogs and cats), but these studies provide limited insights into suman-related fairnsss issues, which often arise due to subtle differences between images (e.g., wrinkles). Expanding our method to other datasets remains an avenue for future work.
6 Societal impact
Designing fair and unbiased image restoration algorithmss is critical for various AI applications and downstream tasks that rely on them, such facial recognition, image classification, and image editing. By proposing practically useful and well-justified fairness definitions, we can detect (and mitigate) bias in these tasks, ultimately leading fairer societal outcomes. This fosters increased trust and adoption of AI technology, contributing to more equitable and responsible use of Al in society.
Acknowledgments
This research was partially supported by the Israel Science Foundation (ISF) under Grant 2318/22 and by the Council For Higher Education Planning & Budgeting Committee.
References
[1] Theo Joseph Adrai, Guy Ohayon, Michael Elad, and Tomer Michaeli. Deep optimal transport: A practical algorithm for photo-realistic image restoration. In Thirty- Conference on Neural Information Processing Systems, 2023. URL httL:htts::////t / wwtprrewiiiieeeettuunueee net/forum?
id=bJJY91Pfee
[2] Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-basse algorithms. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 120-129. PMLR, 09-15 Jun 2019. URL https://proceedinn mml.inggg.mm.......mmmmmm.. www.srresssvvv /vtess///v///e http////ggggaaa11....a html
[3] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. arXiv, 1803.08664, 2018.
[4] Arturs Backurs, Piotr Indyk, Krzysztof Onak, Baruch Schieber, Ali Vakilian, and Tal Wag- ner. Scalable fair clustering. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pages 405-413. PMLR, 09-15 Jun 2019. URL https: http:ss/pprceedinngg mlr, wtw.:ggss..rrr..e//r1.... https///9bbbaauurrbeeee html
[5] Suman Bera, Deeparnab Chakrabarty, Nicolas Flores, and Maryam Negahbani. Fair algorithms for clustering. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates. Inc 2019. URL htt.:: htppittttts//prrcceinggg newrips hcc/paper_filee/// B0B//Fi00EE520000770006866666622 Paper pdf
[6] Ioana O. Bercea, Martin Groß, Samir Khuller, Aounon Kumar, Clemens Rösner, Daniel R. Schmidt, and Melanie Schmidt. On the cost of essentially fair clusterings. arXiv, 1811,10319, 2018.
10

---

[7] Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgen- stern, Seth Neel, and Aaron Roth. A convex framework for fair regression. arXiv, 1706.02409) 2017.
[8] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. arXiv, 7703.09207, 2017.
[9] Mikotaj Binkkwski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton, Demystifying MMD GANs. In International Conference Learning Representations, 2018. URL https:
//openreviee ew wwtp:viww..eeeeeeeeeeeeeeee
[10] Yochai Blau and Tomer Michaeli. The perception-die -ispttttonnntttttiionn tradcoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[11] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors, Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77-91. PMLR, 23-24 Feb 2018. URL sttps://proceediie wwt.::/www.........t.......e..ee
mlr. press/ http:///ww//ooolaiiiiiiaama.aaaa 18a. html
[12] Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang. Controlling attribute effect in linear regression. In 2013 IEEE 13th International Conference Data Mining, pages 71-80, 2013. doi: 10.1 2..1009/CMM..013.....
[13] Flavio Chierichetti, Ravi Kumar, Silvio Lattanzi, and Sergei Vassilvitskii. Fair clustering through fairlets. In I. Guyon, Von Luxburg, Bengio. H. Wallach, R. Fergus, Vish- wanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://procceeeinngg neur ips cc/ paper fww.fillllppppe /1ls://ppp27771111//cccccccccce3
pdf
[14] Jun-Ho Choi, Huan Zhang, Jun-Hyuk Kim, Cho-Jui Hsieh, and Jong- long-Seok Lee. Evaluating robustness of deep image super-resolution against adversarial attacks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[15] Yujin Choi, Jinseong Park, Hoki Kim, Jaewook Lee, and Saerom Park. Fair sampling in diffusion models through switching mechanism. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Chirty-Eightl AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 21995-22003 22003. AAAI Press, 2024. doi: 10. 10.1609/AAAI. V38120..0000 URL buttps://ddat (rg.ooooorg/// 10. 1609/ 00009//aai... v38120.30200
[16] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann. Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023. URL https
//penreview.
[17] Sam Corbett-Davice Johann D. Gaebler, Hamed Nilforoshan, Ravi Shroff, and Sharad Goel. The measure and mismeasure of fairness. arXiv, 1808.00023. ...000003,,. 2023.
[18] Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel recursive super resolution. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
[19] Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. Transactions on Machine Learning Research, 2023, Featured
ISSN 2835-8856. URL https://openreviiet Certification.
[20] Emily L Denton. Soumith Chintala, arthur szlam, and Rob Fergus. Deep generative image models using laplacian pyramid of adversarial networks. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: http:t::/ppoceedngggsssssss///
cc/

---

[21] Chao Dong, Chen Change Loy. Kaiming He, and Xiaoou Tang, Image super- using deep convolutional networks. 1501.00092 2014.
[22] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. Proceedings of the 3rd Innovations Theoretical Computer Science Conference, ITCS 12, page 214-226, New York, NY. USA. 2012. Association for Computing Machinery. ISBN 0781450311151
doi: 00.144522000236.000005555 URL postps:/ddbr
org/ 10. 101111112299000 2090236 2
[23] Dror Freirich, Tomer Michaeli, and Ron Meir. A theory of the intorttion-prceptioon tradeoff in wasserstein space. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages neurips. cc/ paper httpa////wpppoo.ooo..o//////62a2
25661-25677 Curran Associates, Inc., 2021.
pdf
[24] Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing eex-to--mmage generation models on fairness. arXiv, 2302.10893, 2023.
[25] Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, and Yongfeng Zhang. Towards long-term fairness in recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM '21, page 445-453 New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450382977.
doi: 10. 1 10.1145/3437963.34418444 URL
https://doi.org//
00445//337963.336 3441824
[26] Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi. Faimness-aware ranking in search & recommendation systems with application to linkedin talent search. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019. /ttpthtpps///giesssstttttttooooo
www.seeeasssseeeeeeeeeeeeee...tt www.nticschollrrr org/CorpusID : 146121159
[27] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan, and Ming-Ming Cheng. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In ECCV, 2022.
[28] Sergio Guadarrama, Ryan Dahl, David Bieber, Jonathon Shlens, Mohammad Norouzi, and Kevin Murphy, Pixcolor: Pixel recursive colorization. In British Machine Vision Conference 2017, BMVC 2017, London. UK, September 4-7, 2017. BMVA Press, 2017. URL https //www. www..dropbos. com/ http///nkkkkmmrreem wwtpe/dddxxeee/e/..ooo 04400044700 pdf
[29] Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in su- pervised learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL
wwtpeiips.ccppppppeeeeee cc/paper....... http:/ppprfflleeppppppeee .df
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran http:ppprrfffl//speeppppe Associates, Inc., 2017. URL http:tss//ppccceeiiggg..ooo/ wwwp::cceeeeeiinnneeneeeeeeeeeee pdd 2017///aeeaaa1866777000066668662
[31] Marco Huber, Anh Thi Luu, Fadi Boutros, Arjan Kuijper, and Naser Damer. Bias and diversity in synthetic-based face recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages
January 2024.
[32] Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there Color!: Joint End- to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (Proc. of SIGGRAPH 2016), 35
(4), 2016.
12

---

[33] Phillip Isola, Jun- Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to inage-to-image translation with conditional adversarial networks. CVPR, 2017.
[34] Ajil Jalal, Sushrut Karmalkar, Jessica Hoffmaann, Alex Dimakis, and Eric Price. Fairness for image generation with uncertain sensitive attributes. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 Proceedings of Machine Learning Research, pages 4721-4732. PMLR, 18-24 Jul 2021. httLttts://:pcceeeeee
mlr www.pprsssseeeeee wwwress/v////// jalalllb.b..... html
[35] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548-1558, 2021.
[36] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Repre- sentations, 2018. URL pttpps wttp://wwweee/ommmii..c.m HK992CeAA
[37] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for genera- tive adversarial networks. In 2019 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4396 4405, 2019. doi: 10.1109/VVR..20190000
[38] Sergey Kastryulin, Dzhamil Zakirov, and Denis Prokopenko. PyTorch Image Qual- ity:
Metries and measure for image quality assessment, 2019. URL https
Open-source software available at
[39] Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry V. Dylov. Pytorch image quality: Metrics for image quality assessment. 2022. doi: 10.48550/ARXIV.2208..85 14818. URL
https://arxiv.c org/ abs/ 2208 14818
[40] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21757-21769, Curran Associates, Inc., 2021. URL http: hppppe: neurips. cc/ paper Saape_ffles//ee
pdf
[41] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Stochastic image denoising by sampling from the posterior distribution. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 1866-1875, 2021, doi: 10. 1091099C5CV555222202222 .00213.
[42] Babjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restora- tion models. Advances in Neural Information Processing Systems, 2022.
[43] Junpei Komiyama, Akiko Takeda, Junya Honda, and Hajime Shimao. Nonconvex optimiza- tion for regression with fairness constraints. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages 2737-2744 PMLR, 10-15 Jul 2018. URL https: https://proon httpses//proceeiingss mlr. wwwppppeepppeppp. prressv80//es http:///mmimmmmmaaaa hatml
[44] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL httpp:::/pooceediinnneeooooe//ee wtwpeeeeeeddiinnenenneeeeeeeeeee neurips. wwtppppppppp..eee faapperrfilesppppeeee .ppp
[45] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In H. Wallach, Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Ad- vances in Neurul Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL /ttttpp://prooceedinngs neurip cwtpurips.ccppppeeee httpc/papee..llllpppeeeeeeeeeeee /tt.le///ppper////////eeee (224651Dcc4000028881114377777330
pdf
13

---

[46] Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. User-oriented fairness in recommendation. Proceedings of the Web Conference 2021. wwW '21, page 624-632, New York, NY, USA, 2021. Association for Computing Machinery. ISBN doi: 10.1145/3442381 .3449866.. URL. fttps://ddi.org/bie
9781450383127.
8442381 3449866)
[47] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Tim- ofte. Swinir: Image restoration using swin transformer. In 2021 IEEE/CVF Interna- tional Conference on Computer Vision Workshops (ICCVW), pages 1833-1844, 2021. doi: 10. 2000CCCCWWA111022220.0002222
[48] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132-1140, 2017. doi:
10.1 109/CCPRR.201..1555
[49] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu. Wanli Ouyang. Yu Qiao, and Chao Dong. Diftbir: Towards blind image restoration with generative diffusion prior. arXiv, 2308.15070, 2024.
[50] Vongani H. Maluleke, Neerja Thakkar, Tim Brooks, Ethan Weber, Trevor Darrell, Alexei A. Efros, Angjoo Kanazawa, and Devin Guillory. Studying bias gans through the lens of race. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII, page 344-360, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN 978-3-031--19777-2 doi: 10.
20. URL https
[51] Sean Man, Guy Ohayon, Theo Adrai, and Michael Elad. High-percepan quality jpeg decoding via posterior sampling. In 2023 IEEE/CVF Conference Computer Vision and Puttern Recognition Workshops (CVPRW), pages 1272-1282, 2023. doi: N0.1109/CVPRW59228.. 00134.
[52] Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fair- ness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM '18, page 2243-2251, New York. NY. USA, 2018. Association for Computing Machinery. ISBN 978145036 doi: 10.1 11.1445522692003222227.,., URL https: org/ 10. 1145/ 004111114122000 /ttp//222222... 3272027
[53] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
[54] Anish Mittal, Anush K. Moorthy, and Alan C. Bovik. Blindtreferenceeess image spatial quality evaluator. In 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pages 723-727, 2011. 6190099.
doi: 10. 10.1109/ACSSC.2011
[55] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-213 2013. doi: 10. 2012.2227726.
[56]
Nate Raw. vit-age-classifier (60444)) 2023. doi: 10.57967/ht/1259, URL https / //huggingface co/
[57] Anton Obukhov, Maximilian Seitzer, Po-Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu-Jing Lin. High-fidelity performance metrics for generative models in pytorch, 2020. URI. https:tts:/ het.abb..com/eehaa//rrrr---i--i- odo.4957738.
Version: 0.3.0, DOI: 10.5281/zen-
[58] Guy Ohayon, Theo Adrai, Gregory Vaksman, Michael Elad, and Peyman Milanfar. High perceptual quality image denoising with a posterior sampling cgan. In 202/ IEEF/CVF International Conference on Computer Vision Workshops (ICCWW), pages 1805-1813, 2021.
doi: 10. 10.109//CCVW5542200022.0000
14

---

[59] Guy Ohayon, Theo Joseph Adrai, Michael Elad, and Tomer Michaeli. Reasons for the superiority of stochastic estimators over deterministic ones: Robustness, consistency and perceptual quality. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26474-26494. PMLR, 23-29 Jul 2023. URL httpt:tt/tptoseeee ohayon23a. html
mlr press/2202/
[60] Guy Ohayon, Tomer Michaeli, and Michael Elad. The deterministic image restoration. arXiv, 231 1.09239,,, 2024.
tradeoff in
[61] Guy Ohayon, Tomer Michaeli, and Michael Elad. Posterior-mean rectified flow: Towards minimum mse photo-realistic image restoration. arXiv preprint arXiv:2410.00418. 2024. URL
https://arxiv.. www.:/ogggg/bbabeo 2000////011000101188
[62] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou. Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. B net weeet/forum?iddaaaae htt.:/eeeeeettoormmmodaaaa6ao6a.
[63] Roy Or-EI, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher- Shlizerman. Lifespan age transformation synthesis. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
[64] Emanuel Parzen. On estimation of a probability density function and mode. The Annals of Mathematicaa Statisties, 1365):1065-1093 1962. ISSN 00034851. URLhttp: http: //www. jstor
org/ stt/stleeee22/8880
[65] Adrián Pérez-Suay, Valero Laparra, Gonzalo Matco-García, Jordi Muñoz-Mari, Luis Gómez- Chova, and Gustau Camps-Valls, Fair kernel learning. In Michelangelo Ceci, Jaakko Hollmén, Ljupčo Todorovski, Celine Vens, and Sašo Džeroski, editors, Machine Learning and Knowl- edge Discovery in Databases, pages 339-355, Cham, 2017. Springer International Publishing. ISBN 378-3-319-71249-1
[66] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Guyon, Von Luxburg. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL /tttpss: neur ec/ paper http://///00//211.0000066/662222
.ddf
[67] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. The Twelfth International Conference on Learning Representations, 2024. URL httphtt///oooeeeeeeeeeee net/ http://ormmmdddddd d1222288x
[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021. URL httpps://proceeding: mlr wttppp//s///aaafoorrrrrooooo html
[69] Ethan Rudd, Manuel Gunther, and Terrance Boult. Moon: A mixed objective optimization network for the recognition of facial attributes. arXiv, 1603.07007, 2016.
[70] Clemens Rösner and Melanie Schmidt. Privacy preserving clustering with constraints. arXiv, 1802.02497, 2018
15

---

[71] Mchdi S. M. Sajjadi, Olivier Bachem, Mario Lučié, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[72] Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 29. Curran Associates, Inc., 2016. URL https::/ neur cc/ pdf paper httpa//aaapppaeoooaaaaaaaaaaaaaa
[73] Melanie Schmidt, Chris Schwiegelshohn, and Christian Sohler. Fair coresets and streaming algorithms for fair k-means clustering. arXiv, 1812.10854, 2021.
[74] Ashish Seth, Mayur Hemani, and Chirag Agarwal. Dear: Debiasing vision-language models with additive residuals. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages (829-6629, June 2023.
[75] Xudong Shen, Chao Du Tianyu Pang, Min Lin, Yongkang Wong, and Mohan Kankanhalli. Finetuning text-to-image diffusion models for fairness. In The Twelfth International Con- ference on Learning Representations, 2024. https:/:::/ http:///pppeeeeee...eoe nww./foumfffiii hnrB5YHoYu
[76]
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pscudoinverse-guieee diffusion models for inverse problems. In International Conference on Learning Representations, 2023. URL htalltttgee///perrrrrrreeeeeeerr (ttps//rrrrrrdddd._ggggaaarrr
[77]
Sha.liiy....biiiilieaooccccooton (revision 5d4cfe8). 2023. URL https /uw.giaghhee..coaaaaaarr//////ad ine-
[78] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens. and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, 2016. doi:
10. 109/CVPRR.216..22
[79] Hossein Talebi and Peyman Milanfar. Nima: Neural image assessment. IEEE Transactions on Image Processing, 27(8):3998-400 2018. doi: 10.1 18.100000..2111828311111
[80] Hossein Talebi and Peyman Milanfar. Learned perceptual image enhancement. In 2018 IEEE International Conference on Computational Photography (ICCP), pages 1-13, 2018. doi: 10. 10111/CCCHHT.2218.868888
[81] Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the Interna- tional Workshop Software Fairness, FairWare '18 page 1-7, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 978 75785553337433 doi: 10.1145/194777031947766
URL https://doi. .ooo//o...o.//ooooo...oooooooeoo 10. 1145 /1/.//999477000 319777
[82] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald. Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 01138/s1592-019-000
[83] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In The European Conference on Computer Vision Workshops (ECCVW), September 2018
[84] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. Towards real-world blind face restora- tion with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
16

---

[85] Xintao Wang, Liangbin Xie. Chao Dong, and Ying Shan. Real-esrgan: Training woal-world blind super-resolution with pure synthetic data. International Conference on Computer Vision Workshops (ICCVW), 2021.
[86] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffu- sion null-space model. The Eleventh International Conference Learning Representations, 2023.
[87] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-61, 2004. doi: 10.1109/T1P.200..8111 81986..
[88] Zhouxia Wang, Jiawei Zhang, Runjian Chen, Wenping Wang, and Ping Luo. Restoreformer: High-quality blind face restoration from undegraded key-value pairs. 2022.
[89] Zhouxia Wang, Jiawei Zhang, Tianshui Chen, Wenping Wang, and Ping Luo. Restoreformer++: Towards real-world blind face restoration from undegraded key-value paris. 2023.
[90] Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software 6(60):3021, 2021. doi: 10.21 0332)) URL https:://boi.orr/100. 2ttr//10.2105/joss.. 03021
[91] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G. Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16293-16308. June 2022.
[92] Ke Yang and Julia Stoyanovich. Measuring faimess in ranked outputs. In Proceedings of the 29th International Conference on Scientific and Statistical Database Management, SSDBM '17, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 9781450352826. doi: 10. 011144/0883544.085552222 URL htttps:///ol: org/10. 0145/3085500 3085526
[93] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[94] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez. and Krishna P. Gummadi. Fairness Constraints: Mechanisms for Fair Classification. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pages 962-970. PMLR, 20-22 Apr 2017. URI. htt.hhtps: htttps://prooceedings. mll. ppwss//v4/zafaaa..
.mlr.
[95] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi. and Cynthia Dwork. Learning fair repre- sentations. In Sanjoy Dasgupta and David McAllester, editors. Proceedings of the 30th International Conference Learning Research, pages 325-333, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https: attps://proceedings..u mlr. www.srpppesss/ee paress/v2//zemae/ssee w2w/v/eee1.....eeeeeec html
Machine Learning, volume 28 of Proceedings Machine
[96] Cheng Zhang, Xuanbai Chen, Siqi Chai, Henry Chen Wu, Dmitry Lagun, Thabo Beeler, and Feraando De la Torre. ITI-GEN: Inclusive text-to-image generation. ICCV, 2023.
[97] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, (177)31142-1155 2017.
[98] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. IEEE Conference on Computer Vision and Pattern Recognition, pages 3929-3938, 2017.
[99] Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Lue Van Gool, and Radu Timofte. Plug- and-play image restoration with deep denoiser prior. arXiv, 2008 13751, 2021.
17

---

[100] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In IEEE International Conference on Computer Vision, pages 4791-4800, 2021.
[101] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
[102] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. Real-time user-guided image colorization with learned deep priors. ACM Transactions on Graphics (TOG), 94), 2017.
[103] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang, The unreason- able effectiveness of deep features a perceptual metric. In CVPR, 2018.
[104] Zhou, Shangehen, Chan, Kelvin C.K., Li, Chongyi, and Chen Change Loy. Towards robust. blind face restoration with codebook lookup transformer. In NeurlPS, 2022.
18

---

Name / Notation
Meaning Formal definition
X
Ground truth image (a random vector)
Y
Degraded measurement (a random vector)
X
Reconstructed image (a random vector) P.d.f of the ground truth images
Px
PY
P.dff of the degraded measurements
Pg
P.d.f the reconstructed images
Perceptual Index (PIa or PI)
d(px.Px'
A
Sensitive attribute (a random vector) P.d.f of the ground truth images of A
Pxla (-la)
PYJA
P.d.f of the degraded measurements A = a
PAIA (-la)
P.d.f of the reconstructed images of A
Na
supp P
y
suppPy]A(-10
&a
suppPxia("1a
Group Perceptual Index (GPIa(a). GPI,, or GPI)
Group Precision (GP(a) or GP)
P(X E x=
Group Recall (GR(a) or GR)
P(X E xo|A=aa
Representation Demographic Parity (RDP)
Val, 02 GP(a1) = =G((a2)
Proportional Representation (PR)
Va: P(X E Na) = P(f E Xa)
Conditional Proportional Representation (CPR) Perceptual Fairness (PF₂ PF)
Va, y P(X E No = y) = P( E Ya =
Val, 02 GPIa(a1) (Plx(a2)
Table 1: Summary of mathematical notations and fairness definitions used in this paper.
A Summary of mathematical notations and fairness definitions
We summarize in TableItth the mathematical notations and fairness definitions used in this paper.
B
Toy signal restoration example
The following toy signal restoration example demonstrates that common estimators (e.g., the stochas- tic estimator which samples from the posterior distribution Pxlx) do not trivially achieve perfect PF.
1.00
PFágy
1.00
PFW,
GPlaTy (0)
0.25
Npostocior
XPosterior
0.50
8MSs--i
GPIws
0.50
0.00
0.25
0.25
PX (a
Pasterioe La (-1a)
0.00
0.00
P.R. Miggl (ila)
PAMSRRPIIa (1a)
0.0
0.5
0.3
1.0
GPIapy (1)
GPIw,
probability density functions Pxja(-la), P PXysela((lo)) Px ia (-laa,, and PA (-la), where = (left plot) or plot). Right: The GPLay, and GPIw; of each group (associated with a = 1 or 0). The dotted lines PF PFw, correspond to the points where perfect PF or perfect PFw, is achieved, respectively. It is clear that all three estimators achieve sub-optimal PFaty and sub-optimal PFw, See Appendix [ooo more details.
Figure 5: Illustration of Example 1
Left:
Conditional
0 (right
Example 1. Suppose that X, NO, 1) are statistically independent random variables, and let Y X + N. In this case, it is known that X MSE = = is the estimator that atlains the lowest possible Mean-Sques Seinnt-qurred--Fre Error (MSE), X Penerror 1 + W where W N 1) statistically independent of X and Y, is the estimator that samples from the posterior distribution Px(Y, and X XMSE+P J/2
19

---

is the estimator that attains the lowest possible MSE among all estimators that satisfy Px = Px (perfect Pla) 0O 23y. Now, consider the "sensitive attribute" A xx21. All of these commonly used estimators produce much better (lower) GPI dyy and GPIw, for the group associated with A 0, which, in this case, is majority satisfying P(A = 0) R2 0.8413 (see Figure 5
Conditional density plots
The density (x]a(z)a) is obtained using the closed form solution of a truncated normal distribution,
d(x)
(xIA(T|1) ((x0) +(1)²
(17)
6(x)
PxIA(z(0)= (1) - (-00))
(18)
where (bx) is normal density and (xx) its cumulative distribution,
2T E t=²
6(x)
(19)
1
((x)
2
1 erf
V2
(20)
and px Px (x[1) 0 and (x 0 for every 2 1 and 1, respectively. The densities are obtained by feeding these algorithms with the P (1a)) M (-la) and Px P (-
0), sepa- rately. This is achieved by generating samples x ~ px and ~ prIx [x), and then partitioning these samples into sets of measurements based the value of We then perform Kernel Density Estimation (KDE) (44 on the reconstructions of each group to obtain their density, using the function seaborn. kdeplot [90] with the arguments bw 2, common, norm=False, gridsize=200, The number of samples used to compute the KDE is set to 200,000 for both a = 1 and 0.
degraded measurements corresponding to X 2 1 (for a 1) and X < 1 (for
B.2
Computation of the total variation distance drv and of the Wasserstein distance Wi
The value of GPI (a) for given algorithm X is defined by the total variation distance
GPI (a) div div(ppxi((()) - 1 /
(xxaa(aa)) (x)a) dr.
(21)
To compute this integral, we use the function scipy. integrate. quad [82] with parameters (a=-1000, b=1000, limit=500, points=[1.0]] At each point æ, the integrand
pxia(=]a) - A (xlaa
(22)
evaluated using the closed form solution of (-1a) and the pre-computed KDE density each (.aa.
The value of GPIw, (a) for given algorithm X the Wasserstein 1-distance be- tween Px]a(-la) and (-la). To approximate this distance, we utilize the function scipy.stats stats. wasserstein distance with the previously obtained 200,000 samples from (l)a(-la) and 200,000 samples from (x)aa-laa).
C Proof of Theorem
Theorem 1. The Group Precision and Group Recall of any restoration method satisfy
GP(a) 2 GPIa,,
(13)
GR(a) > 1 GPI (a),
(14)
for all a E SuPPPA-
20

---

Proof. For every a, it holds that
( 2 min {px a(xla), pxIa (x)a())a)
(23)
Moreover, the value of min (xxIa is zero for every x & supp supp Px)a(taa) so
J / pxiA((a)())))
min (xxp = min { px (x)la) Pxx (x) Jdz. dr.
(24)
Thus,
GP(a) = PP E xalA = a)
(25)
P(X supppx/s(Haal4 a)
(26)
-amppxiattt) (=)a(
(27)
/ supppxia((k)
min (pxIa(t(a), PXIA (cola)
(28)
min ( for
(29)
+ (x]a(t)aa) [Pxia(ela) pxia(ola))) dr . (PRilttoa)PXxAatttt
(30)
1 / (Pxia(ola) + +XxIa(=aa) dr - 2 / ( - pxiaatlaa dr
= 2
(31)
( (a
(32)
GPIan (a).
(33)
By replacing the roles of Px A (a and (x)a())a), the result GR(a) 2 1 - GPL (a) can be derived with identical steps using the same mathematical arguments.
D Proof of Theorem 2
Theorem 2. Suppose that 3a, 02 supp PA such that
P(X E Xay n Xny A = a;) < P(Y E Va, nV = az); for both i = 1,2, where Xa, = supp Px a (-(a;) and Va. supp Pr]a Then, GPIa(a)) and GPIa(az) cannot both be equal to zero.
(15)
Proof. Suppose by contradiction that Pxia(11a;) Px(a(-la;) for both i = 1, Thus,
1 F E Xa 14 a+)
(34)
= P( E X AA a;)
(35)
-Je, Xx;
(36)
= =J |
P (olo))dyy
(37)
Xxx,
/ [ PRIAY (x/as (PY (ylaa
(38)
No.
( I (
Jya Xx.
(39)
= -f (yla)) (Jan, PKlx (zlw)de dy
(40)
you
=/0 (( E XailY
y)dy,
(41)
21

---

where Equation (3) holds from the assumption that A and X are statistically independent given Y. and from the fact that (yja(y)a,)
0 for every y Yx.. We will show that P(X E Xa, = = 1 for almost every y E Va. Indeed, if this does not hold, then for some T S Va, with P(Y T=|A= a:) > 0 we have P( E Xa; y) for every y E Tí. Thus,
[ PYIA( E Xa, y)dy
(42)
yo,
VT. J Priaola)PP(E & Ka,lY = y)dy + / Pria(vla)PP(X E Na, Y yddy
(43)
= [ PrIa( CATT = yydy Pria(ula)ddy
(44)
Prin(yla)dy++ + / Prialolaiddy
(45)
VT.
Pyla(yla:)dy
(46)
Vos
=
(47)
which is not possible. So, IP(x E Na, I y) = for almost every y E Voy. Now, from basic rules of probability theory, we have
P(X E No, n N2, A = 01) =P(X E Ka, |A- = a1)
P RaylA= a1) - P E Na, U -azlA=
(48)
where the first and last terms on the right hand side cancel out (from the definition of Xa,, they are both equal to 1). Thus, we have
PPX E Xa, n No = a1) = P(X E A = a1);
(49)
and finally.
P(X E Na, nxz2/4 = a1)= E Xe, AA=
(50)
= =P( E xo
(51)
= =/o, PrIA(Ula)P(( E Xag yddy
(52)
yon
(M E NoolY = yydy
(53)
/ Ming
Pria(yla))dy
(54)
= P(Y E Vo,
01),
(55)
where Equation 51
follows from the contradictory assumption that P ( aa = (+),). Equa- tion (52) follows from the same steps that led to Equation (41), and Equation (54 follows from our pre-
vious finding that PP E Xa, Y y) = for every y E Va, (we have E Da, nya, in the integrand,
so y ). However, it is given that P(X E Xa, O nX AA = a,) < P(Y E ) 1) 1 a1), so we have established contradiction.
E Proof of Theorem 3
Theorem 3. Suppose that A takes discrete values, X attains perfect Pla (px = Px), and 3a, am
supp such that GPIa(a) > 0 and IP(A = am) > 0.5. Then, X cannot achieve perfect PF
Proof. Suppose that GPI (am) = 0. From the assumptions, there exists a * am such that GPLa(a) 0, GPL (a) > 0. This means that PF is not perfect.
22

---

Otherwise, suppose that GPI (am) > 0. Thus, from Theorem we have
P(A am)
GPl (Am)
5
max a'zum
GPI (a')
(56)
IPA = am)
max GPI (a) a' Fum
(57)
GPId (a"),
(58)
where Equation 57
holds since 1-P(A=am) p(Amam)
< 1, and Equation (58) holds by defining
= arg max GPI (a').
(59)
Thus, we have found two groups 0m and a² such that GPIdy (am) < GPIdty (a") s0 PF cannot be perfect.
F Proof of Theorem 4
Theorem 4. Suppose that A takes discrete values and X attains perfect Pla (Px = px)) Then,
(P) S P(A a) a'$a
P(A== al==a')6e d')Glla
(16)
for every with P(A = a) > 0.
Proof. For every let us denote Pa IP(A a). Suppose that X attains perfect perceptual index, so Px = px. From the marginalization of probability density functions, it holds that
Px(=) Σ (xla),
(60)
Px(x)= Σ PaPx)A(Ela))
(61)
and since Px = Px we have
Σ (apx) = E (Tl))
(62)
Let be some group with Pa > By rearranging Equation
662 we get
(x(p (x(a)= = Σ Pa (a (x)a())a))).
(63)
a'yla
Taking the absolute value both sides, we have
Pa px]a(<)()) Px a(x|aa
Σ Pa (x (xla)))aa
(64)
a'ata
Σ Pa (Xla())aa)) (xla())
(65)
a/
where Equation
65 follows from the triangle inequality Thus, it holds that
dry(pxia((e) = - /1xxiaaola))
(ala) dx Σ Par (XIA(2)a')) Px(a(x)aa)) dx
/
1
<
/ P.
(66)
a'ıta
1
1
(/lpxiitlle) - pxiulola()dds
Σ Pa
(67)
Pa
a'Fa
(68)
P
a'za
This concludes the proof.
23

---

G Face image super-resolution www.rresssciiooocoomemeeeta details and results
G.1 Synthetic data sets
All the Ce images we use are of size 512 x 512. The image-to-image translation model we O, is sourced from Hugging Face [77] and boasts over 1,200,000 downloads (at the time writing this paper). This model integrates SDXL [67] with SDEdit [53]. For all groups,
utilize, stabilityai
adjust the hyperparameters strength and guidance seceance__caleeeeeee_ee scale from their default settings, with strength set to 0.4. When translating a CelebA- HQ image into a group image using its specified text instruction (see Table 2), choose the smallest value from [8.5, 9.5, 10.5, 11.5, 12.5) as the guidance quidance_scaae hyperparameter, such that the resulting image is classified belonging the group. Otherwise, if none of these gui dance scale values work for some group (i.e., their class is incorrect), we discard all the translations of æ from all groups. To clarify, this means that the translated images for different groups may use different aadaaaneecaallle_l values, as long as all translations are correctly classified. The text instructions we use for each group are provided in 2 For all groups, we use the same rngative_pronpee text instruction "ugly, deformed, fake, caricature Each of the resulting groups contains 1, 1,356 images of size 512 x 512, In Figures
20 we present 130 image samples from each group.
Group
image-to-image translation text instruction
Old&Asian Young&Asian
120 years old human, Asian, natural image, sharp, DSLR 20 years old human, Asian, natural image, sharp, DSLR 120 years old human, natural image, sharp, DSLR 20 years old human, natural image, sharp, DSLR
Old&non-Asian Young&nonnAAiann
Table 2: Text instructions for the image-to-image translation model to generate images of each faimess group.
See Section 4 and Appendix G.T for more details.
G.2 Visual results
Visual results of all algorithms (the reconstructions of each fairness group) for E {4, 8, 16, 32} and
ON {0, 0.1} are provided in Figures 21 28
G.3 Additional levels of additive noise
Figure 3 presents quantitative results with all scaling factors, and without adding white Gaussian noise ( 0). Here, in Figures 6 and 7 we report the results with ON E (0.1, 0.25). We observe similar trends and conclusions as in Figure (please refer to Section for more details).
G.4 Comparing GPIFID instead of GPTrid
We report in Figures g to 10 the GPIED of each group, where FID is the Fréchet Inception Dis- tance 30 These results show trends similar to those observed in Figure 3 Namely, using the statistical distance FID instead of KID does alter the trends and conclusions of the results.
G.5 Additional group metrics
We report, compare and analyze additional group performance metrics.
GP and GRNN We approximate the GP and GR of each group using 45 method which evaluates the precision and recall between two distributions in their feature space. We denote the results by GP and GRNN, respectively. Note that this approach to approximate GP differs from our previous experiments, where we use the classification hit rate (Figures 6 [nnd]] 77. Similarly
to the experiments where we compute GPIRID (Section 4.2) and GPIED (Appendix G.4). GP and
GR are computed by extracting image features using the last average pooling layer of the FairFace combined age & ethnicity classifier [35].
24

---

GPSNR and GLPIPS For each group we compute the Peak Signal-to-Noise, Ratio (PSNR) and the Learned Perceptual Image Patch Similarly (LPIPS) [03 where these metrics are evaluated by feeding the restoration algorithm only with the group's inputs and with respect to the group's 's ground truth images. Formally, we define the Group PSNR (GPSNR) and the Group LPIPS (GLPPPS) as
(a) = [E[PSNR (X, X) = []. GLPIPS(a) E|IPIPS X, X) a],
(69)
(70)
where the expectation is taken over the joint distribution of a group's ground truth images and their reconstructions, Px, &x, (;-/a)
The results for all noise levels ON E {0.0, 0.1, 0.25] are provided in Figures 8 to 10 First, note that both the GPSNR and the GLPIPS metrics are unreliable indicators of bias. For example, the metrics GP, NP GPIKID. and GPIFD all indicate that the group young&non-Asian receives better treatment than the group young&Asian (e.g., the GP of the former group is clearly higher than that of the latter group across all noise levels and scaling factors). However, both groups exhibit roughly similar GPSNR and GLPIPS scores. This highlights why assessing the faimess of image restoration algorithms solely based on GPSNR, GI or similar metrics (MSE, SSIM [8 etc.) might not be sufficient. This result regarding GPSNR is not surprising,
it is well known that such metric often does not correlate with perceived image quality [10]. Regarding GLPIPS, it might be more effective to use image features extracted by a classifier trained to identify the sensitive attributes in question. We leave exploring this option for future work. Second, the GP values in Figures g to 10 are almost identical the GP scores reported in Figures 6 and
This suggests that approximating the true GP
either through the classification hit rate (as in Figures
G oand 7 or via 45 (as done in this section),
are consistent. Third, the GRyn scores suggest potential unfairness in the perceptual variation across different groups. For example, when = 16, ON = 0, we observe that all algorithms consistently produce higher Ge scores for the young&non-ASet Asian group compared to the young& Asian group.
G.6 Feature extractors ablation
We employ the dinov2-vit-g-14 662 clip-vit-1-14 668, and inception-v3-comge [78
feature extractors via torch-fidelity [57 compute the GPIKID for each fairnsss group (previ- ously, we used the image features extracted from the FairFace classifier's 's final average pooling layer). The results are presented Figures 11 to 13
The outcomes from both the 2inov2-vit-g-2 and clip-vit-1-1- feature extractors generally align with those of the FairFace image classifier, though the biases exposed by these extractors less pronounced. Put differently, computing GPIKID with either of these general-purpont feature extractors leads smaller disparity in the GPIRID of the different fairess groups. Moreover, the ceapption-v3-oooe
image feature extractor yields inconsistent results, suggesting that the old& Asian group receives more favorable treatment compared the old&non-Asian group (contrary to the biases indicated by the other feature extractors). The following section strengthens our argument that this behavior of inception-Ve comppat is undesirable. Overall, relying on such image feature extractors seems unsatisfactory for the purpose of uncovering nuanced biases in face image restoration methods.
G.7 Considering age and ethnicity as separate sensitive attributes
we reveal a significant discrepancy between PF and RDP regarding whether the groups Ald&Asiin and oldinon-Asian are treated equally. Specifically, both groups achieve similar GP, while the GPIRID of the latter group cold&non-Asian) is notably better (lower) than that of the former group (old&Asian Asian). In other words, GPIKID indicates that the non-saann group enjoys better preservation of ethnicity.
In Section 4.2
Let us support our claim in Section 4.2 that this outcome of PF is the desired one, by showing that RDP may obscure the fact that some sensitive attributes are treated better than others. Indeed, as shown in Figure 14 the ethnicity of the old&non-Asian group is better preserved than that of the old&Asian group, while Figure confirms that the age of these two groups is equally preserved.
Future work may investigate the utility of no- no-reference perceptual quality measures (c.g.. assess fairness in image restoration.
54
55
79
to
25

---

While RDP fails to uncover this ethnicity bias when the fairness groups are determined based on both age and ethnicity, PF clearly reveals it.
G.8
Final details
All algorithms are evaluated using the official codes and checkpoints provided by their authors. We use the torch-fidelite ty package [57] (GitHub commit a614 a61422f) compute the KID [9], FID 30)
precision and recall [4)]. The GPSNR and the GLPIPS are computed using the piq package 38 39 (version 0.8.0 in pip).
Finally, note that some of the evaluated algorithms generate output images of size 256 x 256 (e.g., DDNM), while others produce images of size 512 512 (e.g., RestoreFormer). Consequently. for fair quantitative evaluations, we resize the outputs of the latter algorithms, along with the ground truth images, 256 256. To clarify, the super-resolution scaling factors calculated based on the 256 x 256 image size. For instance, when s = 4, the resolution of the input images is 64 x 64.
H Adversarial attacks - complementary details
The degradation we apply consists of three consecutive steps: (1) Average pooling lown-sampling with scale factor of = 4, (2) additive white Gaussian noise with standard deviation of σы = 0.1. and then (3) JPEG compression with quality factor of 50. We attack each degraded image using a tweaked version of the I-FGSM basic attack [14] with = 6/255 and T = 200. In particular, instead of using the L2 loss I-FGSM like in [14), we forward each attacked output through a classifier that predicts the age category of the output face image (56), and then maximize the tog-probability of the oldest age group category. other words, we forward each degraded image through RestoreFormer++ and then feed the result to the age classifier. We then the I-FGSM update rule to maximize the soft-max probability of the oldest age category (this adversarial attack technique was employed in (60)).
I Additional experiments image denoising and deblurring
We conduct additional experiments on image denoising and deblurring to further demonstrate the utility of the proposed notion of perceptual fairness. Specifically, for image denoising we use additive white Gaussian noise with standard deviation ON = 0.5, and for image deblurring we use a Gaussian blur kernel of size k and
10, and add the blurred image white Gaussian noise of standard deviation ON E {0.1, 0.25, 0.5}. Since these degradations are not handled well by the GAN-based methods, we only compare DPS, DDNM DDRM, and PiGDM.
Quantitative results are reported in Figure I6 and visual comparisons are provided in Figures 29 32 As in the super-resolution experiments, PF is able expose bias (which is also visually clear) when RDP fails to do so, and not vice versa.
J Computational resources
All our experiments are conducted on NVIDIA RTX A6000 GPU,
26

---

4,0% 0.1
ON 0.1
32, n. 0.1
Young
Asian
Young
Non-
Aslan
Old
Asian
0.0
1 GP(a)
1 GP(a)
1 GP(a)
GP(e)
Young Asian
Young
Non-Asian
Asian
Old
Non-Asian
GPIgID (e)
(P)
CPPllD (u)
GP]IID
DDNM DDRM
RestoreFormer
DIBBIR GFPGAN
CPEN
VQPR
PIGDM
CodeFormer
RestoreFormer
Figure 6: Experiments similar Figure but when the standard deviation of the additive white Gaussian noise
dN 0.1.
0.25
0.25
16, O. 0.25
32, 0x 0.25
Young Asian
Young
Non-Asian
Asian
Asian
t GP(n)
1 GP(a)
GP(s)
GP(e)
Young
Asian
Young
Non- Asian
Old
Asian
AsiaA
(P)
(PIgID(a)
(PlgID(a)
G.PIIID(e))
DPS PiGDM
DDNM DDRM
RestoreFormer CodeFormer
DiEBIR GFPGAN
GPEN
VQPR
RestoreFormer
Figure 7: Experiments similar Figure but when the standard deviation of the additive white Gaussian noise
N = 0.25.
27

---

N 0.0
8, oN 0.0
16,0x dN
32, ON
Young
Asian
Young
Non-Asian
Old
Asian
Old
(PIrI((a)
(Plen(
GPipin(a)
Young
Asian
Young
AonnAniinn
Old
Asian
Old
0.0
(P)
GP
GPNN(a)
GPNN
Young
Asian
Young
AonnAsiinn
Old
Asian
Old
0.5
1.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
(R)
(R)
GRNN(()
(G)x(())
Young
Young
Non-Asian
Old
Asian
Old
Aon-A
10
10
30
30
GPSNR(()
GPSNR(c)
GPSNR(a)
GPSNN(a)
Young
Non-Asian
Old
Asian
Old
Aoinn
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
GLPIPS(A)
GLPIPS(s)
GLPIPS(())
GLPIPS((a)
DPS
DDNM DDRM
RestoreFormer
DIABIR GFPGAN
CPEN
VQPR
PIGDM
CodeForner
RestoreFormer
Figure 8: Evaluation of additional group metrics where the additive noise level is ox = 0.0 and the super- resolution scaling factor is E {4,8, 16, 32). Please refer Appendix G for details.
28

---

N, 0.1
8, oN 0.1
== 16, av 0.1
32, ON 0.1
Asian
Young
Non-Asie Asian
Old
Asian
Old
600
400
200
(PIpin(a)
(Plpm(()
(P)p
(P)
Young
Asian
Young
Non- Asian
Old
Asian
Old
1.0
0.0
(M)
GPN
GPNN(()
1 (P)
Young
Asian
Non- Acian
Old
Asian
Old
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
(R)
GRNN(1)
(G)N
1 (R)
Young
Non-Asian
Old
Asian
Old
Non-
10
20
20
GPSNN(a)
GPSNR((e
GPSNR(a)
GPSNR(a)
Young
Non-Asian
Old
Asian
Old
0.2 0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.0
0.2
0.4
0.6
GLPIPS(a)
GLPPIS((c)
GLPIPS((a)
(.LPIP((a)
DPS
DDNM DDRM
RestoreFormer
DIABIR GFPGAN
CPEN
VQPR
PIGDM
CodeForner
RestoreFormer
Figure 9: Evaluation of additional group metrics where the additive noise level is ox = 0.1 and the super- resolution scaling factor is E {4,8, 16, 32). Please refer Appendix G for details.
29

---

1,0N 0.25
8, ON 0.25
16, ox 0.25
32, ON 0.25
Asian
Young
Non-Asian
Old
Asian
Old
200
(a)
(P)
(PIFID(a)
GPIpID(a)
Young
Asian
Young
Non- Asian
Old
Asian
Old
Non
0.25
0.25
1 (PNN(()
1 Gx
(P)N(()
GPNN(
Young
Asian
Sott-Asian
Old
Asian
Old
0.00 0.25
0.50 0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
(R)
(R
(a)
GRNN
Young
Non-Asian
Old
Asian
Old
Non-Asian
10
10
20
GPSNR(a)
GPSNR(s)
1 GPSNN(aa
GPSNN(a)
Young
Non-Asian
Old
Asian
Old
Non-Asian
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
GLPIPS(()
GLPIPS(a))
GLPIPS(a)
G.LPIP((a)
DPS
DDNM DDRM
RestoreFormer
DIABIR GFPGAN
CPEN
VQPR
PIGDM
CodeFormer
RestoreFormer
Figure 10: Evaluation of additional group metrics where the additive noise level is on 0.25 and the super- resolution scaling factor is E {4,8, 16, 32). Please refer Appendix G for details.
30

---

4, 0.0
8, 0.0
16, 0.0
32, 0.0
Young
Young
Non
Old
Old
Non-Asian
GPIKa(()) (n)
4 (P]
(P]RID(e)
GPIIiD (o)
DPS
DDNM
RestoreFormer
DiTBIR
CPEN
VQFR
PIGDM
DDRM
Codeeoorerr
GFPGAN
KostoroForree
Figure 11: Using the dinov2-vit-g-L 14 feature extractor 62 via torch-fidelity [57
compute the
GPIRDD of each group. This general-purpose feature extractor network is somewhat able detect bias between the Asian and old&non-Asian (as detected before by extracting features from FairFace image classifier). However, the bias is significantly less pronounced in this case.
0, 0.0
8 0.0
16,
0.0
32,
0.0
Young
Young
Asinn
Old Asian
Old
Non-Asian
4. (M)
(PP
(PPgID(())
DPS
DDNM
RestoreFormer
DiEBIR
GPEN
VQFR
PIGDM
DDRM
CodePormer
GRPGAN
RestoreFormer
Figure 12:
Using the clip-vit-1-2 feature extractor 688 via torch-fidelle 57
compute the GPlaD
of each group. Even this general purpose feature extractor network somewhat able detect some bias between the Asian and old&non-Asian (as detected hefore by extracting features from FairFace image classifier). However, the hias is significantly less pronounced in this case.
31

---

N
8, NN
16, ON
Young
Asian
Non-Asian
Asian
Non-Asian
0.000 0.025 0.050 0.075
0.000 0.025 0.050 0.075 0.000 0.025 0.050 0.075
0.000 0.025 0.050 0.075
GPlgi(()
(Plgi
GPIgiD(a)
(GIggi(()
RestorePormer
PiGDM
DDRM
CodeFormer
GFPGAN
Rormere
compute GPIRD of each group. These results of Dnception-v3-comge hint that the old& Asian group in some cases receive better treatment than the oldınon-Asian group, while all the other feature extractors suggest the opposite bias. This outcome inception-v8 compat is also inconsistent with the experiments in Appendix G.7 which demonstrate that the old&non-Asiin Asian group is the one receiving the better treatment.
Figure 13: Using the inception- v----v--coa- -o-ppppptttctoccmmmmo feature extractor
torch-fidelity St
32

---

Ethnicity is the only considered sensitive attribute
ov 0.0
8,
ON
0.0
32, on
0.0
(Young)
(Young)
Non-Asian
(Old)
(Old)
Non-Asian
0.0
0.5
1.0 0.0
0.5
1.0 0.0
1.0 0.0
0.5 GP())
(P(s)
GP(a)
GP())
4,
8.0N
16,
ON
(Young)
Asian
(Young)
Non-Asian
(Old)
Asinn
(Old)
Asian
0.0
0.5
1.0 0.D
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0
GP(a)
GP(a)
GP(a)
GP(a)
4, .. 0.25
16, ON
32, aN
(Young)
Asian
(Young)
Non-Asian
(Old)
(Old)
Non-Asian
0.0
0.5
1.00.0
0.5
1.00.0
0.5
1.00.0
0.5
1.0
GP(s)
GP(u)
GP(s)
GP(u)
DPS
DDNM DDRM
RestoreFormer
DiBBIR
CPEN
VQPR
PIGDM
CodeForrer
GFPGAN
Reatore
Figure 14: Evaluating the GP of each group. where ethnicity is the only considered sensitive attribute. Here, the groups Asian and young& Asian each considered Asian, and the groups old&non-Asian and young&non- Assann are each considered as non-Asian. For clarity, we still specify in each bar plot the correspond- ing age of each group, but the classifier operates solcly on ethnicity (i.e., the GP is approximated with respect to ethnicity alone). As we claim in Section 4.2 the ethnicity of the old&non-AAse Asian group is clearly preserved better than that of the Alda group.
33

---

Age is the only considered sensitive attribute
0.0
16, ON
0.0
32, on
0.0
Young
(Asian)
Young (Non Asian)
Old
(Arian)
(Non-Asian)
0.0
0.5
1.0 0.0
0.5
1.0 0.0
1.0 0.0
0.5 GP())
GP(N)
GP(a)
GP(n)
4,
8.0N
32, ON
Young (Asian)
Young
[Non-Asian)
Old
(Asian)
Asian)
0.0
0.5
1.0 0.D
0.5
1.0 0.0
0.5
1.0 0.0
0.5
1.0
GP(a)
GP(a)
GP(a)
GP(a)
4, .7
0.25
16, mM
32, AN
Young
(Asian)
Young
Non-Asian)
Old
(Asinn)
Old
(Non-Asian)
0.0
0.5 CP(s)
1.0 0.0
0.5
1.0
0.5
1.0 0.0
0.5
1.0
GP(a)
GP(x)
GP(u)
DPS
DDNM DDRM
RestoreFormer
DiBBIR GFPGAN
CPEN
VQPR
PIGDM
CodeFormer
Reatore Former
Figure 15: Evaluating the GP of each group, where age is the only considered sensitive attribute. Here, the groups Alian and old&non-Asian each considered as old, and the groups y and young&non- Asian are each considered as young. For clarity, we still specify in each bar plot the corresponding ethnicity of cach group, but the classifier operates solely on age (i.e., the GP is approximated with respect to age alone). As we claim in Section 4.2 the age of both the old&non-Asian and old& Asian groups is (roughly) equally preserved.
34

---

Denoising, ON = 0.5
Deblurring, ON 0.5
Deblurring, ay 0.25
Deblurring, ox 0.1
Young
Non-Asian
Old
Asian
Old
Non-Asian
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 1 GP())
1.0
0.0
0.5
1.0
t GP(s)
t GP(s)
1 GP())
Young
Asian
Young
Non- Asian
Asian
Old
0.0
2.0
5.0
7.5
10
0.0
2.5
0.0
7.0
GPlgD(()
GP]RD(()
GPigin (a)
(P)
PiGDM
DDNM
DDRM
Figure 16: Experiments similar to Figure 3 but on the image denoising and deblurring tasks described in Appendix[] We observe similar trends in these tasks as well. Namely, in the super-resolution tasks, PF exposes clear bias when RDP does not (but not vice versa).
35

---

B
Examples of generated images for the old& Asian user group. These samples were generated by passing images from the CelebA-HE -QQ test partition (36 through the SDXL image- image-to-image to-image model. The text instruction used was "120 years old human, Asian, natural image, sharp, DSLR' The FairFace ethnicity and age classifier [35] categorizes all these images as belonging to either Southeast Asian or East Asian ethnicities, and to the 70+ age group.
Figure 17:
36

---

KEI
Figure 18: Examples of generated images for the young& user group. These samples were generated by passing images from the CelebA- HQ test partition (36 through the SDXL image-to-image model. The instruction used was '20 years old human, Asian, natural image, sharp, DSLR" The FairFace ethnicity and age classifier [35] categorizes all these images belonging to either Southeast Asian or East Asian ethnicities, and to any age group younger than 70 years old.
37

---

T
Figure 19: Examples of generated images for the Lalknon-Asiial user group. These samples were generated by passing images from the CelebA-HO HQQ test partition (36) through the SDXL image-to to-image model. The text instruction used was "120 years old human, natural image, sharp, DSLR". The FairFace ethnicity and age classifier (35] categorizes all these images as belonging ethnicities other than Southeast Asian or East Asian, and to the 70+ age group.
38

---

G
NNN
Figure 20: Examples of generated images for the young&non-Asinn user group. These samples were generated by passing images from the CelebA HQ test partition (36) through the SDXL image- to-itage model. The text instruction used was "20 years age classifier (35] categorizes all of these images as helonging to ethnicities other than Southeast Asian Fast Asian, and to any age group younger than 70 years old.
human, natural image, sharp, DSLR" The FairFace ethnicity and
39

---

Visual results for == 4, dv 0.0
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(3)
(4)
(6)
(8)
(9)
Figure 21: Face image super- resolution for each fairness group, where = 4, d = 0. (0) DDRM, (1) VQFR. (2) CodeFormer, (3) DDNM (4) RestoreFormer + +, (5) GPEN, (6) DPS, (7) GFPGAN, (8) PiGDM. (9)
RestoreFormer, (10) DifiBIR. Zoom in for best view.
40

---

Visual results for == 4, av 0.1
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(3)
(4)
(6)
(8)
(9)
Figure 22: Face image super-resolution for each fairness group. where 4, = 0.1. (0) DDRM. (I)
VQFR, (2) CodeForme, (3) DDNM (4) RestoreFormer + +. (5) GPEN, (6) DPS, (7) GFPGAN, (8) PIGDM, (9) RestoreFormer, (10) DifBBIR. Zoom in for best view.
41

---

Visual results for == 8, dv 0.0
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(3)
(4)
(6)
(8)
(9)
Figure 23: Face image super- resolution for each fairness group, where s = 8, d = 0. (0) DDRM, (1) VQFR. (2) CodeFormer, (3) DDNM (4) RestoreFormer + +, (5) GPEN, (6) DPS, (7) GFPGAN, (8) PiGDM. (9)
RestoreFormer, (10) DifiBIR. Zoom in for best view.
42

---

Visual results for == 8, av 0.1
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(3)
(4)
(5)
66)
(7)
(8)
(9)
Figure 24: Face image super-resolution for each fairness group. where 8, = 0.1. (0) DDRM. (I)
VQFR, (2) CodeFormer, (3) DDNM*. (4) RestoreFormer + +. (5) GPEN, (6) DPS, (7) GFPGAN, (8) PIGDM, (9) RestoreFormer, (10) DiftBIR. Zoom in for best view.
Visual results for 16, ON 0.0
Young & non-Asian
Old non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 25: Face image super-resolution for each faimess group, where a = 16, ox (0) DDRM, (1)
DDNM (2) DPS, (3) PiGDM. Zoom in for best view.
43

---

Visual results for s 16, ON 0.1
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 26: Face image super-resolution for each faimess group, where 8 = 16, ox 0.1. (0) DDRM, (1)
DDNM (2) DPS, (3) PiGDM. Zoom in for best view.
Visual results for = 32, ON 0.0
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 27: Face image super-resolution for each fairness group, where 32, ax = (0) DDRM, (1)
DDNM (2) DPS, (3) PiGDM. Zoom in for best view.
44

---

Visual results for = 32, ON 0.1
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(8)
Figure 28: Face image super-resolution for each faimess group, where 8 = 32, .x 0.1. (0) DDRM, (1)
DDNM (2) DPS, (3) PiGDM. Zoom in for best view.
Visual results for dennising, av 0.5
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 29: Face image denoising for each fairness group, where dx = 0.5. (0) DDRM, (1) DDNM*, (2) DPS,
(3) PiGDM. Zoom in for best view.
45

---

Visual results for deblurring, ox 0.1
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 30: Face image deblurring for each fairness group, where ox = 0.1. (0) DDRM, (1) DDNM (2) DPS.
(3) PiGDM. Zoom in for best view.
Visual results for dehlurring, ag 0.25
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 31: Face image deblurring for each faireess group, where = 0.25. (0) DDRM, (1) DDNM", (2)
DPS, (3) PiGDM. Zoom in for best view.
46

---

Visual results for deblurring, ag 0.5
Young & non-Asian
Old & non-Asian
Old & Asian
Young & Asian
(0)
(1)
(2)
(3)
Figure 32: Face image deblurring for each fairness group, where 0.5. (0) DDRM, (1) DDNM (2) DPS.
(3) PiGDM. Zoom in for best view.
47

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe our paper's contributions and scope is accurately reflected in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
48

---

Justification: We provide 4 theorems in our paper (Theorems !
- and we state the full
set of assumptions in each of them. We rigorously prove our results in Appendices to
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our experiments involve evaluating existing face image super-resolution algorithms (using their official code and checkpoints) and generating synthetic image datasets. We carefully detail the evaluation procedures for the algorithms and the data generation process in both the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
49

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We evaluate existing face image super-resolution algorithms using their of- ficial codes and checkpoints. We employ well- metrics like KID, FID, and PSNR, leveraging the torch- forch-fidelity fidelity and piq packages for their calculation (all the details are in the appendix). To avoid potential licensing issues, we refrain from publicly sharing the evaluation datasets, but we provide thorough explanation of their construction process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wwwssssiioiiiyyooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our evaluation involves existing face image super-resolution algorithms, lever- aging their official code, checkpoints, and hyper-parameters provided by the authors. We do not optimize these algorithms within this work. However, we do conduct adversarial attacks, which require optimization. We disclose the parammtrrrssterss used in such experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report results averaged over 1,356 images. For the metrics we evaluate (KID, PSNR, etc.), such large number of images eliminates the need for error bars.
Guidelines:
50

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the appendices.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htinss/hhhrs////rrrp....oooo/bbb wtw.:ehhhhhccucceeeeeeeen
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics in every aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We dedicate Section 6 to discuss the socictal impacts of our paper.
51

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release data models. The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the use of publicly available datasets and conform to their license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
52

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaa datasess package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
53

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
54


Reviews:
[{"review_id": "EXnnvlY4JN", "reviewer": "Reviewer_xBjq", "summary": "This paper reveals that the conventional definition of fairness for image restoration is restrictive and often causes controversy. To address this issue, the authors introduce a new approach to measure fairness in image restoration tasks by proposing the Group Perceptual Index (GPI). Specifically, they propose assessing the fairness of an algorithm by comparing the GPI of different groups, where perfect Perceptual Fairness (PF) is achieved if the GPIs of all groups are identical. They theoretically study this notion of fairness and demonstrate its utility on state-of-the-art face image super-resolution algorithms.", "questions": "1. The paper shows that achieving perfect GPI for all groups is often not feasible, especially under severe degradation conditions.  it leaves an open question of how to balance fairness among different groups in practice best when perfect GPI cannot be achieved.\n2. In addition to face synthesis, is the proposed GPI suitable for natural images with complex structures?\n3. The application scenes are not clear. How do we apply the proposed GPI to real-world applications?", "limitations": "The paper rethinks the fairness in image restoration and proposes a novel method, called Group Perceptual Index (GPI), to measure the fairness for image restoration models. The proposed method can effectively detect subtle and malicious biases enhances the robustness and security of image restoration systems", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper reveals the existing fairness measures such as Representation Demographic Parity (RDP) and highlights their limitations. It shows that these measures can be overly simplistic and may not detect subtle biases that affect different groups.\n2. The paper proposes the Group Perceptual Index (GPI) as a measure of fairness in image restoration, which is a novel and significant contribution.\n3. It provides a theoretical analysis of the properties of GPI and its relationship to other fairness measures.\n4. The authors use a variety of datasets and experimental setups to demonstrate the effectiveness of GPI, which are convincing.", "weaknesses": "1. Group Perceptual Index (GPI) also increases the complexity of the evaluation process of image restoration algorithms, compared with the traditional fairness method, because it involves comparing the distributions of different groups.\n2. The experiments use synthetic datasets generated from high-quality, aligned face image datasets like CelebA-HQ. \n3. The paper only evaluate the proposed method on the face dataset and does not provide the results on other kinds of image data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Complexity of evaluating perceptual fairness\nThank you for raising this important point. We acknowledge that computing the GPI of each group increases the complexity of evaluating fairness compared to previous methods, which typically compute the classification hit rates (e.g., counting the reconstructed images classified as having a specified sensitive attribute). However, it is important to note that both our method and previous methods require each sample to be processed through a classifier. In our method, the classifier is used to extract deep image features to compute metrics such as FID and KID, whereas in previous methods the classifier evaluates the predicted sensitive attributes of each image.\nThus, the additional computational overhead of our method arises from two factors:\n1. Extracting deep features from the source images in addition to the reconstructed images, effectively doubling the computation required for feature extraction compared to previous methods.\n2. Approximating a statistical divergence (e.g., FID) between the extracted features of the ground truth images and the extracted features of the reconstructed images. This additional step introduces some computational overhead (e.g., computing the empirical mean and covariance matrix of the samples, as in FID), but it is relatively minor in practice compared to the benefits of our more nuanced fairness evaluation. We will point this limitation in our paper.\n\n### Why do we use synthetic datasets?\nWe appreciate the reviewer's concern regarding our use of synthetic data sets. Please note that, as discussed in Section 4.1 of our paper, we opted to use synthetic, high-quality datasets because existing face datasets often lack ground truth labels for sensitive attributes such as ethnicity, age, and gender. Moreover, the datasets that do include such labels are typically imbalanced w.r.t. these attributes. To adequately approximate and compare the GPI of different groups, it is essential to have equal amounts of data from each group. Otherwise, approximating the GPI with metrics like FID would lead to completely different scoring scales for varying sample sizes (see, e.g., Figure 1 in [1], which shows that FID suffers from bias when using small sample sizes). Generating synthetic datasets allows us to control the number of images from each group, and therefore allows to ensure balanced and fair evaluations.\n\n### Applicability to other types of data besides facial images\nThis is a very interesting point that definitely deserves an explanation. While the proposed GPI is indeed suitable for evaluating fairness in natural images with complex structures, fairness issues are particularly critical when dealing with human images due to their societal implications (which is why we only evaluated facial images). For example, if a general-content image restoration algorithm performs better on images with complex structures than on images of clear skies, this discrepancy is unlikely to be problematic for practitioners, as long as the algorithm attains good performance overall. Moreover, previous works (e.g., [2]) evaluated fairness with respect to non-human subjects (e.g., dogs and cats), but these studies provide limited insights into human-related fairness issues, which often arise due to subtle differences between images. Expanding our method to other datasets remains an avenue for future work. We will include a discussion about this point in the paper.\n\n### How to balance fairness among different groups in practice?\nPlease let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an \"adversarial\" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.\n\n### How do we apply the proposed GPI to real-world applications?\nMany practical imaging systems should ensure that their algorithms do not introduce or amplify biases against any particular group. Mobile phones, for example, are used by everyone, and all of them incorporate image restoration algorithms within the Image Signal Processing (ISP) pipeline to produce a high-quality image from the given sensor measurements. When the degradation is sufficiently severe (e.g., in low-light conditions), even a well-performing image restoration algorithm (e.g., posterior sampler) may treat some groups better than others. Thus, it is important to evaluate the fairness of such algorithms. For example, the GPI can help identify subtle biases that traditional methods might overlook, thereby alerting for fairness issues in these systems.\n\n#### References\n[1] Mikołaj Bińkowski et al. \"Demystifying MMD GANs.\" Proceedings of the International Conference on Learning Representations (ICLR), 2018.\n\n[2] Ajil Jalal et al., \"Fairness for Image Generation with Uncertain Sensitive Attributes.\" Proceedings of the International Conference on Machine Learning (ICML), 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BmAUVqSiVE", "reviewer": "Reviewer_HsYQ", "summary": "This study presents a novel method to evaluate fairness in image restoration using the Group Perceptual Index (GPI). GPI quantifies the statistical disparity between a group's original images and their restored versions. Fairness is assessed by comparing GPIs across multiple groups, striving for perfect Perceptual Fairness (PF) where all GPI values are identical. The research provides theoretical insights into this innovative fairness concept, drawing comparisons to existing frameworks, and showcases its practical implementation through advanced face image super-resolution algorithms.", "questions": "1. What is the primary difference between fairness evaluation and standard image quality assessment metrics like PSNR and SSIM?\n2. How does this method contribute to the design of fair and unbiased image restoration algorithms?", "limitations": "Limitations have been thoroughly discussed and adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and includes sufficient theoretical explanations.\n2. The concept of GPI is logically sound.\n3. The paper demonstrates that the proposed method outperforms other baseline methods.\n4. The paper thoroughly discusses both the advantages and limitations of the proposed method. The advantages highlight the method's effectiveness and potential benefits, while the limitations are clearly outlined, providing a balanced view of its capabilities and areas for improvement.", "weaknesses": "1. The authors introduce a novel method to evaluate the fairness of image restoration. However, it is important to note that this method has been validated exclusively on image super-resolution tasks. Further validation on other types of image restoration tasks would be beneficial to demonstrate its broader applicability and robustness.\n2. How can sensitive attributes be detected and acquired? The impact of sensitive attributes deserves an in-depth discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Demonstration only on image super-resolution tasks\nWe opted to illustrate our approach on 12 different super-resolution tasks (4 scale factors and 3 noise levels) simply because of the availability of many methods to compare against on these tasks. Note that this choice aligns with common practice in the field, as related works also focus on image super-resolution [1,2,3], which serves as a standard benchmark due to the availability of numerous algorithms for comparison. Nonetheless, we agree that validating our method on other types of image restoration tasks, such as image denoising or deblurring, or on different modalities like audio, video, or text restoration, would further substantiate its robustness and broad applicability. To address this, we will add experiments with image denoising and deblurring to the appendices and discuss the potential for future work on additional types of degradations and data modalities.\n\n### How can sensitive attributes be detected and acquired?\nThank you for raising this interesting question. Please note that our work assumes that the sensitive attributes are prespecified, which is the case handled by most works that tackle fairness concerns. It should be noted that the term \"sensitive\" here is application dependent. For example, in one application gender may be considered the only sensitive attribute, whereas in another application age may be considered sensitive as well. This is despite the fact that both applications use the precise same data for training. Thus, sensitive attributes cannot be detected automatically from the data. Rather, they should be specified by the user according to the particular societal concerns in question. We will discuss this interesting point in the manuscript.\n\n### The difference between GPI and standard metrics like PSNR\nWe appreciate this question and address it in Appendix G.5 (L794 onwards), where we show that such metrics are not good indicators of perceptual bias. Indeed, our experiments (Figures 8-10 in the appendix) show that, while the different groups attain roughly the same distortion (e.g., PSNR, LPIPS), their GPIs differ significantly, a discrepancy that is also visually evident.\n\n### How can our method contribute to the design of fair and unbiased image restoration algorithms?\nPlease let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an \"adversarial\" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.\n\n#### References\n\n[1] Ajil Jalal et al., \"Fairness for Image Generation with Uncertain Sensitive Attributes.\" Proceedings of the International Conference on Machine Learning (ICML), 2021.\n\n[2] Yochai Blau and Tomer Michaeli, \"The Perception-Distortion Tradeoff.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[3] Guy Ohayon et al., \"The Perception-Robustness Tradeoff in Deterministic Image Restoration.\" Proceedings of the International Conference on Machine Learning (ICML), 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGwtpV1quE", "reviewer": "Reviewer_msyK", "summary": "This work introduces a new method for assessing fairness in image restoration, called the Group Perceptual Index (GPI). This measure quantifies the statistical difference between the distribution of a group's original images and the distribution of their restored versions. The authors illustrate the effectiveness of GPI by applying it to advanced face image super-resolution algorithms.", "questions": "Please address my comments under weaknesses", "limitations": "Authors has addressed the limitations to a satisfactory extent.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- the problem tackled in this paper is of practical importance\n- paper is written well\n- the proposed method is theoretically sound and is shown to work in meaningful ways when used on the problem space of image super-resolution", "weaknesses": "- Usefulness of the method is validated only on the super-resolution solution. Given the fact that the proposed method has potential to impact various image restoration algorithms, it would have been interesting to see how well it does on other image restoration application such as image denoising, deblurring etc.\n- It is also not clear what kind of changes to the existing super-resolution methods might result in better fairness handling. Some insights into why certain methods are not good at fairness handling as compared to others might have helped the future works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Demonstration only on image super-resolution tasks\nWe opted to illustrate our approach on 12 different super-resolution tasks (4 scale factors and 3 noise levels) simply because of the availability of many methods to compare against on these tasks. Note that this choice aligns with common practice in the field, as related works also focus on image super-resolution [1,2,3], which serves as a standard benchmark due to the availability of numerous algorithms for comparison. Nonetheless, we agree that validating our method on other types of image restoration tasks, such as image denoising or deblurring, or on different modalities like audio, video, or text restoration, would further substantiate its robustness and broad applicability. To address this, we will add experiments with image denoising and deblurring to the appendices and discuss the potential for future work on additional types of degradations and data modalities.\n\n### What kind of changes to the existing super-resolution methods might result in better fairness handling?\nPlease let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an \"adversarial\" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.\n\n### Insights into why certain methods are not good at fairness handling compared to others\nThank you for raising this important point. Please note that our theoretical section (Section 3) attempts to provide such insights. For example, we show that common image restoration algorithms, such as the MMSE point estimate or the posterior sampler, may often lead to poor perceptual fairness. This means that perceptual fairness is not \"trivially\" acquired by common algorithms. Thus, it is interesting to ask: \n1. Under which circumstances can some algorithm achieve perfect GPI for all groups simultaneously (Theorem 2)? \n2. Otherwise, when perfect GPI cannot be achieved for all groups simultaneously, under which circumstances can some algorithm achieve perfect perceptual fairness? (Theorems 3 and 4). For example, from Theorem 4 we learn that, among all the algorithms that achieve perfect Perceptual Index (PI), the better ones in terms of perceptual fairness are those that excel on the \"toughest\" groups, and this is not directly achieved by methods that simply attain perfect PI (e.g., posterior sampling).\n\n#### References\n\n[1] Ajil Jalal et al., \"Fairness for Image Generation with Uncertain Sensitive Attributes.\" Proceedings of the International Conference on Machine Learning (ICML), 2021.\n\n[2] Yochai Blau and Tomer Michaeli, \"The Perception-Distortion Tradeoff.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.\n\n[3] Guy Ohayon et al., \"The Perception-Robustness Tradeoff in Deterministic Image Restoration.\" Proceedings of the International Conference on Machine Learning (ICML), 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OA8PgmzYGk", "reviewer": "Reviewer_ziAr", "summary": "This paper considers fairness issue of image restoration and proposes Group Perceptual Index to measure the distance between restoration distribution and gt distribution. Experimental and theoretical results demonstrate that the superiority the proposed perceptual fairness over previous method.", "questions": "Can the method be extended to measure other fields, like image generation?", "limitations": "Yes, the authors discuss limitations carefully.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe fairness is important for the image restoration community and the topic is interesting to study. The proposed Group Perceptual Index can be reasonable to measure the fairness properly.\n2.\tThe paper includes solid theoretical and experimental results which provides evidence for fairness measurement.\n3.\tThe paper is well-written and easy to follow.", "weaknesses": "1. The main results are mainly based face restoration. Can this method be useful for general scenario of image restoration?\n2. The paper proposes a measure to detect the fairness issue. But can you suggest some potential solutions to address this problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###  Can our method be useful for general-content image restoration?\nThis is a very interesting point that definitely deserves an explanation. While the proposed GPI is indeed suitable for evaluating fairness in natural images with complex structures, fairness issues are particularly critical when dealing with human images due to their societal implications (which is why we only evaluated facial images). For example, if a general-content image restoration algorithm performs better on images with complex structures than on images of clear skies, this discrepancy is unlikely to be problematic for practitioners, as long as the algorithm attains good performance overall. Moreover, previous works (e.g., [1]) evaluated fairness with respect to non-human subjects (e.g., dogs and cats), but these studies provide limited insights into human-related fairness issues, which often arise due to subtle differences between images. Expanding our method to other datasets remains an avenue for future work. We will include a discussion about this point in the paper.\n\n### How can we train models to achieve perceptual fairness?\nPlease let us propose a method to train algorithms to achieve the best possible perceptual fairness when the sensitive attributes of each ground truth image in the data is known. In particular, one can train a conditional discriminator that takes the sensitive attribute as an additional input, namely, the discriminator is trained to distinguish between the ground truth images of a group (e.g., women images) and their reconstructed images (e.g., the reconstructions produced for degraded images of women). While training such a discriminator, one can add a regularization to the objective of the restoration algorithm (in an \"adversarial\" fashion) to minimize the deviation in the discriminator scores for different groups. For example, at each training step of the restoration algorithm, one can average the discriminator scores produced for each group and then minimize the standard deviation of the results. This will ensure that the average discriminator scores produced for the reconstructions of each group would be similar for all groups. While this proposed approach was not explored in our paper, it represents an interesting direction for training fair image restoration algorithms. We will thoroughly describe this proposed approach in our manuscript, as a suggestion for future work.\n\n### Can our method be extended to other fields such as image generation?\nYes! Thank you for raising this interesting question. The concept of GPI can indeed be extended to other fields, and in particular to image generation. For example, one can regularize a diffusion model (a \"time\"-dependent image denoiser) during training to attain similar GPIs for all groups. This can be achieved, e.g., by regularizing the denoiser at each time-step using the \"adversarial\" technique we described above (\"*How can we train models to achieve perceptual fairness?*\"), while making the discriminator time-dependent as well. Incorporating such a regularization would ensure that the output distribution at each time-step of the diffusion would be balanced with respect to the specified sensitive attributes. This implies that the output distribution at the end of the generation process would also be balanced. We will discuss this interesting potential avenue in our paper.\n\n#### References\n\n[1] Ajil Jalal et al. \"Fairness for Image Generation with Uncertain Sensitive Attributes.\" Proceedings of the International Conference on Machine Learning (ICML), 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Thank you!
We are deeply grateful to all the reviewers for dedicating their time to evaluate our paper. The feedback we received has been highly encouraging and has helped us improve the quality of our work.

Meta-review: Overall, the reviewers agree that the paper makes a novel and significant contribution by proposing the Group Perceptual Index (GPI) as a measure of fairness in image restoration. The reviewers commend the convincing experiments, logically sound concept, theoretical analysis of GPI's properties, thorough discussion of both the advantages and limitations, and the overall quality of the presentation. 

The main concerns are whether the proposed metric is applicable to general images beyond human faces and the sole validation on the image super-resolution task. The rebuttal addresses the first issue and promises to include additional experiments on image denoising and deblurring in the appendices. It also commits to discussing the potential for future work on additional types of degradations and data modalities to address the latter concern. Additionally, to address the ethics issue, the authors agree to highlight the choice of sensitive attributes in the study. Please take these commitments into account in your revision.
Decision: Accept (poster)