Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces a method for applying popular 'beautification' filters to an open dataset of facial images. This results in further datasets of modified images (one for each filter). The authors use these datasets to analyze the amount to which the filters homogenize faces and whether the filters affect the accuracy of facial recognition algorithms. The datasets and code are released under GPL and CC licenses.
strengths: The paper clearly describes the methodology and results. The strongest contribution is in the analysis of the comparative homogenization of faces and the analysis of how filters affect facial recognition.

The datasets of filtered facial images may be of interest to other researchers.
weaknesses: The method of scripting use of online platforms to apply filters may not be reproducible. The authors automated application of filters to run them on up to 22,000 images per day but did not report working with the platforms to ensure that this was an acceptable use - if platforms begin to limit the number of applications per day per user, this technique will no longer work.

The authors rightly note that 'beautification' filters are constantly changing and rising and falling in popularity. Their datasets use several of the most popular filters at the time of writing, but this will quickly date the work as new filters become popular over time. The intent is to allow other researchers to use the open-source toolset to create new datasets with new filters, but this relies on some degree of cooperation (or at least non-interference) by the platforms being used to run the filters across large numbers of images.

It is not clear whether the use of scripting to automate filtering thousands of images is allowed under the terms of service of the various platforms. 
correctness: The claims and analysis seem sound - another reviewer remarked on some methodological concerns about the use of t-tests and these seem to have been addressed.
clarity: In table 1, the p-values are certainly low, but more discussion would help to clarify the significance of this result. There is clearly a statistically sound difference between the values, but it's not clear whether it's a meaningful difference. 
relation to prior work: Yes, the authors are clear where they build on and extend prior work.
additional feedback: none.
review_title: Generation and analysis of open sets of 'beautification' filter images

final_score: rating: 3: Clear rejection
review_content: summary and contributions: The authors publish a tool for applying social media filters to facial images, as well as 2 datasets to which this tool was applied. The datasets FairBeauty and B-LFW were generated by applying the OpenFilter tool to the pictures from the open source datasets FairFace and LFW, respectively.
The paper describes 2 experiments, one for each dataset. The FairBeauty dataset was used to investigate the similarity of faces post-application of the social media filters. The B-LFW dataset was used to compare the performance of face recognition algorithms pre- and post-application of the social media filters.
strengths: 1. Significance: the authors identify a social phenomenon which has lacked quantitive research due to a lack of data. By producing a large-scale dataset, further research into social media filters and their social impact could be enabled.
weaknesses: 1. Sensitivity to ethical and social implications: described in more detail in the Ethics section below.
2. Accessibility: even though the stated intent of the work is to enable interdisciplinary research into social media filters, reproducing the datasets requires non-trivial computer science skills.
correctness: The construction of the dataset is sound. Conclusions from the 2 experiments seem correct.
clarity: The work is clearly described.
relation to prior work: Yes, the authors discuss the need for quantitative research into social media filters and how the construction of this dataset enables that.
additional feedback: If the intent of this work is to enable quantitative, interdisciplinary research regarding the properties of social media filters, and how they impact social subgroups, then I strongly suggest to include that as part of the work given the sensitive nature of the dataset.
review_title: Social impact of dataset not sufficiently addressed

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper considers Augmented Reality filters for faces, which are used on many social media sites. The authors argue these filters are of interest to scientific researchers from psychological, social, cultural, and artistic perspectives but that researchers are currently limited in their ability to study these filters at scale because they are often proprietary and cannot ethically be scraped from the platforms. To provide researchers with access to real-world AR filters from social media, the authors have developed a tool that can apply AR filters to an existing dataset of faces. The authors also contribute two new datasets that result from applying a collection of 8 popular beauty filters to two existing open access face datasets: FairBeauty (applied to FairFace) and B-LFW, applied to Labeled Faces in the Wild (LFW). These are meant to illustrate potential uses of their tool but also provide resources for studying the role of beauty filters specifically. The authors present two example experiments: One which compares FairBeauty to FairFace to look at whether beauty filters make faces more homogeneous (lowers distance between randomly selected pairs) and one which uses LFW and B-LFW to study effect of beauty filters on accuracy of facial verification algorithms (they find little decline in performance).
strengths: •	well-written

•	authors make a fairly convincing case that their tools could help researchers access, probe, and understand a usually proprietary and inaccessible feature of the contemporary social media landscape – “democratizing” access, as they say. 

•	motivating questions, technical set-up of the two illustrative experiments were easy to grasp, though the analysis of experiment 1 is flawed (see weaknesses). 

•	authors do a good job of situating their technical contribution within the societal and broader research context: provide some background on selfies and AR filters on social media, mention  potential research topics that could benefit from their tool and datasets, and discuss ethical issues. 

weaknesses: The authors take some good steps to address the ethical stakes of this research. Still, some important considerations are unaddressed – see ethics section of review.

The analysis and presentation of experiment 1 data is flawed – see correctness section of review

correctness: I cannot really technically evaluate the Open Filter tool itself, but the set-up and dataset creation processes sound plausible. The experimental designs make sense for investigating the authors’ questions about homogenization and face verification performance. However, the approach to analyzing experiment 1 is at best, suboptimal.

In experiment 1 (Algorithm 1), the authors :

* randomly sample pairs of images from FairFace 
* apply four different ‘treatments’ (beautification, blurring 1, blurring 2, pixelation) to the images
* for the originals and each of the treated pairs, calculate the distance between the pairs of faces. 

The idea is that if the distance between pairs of beautified faces is lower than for the originals, this is evidence that the beautification filters create more homogeneity in the faces. 

This is essentially a blocked experimental design *at the level of the (dis)similarity measurements* -- that is, we have groups of similarity measurements which are all for the same pair of original images. However, in their analysis, the authors ignore this paired design and simply calculate the average distance for each treatment group and look at the differences in group averages. This is an inefficient way to analyze these data that throws away information and results in less power to detect a treatment effect.

Essentially, this is the difference between doing a two-sampled t-test (is the **difference in the means** of two independent samples different from 0) and doing a paired t-test (is the **mean difference** different from 0?). When we have paired data but act as if they are not paired, we ignore the fact that observations from the same pair may tend to be more similar (intraclass correlation). In the case of experiment 1, distance measurements that are all applied to images derived from the same baseline sets of faces will presumably tend to be more similar to each other than distance measurements randomly selected across baseline sets of faces. Looking at the mean difference controls for that varying baseline by asking, “given each baseline set of faces, are $x_b$ and $x'_b$ more similar than $x$ and $x’$?” while looking at the difference in means does not account for the varying baseline and thereby introduces more noise into the comparison. Comparing the group averages gives some indication, but it’s not the best you can do. 

Instead of the current analysis, I recommend that the authors look at the pairwise differences in similarity for each of the treatments compared to the original. In the paper’s notation, for each randomly sampled pair ($x$,$x'$ ), one would calculate $∆_b=m_b-m$,  $∆_g'=m_g'- m$, $∆_g''=m_g''-m$, $∆_s=m_s-m$. I recommend the authors then use the boxplot approach of Figure 4 to show the distribution of $∆_b$,$∆_g'$,$∆_g''$, and $∆_s$ (note: there would be no boxplot for the original group anymore since the others are relative to it). If, on average, the $∆_b$'s are less than 0, this indicates that the beautified faces tend to be more similar (feel free to flip it so positive indicates effect). You could also do pair t-tests and report the results – this will allow you to do more than say the result is “positive.” Doing all this could still yield a result that suggests homogenization – it could even make the case more strongly.

A few other things on experiment 1:

* I see no reason to sample different sets of 500 images for each of the models. This only introduces an additional source of variation that hinders our ability to compare the models. Ideally, you could apply all the models to 3000 images but if this is infeasible, I would prefer all models applied to the same 500
* Figure 4 is not accessible to those who are color blind. This could be improved by providing a label or at least stating in the caption that the order of the key at the bottom is also the order of the box plots
* It would be good to report a table with numbers for experiment 1
* Something you might look at or mention as a future extension: it would be interesting to examine whether $∆_b$ varies with the baseline distance of the original untreated images (could plot a scatterplot). I imagine there could be a saturation effect. For example, take faces of two siblings of similar age and same sex and beautify them and the original images might already be so similar that there isn’t much left to increase. Take photos of people of different ages, sexes, and/or ethnicities and there could be much more room to homogenize.

clarity: Clearly written, good organization, no issues.
relation to prior work: Authors emphasize that while other research has worked with simpler filters that cover some part of the face, their tool and datasets better reflect the real-world social media ecosystem and therefore could support more ecological validity 
additional feedback: A few miscellaneous questions and thoughts:


1. Has the apply-each-filter-to-LFW dataset from experiment 2 also been released? This isn’t clear to me.

2. On pg. 4, there are some potentially contentious claims about how beauty filter transformations align with different ethnicities’ associated characteristics – are these fairly well established? Are they the authors’ own impressions? I notice there is no source cited for this. 

3. The discussion of beauty standards mainly speaks of femininity and female faces. Do we know anything about the balance of sexes present in the benchmark datasets or the filters used? Any existing research on filter choices and how it might be gendered? Are all the filters ‘feminine’ oriented or are there also masculine filters included among the beauty filters? Potential implications for research?

4. In addition to the technical limitations mentioned in the Discussion, it is worth considering  remaining limitations to ecological validity or what can be studied with the data. For example, we do not know whether the photos to which beauty filters have been applied here actually reflect the kinds of photos (poses, expressions etc.) people tend to use beauty filters on. That is: the dataset doesn’t reflect actual human choices to apply or not to apply beauty filters in real world settings. That seems worth emphasizing, as it means we cannot use these data to make claims like, "people tend to apply filters to photos with characteristics X".

Overall, with fixes to experiment 1’s analysis and some more discussion of the ethics around facial verification, I would recommend for this to be published.
review_title: Potential to increase access to hard-to-study aspect of social media; ethically contentious area requiring careful consideration; some fixable methodological flaws

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper presents a new data set of filtered selfie images (of faces) constructed by automatically applying a range of filters to two pre-existing face data sets.



strengths: Overall this is a well written paper. The experiments in distance and facial recognition are interesting and the authors provide an interesting analysis of some of the ethical considerations, and apply an appropriate open source license to address some of that. 

- The data set created by the authors is a valuable addition to the existing data sets for facial analysis
- The analysis done by the others is interesting and valuable in its own right and helps to provide some insight about the filters. 

weaknesses: I don't see any significant weaknesses, outside of the correctness question I raise below.
correctness: It seems to be constructed in a sound way. However, the details on which filters were selected and how were a bit sparse. It would be nice to see more information about how the filters were sampled and what the rationale for that sample was.
clarity: There are the following minor writing issues:

- There is unnecessary jargon in this paper. An example is the introduction of the word "automatizes" "heterosexy" and so on. 
- There are a lot of unnecessary italics in this paper.
relation to prior work: This is not an area of expertise for me and I hope other reviewers will address this.
additional feedback: No
review_title: A good paper with a lot of jargon

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a custom framework called OpenFilter to apply AR filters from social media onto images of human faces. The output of the framework is demonstrated on two well-known and publicly available datasets, FairFace and LFW. The modified versions of these datasets are titled "FairBeauty" and "B-LFW". The authors specifically focus on eight filters that "beautify" faces by some sort of lighting, contouring, or makeup effect, but their framework is able to directly take filters from social media and thus should be able to apply any type of AR filter.

The paper specifically highlights the popularity of AR filters on social media, but the lack of available data for researchers to study the filters and their societal, psychological, and artistic effects. Most filters are applied by users on personal images and is thus unethical to scrape from social media. The only method to obtain AR-modified data so far has been to conduct surveys and user studies, asking users to submit images with AR filters applied to them. The paper thus presents a way to simply apply AR filters to existing human facial image datasets that are already ethically approved. 

The authors also demonstrate the applicability of their dataset by addressing two research questions: whether AR filters make all images more similar, and how facial recognition technology performs on their modified dataset. They address the first question with FairBeauty, and the second with B-LFW. They find that the distance between faces does indeed decrease with the applied filters, but facial recognition technology still performs quite well despite the added filters. 

The paper also acknowledges that the usage of OpenFilter does require software skills and that there are some resolution restrictions due to social media constraints.
strengths: The strengths of the paper (both the framework and the dataset) are summarized as follows: 

1. The paper creates a streamlined process to swiftly apply AR filters to images, thus creating an easy way to obtain a dataset with AR filters for research studies that have so far been bottlenecked by the data collection process. The filter is well documented in the supplemental material and in the Github, and the authors provide good instructions for installation, set-up, and even error-handling. 

2. The paper demonstrates the usability of the framework by applying it to two renowned datasets, thus avoiding any new ethical issues since the datasets have already been published. The chosen datasets are also unbiased in terms of facial traits and thus any study involving the dataset is truly able to focus on the impact of the AR filters.  The dataset is well documented in the supplemental material with clear instructions on the organization and construction. The authors also provide regular expressions to facilitate easy access and usage of their dataset. 

3. The experiments and the results in the paper seem reasonable and reproducible. The paper dedicates a subsection to explain the mathematical foundation behind their experiments. Specifically for the FairBeauty homogeneity experiment, the authors also compare results to gaussian blurring effects and down-sampling, which is a helpful parallel and a reference to interpret the results. Thus, the results seem robust and irrefutable. 

4. The problem addressed by the datasets and the framework is relevant to modern society due to the current popularity of AR filters, and should allow for better societal studies on the impacts of social media. The authors themselves are able to identify various areas of research where their dataset can be applicable, demonstrating their thorough knowledge and thought. 
weaknesses: The weaknesses of the paper (both the framework and the dataset) are summarized as follows: 

1. The filter, while well documented, does not seem easy to use. There are many steps to be followed in terms of set-up and installation, and the authors themselves acknowledge the need for software skills. The authors intend for their framework to be used in various forms of research that may or may not include software-experienced individuals. Strictly following the long documentation may prove to be intimidating or daunting. The authors could consider posting a demo video of the framework to assist all interested users. 

2. For the B-LFW experiment, the authors also provide results on the filtering the entire dataset. The final evaluation and discussion seems to focus on singular filters, which is not the dataset that they are presenting. The results from this experiment thus seem slightly confusing and there is not quite enough information presented on the actual modified dataset, B-LFW. A different form of experiment on the B-LFW dataset, such as perhaps evaluating the average per-person facial recognition performance sectioned by filter, may have been more informative. 

3. Although the authors chose unbiased datasets, since all the filters are not applied on all images, there is room for imbalance and bias to be introduced on the portions of the dataset with the filters. The authors do not provide evidence that they maintain the unbiased nature of the original dataset.
correctness: Yes, the claim made by the paper is correct. The dataset construction is thorough and well-detailed, and is sound. 
clarity: Yes, the paper is clear. 
relation to prior work: Yes, the authors make it clear that the framework is a novelty and there is no existing AR filter dataset. 
additional feedback: 1. Has the framework been tested on any other device? Curious what comparable speeds are, especially if a GPU is not readily available. 
2. What is meant by "heterosexy" in Lines 128-129?
3. How exactly does the selfie culture encourage a "diverse and inclusive" view on beauty? (Line 141)
review_title: A method to apply AR filters to images of human faces, and two datasets produced by this method 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: ### Summary
- This paper proposes an Openfilter framework that visually transforms human faces on social media platforms.
- This paper creates the beautified face dataset FairBeauty and B-LFW.
- Diversity analysis was performed on whether the beauty filter homogenizes the face images. And, correlation between beauty filter and face identity were analyzed.


### Contributions
- AR filters on social media are a novel (research) field, and it is true that they are receiving attention recently in face-related vision tasks.
- Unlike previous filters that are simply superimposed on 2D face images, the proposed framework can create a more realistic and aesthetic face image through the interlocking of AR filter and social media application.
- It deals well with the possibility of extension to various face-related vision tasks.

strengths: ### Significance
- The proposed framework is valuable in terms of considering all aspects of cultural ecosystems, face aesthetic, and face identity.
- The experimental results that beautified images show lower average distance than original images are encouraging.


### Applicability
- Since the various human-related vision database is built through social media, it seems that the Openfilter framework can be grafted from various perspectives.
- The proposed FairBeauty and B-LFW are easy to access.
- The continued mention of a public license increases the reliability of this dataset.
weaknesses: - Although the appendix describes the operation of Openfilter framework in detail, it is true that it is difficult to understand the details of the pipeline in Fig. 1 in main body.

- One of the key aspects in social media is the emotional state of the subject. Therefore, the relationship between facial expression recognition and the beautified dataset should be dealt with (at least a brief consideration is necessary).
correctness: The explanation of the dataset acquisition process and software script seems reasonable.
clarity: This paper explains the research background and existing trends really well. It also explains how the Openfilter framework can be used.
relation to prior work: A description of the prior work on the beautified dataset seems sufficient. In addition, even the most recent face recognition techniques were used for benchmarking.
additional feedback: In order to increase the reliability of Table 1, it would be good to use an evaluation metric such as Rank-K in addition to verification accuracy.
review_title: This paper proposes a novel AR filter-based image transformation framework and answers are required in some respects.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper presents the OpenFilter framework for the application of public AR filters to any set of images, which allows to analyze how AR filters affect models performance and / or human perception. The OpenFilter framework is applied to the publicly available Fairface and Labeled Faces in the Wild datasets, resulting in their beautified versions FairBeauty and B-LFW, which are also published and can be used for further research.

The authors provide a well-documented code repository with both the datasets and the framework available.
strengths: The proposed framework is a novel research instrument for automatic beautification of large datasets. It is outlined that previous user studies consider only small amounts of data and cannot scale effectively. And the existing datasets contain approximations of few AR filters instead of arbitrary user-generated filters.

The authors explain the framework pipeline in detail and provide appropriate evaluation methods for the generated datasets. The framework is described thoroughly and can be used for further research of AR filters applications.
weaknesses: [resolved] The filters used in the datasets generation and evaluation look very similar, which leads to similarities in quantitative results for the most of them. Hence, the diversity in filters could be improved.

[resolved] The study of the homogenization of faces lacks quantitative results. For more clarity, statistical tests could be reported along with the boxplots.
correctness: The datasets are constructed and evaluated reasonably, the user-generated filters are evaluated against simpler transformations like blurring and rescaling.

[resolved] The choice of filters is questionable: most of them look similar and there is no evidence why more diverse filters could not be used. Therefore, it is not clear if the proposed datasets represent real-world filters well enough.
clarity: Lines 154-156: The choice of the beauty filters is based on their popularity, which we assessed through articles in women’s magazines and relevant trends on Instagram.
[resolved] The paper lacks explanation how the sources are credible and not biased, which puts the choice of the analyzed filters in question.
relation to prior work: The authors observe existing datasets and studies, and describe drawbacks of previous approaches which the proposed method aims to overcome.
additional feedback: .
review_title: The new and promising framework for analysis and application of AR filters

Meta-review: metareview: This paper contributes a tool as well as a dataset to study the application of social media AR filters to enable interdisciplinary research  from psychological, social, cultural, and artistic perspectives. The reviews are largely positive (6 positive, 1 negative) and the authors have successfully  addressed some of the weaknesses that the reviewers raised, eg. the reviewers addressed the concern that the tool is non trivial to use without computer science skills, though a tutorial video. The authors have also addressed the ethical concerns raised by the reviewers. I think this paper meets the bar for the track and should be accepted
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept