Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-attribute selective suppression (MaSS) framework for the multi-attribute classification task, where there is some targeted attribute to be suppressed. The authors apply a MLP based Encoder-decoder with skip link architecture to achieve the task, where it is decomposed into mainly three parts, the suppression branch, the data modifier (the architecture), and the preservation branch. Their method is further evaluated across a set of experiments, leveraging the face attribute of gender and age while suppressing ID, the audio sound of MNIST for digit classification while suppressing speaker ID, and the PA-HMDB for action classification while suppressing skin color, gender and other attributes. The overall performance has shown clear improvement over baselines and some SOTA methods. 
strength_and_weaknesses: Strength:
1. the technical pipeline of the selective suppression is clearly illustrated with adequate figures and formulated equations and losses.

2. the experiments are designed to cover wider range of attribute classifications including face, audio and action recognition.

3. the experiments are towards extensive to check each of the ablative components, and the hyper-parameters for a in-depth comparison for better understanding.

Weakness:
1. the scalability of dealing with many attributes is not sufficiently verified. Through all the evaluation datasets, the attributes considered is less than 10, which cannot directly expand to large scale scenario.

2. the compared methods are mostly focusing on MaSS's ablative variations. There significantly lacks the comparison to other state-of-the-art methods, thus not so clear how better it is against the literature.
clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear way where most of the illustration, statement and equations are in a clear way with little confusion. Regarding some designing, there are some points to be further discussed.

1. in suppression branch, for the similarity definitions, the authors propose it could be cosine similarity or negative KL-divergence. Firstly, it should be specific enough, i.e., for all the experiments, what exact similarity measure is utilized?

Secondly, consider the suppression, one would like to deduct the information from the original embedding that is related to the specific attribute to be suppressed. So, the original x and the suppressed x' can not be similar.

2. the formulation in Eqn. (9) allows both multiple attributes for suppression and multiple attributes for reservation. However, across all the experiments, the demonstration is only one attribute for suppression. This can hardly justify the scalability of the method.

3. for the joint loss, there is loss weights designed. However, there is no empirical value or suggested value for the replication.

4. In the ablation study, those loss terms are incrementally increasing. However, it is not clear enough to highlight each of the loss component. For example, in table 2, would all loss terms but without $L_{r^{\*}}$ a valid ablation baseline? This is to highlight how important the $L_{r^{\*}}$ w.r.t. the joint loss.

5. Across all the experiments, the proposed method is mostly compared to its ablative baselines. Only Table 5 is compared to CIAGAN on one dataset, Adience. 

6. In the experiment, the last second paragraph before "Conclusion" section, the authors mentioned:
"Lastly, comparing to SPAct (Dave et al., 2022) on PA-HMDB1, MaSS achieved competitive suppression ratio of cMAP on both datasets (VISPR: 55.6% vs. 57% and PA- HMDB: 20.7% vs. 16%) for the other 5 non-action attributes."
However, in Table 4, the P-HMDB dataset evaluation, we did not see the comparison to SPAct.


summary_of_the_review: Overall the paper presents a technically valid framework for interested attribute suppression based on the entropy maximization loss design. However, from the experiment, it does not show the ability to scale up to multiple attributes suppression, neither was it sufficiently compared to state-of-the-art methods to demonstrate its effectiveness. Thus, I would like to see the enrichment of the experiment for the final rating. For now, it is borderline towards reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Research Question: Solve problems where simple or complete deletion of data attributes results in severe degradation of data quality. For example, having to lose data for user privacy.

Research Target: The paper proposes a method to suppress any selected attribute in a multi-attribute dataset. The paper highlights that the proposed method can leave the remaining properties unchanged for potential ML-based downstream analysis tasks.

Research Solution: The solution proposed in the paper is to learn data modifiers through an adversarial game between two sets of networks, one designed to suppress selected attributes and the other to ensure the remaining attributes are preserved through a general contrastive loss and explicit classification metrics.

The paper validates the proposed method in three domains: Facial Images, Voice Audio, and Video Clips.
strength_and_weaknesses: Advantages:
1. The research topic is helpful for data protection and privacy.
2. The paper tries to verify the proposed method in different domains.
3. The paper elaborates on the research question and background in detail.

Disadvantages:
1. The paper does not describe the details of the method, and it is not easy to reproduce the paper.
2. The paper lacks some very important references and related work.
3. The paper lacks some necessary experiments and analysis.

Questions: 1. Is the role of the DATA MODIFIER to normalize the data? What is the value of n in Equation 1? How to set the residual shortcut here?

2. Does the loss in Equation 2 compute the similarity of each pair of data? Are the same settings used here for all three tasks?

3. How are the attribute-agnostic logits in Equations 5 and 6 estimated?

4. Which formula does $L^{r_1}$ and $L^{r_2}$ in Experiment Table 2 equal? I didn't find it in the paper.

5. The author's method in Table 5 seems to perform similarly to the Downsample. What is the reason for this?

6. Has the author performed other verifications? Does the paper only achieve attribute recognition?

7. Did the author test directly on the original data? Is it using the features already extracted from the dataset for testing?

8. The authors leave out some work on multitask attribute recognition, eg.[1][2]. Could the proposed approach be extended to these multi-attribute recognition methods?

[1] Huang, Siyu, et al. "Gnas: A greedy neural architecture search method for multi-attribute learning." Proceedings of the 26th ACM international conference on Multimedia. 2018.
[2] Cheng, Zhi-Qi, et al. "Learning to transfer: Generalizable attribute learning with multitask neural model search." Proceedings of the 26th ACM international conference on Multimedia. 2018.
clarity,_quality,_novelty_and_reproducibility: 1.Clarity:
Some methodological and experimental setup details are unclear.

2.Quality:
Papers can add details and supplement more analysis.

3.Novelty:
The research question is innovative. The research methodology needs to discuss how it differs from previous work.

4.Reproducibility:
Some details of the paper are unclear. I am concerned that this paper is challenging to reproduce.
summary_of_the_review: This paper investigates an interesting question and has practical value. But some details of the methodology and experiments confuse me. I would like the author to reply in the rebuttal.

Because of the problems existing in the current paper, my initial score is: marginally above the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper designs a data obfuscation framework that is able to suppress an arbitrary set of attributes while preserving all other attributes. The proposed framework trains a neural data (additive) modifier to minimize the similarity between the original data and the modified data for the attributes to be suppress while maximizing the similarity between the original data and the modified data for other attributes. The proposed method demonstrates promising results for suppressing targeted attributes while preserving others on multiple datasets.
strength_and_weaknesses: Pros:
- The topic is interesting and the motivation is strong. Removing sensitive attributes from data is crucial for real-world data usage, and (as shown in Table 5) vanilla or prior data obfuscation methods either cannot successfully destroy the performance on targeted attributes or cannot preserve satisfiable performance on other attributes. This paper aims at reaching the two goals simultaneously.
- The proposed method is straightforward, flexible and effective. Each components of the proposed loss function directly relates to a desired purpose. The proposed method can suppress an arbitrary set of attributes and can preserve information in both attribute-agnostic and attribute-specific situation. Experiments show the proposed method can decrease the performance on targeted attributes drastically with little impact on the preserved attributes. 

Cons:
- The abstract is too long. It should be more concise.
- The method section and Figure 2 are a little bit confusing. It is not clear how many pre-trained models are used and where they are. It is neither clear which loss is penalty and which loss is award. For example, both the penalty and the award similarity loss for the two branches are called "Attribute-Specific Similarity loss", which is very confusing.  Also, why is there only one Prediction Entropy Loss in the Suppression Branch in Figure 2? Shouldn't it be one for each attributes and thus S in total? Besides, what does it mean by "the data modifier tries to minimax the loss"? Isn't the data modifier the only updating network which is minimizing the loss while others are pre-trained and fixed models?
- All data points are converted into feature embeddings in the experiments. It not only requires a good feature extractor in advance when applying the proposed method, but also prevent the usage of domain-specific techniques (e.g. data augmentations, neural architectures) when data users uses the processed data. 
- The ablation of the proposed loss function is not enough. Since the proposed loss function is composed of so many terms, it will be considered as validated only if each component shows effect. Currently, there's no experiments demonstrating the benefit from the reconstruction loss. For the suppression branch, there's no experiments showing how many benefits come from the Attribute-Specific Similarity loss and how many benefits come from the Prediction Entropy Loss. For the preservation branch, there's no experiments testing whether the Attribute-Agnostic Loss still contributes to preserving the information when the Attribute-Specific Loss is available.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper can be improved (see the first and second points in Cons). 

The novelty of the proposed method is moderate. It notices the drawbacks of previous methods (e.g. not preserving useful information, not flexible enough) and addresses them, but the loss function just combines several natural and standard terms.

The proposed method should be reproducible since most of the details are provided.
summary_of_the_review: This paper proposes a data obfuscation framework to selectively suppress attributes while preserving other attributes. The proposed method is straightforward, flexible and effective and the experimental results are great. However, it has some evident drawbacks (e.g. all data points should be converted into feature embeddings) and the paper writing can be improved (e.g. too long abstract, confusing method section, lack of ablation experiments). Also, the proposed method just combines several natural and standard terms, and the ablation study does not convincingly show the efficacy of each (e.g. the reconstruction loss). Therefore, the paper still needs some work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Multi-attribute Selective Suppression, or MaSS, a general framework for performing precisely targeted data surgery to simultaneously suppress any selected set of attributes while preserving the rest for downstream machine learning tasks. The end goal is to achieve a sophisticated mechanism that gives data owners fine-grained control while retaining the maximal degree of data utility.
strength_and_weaknesses: strength:
1. the paper is easy to follow and written well.
2. the motivation of this paper is valid and important to me, i.e., suppressing the data properly without affecting other attributes.
3. the results seem to be effective from the main tables on main benchmarks.

weakness:
1. in section 3.2.2, the author mention that "It utilizes the corresponding set of inference models pretrained on the original dataset X". It is not clear how the pretrained models are trained such that it only focuses on one specific attribute.
2. in section 3.2.3, for the agnostic case, it is not clear why a specific self-supervised pretraining can help preserve the non-selected attributes. If the selected attributes are forced to have a similar embedding after transformation, isn't it harmful to the final goal?
3. the comparison in the experiment section is not consistent, for example, the author compares with the SPAct in PA-HMDB dataset, but they did not compare with this method on the ADIENCE dataset, which is their main result. 

clarity,_quality,_novelty_and_reproducibility: The paper is generally written clearly. I am not an expert in this area and cannot evaluate the originality of this work in great detail. The method makes sense to me.
summary_of_the_review: The paper utilizes an adversarial training-based learning algorithm to fulfill their goal-simultaneously suppressing any selected set of attributes while preserving the rest for downstream machine learning tasks. There are some vague explanations in the paper that need to be clarified. Will consider changing my score after I see some of the other reviewers' opinions and the authors' responses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies an approach that enables removal of specific attributes from data while preserving the remaining features of the data for downstream machine learning applications. The paper evaluates the efficacy of the proposed approach in experiments that suggest removed attributes are difficult to infer while other machine-learning predictions can still be performed quite well.

The reviewers are generally lukewarm about the paper, expressing around scalability of the approach, comparisons with alternative methods, and clarity of the presentation. While some of these concerns were addressed in the discussion phase, all reviewers remain lukewarm about the paper.

For that reason, the AC decided to read the paper themselves. They noticed a severe weakness in the paper that was not mentioned by the reviewers. While the paper focuses on preserving privacy of some aspects of the data, the paper does not formalize the privacy definition or threat model it adopts. As a result, it is unclear what guarantees the paper actually provides. For example, is it truly impossible to infer DataID in the Adience dataset? Or is it just not possible to infer DataID with the model that was used, but would another model family infer it just fine? Without rigorous privacy guarantees, it is impossible to answer this questions as a result of which the true value of the proposed approach is difficult to gauge. The proposed approach would have much more real-world relevance if it provided an attribute-level version of privacy guarantees on membership inference (differential privacy) or data reconstruction (Fisher information loss). Earlier methods without such guarantees were rapidly broken rendering them useless (for example, InstaHide was broken by Nicolas Carlini) and the approach proposed in this paper may suffer a similar faith without robust privacy guarantees.
justification_for_why_not_higher_score: This paper presents yet another privacy approach without clear privacy guarantees, rendering the approach completely useless and potentially dangerous because it is misleading.
Decision: Reject