Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new and useful direction: AI model audit. The proposed solution (framework) is very interesting. The experiments show the effectiveness of the proposed method.
main_review: Pros
1. This paper proposes a new direction AI model audit. I think this is very useful direction, in particular, for industry. Rejecting making a decision (e.g. reject making decision for an image of bad quliaty and ask the user to re-submit a better image) is much better than making a wrong decision. 
2. The proposed solution is convincing for me and the experiments show the efficacy of the solution.

Cons
1. The classification model and the generative model share the same encoder. You cannot directly apply the proposed method to any exsiting network. For example, the classification backbone is ResNet-50. You cannot connect an encoder (for disentanglement learning) to this ResNet-50, or first few layers of ResNet50 works as the encoder.
2. The key theoritical barries are addressed by an off-the-shelf method, IBP.
3. The method cannot be applied to other tasks, e.g. object detection, segmentation, etc. At least, no supportive experiments for other tasks.

Questions and Suggestions
1. Can you translate the face rotation angels are within 30 degree and maintain 95% accuracy to the representation with  F(x,y) <= 0 and S_{i, in}.
2. In practice, many people use the uncertainty/quality/confidence score to do AI model audit. I would suggest to compare with them in Related Work section.
summary_of_the_review: Despite my concerns, I think this is a VERY valuable work for the society. The paper is well written and organized.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents, AuditAI, a system for verifying that a trained model meets a particular `specification', and also for certified training. The system uses a generative model to construct embeddings for inputs and then tests whether these embedding vectors belong to a particular set. Membership in the pre-specified set is reduced to satisfying the test. The key technical insight in this work is applying the interval bound propagation technique to embedding/latent codes in order to verify semantic behaviors of trained models.
main_review: This paper tackles an important problem and provides a comprehensive discussion and evaluation of the approach. Certified training and verification is quite important for developing reliable models going forward. I detail key strengths, weaknesses, and additional questions below.


### Strengths
- Extending IBP to the latent code is straightforward and simple. 
- The breadth/scope that this work tackles is impressive. The paper shows how to verify a particular behavior like brightness, etc, and also how to train in a certified manner. 
- The scope of the empirical assessment in the work is also impressive. The paper tests on domain shift, ImageNet, Chest Xrays, and FFHQ. In each case, demonstrating convincing performance of the modified IBP.


### Weaknesses
- Toy setting with known ground truth. There is a simple experiment that the authors can run to provide a sanity check that their variation of IBP 'works'. Why not train a toy model that specifically violates a unit test of interest and show that the entire scheme here recovers the violation? 
- The principal assumption in this work is that the latent code and the corresponding generator can decompose the input into semantically meaningful concept. Even further, the assumption is that each dimension of the latent code only encodes for a single concept. Contemporary evidence shows that this assumption might not exactly be true. While this issue doesn't invalidate this work, I think the authors to make this assumption/requirement more explicit.
- User study. The study is interesting, but I am not sure what to take away from this part of the work. First, I don't think Turkers are quite equipped to judge Chest XRays. However, even if we assume that they are, showing indistinguishability between real and generated images does not demonstrate the effectiveness of AuditAI.
- I am confused that Theorem 1 does not have assumptions on $f$. It seems like what IBP does (not an expert on this technique) is to use a bound on the logits to bound the output of the network. Wouldn't this require assumptions on the activations or some form of bound on the Lipschitz constant of the function? 
- It is unclear to me how the search procedure in IBP actually works in practice. Can this information be added to the appendix?
- What does, "the corresponding convex bound of the variations of $\mathcal{S}^{0}_{i, in}$ in image space" mean?
summary_of_the_review: I lean towards an acceptance for this work since it provides a convincing demonstration of IBP for latent embeddings, which means more interesting properties beyond adversarial robustness on the input space can be verified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents AuditAI that can do certified training, verification under semantic specifications based on a generative model. 
The authors extended IBP to latent space variations and evaluate on a wide range of datasets. 

main_review: Strength:
- The paper is written clearly.

Weakness:
- The paper seems to be a natural extension of IBP, Advmix [1]
- I am not sure whether the distinction with [2] is clear (in appendix A.6). I think modeling perturbation sets for images with perturbations (like adversarial corruptions) explicitly can capture the specification better. 
- For the perturbation in the semantic space of a generative model, it is not clear to me whether this is a semantic preserving perturbation. Also it is not clear whether the latent for a particular GAN model can capture all the variations. Usually, the diversity of the GAN is quite limited. 
- I don’t find the unit test arguments to be very novel as it is a standard verification protocol. 
- The name AuditAI is not formal. 
- Minor: for few references, the use of citep, citet is mixed.
- Minor: theorem 1 “(“ should be removed.

[1] https://arxiv.org/abs/1912.03192

[2] https://openreview.net/pdf?id=MIDckA56aD

summary_of_the_review: Overall, the novelty of the paper is a bit limited and I am not sure whether the perturbation on the latent space of generative models indeed captures all the possible semantic specifications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a framework for auditing deep learning models in regards to
a specification. The goal is to increase the confidence in a trained model. The
auditing is performed by generating variations in input data to evaluate the
boundaries in the input for an expected output. An evaluation was performed
against a pixel-based training approach and under consideration of 4 different
datasets.
main_review: ## Major Issues
* At times the formal foundation is not precisely specified so a detailed
   understanding is at times difficult, e.g.:
    * Page 4: $L = L_{task} + ɣ L_{spec} + δ L_{gen}$ -> gamma and delta not
      explained
    * Page 4: Variables used in paragraph "Task Performance" are only introduced
      in the next paragraph "Verifiablity" (e.g. $z_K$, $y_{true}$)
* Page 5: Not clear how the paragraph "Interpretability" which discusses
   $L_{gen} is related to interpretability.
* Page 8; Section 4.5: Authors are talking about "certified images"; so far the
   certification was about the training not individual images, the intend of
   this section is not clear; also is not a robustness against adversarial
   attacks also a property which can be seen as a variation which is the target
   of the framework.
* Page 8; Section 5: The authors perform a human study to evaluate whether the
   generated images are realistic for the human eye. However I am unsure whether
   this is a good metric for evaluation since realistic images for the human eye
   are known to not be equivalent to useful images for machine learning datasets
   (c.f.g. adversarial attacks). Also the participants of the study are of
   unknown expertise and the chosen dataset was that of Chest-X-ray images.

## Discussion
The paper contributes to a solution of a problem that is both relevant and in
scope of the conference. The authors motivate the paper and clearly state the
contributions as well as the followed approach. While some rework of the formal
foundation (see `Points Against: 3.) would be good to increase understandability
they are sound and allow a deep understanding of the approach and also convince
the reader that the approach is well thought out. However the usage of a
generative model to create the inputs to check boundaries does limit the
confidence in the framework since the reliability of the generative network is
now at question. Which means the reliability of the framework depends again on
another machine learning component. The evaluation of this component, the
generative network, was done by performing a human study about how realistic the
images are for the human eye and was also performed by participants of unknown
expertise. The specification of the encoding function $e()$ is not clear aside
from the used example of the change in degree and should be discussed further,
as this is also an important aspect in the framework. Without clear
specification of $e()$ and assurance that this in fact represents the variation
which needs to be audited the result of audit is questionable. Non-the-less the
contribution is worth publishing as this represents a sound foundation for
future research.

## Minor issues
* Figure 1: The mapping of the three parts of the figure to the explanation
  could be improved.
* No structural overview of the paper at the end of the introduction.
* Page 2; last paragraph: "… 25%larger …" -> space missing
* across the whole paper: "upto" -> up to
* Page 5; first paragraph: "tigher" -> tighter
* Page 7; Table 4 is referenced before Table 3 in the text, so these two tables
  should be swapped in the tex document.
* Page 8 and 9: "AuditAI" is not in monospace font
* the authors at times claim to certify, however this is in my opinion to
  strong, most of the time they speak of auditing which is the better
  description
summary_of_the_review: ## Points in Favor
+ Relevant problem and in scope of the conference.
+ Presentation of a framework for auditing deep learning models in respect to
   given specification.
+ Formally sound foundation of the framework and its mathematical workings.
+ Generally the introduction of the framework is given in stages with
   relaxing assumptions, so the understandability is good.
+ Experiments show the improvement with the framework.

## Points Against
- Generative Network used to create the variations: while this strengthens the
   confidence in the model it does move the reliability of the framework to
   the GAN as well.
- The framework is dependent on the encoding function e() and it is not clear
   whether this is always specifiable.
- Partly, the formal foundation is not precisely specified
review_title: null

Meta-review: The paper discusses the auditing of AI systems which is clearly a very important topic. The approach targets mainly image classification systems where the main idea is to do the certification not in pixel space but in a latent space of a generative model. For the certification the authors use the existing CROWN-IBP for cubes around test points in latent space. The authors report results on several datasets using existing methods for the generative models.

Most of the reviewers liked this work even though all of them also raised quite similar concerns. The positive reviews were to some extent based on the fact that auditing of AI systems is a very important problem - and I completely agree on this. On the other hand auditing of AI systems, in particular of safety critical systems as discussed with the chest x-ray application in this paper, is a serious problem where a paper has to fulfill very high standards. In particular, being transparent about its limitations is crucial which in my opinion does not hold for the current version of this paper.

My main conerns are:
- the motivating example (varying the angle from which a face is seen by less than 30 degress) is misleading as such a precise threat model is not feasible with the generative models discussed later on (see last bullet point)
- there is no discussion on the potential trade-off regarding test accuracy training the classifier i) in latent space, ii) with CROWN-IBP using a rather shallow model. This is important to judge the utility of this approach - it has to be transparent what the trade-off is regarding test accuracy and what kind of price one has to pay in terms of test accuracy for the certificates.
- the deployment check discussed in Section 2.4 is nowhere implemented as later on only boxes in latent space around individual test points are checked but not a fixed box
- the main problem connected to what I mentioned above with the motivating example is that the latent dimensions have to be interpreted by humans. I doubt that this can be made fully transparent to humans what such a certificate in latent space meand and thus to give very precise labels to the latent dimensions like "brightness" is in my point of view misleading. It suggests a much more finegrained semantic control than what is available and to use such inprecise and potentially wrong ``labels'' is highly problematic for an auditing process as e.g. this paper does not contain any guarantee against "brightness"-changes. For safety critical systems this is an important distinction.

As a minor point there is little technical novelty (certification with CROWN-IBP is done with an existing library) but this plays no role in the decision.

In total the paper is not ready yet for publication. The authors should revise their manuscript by addressing the points above, being fully transparent about what the paper can do and what it cannot do. I suggest to do less examples but provide more details on how the authors envision the auditing/certification process for the given application, in particular also with the discussion about rejection of inputs not lying in certified areas (which has not been done in the present paper).
Decision: Reject