Title: Learning Bregman Divergences with Application to Robustness
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bregman divergence

similarity and distance learning

mirror descent

corruption robustness.
Abstract: We propose a novel and general method to learn Bregman divergences from raw high-dimensional data that measure similarity between images in pixel space. As a prototypical application, we learn divergences that consider real-world corruptions of images (e.g., blur) as close to the original and noisy perturbations as far, even if in $L^p$-distance the opposite holds. We also show that the learned Bregman divergence excels on datasets of human perceptual similarity judgment, suggesting its utility in a range of applications. We then define adversarial attacks by replacing the projected gradient descent (PGD) with the mirror descent associated with the learned Bregman divergence, and use them to improve the state-of-the-art in robustness through adversarial training for common image corruptions. In particular, for the contrast corruption that was found problematic in prior work we achieve an accuracy that exceeds the $L^p$- and the LPIPS-based adversarially trained neural networks by a margin of 27.16\% on the CIFAR-10-C corruption data set.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use input-convex neural networks to learn Bregman divergences as a means to distinguish semantically meaningful image corruptions from random noise perturbations. The approach is linked to classifier robustness by showing how the associated mirror descent algorithm can be used to perform adversarial training against image corruptions coming from a Bregman ball. Experiments on benchmark corruption datasets show that the proposed method outperforms prior learned similarity metrics in distinguishing corruption from noise and in adversarial training. The proposed method is also shown to generalize quite well to corruptions that it is not trained on.\n\nSTRENGTHS:\nThe paper is very well-written, easy to follow, and has great visualizations. The proposed method appears novel and performative, and is certainly of interest to the ML and robustness communities.\n\nWEAKNESSES:\nSee questions below.\n\nQUESTIONS:\n1. Line 42: \"...that has to be convex and with invertible gradient.\" Do you mean strictly convex?\n2. Line 37: The title of this paragraph is \"Bregman divergence and mirror descent,\" yet mirror descent is not discussed at all. Please give a brief description of mirror descent here.\n3. Line 64: Please put the footnote number \"1\" after the punctuation (period). With the footnote number before the punctuation, it makes it look like an exponent. Please also do the same for all other footnotes.\n4. Line 73: In (3), do you mean \"argmin\" instead of \"min\"?\n5. Table 1: Please write out \"ICNN\" completely as \"input convex neural network\" here, or define the acronym somewhere in the text before Table 1 for readers unfamiliar with ICNNs.\n6. Line 79: This sentence looks strange starting with $\\mathbb{B}_h$; I suggest adding \"The ball\" to the beginning of the sentence.\n7. Table 1: Do you mean \"strictly convex\" instead of \"strongly convex\" in the text underneath the base function?\n8. Line 79: \"...but not necessarily convex.\" This is not true; Bregman balls, as you've defined them, are indeed always convex. Since $h$ is convex, its domain $\\mathcal{X}$ is convex, and therefore convexity of the Bregman ball (4) follows from convexity of $\\mathcal{X}$ together with convexity of $D_h$ in its first argument. On the other hand, if you were to have defined the ball with respect to a fixed point in the first argument of $D_h$ and varying second arguments, then the set is not necessarily convex.\n9. Line 102: I don't recall ever seeing an ICNN defined in terms of Hadamard squares of the input feedthroughs. Can you explain why you are choosing to define your ICNN model using these Hadamard squares, and how the properties of this model might differ from just using the standard linear feedthroughs with arbitrary (not necessarily nonnegative) weights? Your model seems somewhat restrictive in how much influence the feedthrough may have, as its contributions to each preactivation vector are always nonnegative.\n10. Line 123: Again, starting a sentence with math looks strange.\n11. Figure 4: How come Moco appears to perform on par with or better than the other two prior learning-based methods in Figure 4b, but it's accuracy is shown as 0 across all noise levels in Figure 4a?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an approach to learn Bregman divergences that capture perceptual image similarities according to a given dataset.\nRelying on two input-convex neural networks, they present a procedure that mimics mirror descent over the learned Bregman divergence.\nThe procedure is used to learn networks that are robust to image corruptions. Results on CIFAR-10-C subsets are presented.\n\nSTRENGTHS:\nThe idea to (attempt to) do mirror descent over learned Bregman divergences in order to train networks robust to corruptions is novel and interesting.\n\nWEAKNESSES:\nWhile the idea in itself is interesting, I do not find neither the technical presentation, nor the provided results convincing.\n\nMost of the motivation of the work derives from the use of Bergman divergences, which come with an associated mirror descent. However, in practice, I do not think the authors can be claiming to do mirror descent, because of the approximation of the inverse map, and because of the lack of a projection operator. Given the two above limitations, I do not think what the authors do would have convergence guarantees even in the convex case. Taking this into account, the stress on the mathematical motivation of the approach seems to be a bit fragile. Sometimes mathematical concepts are introduced without a clear purpose (for instance, the Legendre type, which is then not really necessary to justify their approximation of the inverse map). I would urge the authors to tone down these claims and refrain from saying they are doing mirror descent: I'd rather call it an approach \"inspired by mirror descent\".\n\nFurthermore, the work assumes that a dataset describing the corruptions is available to learn the divergence: is this a reasonable assumption for datasets such as CIFAR-10-C, which are designed as benchmarks for OOD generalization?\n\nConcerning the results: I do not think the proposed comparisons are fair. Both l2 PGD and RLAT use a threat model which is general and not targeted at specific perturbations. As such, they attain good performance over the entirety of the CIFAR-10-C corruptions. The authors, instead, focus on a small set of perturbations and show results for an algorithm that is explicitly aware of the perturbations the network need to be robust against.\n\nQUESTIONS:\n1) How is the dataset for the training of the Bergman divergence obtained? Is this a holdout from the original CIFAR-10-C?\n2) Could the authors add the performance against noise-like perturbations in the comparison against PGD and RLAT? \n3) Would the proposed approach scale to ImageNet-C? I do not think scaling to ImageNet-C is necessary, but I think such discussions should be included.\n4) The employed ICNN is different from the original work [1] as it displays quadratic terms. I understand that the squared term on x added on top of $z^l$ is useful for strong convexity. What about the quadratic terms in equation (5)?\n\nLIMITATIONS:\nThe limitations paragraph is satisfactory.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method to learn Bregman divergences from raw, high-dimensional data. This method measures similarity between images in pixel space, and considers two images as similar even if one image is corrupted by real-world corruptions, such as blur, changes in contrast, or weather conditions such as fog. The method does this in-part by simultaneously considering real-world corruptions as close to the original image, while noisy perturbations as far from the original image, even when the $L^p$ distance considers noisy perturbations as close. The authors then define adversarial attacks by replacing the projected gradient descent with mirror descent using the learned Bregman divergence. Through adversarial training on this new learned Bregman divergence, they improve the state-of-the-art in robustness.\n\nSTRENGTHS:\n- The authors clearly explain the pipeline of the algorithm and give great explanations for the choices they made (e.g. using equation (7) to approximate $\\nabla \\bar{\\phi}$.)\n\n- The authors make a good case for using Bregman divergences and learning the metric, and it seems like an interesting direction.\n\n- The algorithm seems well-motivated at each step of the pipeline, and it looks like the authors took care to make sure each step follows theory. Figures 1, 2, and 3 are helpful in explaining the motivation.\n\nWEAKNESSES:\n- A big weakness is using only one dataset for comparison. Perhaps the authors could show more experiments on ImageNet-C, and/or the Berkeley-Adobe Perceptual Patch Similarity (BAPPS) that was introduced with one of the methods the authors compared against, LPIPS.\n  - On the above note, there are a lot of parts to the algorithm, and it's unclear how hard one has to tune the algorithm to make sure each approximation lines up to get an overall, well-performing model. I would have wanted to see a stress test on the pipeline with larger images.\n  - If the authors had comparisons on more datasets and of more difficulty than CIFAR10-C, then I would be inclined to raise the score to an accept.\n\n- The proposed method takes longer to train than other standard adversarial training methods, as mentioned in the appendix.\n\nEDIT: After considering the author responses and reading all the reviewers, I have raised my score from a 4 to a 5.\n\nQUESTIONS:\n- Can we see some examples of noisy images that are considered not semantically the same as the original? Under some noise threshold, it seems reasonable for a human to classify a noisy image as the original label, right?\n\n- In Table 3, when using the proposed method, training for one corruption doesn't necessarily perform the best for that corruption (as acknowledged by the authors in the paper). For example, $\\text{MD} \\thinspace D_{\\phi}^{\\text{contrast}}$ does not perform the best on contrast corruptions, but rather $\\text{MD} \\thinspace D_{\\phi}^{\\text{zoom-blur}}$ does the best. Do you have an explanation for that? I would have liked to see this explored and explained, as it goes against intuition.\n\n- There are a lot of approximations, like computing the inverse map. How do you expect your algorithm to perform with higher resolution images?\n\nLIMITATIONS:\nThe authors appropriately acknowledge limitations where appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}