Year: 2021

Abstract:
Adversarially robust learning methods require invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity. Learning multiple sub-models in an ensemble can mitigate this insufficiency, further improving both generalization and robustness. However, an ensemble still wastes the limited capacity of multiple models. To optimally utilize the limited capacity, this paper proposes to learn a collaboration among multiple sub-models. Compared with the ensemble, the collaboration enables the possibility of correct predictions even if there exists a single correct sub-model. Besides, learning a collaboration could enable every sub-model to fit its vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas. To implement the idea, we propose a collaboration framework—CDA the abbreviation for Collaborate to Defend against Adversarial Attacks. CDA could effectively minimize the vulnerability overlap of all sub-models and then choose a representative sub-model to make correct predictions. Empirical experiments verify that CDA outperforms various ensemble methods against black-box and white-box adversarial attacks.

text: Adversarially robust learning methods require invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity. Learning multiple sub-models in an ensemble can mitigate this insufficiency, further improving both generalization and robustness. However, an ensemble still wastes the limited capacity of multiple models. To optimally utilize the limited capacity, this paper proposes to learn a collaboration among multiple sub-models. Compared with the ensemble, the collaboration enables the possibility of correct predictions even if there exists a single correct sub-model. Besides, learning a collaboration could enable every sub-model to fit its vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas. To implement the idea, we propose a collaboration framework—CDA2 the abbreviation for Collaborate to Defend against Adversarial Attacks. CDA2 could effectively minimize the vulnerability overlap of all sub-models and then choose a representative sub-model to make correct predictions. Empirical experiments verify that CDA2 outperforms various ensemble methods against black-box and white-box adversarial attacks.

heading: 1 INTRODUCTION
text: Safety-critical applications (such as in medicine and finance) require the adversarial robustness of deep models (Goodfellow et al., 2015; Szegedy et al., 2014). An adversarially robust learning method requires invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity (Zhang et al., 2021; Yu et al., 2021a). This limits the further improvement of robustness and has the undesirable degradation of generalization (Madry et al., 2018).
Learning multiple sub-models in an ensemble (Breiman, 1996; Freund et al., 1996) can mitigate this insufficiency (Pang et al., 2019; Kariyappa & Qureshi, 2019; Yang et al., 2020a). Remarkably, Pang et al. (2019), Kariyappa & Qureshi (2019) and Yang et al. (2020a) minimized the vulnerability overlaps between each pair of sub-models and improved both robustness and generalization over a single model.
However, an ensemble wastes the limited capacity of multiple models. In the example of three sub-models (see Figure 1(b)), the adversarial input that lies in the black areas can fool the ensemble successfully, i.e., more than half of sub-models must correctly classify the adversarial input. Therefore, the ensemble’s voting-based strategy excludes the possibility that true predictions remain with the minority. Besides, learning an ensemble requires more than half of the sub-models to fit the same vulnerability areas, which leaves the following question unanswered whether we could only leverage a single sub-model to fit a vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas.
To optimally utilize the limited capacity, this paper proposes to learn a collaboration among multiple sub-models. As shown in Figure 1(c), the adversarial input that lies in the vulnerability overlaps of all sub-models can undoubtedly fool the collaboration. Compared with the ensemble in Figure 1(b)), collaboration enables the possibility of correct predictions even if there exists a single correct submodel merely. Besides, learning a collaboration could enable every sub-model to fit its vulnerability area, which could collectively fix broader vulnerability areas than the ensemble does. Then, submodels could collaboratively choose trustworthy ones to make the final predictions.
To realize the idea, we propose a collaboration framework—Collaborate to Defend against Adversarial Attacks (CDA2) (Algorithms 1 and 2). In CDA2, each sub-model has dual heads: one outputs a vector of predicted probability fθ(·); another outputs a scalar that measures posterior probability density (PPD) of the prediction. In the training phase, given a natural or adversarial input x, each sub-model chooses an easy one(s) to feed itself. The PPD head is meanwhile updated by comparing the predicted probability on the true label—fyθ (·) (a scalar). In the inference phase, given an input, CDA2 chooses a sub-model with the largest PPD value as the representative to output the prediction.
We highlight our key contributions as follows.
• We provide a new perspective on learning multiple sub-models for defending against adversarial attacks. We theoretically show the collaboration makes better decisions than the ensemble, which implies collaboration may fix broader vulnerability areas.
• We propose a novel collaboration framework—CDA2 (see Section 3.2). In the training phase, CDA2could effectively minimize the vulnerability overlap of all sub-models; In the inference phase, CDA2 could effectively choose a representative sub-model to make correct predictions. We also provide a comprehensive analysis illustrating the rationale of CDA2.
• Empirical experiments verify that CDA2 outperforms various ensemble methods against black-box and white-box adversarial attacks.

heading: 2 RELATED WORKS
text: Adversarial attack Adversarial attacks aim to craft the human-imperceptible adversarial input to fool the deep models. Adversarial attacks could be roughly divided into white-box attacks in which the adversary is fully aware of the model’s structures (Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017b; Chen et al., 2018; Athalye et al., 2018; Xiao et al., 2018; Zheng et al., 2019; Wong et al., 2019; Mopuri et al., 2019; Alaifari et al., 2019; Sriramanan et al., 2020; Wu et al., 2020b; Croce & Hein, 2020; Yu et al., 2021b) and black-box attacks in which the deep models are treated as black boxes to the adversary (Cheng et al., 2019; 2020; Wu et al., 2020a; Chen et al., 2020a; Li et al., 2020a; Rahmati et al., 2020; Yan et al., 2021b; Hendrycks et al., 2021; Dong et al., 2018; Xie et al., 2019). This paper focuses on building effective defense and select both white-box and black-box attack methods as our robustness evaluation metrics.
Adversarial defense Defending adversarial attacks is a challenging task and researchers have proposed various solutions. Certified defense tries to learn provably robust deep models against norm-bounded (e.g., `2 and `∞) perturbations (Wong & Kolter, 2018; Tsuzuku et al., 2018; Weng et al., 2018; Mirman et al., 2018; Hein & Andriushchenko, 2017; Lécuyer et al., 2019; Xiao et al., 2019; Cohen et al., 2019; Balunovic & Vechev, 2020a; Zhang et al., 2020a; Singla & Feizi, 2020; Balunovic & Vechev, 2020b; Zou et al., 2021). Empirical defense leverages adversarial data to build effective defense such as adversary detection (Metzen et al., 2017; Li & Li, 2017; Carlini & Wagner, 2017a; Tian et al., 2018; Ma et al., 2018b; Lee et al., 2018; Pang et al., 2018; Smith & Gal, 2018; Roth et al., 2019; Liu et al., 2019; Yin & Rohde, 2020; Sperl et al., 2020; Cohen et al., 2020; Sheikholeslami et al., 2021; Chen et al., 2021a; Yang et al., 2020b; Qin et al., 2020; Tian et al., 2021; Wu et al., 2021) and adversarial training (AT), in which AT stands out as the most effective defense. Researchers have investigated various aspects of AT, such as improving AT’s robustness or generalization (Madry et al., 2018; Yan et al., 2018; Wu et al., 2018; Cai et al., 2018; Najafi et al.,
2019; Alayrac et al., 2019; Carmon et al., 2019; Farnia et al., 2019; Song et al., 2019; Zhang et al., 2019b; Wang et al., 2019; Tramèr & Boneh, 2019; Zhang & Wang, 2019; Stutz et al., 2020; Pang et al., 2020; Gan et al., 2020; Dong et al., 2020; Zhang et al., 2020b; Chen et al., 2020b; Song et al., 2020; Ding et al., 2020; Wang et al., 2020b; Zhang et al., 2021), fixing AT’s undesirable robust overfitting (Rice et al., 2020; Chen et al., 2021b), improving AT’s training efficiency (Zhang et al., 2019a; Shafahi et al., 2019; Zheng et al., 2020; B.S. & Babu, 2020; Andriushchenko & Flammarion, 2020; Wong et al., 2020), understanding/interpreting AT’s unique traits (Nakkiran, 2019; Yin et al., 2019; Gao et al., 2019; Cranko et al., 2019; Zhang et al., 2019c; Liu et al., 2020; Roth et al., 2020; Wang et al., 2020a; Zhang et al., 2020c; Li et al., 2020b; Zou et al., 2021; Mehrabi et al., 2021; Xu et al., 2021), etc. Besides, researchers have alao actively investigated robust-structured models (Cisse et al., 2017; Xie et al., 2020; Moosavi-Dezfooli et al., 2019; Xie & Yuille, 2020; Yan et al., 2021a; Du et al., 2021; Pang et al., 2021). Nevertheless, the above research thoroughly investigated a single model; this paper focuses on the collaboration among multiple models for adversarial defense.
Ensemble methods for adversarial robustness The most relevant studies are the ensemble methods. Ensemble methods such as bagging (Breiman, 1996) and boosting (Freund et al., 1996) have been investigated for significantly improving the model’s generalization. Motivated by the benefits of ensemble methods in improving generalization, researchers introduced an ensemble to improve the model robustness (Yang et al., 2020a; Kariyappa & Qureshi, 2019; Pang et al., 2019; Tramèr et al., 2018). Tramèr et al. (2018) proposed to reduce the adversarial transferability by training a single model with adversarial examples from multiple pretrained sub-models. Pang et al. (2019) introduce a regularization method—ADP—to encourage high diversity in the non-maximal predictions of sub-models. Kariyappa & Qureshi (2019) improved the ensemble diversity by maximizing the introduced cosine distance between the gradients of sub-models with respect to the input. Yang et al. (2020a) proposed to distill non-robust features in the input and diversify the adversarial vulnerability. These methods reduced overlaps of vulnerability areas between sub-models (Yang et al., 2020a).
To further improve the ensembles, mixture-of-experts (MOE) assume that the problem space can be divided into multiple sub-problems through a gate module; the gate module specifies each sub-model on a specific sub-problem (Jacobs et al., 1991; Ma et al., 2018a).
Nevertheless, to the best of our knowledge, MOE-based methods have been not applied to help adversarial robustness. Inspired by MOE, we propose the collaboration framework to defend against adversary attacks.

heading: 3 COLLABORATION TO DEFEND AGAINST ADVERSARIAL ATTACK

heading: 3.1 SUPERIORITY OF COLLABORATION
text: This section shows a collaboration, in theory, could make better decisions than an ensemble.
Ensemble Suppose that there are M learned sub-models {fθ1 , fθ2 , ..., fθM }, given an input x, M sub-models make predictions { fθ1(x), fθ2(x), ..., fθM (x) } . The ensemble outputs a final prediction ensemble(x, fθ1 , ..., fθM ) by the voting-based strategy:
ensemble(x, fθ1 , ..., fθM ) = arg maxy∈{1,...,K} ( M∑ i=1 1y=fθi (x) ) , (1)
where 1 is the indicator function. Note that the ensemble outputs the predicted label y that agrees with the majority predictions of the sub-models.
Definition 1 (best-performing sub-model). Given an input x and its label y, the best-performing sub-model achieves the lowest objective loss on the data (x, y) among all M sub-models:
fθbest(x) = min fθi∈{fθ1 ,.,fθM } `(fθi(x), y). (2)
Note that the best-performing sub-model is w.r.t. the input data (x, y), i.e., different input data correspond to different best-performing sub-models.
Under review as a conference paper at ICLR 2022
Collaboration Suppose that there are M learned sub-models {fθ1 , fθ2 , ..., fθM }. Given an input x, sub-models make predictions { fθ1(x), fθ2(x), ..., fθM (x) } . The collaboration tries to output a final prediction collaboration(x, fθ1 , ..., fθM ) by the best-performing sub-model:
collaboration(x, fθ1 , ..., fθM ) = fθbest(x). (3)
Proposition 1. Given M learned sub-models, the predicted accuracy of the collaboration is upperbounded that of the ensemble, i.e., E(x,y)∈D [ 1collaboration(x,fθ1 ,...,fθM )=y ] ≥ E(x,y)∈D [ 1ensemble(x,fθ1 ,...,fθM )=y ] . (4)
Proof. Given an (x, y) ∈ D, if the ensemble’s prediction is correct, at least one sub-model makes correct prediction, i.e., 1fθbest (x)=y holds; therefore, the collaboration’ prediction is correct. If the collaboration’s prediction is correct, there exists a case that the majority of sub-models make consistent but wrong predictions, while a single sub-model’s prediction is correct; then, ensemble’s prediction is wrong. Therefore, Proposition 1 holds.
From Proposition 1, a collaboration can theoretically achieve an equal or higher performance than an ensemble. Next, we will introduce a realization of our collaboration framework.

heading: 3.2 REALIZATION OF COLLABORATION FOR DEFENDING AGAINST ADVERSARIAL ATTACK
text: This section realizes the framework of Collaboration for Defending against Adversarial Attack (CDA2).
Notation We firstly introduce the needed notations. Suppose X and Y denote input space and output space, where Y = {1, ...,K} for a K-class classification problem. There are N samples in the dataset D = {(x, y)}, where x ∈ X and y ∈ Y . Let dinf(x, x′) = ‖x − x′‖∞ denotes the infinity distance metric, and B [x] = {x′ ∈ X | dinf (x, x′) ≤ } is the closed ball of of radius > 0 centered at x. To search for adversarial data within norm ball B [x], Madry et al. (2018) proposed a projected gradient descent (PGD) method that iteratively searches for adversarial data x̃ (x refers to natural data). fθ(x) outputs a K-dimensional predicted probability, i.e., p̂(x) = [p̂1(x), ..., p̂k(x)].
<latexit sha1_base64="Pbc+1s8Sc82XeqeLL9Ole7+uVlw=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LIoiMsW7AO0SJJO69DJg8lELEV/wK1+m/gH+hfeGVNQi+iEJGfOvefM3Hv9RPBUOc5rwZqbX1hcKi6XVlbX1jfKm1vtNM5kwFpBLGLZ9b2UCR6xluJKsG4imRf6gnX80amOd26ZTHkcXahxwnqhN4z4gAeeIqp5d12uOFXHLHsWuDmoIF+NuPyCK/QRI0CGEAwRFGEBDyk9l3DhICGuhwlxkhA3cYZ7lEibURajDI/YEX2HtLvM2Yj22jM16oBOEfRKUtrYI01MeZKwPs028cw4a/Y374nx1Hcb09/PvUJiFW6I/Us3zfyvTteiMMCxqYFTTYlhdHVB7pKZruib21+qUuSQEKdxn+KScGCU0z7bRpOa2nVvPRN/M5ma1fsgz83wrm9JA3Z/jnMWtA+qbq162KxV6if5qIvYwS72aZ5HqOMcDbSM9yOe8GydWcJKrewz1Srkmm18W9bDB220j4U=</latexit>
<latexit sha1_base64="KXCfkoyAXpEn449EMfAWF4Xk03o=">AAACy3icjVHLTsJAFD3UF+ILdemmkZi4agpihB3RjRsTTOSRACFtGWBiX2mnJogs/QG3+l/GP9C/8M5YEo0hOk3bM+fec2buvXbo8liY5ltGW1peWV3Lruc2Nre2d/K7e804SCKHNZzADaK2bcXM5T5rCC5c1g4jZnm2y1r27YWMt+5YFPPAvxGTkPU8a+TzIXcsQVR71J92wzGf9fMF0yhVK+ZJVf8NioapVgHpqgf5V3QxQAAHCTww+BCEXViI6emgCBMhcT1MiYsIcRVnmCFH2oSyGGVYxN7Sd0S7Tsr6tJeesVI7dIpLb0RKHUekCSgvIixP01U8Uc6SXeQ9VZ7ybhP626mXR6zAmNi/dPPM/+pkLQJDVFQNnGoKFSOrc1KXRHVF3lz/VpUgh5A4iQcUjwg7Sjnvs640sapd9tZS8XeVKVm5d9LcBB/yljTg+RT1xaBZMopl4/S6XKidp6PO4gCHOKZ5nqGGS9TRUHN8wjNetCst1u61h69ULZNq9vFjaY+fM6eTCw==</latexit>
<latexit sha1_base64="b3+wLcOAmLz+Yl4QyP2tsqMzHLo=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LKoC5cp2Ae0RZJ02obmxWSilFJx6w+41b8S/0D/wjtjCmoRnZDkzLn3nJl7rxP7XiIM4zWnLSwuLa/kVwtr6xubW8XtnUYSpdxldTfyI95y7IT5XsjqwhM+a8Wc2YHjs6YzOpfx5g3jiReFV2Ics25gD0Kv77m2IKrTCWwx5MHEsi6m18WSUTbU0ueBmYESsmVFxRd00EMEFykCMIQQhH3YSOhpw4SBmLguJsRxQp6KM0xRIG1KWYwybGJH9B3Qrp2xIe2lZ6LULp3i08tJqeOANBHlccLyNF3FU+Us2d+8J8pT3m1MfyfzCogVGBL7l26W+V+drEWgj1NVg0c1xYqR1bmZS6q6Im+uf6lKkENMnMQ9inPCrlLO+qwrTaJql721VfxNZUpW7t0sN8W7vCUN2Pw5znnQOCqblfJxrVKqnmWjzmMP+zikeZ6giktYqJN3jEc84VmrabfanXb/marlMs0uvi3t4QNWGpQ8</latexit>
<latexit sha1_base64="Iy5U2YnYdoQPvVfumLtQRB87WzE=">AAACzXicjVHLTsJAFD3UF+ILdemmkZi4IgUxwo7oxp2YyCMCIe0wQENfaacmBHHrD7jV3zL+gf6Fd8aSaAzRadreOfecM3PvtQLHjoRhvKW0peWV1bX0emZjc2t7J7u714j8OGS8znzHD1uWGXHH9nhd2MLhrSDkpms5vGmNL2S+ecfDyPa9GzEJeNc1h549sJkpCLod9KYdMeLCnPWyOSNfrJSNk4r+OyjkDbVySFbNz76igz58MMRwweFBUOzARERPGwUYCAjrYkpYSJGt8hwzZEgbE4sTwyR0TN8h7doJ6tFeekZKzegUh96QlDqOSOMTL6RYnqarfKycJbrIe6o85d0m9LcSL5dQgRGhf+nmzP/qZC0CA5RVDTbVFChEVscSl1h1Rd5c/1aVIIeAMBn3KR9SzJRy3mddaSJVu+ytqfLviilRuWcJN8aHvCUNeD5FfXHQKOYLpfzpdSlXPU9GncYBDnFM8zxDFZeooU7eHp7wjBftSou1e+3hi6qlEs0+fizt8RN955Pz</latexit>
<latexit sha1_base64="vHf4JuwSfHN9bcLOxGS5EE+IepE=">AAAC3nicjVHLSsNAFD2Nr1pfVVfiJliEuimpVHRZdOOygn1AW2qSTtvQvEgmYgnBnTtx6w+41c8R/0D/wjtjBLWITsjMmXPvOTN3ruHbVsg17SWjzMzOzS9kF3NLyyura/n1jUboRYHJ6qZne0HL0ENmWy6rc4vbrOUHTHcMmzWN8YmINy9ZEFqee84nPus6+tC1Bpapc6J6+a24M9J53DE8ux9OHFpiP0mS4tVeL1/QSpoc6jQop6CAdNS8/DM66MODiQgOGFxwwjZ0hPS1UYYGn7guYuICQpaMMyTIkTaiLEYZOrFjmoe0a6esS3vhGUq1SafY9AekVLFLGo/yAsLiNFXGI+ks2N+8Y+kp7jah1Ui9HGI5RsT+pfvM/K9O1MIxwJGswaKafMmI6szUJZKvIm6ufqmKk4NPnMB9igeETan8fGdVakJZu3hbXcZfZaZgxd5McyO8iVtSg8s/2zkNGvulcqV0cFYpVI/TVmexjR0UqZ+HqOIUNdTJ+xoPeMSTcqHcKLfK3Ueqkkk1m/g2lPt3Z3Waew==</latexit>
Goal of collaboration 1) ensure the correct prediction of the best-performing sub-model for a given input, and 2) select the best-performing sub-model among all sub-models to make prediction.
First, intuitively, every sub-model in a collaboration should maximize its expertise to fit its areas and leave the remaining areas fitted by others. As a result, the collaboration can minimize the vulnerability overlaps of all sub-models. Section 4 shows “minimizing the vulnerability overlap of all sub-models” is “minimizing the objective loss of the best-performing sub-models”. Therefore, during the training phase, the given data should always be allocated to the sub-model that has the lowest objective loss. In other words, the sub-models always choose the easiest data to learn. In the example of Figure 2(a), i) Data 3© is misclassified by both sub-models. The blue sub-model is near Data 3© and has the lowest objective loss. We assign the blue sub-model to fit Data 3©. ii) Data 2© is correctly classified by the pink model but wrongly classified by the blue model; for ease of effort, we assign the pink model to
Under review as a conference paper at ICLR 2022
<latexit sha1_base64="Pbc+1s8Sc82XeqeLL9Ole7+uVlw=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LIoiMsW7AO0SJJO69DJg8lELEV/wK1+m/gH+hfeGVNQi+iEJGfOvefM3Hv9RPBUOc5rwZqbX1hcKi6XVlbX1jfKm1vtNM5kwFpBLGLZ9b2UCR6xluJKsG4imRf6gnX80amOd26ZTHkcXahxwnqhN4z4gAeeIqp5d12uOFXHLHsWuDmoIF+NuPyCK/QRI0CGEAwRFGEBDyk9l3DhICGuhwlxkhA3cYZ7lEibURajDI/YEX2HtLvM2Yj22jM16oBOEfRKUtrYI01MeZKwPs028cw4a/Y374nx1Hcb09/PvUJiFW6I/Us3zfyvTteiMMCxqYFTTYlhdHVB7pKZruib21+qUuSQEKdxn+KScGCU0z7bRpOa2nVvPRN/M5ma1fsgz83wrm9JA3Z/jnMWtA+qbq162KxV6if5qIvYwS72aZ5HqOMcDbSM9yOe8GydWcJKrewz1Srkmm18W9bDB220j4U=</latexit>
<latexit sha1_base64="SRI39zozco4qoxG7gdB0yFEsXdE=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LLoxmVF+4C2liSdtqF5MZkIJRTErT/gVn9K/AP9C++MKahFdEKSM+fec2buvXbkubEwjNectrC4tLySXy2srW9sbhW3dxpxmHCH1Z3QC3nLtmLmuQGrC1d4rBVxZvm2x5r2+FzGm7eMx24YXItJxLq+NQzcgetYgqibQS/tiBETVi81p9NesWSUDbX0eWBmoIRs1cLiCzroI4SDBD4YAgjCHizE9LRhwkBEXBcpcZyQq+IMUxRIm1AWowyL2DF9h7RrZ2xAe+kZK7VDp3j0clLqOCBNSHmcsDxNV/FEOUv2N+9Uecq7TehvZ14+sQIjYv/SzTL/q5O1CAxwqmpwqaZIMbI6J3NJVFfkzfUvVQlyiIiTuE9xTthRylmfdaWJVe2yt5aKv6lMycq9k+UmeJe3pAGbP8c5DxpHZbNSPr6slKpn2ajz2MM+DmmeJ6jiAjXUyZvjEU941q60iXan3X+marlMs4tvS3v4APSFlUY=</latexit>
<latexit sha1_base64="T3+vY9iQSc0/5AjFO1QkFzkmcoI=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl040aoaB/Q1pKk0zY0LyYToYSCuPUH3OpPiX+gf+GdMQW1iE5Icubce87MvdeOPDcWhvGa0+bmFxaX8suFldW19Y3i5lY9DhPusJoTeiFv2lbMPDdgNeEKjzUjzizf9ljDHp3JeOOW8dgNg2sxjljHtwaB23cdSxB10++mbTFkwuqmF5NJt1gyyoZa+iwwM1BCtqph8QVt9BDCQQIfDAEEYQ8WYnpaMGEgIq6DlDhOyFVxhgkKpE0oi1GGReyIvgPatTI2oL30jJXaoVM8ejkpdeyRJqQ8Tliepqt4opwl+5t3qjzl3cb0tzMvn1iBIbF/6aaZ/9XJWgT6OFE1uFRTpBhZnZO5JKor8ub6l6oEOUTESdyjOCfsKOW0z7rSxKp22VtLxd9UpmTl3slyE7zLW9KAzZ/jnAX1g7J5WD66PCxVTrNR57GDXezTPI9RwTmqqJE3xyOe8KxdaWPtTrv/TNVymWYb35b28AE3TJVi</latexit>
<latexit sha1_base64="HJ8Aaph3DpBoOP6hNck7TudISXQ=">AAAC23icjVHLSsNAFD2Nr1pfVcGNm2ARXJVEKrosunFZwT6grSVJp+3QNAnJRCyxK3fi1h9wq/8j/oH+hXfGCGoRnZDkzLn3nJl7rx24PBKG8ZLRZmbn5heyi7ml5ZXVtfz6Ri3y49BhVcd3/bBhWxFzuceqgguXNYKQWSPbZXV7eCLj9UsWRtz3zsU4YO2R1fd4jzuWIKqT32oJ7nZZcjW5SFpiwITVSczJpJMvGEVDLX0amCkoIF0VP/+MFrrw4SDGCAweBGEXFiJ6mjBhICCujYS4kBBXcYYJcqSNKYtRhkXskL592jVT1qO99IyU2qFTXHpDUurYJY1PeSFheZqu4rFyluxv3onylHcb099OvUbECgyI/Uv3mflfnaxFoIcjVQOnmgLFyOqc1CVWXZE3179UJcghIE7iLsVDwo5SfvZZV5pI1S57a6n4q8qUrNw7aW6MN3lLGrD5c5zToLZfNEvFg7NSoXycjjqLbexgj+Z5iDJOUUGVvK/xgEc8aW3tRrvV7j5StUyq2cS3pd2/Ay3bmT4=</latexit>
<latexit sha1_base64="HBODfQKct7k8LzziKDMpDrEIVX8=">AAAC23icjVHLSsNAFD2Nr1pfVcGNm2ARXJVUKrosunEjVLAPaGtJ0mk7NE1CMhFL7MqduPUH3Or/iH+gf+GdMQW1iE5Icubce87MvdfyHR4Kw3hNaTOzc/ML6cXM0vLK6lp2faMaelFgs4rtOV5Qt8yQOdxlFcGFw+p+wMyh5bCaNTiR8doVC0LuuRdi5LPW0Oy5vMttUxDVzm41BXc6LL4eX8ZN0WfCbMdn43E7mzPyhlr6NCgkIIdklb3sC5rowIONCEMwuBCEHZgI6WmgAAM+cS3ExAWEuIozjJEhbURZjDJMYgf07dGukbAu7aVnqNQ2neLQG5BSxy5pPMoLCMvTdBWPlLNkf/OOlae824j+VuI1JFagT+xfuknmf3WyFoEujlQNnGryFSOrsxOXSHVF3lz/UpUgB584iTsUDwjbSjnps640oapd9tZU8TeVKVm5t5PcCO/yljTgws9xToPqfr5QzB+cF3Ol42TUaWxjB3s0z0OUcIoyKuR9g0c84VlrabfanXb/maqlEs0mvi3t4QNwk5la</latexit>
<latexit sha1_base64="SRI39zozco4qoxG7gdB0yFEsXdE=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LLoxmVF+4C2liSdtqF5MZkIJRTErT/gVn9K/AP9C++MKahFdEKSM+fec2buvXbkubEwjNectrC4tLySXy2srW9sbhW3dxpxmHCH1Z3QC3nLtmLmuQGrC1d4rBVxZvm2x5r2+FzGm7eMx24YXItJxLq+NQzcgetYgqibQS/tiBETVi81p9NesWSUDbX0eWBmoIRs1cLiCzroI4SDBD4YAgjCHizE9LRhwkBEXBcpcZyQq+IMUxRIm1AWowyL2DF9h7RrZ2xAe+kZK7VDp3j0clLqOCBNSHmcsDxNV/FEOUv2N+9Uecq7TehvZ14+sQIjYv/SzTL/q5O1CAxwqmpwqaZIMbI6J3NJVFfkzfUvVQlyiIiTuE9xTthRylmfdaWJVe2yt5aKv6lMycq9k+UmeJe3pAGbP8c5DxpHZbNSPr6slKpn2ajz2MM+DmmeJ6jiAjXUyZvjEU941q60iXan3X+marlMs4tvS3v4APSFlUY=</latexit>
<latexit sha1_base64="53JzsdwLkD15BrTWXnxxBGdSqeI=">AAAC63icjVHLSsNAFD3GV62vqks3waLUTUmkokvRjcsKthVaLUk6bQfzIpmIEvoH7tyJW3/Arf6H+Af6F94ZU6gW0QlJzpx7z5m599qhy2NhGG8T2uTU9Mxsbi4/v7C4tFxYWa3HQRI5rOYEbhCd2VbMXO6zmuDCZWdhxCzPdlnDvjyS8cYVi2Ie+KfiJmTnntXzeZc7liCqXdjqttOW6DNhtVNzMCi1BHc7LL0eXIzS2+1C0SgbaunjwMxAEdmqBoVXtNBBAAcJPDD4EIRdWIjpacKEgZC4c6TERYS4ijMMkCdtQlmMMixiL+nbo10zY33aS89YqR06xaU3IqWOTdIElBcRlqfpKp4oZ8n+5p0qT3m3G/rbmZdHrECf2L90w8z/6mQtAl3sqxo41RQqRlbnZC6J6oq8uT5SlSCHkDiJOxSPCDtKOeyzrjSxql321lLxd5UpWbl3stwEH/KWNGDz5zjHQX2nbFbKuyeV4sFhNuoc1rGBEs1zDwc4RhU18r7FE57xonnanXavPXylahOZZg3flvb4CZ5Ln+Y=</latexit>
<latexit sha1_base64="T3+vY9iQSc0/5AjFO1QkFzkmcoI=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl040aoaB/Q1pKk0zY0LyYToYSCuPUH3OpPiX+gf+GdMQW1iE5Icubce87MvdeOPDcWhvGa0+bmFxaX8suFldW19Y3i5lY9DhPusJoTeiFv2lbMPDdgNeEKjzUjzizf9ljDHp3JeOOW8dgNg2sxjljHtwaB23cdSxB10++mbTFkwuqmF5NJt1gyyoZa+iwwM1BCtqph8QVt9BDCQQIfDAEEYQ8WYnpaMGEgIq6DlDhOyFVxhgkKpE0oi1GGReyIvgPatTI2oL30jJXaoVM8ejkpdeyRJqQ8Tliepqt4opwl+5t3qjzl3cb0tzMvn1iBIbF/6aaZ/9XJWgT6OFE1uFRTpBhZnZO5JKor8ub6l6oEOUTESdyjOCfsKOW0z7rSxKp22VtLxd9UpmTl3slyE7zLW9KAzZ/jnAX1g7J5WD66PCxVTrNR57GDXezTPI9RwTmqqJE3xyOe8KxdaWPtTrv/TNVymWYb35b28AE3TJVi</latexit>
<latexit sha1_base64="SRI39zozco4qoxG7gdB0yFEsXdE=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LLoxmVF+4C2liSdtqF5MZkIJRTErT/gVn9K/AP9C++MKahFdEKSM+fec2buvXbkubEwjNectrC4tLySXy2srW9sbhW3dxpxmHCH1Z3QC3nLtmLmuQGrC1d4rBVxZvm2x5r2+FzGm7eMx24YXItJxLq+NQzcgetYgqibQS/tiBETVi81p9NesWSUDbX0eWBmoIRs1cLiCzroI4SDBD4YAgjCHizE9LRhwkBEXBcpcZyQq+IMUxRIm1AWowyL2DF9h7RrZ2xAe+kZK7VDp3j0clLqOCBNSHmcsDxNV/FEOUv2N+9Uecq7TehvZ14+sQIjYv/SzTL/q5O1CAxwqmpwqaZIMbI6J3NJVFfkzfUvVQlyiIiTuE9xTthRylmfdaWJVe2yt5aKv6lMycq9k+UmeJe3pAGbP8c5DxpHZbNSPr6slKpn2ajz2MM+DmmeJ6jiAjXUyZvjEU941q60iXan3X+marlMs4tvS3v4APSFlUY=</latexit>
<latexit sha1_base64="T3+vY9iQSc0/5AjFO1QkFzkmcoI=">AAAC0XicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl040aoaB/Q1pKk0zY0LyYToYSCuPUH3OpPiX+gf+GdMQW1iE5Icubce87MvdeOPDcWhvGa0+bmFxaX8suFldW19Y3i5lY9DhPusJoTeiFv2lbMPDdgNeEKjzUjzizf9ljDHp3JeOOW8dgNg2sxjljHtwaB23cdSxB10++mbTFkwuqmF5NJt1gyyoZa+iwwM1BCtqph8QVt9BDCQQIfDAEEYQ8WYnpaMGEgIq6DlDhOyFVxhgkKpE0oi1GGReyIvgPatTI2oL30jJXaoVM8ejkpdeyRJqQ8Tliepqt4opwl+5t3qjzl3cb0tzMvn1iBIbF/6aaZ/9XJWgT6OFE1uFRTpBhZnZO5JKor8ub6l6oEOUTESdyjOCfsKOW0z7rSxKp22VtLxd9UpmTl3slyE7zLW9KAzZ/jnAX1g7J5WD66PCxVTrNR57GDXezTPI9RwTmqqJE3xyOe8KxdaWPtTrv/TNVymWYb35b28AE3TJVi</latexit>
<latexit sha1_base64="Z2xtlwyAJ8GiQ3JRHC4/wZkRCUI=">AAAC63icjVHLSsNAFD3GV62vqks3waLopiRS0WXRjRuhgq2C1ZKk0zqYF8lELCF/4M6duPUH3Op/iH+gf+GdMYpaRCckOXPuPWfm3muHLo+FYTwPacMjo2PjhYni5NT0zGxpbr4ZB0nksIYTuEF0ZFsxc7nPGoILlx2FEbM822WH9vmOjB9esCjmgX8g+iE78ayez7vcsQRR7dJKt522xBkTVjvdy7LVluBuh6WX2eknbWbZWrtUNiqGWvogMHNQRr7qQekJLXQQwEECDww+BGEXFmJ6jmHCQEjcCVLiIkJcxRkyFEmbUBajDIvYc/r2aHecsz7tpWes1A6d4tIbkVLHMmkCyosIy9N0FU+Us2R/806Vp7xbn/527uURK3BG7F+6j8z/6mQtAl1sqRo41RQqRlbn5C6J6oq8uf6lKkEOIXESdygeEXaU8qPPutLEqnbZW0vFX1SmZOXeyXMTvMpb0oDNn+McBM31ilmtbOxXy7XtfNQFLGIJqzTPTdSwizoa5H2FezzgUfO0a+1Gu31P1YZyzQK+Le3uDeOjoAI=</latexit>
<latexit sha1_base64="9g1z2HDzzbKpV+uB1W8se3BseMY=">AAAC63icjVHLSsNAFD3GV62vqks3waLopiRS0WXRjRuhgq2C1ZKk0zqYF8lELCF/4M6duPUH3Op/iH+gf+GdMYpaRCckOXPuPWfm3muHLo+FYTwPacMjo2PjhYni5NT0zGxpbr4ZB0nksIYTuEF0ZFsxc7nPGoILlx2FEbM822WH9vmOjB9esCjmgX8g+iE78ayez7vcsQRR7dJKt522xBkTVjs1s2y1JbjbYelldvpJ72XZWrtUNiqGWvogMHNQRr7qQekJLXQQwEECDww+BGEXFmJ6jmHCQEjcCVLiIkJcxRkyFEmbUBajDIvYc/r2aHecsz7tpWes1A6d4tIbkVLHMmkCyosIy9N0FU+Us2R/806Vp7xbn/527uURK3BG7F+6j8z/6mQtAl1sqRo41RQqRlbn5C6J6oq8uf6lKkEOIXESdygeEXaU8qPPutLEqnbZW0vFX1SmZOXeyXMTvMpb0oDNn+McBM31ilmtbOxXy7XtfNQFLGIJqzTPTdSwizoa5H2FezzgUfO0a+1Gu31P1YZyzQK+Le3uDeEfoAI=</latexit>
<latexit sha1_base64="AGt4k7VJmkQM5psYLcza9FRQ7aE=">AAAC63icjVHLSsNAFD3GV31XXboJFkU3JZGKLotu3AgVrAqthiSdtoN5kUzEEvoH7tyJW3/Arf6H+Af6F94ZU1CL6IQkZ86958zce53I44kwjNcRbXRsfGKyMDU9Mzs3v1BcXDpJwjR2Wd0NvTA+c+yEeTxgdcGFx86imNm+47FT53Jfxk+vWJzwMDgWvYid+3Yn4G3u2oIoq7jetrKm6DJhW9lhv7/RFNxrsey6f/GV3rSKJaNsqKUPAzMHJeSrFhZf0EQLIVyk8MEQQBD2YCOhpwETBiLizpERFxPiKs7QxzRpU8pilGETe0nfDu0aORvQXnomSu3SKR69MSl1rJEmpLyYsDxNV/FUOUv2N+9Mecq79ejv5F4+sQJdYv/SDTL/q5O1CLSxq2rgVFOkGFmdm7ukqivy5vqXqgQ5RMRJ3KJ4TNhVykGfdaVJVO2yt7aKv6lMycq9m+emeJe3pAGbP8c5DE62ymalvH1UKVX38lEXsIJVbNA8d1DFAWqok/cNHvGEZ83XbrU77f4zVRvJNcv4trSHDyaGoB4=</latexit>
<latexit sha1_base64="67QdplBzFYPayCx6zk8ADSUpLdY=">AAADA3icjVHLTtwwFD2khfIodGiXbKyOKg0SGiVoqrKqULthCRIDSAwdORkPY+E85DiooyhL/qQ7dqhbfqDbVkL9g/IXXJuAeKgCR4mPzz3nxNcOMyVz4/t/J7wXLyenXk3PzM69nl9401h8u5OnhY5EN0pVqvdCngslE9E10iixl2nB41CJ3fDoq63vHgudyzTZNuNMHMT8MJFDGXFDVL/xuSeUarFhv+yZkTCc5pibkY7LUOSmqqpWz0g1EOX36tutJKiq5RU2Zmy532j6bd8N9hgENWiiHptp4wI9DJAiQoEYAgkMYQWOnJ59BPCREXeAkjhNSLq6QIVZ8hakEqTgxB7R95BW+zWb0Npm5s4d0V8UvZqcDB/Ik5JOE7Z/Y65euGTL/i+7dJl2b2OawzorJtZgROxTvhvlc322F4Mh1lwPknrKHGO7i+qUwp2K3Tm705WhhIw4iwdU14Qj57w5Z+Y8uevdni139X9OaVm7jmptgUu7S7rg4OF1PgY7q+2g0/641Wmuf6mvehpLeI8W3ecnrGMDm+hS9g/8wm/88U68U+/M+3kt9SZqzzvcG975Ff56qT8=</latexit>
<latexit sha1_base64="5fN3LiyrngW/JMXxeYQ1hLUTwIc=">AAADA3icjVFPSxtBHH3Z2mq1f9J67GUwCBFK2BRFT0XaSy+CQpMIRsPsZmIGZ/8wOyuGZY/9Jt68lV79Al4VSr+B/Rb+ZtyUtiI6y+68eb/33s5vJkiVzIzv/6p5T2aePpudez6/8OLlq9f1N2+7WZLrUHTCRCV6N+CZUDIWHSONErupFjwKlOgFR59tvXcsdCaT+KuZpGI/4oexHMmQG6IG9Y99oVSTjQZF34yF4TRH3Ix1VAQiM2VZNvtGqqEoTsqDP5Ktslx5zyaMrQzqDb/lu8HugnYFGqjGdlL/iT6GSBAiRwSBGIawAkdGzx7a8JESt4+COE1IurpAiXny5qQSpODEHtH3kFZ7FRvT2mZmzh3SXxS9mpwMy+RJSKcJ278xV89dsmXvyy5cpt3bhOagyoqINRgT+5Bvqnysz/ZiMMKG60FST6ljbHdhlZK7U7E7Z391ZSghJc7iIdU14dA5p+fMnCdzvduz5a5+7ZSWteuw0ub4bXdJF9z+/zrvgu6HVnu1tbaz2tj8VF31HN5hCU26z3Vs4gu20aHsU1zgElfeN+/M++79uJV6tcqziH+Gd34DQgWpWw==</latexit>
<latexit sha1_base64="q2eCfEkXjhGZ0qFnP6LvTpWDLtY=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LIoiMsW7ANqkWQ6rUMnD5KJUIr+gFv9NvEP9C+8M6agFtEJSc6ce8+Zuff6sRSpcpzXgrWwuLS8Ulwtra1vbG6Vt3faaZQljLdYJKOk63splyLkLSWU5N044V7gS97xx+c63rnjSSqi8EpNYt4PvFEohoJ5iqjm5KZccaqOWfY8cHNQQb4aUfkF1xggAkOGABwhFGEJDyk9PbhwEBPXx5S4hJAwcY57lEibURanDI/YMX1HtOvlbEh77ZkaNaNTJL0JKW0ckCaivISwPs028cw4a/Y376nx1Heb0N/PvQJiFW6J/Us3y/yvTteiMMSpqUFQTbFhdHUsd8lMV/TN7S9VKXKIidN4QPGEMDPKWZ9to0lN7bq3nom/mUzN6j3LczO861vSgN2f45wH7aOqW6seN2uV+lk+6iL2sI9DmucJ6rhEAy3j/YgnPFsXlrRSK/tMtQq5ZhfflvXwAXAUj4Y=</latexit>
<latexit sha1_base64="q2eCfEkXjhGZ0qFnP6LvTpWDLtY=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LIoiMsW7ANqkWQ6rUMnD5KJUIr+gFv9NvEP9C+8M6agFtEJSc6ce8+Zuff6sRSpcpzXgrWwuLS8Ulwtra1vbG6Vt3faaZQljLdYJKOk63splyLkLSWU5N044V7gS97xx+c63rnjSSqi8EpNYt4PvFEohoJ5iqjm5KZccaqOWfY8cHNQQb4aUfkF1xggAkOGABwhFGEJDyk9PbhwEBPXx5S4hJAwcY57lEibURanDI/YMX1HtOvlbEh77ZkaNaNTJL0JKW0ckCaivISwPs028cw4a/Y376nx1Heb0N/PvQJiFW6J/Us3y/yvTteiMMSpqUFQTbFhdHUsd8lMV/TN7S9VKXKIidN4QPGEMDPKWZ9to0lN7bq3nom/mUzN6j3LczO861vSgN2f45wH7aOqW6seN2uV+lk+6iL2sI9DmucJ6rhEAy3j/YgnPFsXlrRSK/tMtQq5ZhfflvXwAXAUj4Y=</latexit>
<latexit sha1_base64="SA5SNRso7yH96ZykGBWhr4c8QU4=">AAACzXicjVHLTsJAFD3UF+ILdemmkZi4Iq3B6JLoxp2YyCMCIW0ZoKG0zXRqQhC3/oBb/S3jH+hfeGccEpUYnabtmXPvOTP3XjcO/ERY1mvGWFhcWl7JrubW1jc2t/LbO7UkSrnHql4URLzhOgkL/JBVhS8C1og5c0ZuwOru8FzG67eMJ34UXotxzNojpx/6Pd9zBFE3vc6kJQZMONNOvmAVLbXMeWBrUIBelSj/gha6iOAhxQgMIQThAA4SepqwYSEmro0JcZyQr+IMU+RIm1IWowyH2CF9+7RrajakvfRMlNqjUwJ6OSlNHJAmojxOWJ5mqniqnCX7m/dEecq7jenvaq8RsQIDYv/SzTL/q5O1CPRwqmrwqaZYMbI6T7ukqivy5uaXqgQ5xMRJ3KU4J+wp5azPptIkqnbZW0fF31SmZOXe07kp3uUtacD2z3HOg9pR0S4Vj69KhfKZHnUWe9jHIc3zBGVcoIIqeYd4xBOejUsjNe6M+89UI6M1u/i2jIcPp2GTlg==</latexit>
<latexit sha1_base64="Pbc+1s8Sc82XeqeLL9Ole7+uVlw=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LIoiMsW7AO0SJJO69DJg8lELEV/wK1+m/gH+hfeGVNQi+iEJGfOvefM3Hv9RPBUOc5rwZqbX1hcKi6XVlbX1jfKm1vtNM5kwFpBLGLZ9b2UCR6xluJKsG4imRf6gnX80amOd26ZTHkcXahxwnqhN4z4gAeeIqp5d12uOFXHLHsWuDmoIF+NuPyCK/QRI0CGEAwRFGEBDyk9l3DhICGuhwlxkhA3cYZ7lEibURajDI/YEX2HtLvM2Yj22jM16oBOEfRKUtrYI01MeZKwPs028cw4a/Y374nx1Hcb09/PvUJiFW6I/Us3zfyvTteiMMCxqYFTTYlhdHVB7pKZruib21+qUuSQEKdxn+KScGCU0z7bRpOa2nVvPRN/M5ma1fsgz83wrm9JA3Z/jnMWtA+qbq162KxV6if5qIvYwS72aZ5HqOMcDbSM9yOe8GydWcJKrewz1Srkmm18W9bDB220j4U=</latexit>
<latexit sha1_base64="n0Da2N1WXFW9CtPNGdUvABipzkQ=">AAAC63icjVHLSsNAFD2Nr/quunQTLIpuSioVXRbduFSwVWhLmaRjO5gXyUQsIX/gzp249Qfc6n+If6B/4Z0xglpEJ2TmzLn3nJk71w5dEUvLeikYY+MTk1PF6ZnZufmFxdLScjMOksjhDSdwg+jMZjF3hc8bUkiXn4URZ57t8lP74kDFTy95FIvAP5HDkHc81vfFuXCYJKpb2kjbAybTth24vXjo0ZKGWZZ107YccMmyzatuKrKtbqlsVSw9zFFQzUEZ+TgKSs9oo4cADhJ44PAhCbtgiOlroQoLIXEdpMRFhISOc2SYIW1CWZwyGLEXNPdp18pZn/bKM9Zqh05x6Y9IaWKdNAHlRYTVaaaOJ9pZsb95p9pT3W1Iq517ecRKDIj9S/eZ+V+dqkXiHHu6BkE1hZpR1Tm5S6JfRd3c/FKVJIeQOIV7FI8IO1r5+c6m1sS6dvW2TMdfdaZi1d7JcxO8qVtSg6s/2zkKmtuVaq2yc1wr1/fzVhexijVsUj93UcchjtAg72s84BFPhmfcGLfG3UeqUcg1K/g2jPt3KOyghg==</latexit>
<latexit sha1_base64="b3+wLcOAmLz+Yl4QyP2tsqMzHLo=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkVwVRKp6LKoC5cp2Ae0RZJ02obmxWSilFJx6w+41b8S/0D/wjtjCmoRnZDkzLn3nJl7rxP7XiIM4zWnLSwuLa/kVwtr6xubW8XtnUYSpdxldTfyI95y7IT5XsjqwhM+a8Wc2YHjs6YzOpfx5g3jiReFV2Ics25gD0Kv77m2IKrTCWwx5MHEsi6m18WSUTbU0ueBmYESsmVFxRd00EMEFykCMIQQhH3YSOhpw4SBmLguJsRxQp6KM0xRIG1KWYwybGJH9B3Qrp2xIe2lZ6LULp3i08tJqeOANBHlccLyNF3FU+Us2d+8J8pT3m1MfyfzCogVGBL7l26W+V+drEWgj1NVg0c1xYqR1bmZS6q6Im+uf6lKkENMnMQ9inPCrlLO+qwrTaJql721VfxNZUpW7t0sN8W7vCUN2Pw5znnQOCqblfJxrVKqnmWjzmMP+zikeZ6giktYqJN3jEc84VmrabfanXb/marlMs0uvi3t4QNWGpQ8</latexit>
<latexit sha1_base64="oB8+9aBl/ne97C5CBIxvAPxIDJQ=">AAAC0XicjVHLSsNAFD2Nr/quunQTLELdlEQquiy6cVnRPqCtJUmnbWiahMlEDKUgbv0Bt/pT4h/oX3hnTEEtohOSnDn3njNz77VDz42EYbxmtLn5hcWl7PLK6tr6xmZua7sWBTF3WNUJvIA3bCtinuuzqnCFxxohZ9bI9ljdHp7JeP2G8cgN/CuRhKw9svq+23MdSxB13RpYYhxOOuNkUrg96OTyRtFQS58FZgrySFclyL2ghS4COIgxAoMPQdiDhYieJkwYCIlrY0wcJ+SqOMMEK6SNKYtRhkXskL592jVT1qe99IyU2qFTPHo5KXXskyagPE5YnqareKycJfub91h5yrsl9LdTrxGxAgNi/9JNM/+rk7UI9HCianCpplAxsjondYlVV+TN9S9VCXIIiZO4S3FO2FHKaZ91pYlU7bK3loq/qUzJyr2T5sZ4l7ekAZs/xzkLaodFs1Q8uijly6fpqLPYxR4KNM9jlHGOCqrkzfGIJzxrl1qi3Wn3n6laJtXs4NvSHj4AsOOVKQ==</latexit>
<latexit sha1_base64="4wXnKE2Vr1CDw7zoRS05x402TN0=">AAACy3icjVHLSsNAFD2Nr1pfVZdugkVwVRJRdFl040aoYB/QlpKk03ZoXmQmQq1d+gNu9b/EP9C/8M6YglpEJyQ5c+45d+be68Y+F9KyXnPGwuLS8kp+tbC2vrG5VdzeqYsoTTxW8yI/SpquI5jPQ1aTXPqsGSfMCVyfNdzRhYo3blkieBTeyHHMOoEzCHmfe44kqjnoTtrxkE+7xZJVtvQy54GdgRKyVY2KL2ijhwgeUgRgCCEJ+3Ag6GnBhoWYuA4mxCWEuI4zTFEgb0oqRgqH2BF9B7RrZWxIe5VTaLdHp/j0JuQ0cUCeiHQJYXWaqeOpzqzY33JPdE51tzH93SxXQKzEkNi/fDPlf32qFok+znQNnGqKNaOq87Isqe6Kurn5pSpJGWLiFO5RPCHsaeesz6b2CF276q2j429aqVi19zJtind1Sxqw/XOc86B+VLaPyyfXx6XKeTbqPPawj0Oa5ykquEQVNT3HRzzh2bgyhHFn3H9KjVzm2cW3ZTx8AF0hkq4=</latexit>
fit Data 2©, because the collaboration can correctly be classified Data 2© through selecting the pink model as the representative. iii) Data 1© is correctly classified by both models. The blue model is far from Data 1© and takes the lowest effort on fitting it; therefore, we assign the blue model to fit Data 1©. Second, to select the best-performing sub-model, we construct dual-head structured sub-models. As shown in Figure 2(b), our sub-model has dual heads: 1) fθ contains the feature extractor module and the label head and predicts the label probability fθ(x) = p̂(x) = [p̂1(x), ..., p̂K(x)] (a vector); 2) the PPD head gφ approximates the posterior probability density (PPD) (a scalar) of each prediction p̂(x).
Note that the PPD is the likelihood that a given prediction p̂(x) equals the true label distribution p(x). Thus, the PPD value measures the quality of the sub-model’s predictions. In other words, the best-performing sub-models can be decided by the PPD values because the largest PPD value corresponds to the lowest objective loss, and vice versa (see theoretical proof in Proposition 2).
Collaborate to defend against adversarial attack To defend against adversarial attack, the collaboration needs to learn from adversarial data. Algorithm 1 along with Figure 3 articulates how to learn such the collaboration, inculding the natural data training and the adversarial data training of the collaboration. Algorithm 2 articulates how to use the collaboration to make predictions. For the natural data training of the collaboration, as shown in Figure 3(a), given natural training data x and its label y, we use the PGD method to obtain M adversarial variants {x̃θi}Mi=1 of M sub-models as existing baselines do. For each adversarial variant x̃θi , we assign the best-performing sub-model to learn and update its feature extractor and label head. This process corresponds to Lines 2–5 and Lines 9–10 in Algorithm 1. Note that in Line 8, we use a surrogate loss to approximate the sub-model assignment process (reasons see Eq. 6 and 7). As shown in Figure 3(b), given an adversarial variant x̃θi , we propose to use the binary-cross-entropy (BCE) loss between the predicted label probability on the true label (i.e., p̂y(x̃θi)) and the approximated PPD (i.e., gφ(x̃θi)) to update each sub-model’s PPD head. This process corresponds to Lines 6–7 in Algorithm 1.
The natural data training of the collaboration using the most adversarial data of each sub-model may converge without a full exploration of the adversarial samples. For the adversarial data training of the collaboration, we propose to generate the adversarial samples that can worsen the outputs of the collaboration. In particular, for each data sample x, we output the prediction p̂(x) whose PPD value gφ(x) is the highest. We perturb x to x̃ to worsen the prediction using PGD method. Then we minimize a surrogated loss to fit this adversarial data x̃. This process corresponds to Lines 11-18. In our implementation, the natural data training of the collaboration is not necessary. Because the adversarial data training of the collaboration also captures the most adversarial samples of sub-models and can provide more other adversarial samples for the collaboration.
During the inference phase shown in Algorithm 2, once M sub-models are properly learned, CDA2 chooses a representative sub-model whose PPD value is highest among all sub-models, and then outputs this sub-model’s prediction.
3.3 ANALYSES OF CDA2
Optimizing the best-performing sub-models We firstly show that minimizing the vulnerability overlap of all sub-models is equal to minimizing the objective loss of the best-performing sub-models. For ease of optimization of the best-performing sub-model, we provide a surrogate loss.
Algorithm 1 Collaboration for Defending against Adversarial Attack (CDA2) (training phase)
Input: sub-models with duel heads {fθi} M i=1 and {gφi} M i=1, where fθi outputs the label prediction
and gφi outputs the approximated PPD, training dataset D, hyperparameter σ; 1: the natural data training of the collaboration 2: for each data (x, y) ∈ D do 3: for each sub-model fθi , i = 1, 2, ...,M do 4: Obtain the adversarial data x̃θi of the sub-model fθi using PGD method; 5: for each sub-model fθj , j = 1, 2, ...,M do 6: Calculate the approximated PPD, i.e., gφj (x̃
θi); 7: Minimize BCE loss `φ = BCE(gφj (x̃ θi), p̂y(x̃ θi)) to update module gφj ;
8: Collect sub-model i’s cross entropy (CE) loss on data x̃θi : ` ( fθj ( x̃θi ) , y ) ;
9: Calculate surrogate loss on data x̃θi : ˆ̀m = −σ ln ∑M j=1 exp
( −`(fθj (x̃
θi),y) σ
) ;
10: Update {fθi} M i=1 by minimizing ˆ̀m; // choose the best-performing sub-model to fit x̃ θi 11: the adversarial data training of the collaboration 12: for each data (x, y) ∈ D do 13: for each sub-model fθi , i = 1, 2, ...,M do 14: Calculate the approximated PPD, i.e., gφi(x); 15: Calculate the prediction i.e., fθi (x); 16: Output the prediction p̂′(x) with the highest PPD value; 17: Obtain x̃ by perturbing x to worsen the prediction; // generate the adversarial samples of the
collaboration 18: Minimize BCE loss `φ(x̃) to update the module gφ of all sub-models; 19: Update {fθi} M i=1 to fit x̃ by minimizing the surrogate loss ˆ̀m(x̃); 20: return the learned sub-models {fθi} M i=1 with {gφi} M i=1.
The vulnerability overlap of all sub-models refers to the set of adversarial data (x̃, y) that are misclassified by all sub-models, i.e., all sub-models’ objective loss is higher than a certain degree δ:
min θi∈{θ1,...,θM} `(fθi(x̃, y)) > δ, where x̃i ∈ D̃, (5)
where D̃ denotes the vulnerability overlap of all sub-models.
To reduce the vulnerability overlap of all sub-models, we only need to reduce objective loss of a single model, which is equal to minimizing the loss of the best-performing sub-model, i.e.,
min {θ1,θ2,...,θM} E(x,y)∈D
( Eθi∈{θ1,θ2,...,θM} min θj∈{θ1,θ2,...,θM} ` ( fθj (x̃ θi), y )) , (6)
where x̃θi is the adversarial data generated by the sub-model fθi .
While directly performing the outer minimization in Eq.(6) may cause a trivial solution (e.g., there is only one optimized sub-model), for ease of the optimization of Eq.(6) (Corresponding to Lines 8–9 in Algorithm 1), we provide a surrogate objective as follows.
min {θ1,θ2,...,θM} E(x,y)∈D ( Eθi∈{θ1,θ2,...,θM} ˆ̀m(x̃ θi , y) ) , (7)
where ˆ̀m(x̃θi , y) = −σ ln ∑M j=1 exp
( −`(fθj (x̃
θi ),y) σ
) and σ > 0 is a pre-defined hyper-parameter.
In Eq.(7), we approximate the objective minθj∈{θ1,θ2,...,θM} ` ( fθj (x̃ θi), y )
using a smooth surrogated maximum function due to
min θj∈{θ1,θ2,...,θM}
` ( fθj (x̃ θi), y ) − δ · ln(M) ≤ ˆ̀m(x̃θi), y) ≤ min θj∈{θ1,θ2,...,θM} ` ( fθj (x̃ θi), y ) . (8)
The proof of Eq.(8) is in Appendix.
Algorithm 2 Collaboration for Defending against Adversarial Attack (CDA2) (inference phase)
Input: the learned sub-models {fθi} M i=1 with {gφi} M i=1, test input x. 1: for all sub-models fθi , i = 1, ...,M (in parallel) do 2: make label predictions p̂(x) = fθi(x) and output approximated PPD value gφi(x); 3: return prediction p̂(x) whose PPD value gφ(x) is the highest among M sub-models.
The best-performing sub-model has the highest PPD. We show that a sub-model with the highest PPD achieves the minimum of the objective loss among all sub-models, i.e., the best-performing sub-model.
According to Bayesian theory, for a given input x in a K-class classification problem, the true probability of the corresponding label p(x) = [p1(x), ..., pK(x)] is agnostic. Usually we assume p(x) comes from a prior probability distribution, e.g., p(x) ∼ Dir(α), where Dir(α) is the Dirichlet distribution with the pre-defined parameter vectorα = [α1, α2, ..., αK ]. The prior probability density function of Dir(α) is
f(p(x),α) = 1
B(α) K∏ i=1 pi(x) αi−1 where B(α) = ∏K i=1 Γ (αi) Γ (α0) , α0 = K∑ i=1 αi, (9)
where Γ(·) denotes the Gamma function. According to the Bayesian views, when we learn the model in a supervised manner, the label y happens n times after n observations given the data (x, y) ∈ D. Therefore, the posterior probability distribution is Dir(α′) where α′ = [α1, ..., αy + n, ..., αK ], and the posterior probability density (PPD) function is
PPD(p(x),α′) = 1
B(α′) K∏ i=1,i6=y p(x)αi−1 · py(x)αy+n−1, (10)
where PPD corresponds to the likelihood of a given p̂(x) being the probability of the true label p(x). In other words, the predictions p̂(x) from the sub-model fθi with a higher PPD is more likely to be the true label probability. Therefore, we propose to use PPD to measure the quality of predictions. In Eq.(10), when n (i.e., the number of posterior observations) is sufficiently large, py(x)αy+n−1 becomes a dominant part; therefore, the value of PPD monotonically increases w.r.t. the value of py(x). Consequently, we have the following proposition. Proposition 2. Given an input x, the sub-model that has the highest PPD corresponds to the best-performing sub-model, i.e.,
arg max j∈{1,2,...,M}
PPD(fθj (x),α ′) = arg min
j∈{1,2,...M} `(fθj (x), y). (11)
From Proposition 2, the PPD value can decide the best-performing sub-model. Given an input to the collaboration, our dual-head structured sub-models can collaboratively decide the best-performing sub-model by just comparing the values of the PPD head (corresponds to Algorithm 2). To learn the PPD head, we could simply compare its output with the predicted probability on the true label (i.e.,p̂y(x)), then update the PPD head by gradient descent (corresponds to Lines 6–7 in Algorithm 1).
Note that the PPD head may be susceptible to adversarial attacks in the white-box setting. In our implementation, we use a simple linear structure to regress the PPD value.

heading: 4 EXPERIMENTS
text: In this section, we provide a synthetic experiment to illustrate the behavior of our method compared with prior ensemble methods. Then, we provide a series of experiments on a benchmark dataset to verify the effectiveness of our method in defending against adversarial attacks.

heading: 4.1 EXPERIMENTAL SETUP
text: Following the work in (Yang et al., 2020a), we compare our method with various related methods, including ADP (Pang et al., 2019), GAL (Kariyappa & Qureshi, 2019), DVERGE (Yang et al., 2020a). We use ResNet-20 (He et al., 2016) as sub-models in all methods for fair comparisons, and we use CIFAR10 as the data set, a classical image dataset (Krizhevsky et al., 2009) that has 50,000 training images and 10,000 test images.

heading: 4.2 SYNTHETIC DATA EXPERIMENTS
text: To demonstrate the behavior of collaboration, we apply our collaboration framework on the wellknown XOR problem as an example.
In XOR problem, there is a binary training set D = {xi, yi)}ni=1, where yi ∈ {±1}. For the samples with the label yi = 1, the input feature x is independently sampled from the two Gaussian distributions x ∈ N (µ = [1, 1],σ = 0.1 · I2) and x ∈ N (µ = [−1,−1],σ = 0.1 · I2), where N (µ = [1, 1],σ = 0.1 · I2) denotes 2-d Gaussian distribution with the mean vector µ = [1, 1] and covariance matrix σ = 0.1 · I2. For the samples with the label yi = −1, we sample the input feature independently xi ∈ N (µ = [−1, 1],σ = 0.1 · I2) and xi ∈ N (µ = [1,−1],σ = 0.1 · I2). Suppose there are two linear sub-models f1(x) = a1 · x+ b1 and f2(x) = a2 · x+ b2. Ensemble methods output a prediction p̂(x) by a voting-based strategy e.g., averaging the predictions as the output, i.e., p̂(x) = 1M · ∑M i=1 fθi(x).
Collaboration aims to 1). minimize the mean square error (MSE) of the best-performing sub-models; 2). optimize the module g1 and g2 to measure the quality of the multiple predictions during inference. From Figure 4(a), learning multiple linear sub-models and averaging the predictions (ensemble)
is still a linear model, so it cannot tackle XOR problem. Collaboration can address XOR which classifies each sample by learning to specify the sub-tasks to different sub-models. From Figure 4(b), the samples are assigned to two sub-models, in which the blue samples are assigned to f1 and the red samples are assigned to f2. Finally, it output the prediction by identifying the best-performing sub-models.

heading: 4.3 PERFORMANCE ON WHITE-BOX ATTACK
text: As there are mainly two threat modes in the adversarial attack setting: white-box attack and black-box attack. White-box attack refers to that attackers know all the information about the models, including training data, model architectures, and parameters, while black-box attackers have no access to the information about the model’s structures and parameters and rely on surrogate models to generate transferable adversarial examples.
We compare our method with baselines (GAL (Kariyappa & Qureshi, 2019), DVERGE (Yang et al., 2020a) and ADP (Pang et al., 2019)) on defending white-box attacks using a subset of CIFAR10. Following in the setting in Yang et al. (2020a), we use 50-step PGD with five random starts and the step size of /5 to attack all methods. In particular, we randomly select 1000 samples under different . For the PGD attack, we select the cross-entropy loss to update the perturbations to search for adversarial samples. In addition to the accuracy under different , we also report the performance of all methods on clean data with the adversarial training in which = 0.03.
From Table 1, CDA2 achieves a better robustness performance under white-box attack. The results verify that collaboration significantly improves the utilization of the limited model capacity. Therefore, CDA2 can fit more adversarial data and has a relatively smaller vulnerable area.

heading: 4.4 PERFORMANCE ON BLACK-BOX ATTACK
text: Due to the transferability of adversarial examples, transfer adversaries can craft adversarial examples based on surrogate models and perform an attack on the target model. In our experiments, we follow the transfer attack setting in (Yang et al., 2020a) and select 1000 test samples randomly. We select 1000 test samples randomly and use hold-out baseline ensembles with three ResNet-20 sub-models as the surrogate models to generate adversarial samples. In particular, we use three attack methodologies: PGD with momentum (Dong et al., 2018), SGM (Wu et al., 2020a) which adds weight to the gradient through the skip connections of the model, and M-FGSM (Xie et al., 2019) which randomly augments the input images in each step. For each sample, three adversarial variants are using the three attack methods. Only when the model can classify all kinds of adversarial variants can the model successfully defend against adversarial attacks. We show the results of all methods in Table 2. In our experiments, GAL is hard to optimize in adversarial training. From Table 2,
we show the transfer attack robustness of ensemble methods across a wide range of attack radius 0.01 ≤ ≤ 0.07. When 0.01 ≤ ≤ 0.04, CDA2 achieves a comparable performance compared with the SOTA method. With the increase of , the volume of -ball increases exponentially, the performances of all methods get worse significantly because of insufficient model capacity. Since CDA2 addresses more adversarial data using a collaboration mechanism, it achieves a relatively better robustness performance as 0.05 ≤ ≤ 0.07. In addition to transfer attack, query attack can also craft adversarial samples based on the predicted scores of the model. To evaluate the robustness of of CDA2 under query attack, we use Square Attack method (Andriushchenko et al., 2020) to attack all method. Square Attack selects localized square-shaped updates at random positions in each step (Andriushchenko et al., 2020). In particular, we set = 0.03 and learn the models for each method with adversarial training. Then we evaluate the robustness of each method under Square Attack with 5000 iterations and the results are presented in Table 3. From Table 3, CDA2 outperforms the baselines under query attack. CDA2 optimizes the
utilization of the model capacity by specifying each sub-models to handle the "specific" adversarial attacks, which defends against query attacks more efficiently.

heading: 5 CONCLUSION
text: In this paper, we study an essential question in the field of adversarial attacks that when we should collaborate. (i) If a single model can handle everything, there is no need for multiple models. (ii) If a single model can only handle a part of the whole, collaboration among multiple models makes sense. Adversarial defense is a typical task that falls into the circumstance (ii) because a single model hardly fits adversarial data. We provided a collaboration framework—CDA2—as the defense strategy over ensemble methods, and empirical experiments indeed verified the efficacy of CDA2. Future work includes applying our collaboration framework to other areas such as kernel methods, fairness, and global federated model, etc.
ETHICS STATEMENT
In this work, we investigate the robustness issue in deep neural networks (DNNs). We propose a collaboration mechanism for reducing the vulnerability of existing DNNs to enhance their trustworthiness. Having reading the codes of ethics, we make sure that our work conforms to them. In our experiments, we use a public data set which complies with requirements.
REPRODUCIBILITY STATEMENT
Our instructions and experimental settings are illustrated in the maintext. For the theoretical results, we provide rigorous analysis and detailed proofs in maintext and the appendix. For more experimental results and the discussions about CDA2 , please refer to the Appendix. The source code of our method is attached in Appendix.

heading: A THEORETICAL PROOF
text: The proof of Eq.(8) is as follows.
Proof. Since we have
max j exp
( −` ( fθj (x̃ θi), y )
σ
) ≤
M∑ j=1 exp
( −` ( fθj (x̃ θi), y )
σ
) ≤M ·max
j exp
( −` ( fθj (x̃ θi), y )
σ
) ,
(12) considering that −δ ln(x) monotonically decreases w.r.t x, we have
−δ ln ( M ·max
j exp
( −` ( fθj (x̃ θi), y )
σ
)) ≤ ˆ̀m(fθj (x̃θi , y)) ≤ −δ ln max
j exp
( −` ( fθj (x̃ θi), y )
σ
) ,
(13) where −δ ln ( M ·maxj exp ( −`(fθj (x̃ θi ),y) σ )) = minθj∈{θ1,θ2,...,θM} ` ( fθj (x̃ θi), y ) − δ · ln(M)
and −δ ln exp ( maxj −`(fθj (x̃ θi ),y) σ ) = minθj∈{θ1,θ2,...,θM} ` ( fθj (x̃ θi), y ) , so Eq.(8) holds.
The proof of Proposition 2 is as follows.
Proof. The objective loss depends on the label probability (e.g., cross-entropy loss ` = − ln py(x)), and the loss monotonically decreases with respect to py(x), so the best-performing sub-model has the highest label probability. Combined the fact that PPD(p(x)) monotonically increases w.r.t. py(x), the best-performing sub-model corresponds to the highest PPD; therefore, Eq.(11) holds.
B IMPLEMENTATION DETAILS
the design of the PPD head In our experiments, we use ResNet20 as our backbone model. For the additional PPD head, we propose to use a one-layer MLP to model the relationship between the prediction and its confidence. Our design of the PPD head is based on the following aspects: 1). convenient optimization; a simple model structure can be optimized easily and will not bring a significant computation cost ; 2). robustness; a complex PPD head may be vulnerable to the adversarial attack. Therefore, in our implementation, we choose to learn a simple PPD head for each sub-model to improve its robustness.
Training devices We conduct all experiments on the device GeForce RTX 2080Ti.

heading: C TRAINING COST ANALYSIS
text: Training cost is a notable issue. We compare the training cost of all methods from the two aspects; 1). parameters and GFLOPs: all methods have the same model architecture (ResNet20), so all methods have a similar number of parameters and GFLOps. Compared with baselines, our method has an additional head (PPD head), which is a one-layer MLP with 128 parameters and has a negligible computation cost; 2). training manner; all methods except DVERGE achieve adversarial training by generating adversarial samples using PGD attack. The time consumption of all methods using the device Geforce 2080Ti (100 epochs) is in Table 4.
DVERGE distills non-robust features by computing transferable adversarial samples, which have a O(N2) time complexity in which N is the number of sub-models, so it has a relatively large time consumption. Our method outperforms baselines by training an additional PPD head and it could cause an additional small time consumption as shown in the above table.

heading: D MORE EXPERIMENTAL RESULTS
text: For the transfer attack, we also use a more challenging setting following the work in (Yang et al., 2020a). For the attack methods, we use PGD with momentum (Dong et al., 2018) with three random starts, M-FGSM (Xie et al., 2019) and SGM (Wu et al., 2020a). We use hold-out baseline models with 3, 5, and 8 ResNet-20 sub-models as the surrogate models. Meanwhile, we generate adversarial samples with cross-entropy loss and CW loss (Carlini & Wagner, 2017b). For each sample, we generate 30 adversarial variants, and only if the model classifies all the 30 variants can the model defend the transfer attack successfully. The results are shown in the following Table. From Table 5, CDA2 outperforms baselines as 0.01 ≤ .

heading: E MORE DISCUSSION ABOUT THE COLLABORATION
text: E.1 MODEL CAPACITY AND COLLABORATION
A model with sufficient capacity to cover all cases does not need to collaborate with others. To verify this claim, we conduct experiments using the ResNet model with different depths and show the clean accuracy (%) of single/multiple models in the following table.
From the Table 6, with the depth 2, the model has the insufficient model capacity to learn the feature extractor, collaboration can have a relatively large improvement (2.0). As the depth of the model is 20, the model has sufficient model capacity to fit all data samples. Collaboration achieves a slight improvement compared with a single model (0.6).
Compared with standard training, adversarial data are adaptively changed based on the current model to smooth the natural data’s local neighborhoods. The volume of these surroundings is exponentially large. The model often encounters insufficient model capacity especially when there is a relatively large ball. Therefore, it is urgent to improve the utilization of the capacity for adversarial training.
E.2 COLLABORATION WITH S SINGLE BIG MODEL
advantages of the collaboration Compared with collaboration, a single big model may be difficult to fit the adversarial data. For a big single model with a deeper structure, it may face gradient vanish or gradient explosion during optimization. Without a well-designed optimization method, it could be more vulnerable to an imperceptible perturbation compared to a simple model. The collaboration alleviates this problem by learning multiple relatively small models.
disadvantages of the collaboration A single big model may be a direct solution for addressing insufficient capacity in adversarial training. Compared with learning a single big model, the collaboration needs to design a complicated mechanism to improve the utilization of the model capacity.
E.3 COLLABORATION IN ADVERSARIAL TRAINING CAN AVOID A TRIVIAL OPTIMIZATION
As our proposed collaboration mechanism enhances the performance of the best-performing submodel. One may wonder whether it obtains a trivial case, e.g., only one sub-model is properly trained. In fact, our collaboration will not bring such a trivial case. Fitting the adversarial data consumes a tremendous model capacity so the model usually cannot fit all adversarial samples. For a learned sub-model, there still exist adversarial samples that cannot handle. From our proposed collaboration mechanism, these adversarial samples generated from a learned sub-model are more likely to be assigned to other sub-models which perform better. Therefore, in our collaboration, all sub-models will be trained.
To experimentally verify this claim, we present the accuracies on clean data of all three sub-models with adversarial training ( = 0.02) in the Table 7. From the Table 7, all sub-models have a similar performance on the clean data.

Review summaries:
This paper firstly analyzes prior adversarial defense methods using ensemble strategy and claims that this method could cause a waste of model capacity. To improve the utilization of (multiple ) model capacity, the author proposes an interesting collaboration strategy $CDA^2$ to defend against adversarial attacks. Specifically, the author develop a dual-head model structure: one is for making a prediction and the other is for predicting the posterior probability of the input. During training, each model can address the adversarial attacks of other sub-models so that it improves the robustness of the collaboration. The experimental results partially verify the superiority of the proposed methods.

This paper presents a new paradigm for defending against adversarial attacks with multiple sub-models. Different from ensemble, the proposed collaboration paradigm, a representative sub-model is chosen to make the decision, instead of letting all sub-models vote. The proposed method has been validated on CIFAR-10 dataset, against both white-box and transferrability-based black-box attacks.

This paper proposes an ensemble or mixture-of-experts method to defend against adversarial examples. Though the authors prefer to use the term collaboration method to highlight its difference from vanilla ensemble. The main idea is that, during adversarial training, the sub-models are trained on each other's adversarial examples.  Specifically: * for each training image, one adversarial example is generated per sub-model by carrying out an attack on each sub-model. * each adversarial example is (softly) assigned to the sub-model that has the lowest loss on it as a training image. * each sub-model has a second output called PPD that quantifies its confidence. * at inference time, for each input, the sub-model with the highest PPD produces the output.  The rationale is that, because each sub-model only needs to cover part of the adversarial example space, they can do a better job. Experiments on CIFAR-10 with L_inf attacks are reported.