Title: THEORETICAL AND EMPIRICAL STUDY OF ADVERSAR-IAL EXAMPLES

Abstract: Many techniques are developed to defend against adversarial examples at scale. So far, the most successful defenses generate adversarial examples during each training step and add them to the training data. Yet, this brings significant computational overhead. In this paper, we investigate defenses against adversarial attacks. First, we propose feature smoothing, a simple data augmentation method with little computational overhead. Essentially, feature smoothing trains a neural network on virtual training data as an interpolation of features from a pair of samples, with the new label remaining the same as the dominant data point. The intuition behind feature smoothing is to generate virtual data points as close as adversarial examples, and to avoid the computational burden of generating data during training. Our experiments on MNIST and CIFAR10 datasets explore different combinations of known regularization and data augmentation methods and show that feature smoothing with logit squeezing performs best for both adversarial and clean accuracy. Second, we propose an unified framework to understand the connections and differences among different efficient methods by analyzing the biases and variances of decision boundary. We show that under some symmetrical assumptions, label smoothing, logit squeezing, weight decay, mix up and feature smoothing all produce an unbiased estimation of the decision boundary with smaller estimated variance. All of those methods except weight decay are also stable when the assumptions no longer hold.

1. INTRODUCTION:

Machine learning models are often vulnerable to adversarial examples, which are maliciously designed to cause misclassification. In the area of computer vision, for instance, object recognition classifiers are much more likely to incorrectly classify images that have been modified with small, often inpreceptible perturbations. Similar problems also occur in natural language processing area, see (Miyato et al., 2017) , where small perturbations of text can easily fool a label classification model. It is therefore important to develop machine learning models that are resistant to adversarial examples in situations where attacker may attemp to interfere, for example with autonomous vehicles (Papernot et al., 2017) . Understanding the design mechanisms of adversarial examples can also help researchers to gain a better understanding of the performance of machine learning, especially deep learning models. In this paper, we introduce an efficient feature smoothing method to improve the adversarial robustness of neural networks and also build a theoretical framework to understand how different approaches help with the adversarial accuracy.

Different adversarial training methods have been proposed to increase robustness by augmenting training data with adversarial examples. Goodfellow et al. (2015) developed the fast gradient signed method (FGSM), which efficiently generated adversarial example by a "single-step" attack based on a linearization of the model's loss. Their trained model is robust to single-step perturbations but remains vulnerable to more costly "multi-step" attacks. Madry et al. (2017) extended FGSM by proposing a multi-step variant FGSM, which is essentially projected gradient descent(PGD). They suggested that adversarial training with the PGD attack is a universal first order adversary defense, which means that models trained against PGD attacks are also resistant against many other first order attacks. Their PGD attacks consists of initializing the search for an adversarial examples at a random point within the allowed norm ball, then running several iterations of the basic iterative method to find an adversarial examples. Kannan et al. (2018) then introduced a logit pairing method (ALP) which encourages the logits for pairs of examples and their corresponding adversarial examples to be similar. Logit pairing improves accuracy on adversarial examples over trainings based on PGD.

The above successful approaches performed data augmentation by generating adversarial examples during each training step, which will unfortunately bring significant computational burden to the training process. In contrast, more "efficient" training methods without hindering the training speed have also been shown to improve adversarial robustness (In this paper we refer "efficient" methods as data augmentation and regularization methods including mixup, label smoothing, logit squeezing, weight decay, and our proposed feature smoothing). Szegedy et al. (2016) proposed label smoothing, which trains a classifier using soft targets for the cross-entropy loss rather than the hard targets. The correct class is given a target probability of 1 -α and the remaining α probability mass is divided uniformly among incorrect classes. Label smoothing reduces overfitting by preventing a network from assigning full probability to each training data, and also offers a small amount of robustness to adversarial examples (Kannan et al., 2018) . Kannan et al. (2018) proposed a logit squeezing method which penalizes the logit of each input example. It is showed that combined with adding Gaussian noise into input examples, logit squeezing gave even better results than ALP in some datasets, for example MNIST and SHNV. Zhang et al. (2018) (Szegedy et al., 2014; Pei et al., 2017) , decision boundary too close to the training data (Tanay & Griffin, 2016) . However, there are few papers theoretically explaining the similarities and differences between those methods, especially based on their estimation of decision boundaries. Goodfellow et al. (2015) discussed the differences between weight decay and adversarial training by comparing their loss functions in logistic regression, but didn't show how these two methods affect the estimation and accuracy.

The above discussion leaves us two questions:

• Without adding any computational burden during training, these "efficient" methods mainly benefit from data augmentation and regularization, and as a result, resist against adversarial examples to some extent. As most of them are not specifically designed for resisting against adversarial examples, can we develop an "efficient" approach specifically designed to be robust to adversarial examples? • What are the connections and differences among these "efficient" methods? Can we build a unified framework to analyze them?

Motivated by these two questions, we investigate defenses against adversarial attacks, and the contribution is two-fold. We first propose feature smoothing, a data interpolation method that softens the features of input. We show that feature smoothing obtains better performance than other "efficient" approaches on both MNIST and CIFAR10. We also observe the best performance when combining our feature smoothing method and logit squeezing strategy, among all "efficient" methods.

We also propose a unified framework to understand how different "efficient" approaches influence the estimation of decision boundary. In particular, based on both simulations and theoretical analysis of logistic regression, we show that under some symmetrical assumption, label smoothing, logit squeezing, weight decay, mixup, feature smoothing and data extrapolation all give an unbiased estimation of boundary with smaller estimation variance. But regularization with weight decay is more sensitive when the assumption may not hold. We believe it is the reason weight decay can hurt the accuracy in clean test data. Our framework are also partially extended to deep convolutional neural networks.

The paper is organized as follows. Section 2 presents our proposed method and other related "efficient" methods. Section 3 reports the performance of feature smoothing against other "efficient" methods.

We conduct theoretical analysis and explore the connections and differences among different methods in Section 4. The last section concludes. Based on this idea, if we are able to generate 'low density' data directly based on the original training set, we can also improve the estimation as what adversarial training does but with much smaller computational cost. We now introduce feature smoothing, a simple data augmentation approach which generates new virtual training data as interpolation of features from a pair of random samples.

Virtual training data are constructed as follows:

x(i) = (1 -α)x (i) + αx (j) , ỹ(i) = y (i) ,

where (x (i) , y (i) ) and (x (j) , y (j) ) are two examples drawn randomly from our training data, and 0 ≤ α < 0.5. When x (i) and x (j) belong to different classes, and the interval between these two data points intercept with the decision boundary only once, xi is closer to the boundary than x i or x j . Figure 1 (c) shows that adding new data interpolated between classes can help with the estimation of decision boundary.

Furthermore, Gaussian noise also helps extend the range of x. Figure 1 (d) shows that adding Gaussian random noise with proper variance into input can also push the estimated boundary closer to the true boundary compared against original clean data. Hence we add Gaussian noise into our feature smoothing method as well:

x(i) = P (x (i) + ), ỹ(i) = y (i) ,

where ∼ N ormal(0, σ 2 ) and P (x) projects x to the range of original data. To distinguish data interpolation part and Gaussian noise part, we use 'feature smoothing' only referring to data interpolation and 'noise' for the Gaussian noise part in the following. A detailed illustration of how feature smoothing helps the estimation of boundary is discussed in Section 4.

2.2. RELATED METHODS:

Though starting from different intuitions, feature smoothing turns to be very similar with mixup (Zhang et al., 2018) . In mixup, additional virtual data points are generated by interpolating both features and labels of the original training data:

x = (1 -α)x (i) + αx (j) , ỹ = (1 -α)y (i) + αy (j) ,

where α ∈ [0, 1]. Mixup can be understood as a form of data augmentation that encourages the model to behave linearly in-between training examples. Zhang et al. (2018) argued that this linear behavior reduces the amount of undesirable oscillations when predicting data outside the training examples. On the contrary, our feature smoothing method includes the interpolations with new label remaining the same as the dominant data point, which maintains the S-shaped curve of logistic model and also allow feature smoothing easier to be combined with regularization methods. More detailed comparison can be found in Sec. 4.

Label smoothing (LaS) and logit squezzing (LoS) are other two efficient ways which improve the adversarial accuracy. Let y ∈ R K be one-hot label for K classes, label smoothing (Szegedy et al., 2016) softens the target by replacing y with

ỹ = δ K -1 (1 -y) + (1 -δ)y,

where δ = 0.1 is shown to be the best choice (Pereyra et al., 2017) . Assume we train a model with parameters θ on a batch of m data points {(x (i) , y (i) ), i = 1, 2, . . . , m}, y (i) ∈ {0, 1} K . Let f (x; θ) denote the mapping function from x to logits of the model. Let L (clean) denote the cross entropy loss for the batch of data points as:

L (clean) = - 1 m m i=1 K j=1 y (i) j log(p θ (y (i) j |x (i) )).

The loss function of label smoothing can also be achieved by some calculation:

L LaS = - 1 m m i=1 K j=1 ỹ(i) j log(p θ (y (i) j |x (i) )) = L (clean) - 1 m m i=1 K j=1 1 -Ky (i) j K -1 δf j (x (i) , θ).

Notice that if we assume our model obtains a good estimation of f (x, θ), then when y j = 0, f j (x, θ) < 0 and when y = 1, f j (x, θ) > 0. In a binary classification case, L LaS can be written as L (clean) + δ|f (x, θ)|, which further indicates that label smoothing predicts logits with smaller magnitude and therefore avoids overfitting.

Similarly, logit squeezing (Kannan et al., 2018) applies a L 2 norm on the logits directly as a penalty of over-confidence:

L LoS = L clean + λ m m i=1 ||f (x (i) )|| 2 ,

where L clean is the original loss of neural networks and f (x (i) ) is the logit of image x (i) as above.

Weight decay is another well known regularizer which efficiently reduces overfitting of neural networks by adding L 1 or L 2 penalty of weight w, L wd = L (clean) + λ||w|| 2 2 . However, weight decay is shown to be not very helpful for adversarial examples compared to label smoothing and logit squeezing, which will be discussed in Sec 4.

Combination of different approaches:

In feature smoothing and mixup, we generate new data points as linear combination of x i and x j . For generating the exact y value of these virtual points, mixup uses a linear interpolation for estimation, while feature smoothing chooses the dominant label. Nevertheless, it is also possible that feature smoothing or mixup also adds mislabeled noises into the training data, especially when x i and x j are not symmetric to the boundary. In that situation, label smoothing and logit squeezing are better ways to avoid overfitting. So we also consider to combine these methods together to gain a better test and adversarial accuracy.

3. EXPERIMENTS:



3.1. RESULTS ON MNIST:

We experiment feature smoothing, label smoothing, mix up, logit squeezing and their possible combinations on MNIST, with the results summarized in Table 1 . We find that combining feature smoothing and logit squeezing give the best performance in both clean test data and adversarial examples. For all experiments in this section, we train our models for 200 epochs and use Adam for our optimizer with a learning rate at 10 -4 . Random noises are added into the training data in several methods, with the same σ value of 0.5.

For MNIST, when α ranges between 0.2 and 0.4 we observe similar performance for feature smoothing, whereas for large α at 0.5, too much noise in data label brings underfitting for feature smoothing. We use a final α value of 0.3 for reporting results in Table 1 . Chosen by cross validation, we use α ∈ Beta(8, 8) for mixup, and δ = 0.1 for label smoothing. In logit squeezing, we use the weight λ of 0.2 as experimented in Kannan et al. (2018) . In feature smoothing and mixup, 10 new data points are generated on each batch with batch size m = 50.

We use the LeNet model as Madry et al. (2017) and also apply the same attack parameters as they provided. After scaling the range of images pixels into [0, 1] (divided by 255), we apply perturbation per step of 0.01, 40 total attack steps with 1 random start, and the total adversarial perturbation threshold set as 0.3. Similar with Madry et al. (2017) , we also generate black box examples for MNIST by independently initializing and training a copy of the LeNet model, then generate PGD attack based on that model. Both cross entropy loss and correct-wrong loss are used.

Each single method improves a small amount of the adversarial accuracy, but combinations of them lead to a much better performance (Table 1 ). Logit squeezing combined with feature smoothing and Gaussian random noise achieves the best performance among all those "efficient" methods. 

3.2. RESULTS ON CIFAR10:

We follow Madry's lab for the experiments in CIFAR10. For all experiments in this section, we train our models for 80000 global steps with batch size m = 128 in each step. We use Momentum at 0.9 for our optimizer with a learning rate at 0.1 for the begining, 0.01 after 40000 global steps and 0.004 after 60000 global steps. Weight decay with λ = 0.0002 is also applied to all experiments. We use α ∈ Beta(8, 8) for mixup, δ = 0.1 for label smoothing, λ = 0.1 for logit squeezing , α = 0.2 for feature smoothing, and σ = 0.5 for Gaussian random noise. In feature smoothing and mixup, 10 new data points are randomly generated on each batch with batch size m = 128.

We apply the ResNet model and the same attack parameters as they used. We use perturbation per step of 2.0, 20 total attack steps with 1 random start and the total adversarial perturbation threshold set as 8. 

4. THEORETICAL EXPLANATIONS:

In this section, we show that the above "efficient" methods increase neural networks' adversarial robustness by improving the estimation of the decision boundaries. The improvement relies on two components: (1) unbiased estimation of boundary;

(2) smaller estimation of variance. Given the same training data, these methods estimate the boundary closer to the true boundary than the original neural networks. Our simulations and theoretical results mainly focus on logistic regression. The idea is then discussed with deep convolutional neural networks.

To gain some intuitions on how the above methods improve the estimation, we start from logistic regression model with binary classes. Assume a feature vector x follows some distribution P x in R d , w ∈ R d and b ∈ R, then the corresponding label y follows a Bernoulli distribution with probabilities given by: wx+b) , P (y = 0) = 1 1 + e wx+b .

p := P (y = 1) = 1 1 + e -(

(1)

Based on the changing the loss function, we divide these methods into two categories: (1) regularization methods: label smoothing (LaS), logit squeezing (LoS), and weight decay (wd); (2) augmentation methods: mixup and feature smoothing. Regularization methods add penalty term to loss function directly, while augmentation methods modify the loss function by adding new virtual data into it. We analyze the properties of these methods based on the two categories in the following subsections and the proofs of the theorems are included in Appendix.

4.1. REGULARIZATION METHODS:

Our main theorem shows that all of the regularization methods estimate the decision boundary with smaller variance and the estimation is unbiased when x is symmetric with the boundary. With one-dimensional x and binary classes, the variance of decision boundary can be defined as: var( b w ), w = 0. Let p denote the estimated probability. The confidence interval of p, which indicates the confidence interval of boundary, is narrowed down with the regularization methods, especially when the support of the distribution of x is far away from boundary (Figure 2(a) ). As the value of w increases, the corresponding variances for w, b and decision boundary are also better controlled with regularization methods than the vanilla logistic regression (Figure . 3). For the vanilla logistic regression, when w is large enough, the variance of boundary grows in an exponential rate with w. But with these regularization methods, variance keeps decreasing even when w is really large. This observation is also true with higher dimensions and multiple classes (Figure 5 ). Inspired by our observation in the simulation study, we prove the following theorems in one dimension (1-D) to further explain the phenomena in the simulations. The symmetric assumption is not unrealistic for imaging classification problems, since we can always assume the true boundary is in the middle of two classes. However, the stability of those methods when this assumption cannot hold is also important. We also show that label smoothing and logit squeezing is relatively more stable than weight decay when x is asymmetric (Figure 7 ). Following the above explanation, our theorem 4.3 shows that adding data around boundary with labels generated from the true distribution into training can narrow down the variance of boundary, even though the sample size remains the same. Adversarial training, mixup and feature smoothing estimated the labels in different ways. We further show that feature smoothing achieve smaller variance than mixup when α is properly chosen (sec A.4). 

4.3. EXTENSION TO NEURAL NETWORKS:

In more complex models like convolution neural network (CNN), the model can be divided into two parts: hidden layers which transform the input data x → f (x) and the classification model which applies the softmax function (or sigmoid function for binary classification) on f (x). Our results can be extended to CNN for regularization methods since softmax function is just a multi-classes logistic regression. For augmentation methods, we also believe that an interpolation of input data implies an interpolation of transformed data after hidden layers. For simplicity, we assume the nonlinear layers in the CNN only consist of ReLU and max-pooling. Obviously, both ReLU and max-pooling satisfy the following properties: let x = αx (i) + (1 -α)x (j) , then

EQUATION

where max-p represents max-pooling. The first inequality in (2) holds when each argument of x (i) , x (j) is non-negative. Given the pooling layer after the ReLU layer, the assumption is valid. It further implies that

f (x) ≤ αf (x (i) ) + (1 -α)f (x (j) ),

which means augmentation methods on the data can be considered as augmentation on the logits.

Then we may use our framework on the logistic regression.

5. DISCUSSION:

We have proposed feature smoothing, a straightforward data augmentation method as an efficient way to increase adversarial robustness of neural networks. In our experiments, feature smoothing combined with logit squeezing shows the best performance in both MNIST and CIFAR10. We found that α ∈ [0.2, 0.4] shows similar results when we apply PGD attack with total perturbation threshold as e = 0.3. If we use smaller perturbations, smaller α, for example α = 0.1 for e = 0.1, we also observe good results. As a future plan, more possibilities of combinations of different techniques can still be further explored.

We also built a framework to explain how different regularization methods and augmentation methods improve the estimation of decision boundaries for logistic regression. Our main theorems show that all of these methods achieve smaller estimation variance of the decision boundary while keeping the unbiasedness of the estimation. In some extreme cases, for example, correctly labeled data around boundary for one specific class (7), the vanilla logistic regression is incorrectly estimated the boundary for sure but all of the above methods resolve the problem. We also extend the analysis to neural networks based on two facts: (1) the softmax regression is a generalized form of logistic regression in multi-class classification problem; (2)the activation functions like Relu and max-pooling can both keep linear inequalities Eq. (2).

A.2 PROOF OF THEOREM 4.2

When {x} is symmetric to the boundary x = 0, the data set can then be splitted into two groups, {x p } containing positive values and {x n } containing negative values, which are symmetric to each other. We further assume the corresponding labels are also approximately symmetric, which is easy to achieve when the sample size is large enough. The loss function is then automatically divided into: L p = L({x p }) and L n = L({x n }). The minimizer ŵ and b of L p and L n have to be the same since the input data are symmetric. Then ŵ and b are also the minimizers of the whole loss function L = L p + L n . That means for any x in positive part and its corresponding image x in negative part, the estimated p θ (y|x) + p θ (y |x ) is equal one, which indicates to an unbiased estimation of the boundary.

A.3 PROOF OF THEOREM 4.3

As mentioned in section 2, new data points

x = αx (i) + (1 -α)x (j) ,

are added as input in both methods, while the corresponding ỹ are estimated by either linear interpolation or dominant point of x (i) and x (j) . Let p denote the true probability of x, we claim that adding x and ỹ ∼ Bernoulli(p) into input decreases the variance of boundary.

Same as above, the variance of boundary can be estimated as

var(- b ŵ ) = 1 w 2 p(1 -p) ,

where x is generated from a distribution P x . When ỹ are generated from the true probability, the MLE estimation do not change. On the other hand, if x (i) and x (j) are from different classes and symmetric to the decision boundary, x is closer to the boundary than x (i) and x (j) , and therefore p(1 -p) > p x (i) (1 -p x (i) ).

A.4 REMARK OF THEOREM 4.3

As mentioned above, given x from a delta mass distribution, the first derivative of variance with repect to w is given by: g (w) = -2w -w 2 x(1 -2p) w 4 p(1 -p) .

Without loss of generality, we still assume x > 0, b = 0, w > 0, then g (w) = 0 gives w = 2

x(1-2 p) . g (w) < 0 if w < 2 x(1-2 p) while g (w) > 0 otherwise. x close to the boundary (0 when b = 0) also leads to probability p close to 0.5, i.e. (1 -2) close to 0. Therefore in a large range of w ∈ (0, 2 x(1-2 p) ), the variance is decreasing with w. As a results, feature smoothing gives even smaller variance than the original MLE with x.

B SIMULATION RESULTS:



B.1 LOGISTIC REGRESSION WITH HIGH-DIMENSIONAL FEATURE AND MULTIPLE CLASSES B.2 IN-SYMMETRIC FEATURE:

However, in real world, we can never have the perfect scenario that {x} is strictly symmetric distributed with respect to the boundary. We further argue that regularizers based on wx + b including label smoothing and logit squeezing are more tolerant to unbalanced data than weight decay regularized on w only. Figure 6 shows how these four methods perform when {x} are not symmetrically generated, in two scenarios: (a) data size is unbalanced with respect to decision boundary; (b) data distribution is unbalanced with respect to boundary. It is easy to see that label smoothing and logit squeezing are less sensitive to the distribution of x in both scenarios. In contrast, vanilla logistic regression and weight decay are more sensitive. Confidence intervals for vanilla x ∈ [-1, -0.9] ∪ [0.9, 1] with size 300. y = 1 ← p > 0.5. Then 10 another input x = 0.1, y = 1 or 10 input of x = -0.1, y = 0 are added randomly with probability 0.5 to the training.

:



Figures and Tables:

performed data augmentation by training the model on virtual input points as interpolation of two random examples from the training set and their labels, resulting in increasing both the robustness of adversarial examples and the accuracy in clean test data. In parallel, many theorems have also been proposed to understand the power and existence of adversarial examples. Transferability is shown to be a common property of adversarial examples. Szegedy et al. (2014) and Papernot et al. (2016) found that adversarial examples generated based on a specific neural network can fool both the same neural network trained with different datasets and different neural networks trained with the same dataset. The existence of adversarial examples is still an open question. Possible reasons have been suggested in recent papers, such as low density

Figure 1: Toy examples of binary classification problem with blue circles and red rhombus representing two classes of data. The blue dash lines and solid lines show true boundary and estimated boundary respectively. (a) The original data; (b) Adversarial examples added into training data; (c) Virtual data interpolated between classes added in training data; (d) Gaussian random noise added into input.

Theorem 4.1. Label smoothing, logit squeezing, and weight decay all estimate the decision boundary with smaller variance in logistic regression model in 1-D.

When x is symmetric with respect to boundary, label smoothing, logit squeezing, and weight decay have unbiased estimation of boundary in logistic regression model in 1-D.

Figure 2: Mean estimated probabilities (solid lines) with 99% confidence intervals (dash lines) obtained from 1000 realizations for regularization methods, with N = 300 data points sampled from (a) U nif orm([-1, -0.9] ∪ [0.9, 1]), w = 4, b = 0. (b) U nif orm([0, 1]), w = 8, b = 4.

Adding data around boundary narrow down the variance of boundary estimation by making the distribution of x closer to boundary. The estimation is unbiased if all labels for the new data are balanced/correctly assigned.

Figure 4: Mean estimated probabilities (solid lines) with 95% confidence intervals (dash lines) obtained from 1000 realizations for augmentation methods, with N = 300 data points sampled from (a) U nif orm([-1, -0.9] ∪ [0.9, 1]), w = 4, b = 0. (b) U nif orm([0, 1]), w = 8, b = 4. In both cases, we use α = 0.3 for feature smoothing and mixup, and σ = 0.3 for generating random noise.

Figure7: Mean estimated probabilities (solid lines) with 95% confidence intervals (dash lines) obtained from 1000 realizations in the case of in-symmetric data. x ∈ [-1, -0.9] ∪ [0.9, 1] with size 300. y = 1 ← p > 0.5. Then 10 another input x = 0.1, y = 1 or 10 input of x = -0.1, y = 0 are added randomly with probability 0.5 to the training.

Peer review (iclr2019):
 The authors proposed a feature smoothing method without adding any computational burden for defensing against adversarial examples. The idea is that both feature smoothing and Gaussian noise can help extend the range of data. Moreover, the authors combined these methods together to gain a better test and adversarial accuracy. They further proved 3 theorems to try to analyze the biases and variances of decision boundary based on the fisher information and delta method. In my opinion, the main contribution of this paper is to prove that the boundary variance will decrease due to adding one additional regularization term to the loss function. Main comments: 1. The proposed feature smoothing method seems less novel to me. In contrast to the mixup method, the proposed method appears to remove the label smoothing part, so it is better to explain or justify why this could be better theoretically. Moreover, in the PGD and PGD-cw results, the performance is not as good as the Gaussian random noise method. Can the authors offer any discussion or comments on the possible reasons? 2. Some details of the proof of Theorem 4.1 seemed to be omitted. I am a bit confused about this. a. “Without loss of generality, we further assume b = 0 and w > 0.” With smaller magnitude, b=0 is reasonable, but why to assume w>0? b. Could you present the derivation details or the backing theory of the approximation of var(b), when one more regularization term are added? 3. In addition, a method of modifying the network is proposed to adapt to the feature smoothing method. However, no experimental results are reported to support its effectiveness. I would believe some empirical evaluations may further strengthen the paper.