Adversarial Regression with Doubly Non-negative Weighting Matrices

Abstract: Many machine learning tasks that involve predicting an output response can be solved by training a weighted regression model. Unfortunately, the predictive power of this type of models may severely deteriorate under low sample sizes or under covariate perturbations. Reweighting the training samples has aroused as an effective mitigation strategy to these problems. In this paper, we propose a novel and coherent scheme for kernel-reweighted regression by reparametrizing the sample weights using a doubly non-negative matrix. When the weighting matrix is confined in an uncertainty set using either the log-determinant divergence or the Bures-Wasserstein distance, we show that the adversarially reweighted estimate can be solved efficiently using first-order methods. Numerical experiments show that our reweighting strategy delivers promising results on numerous datasets.


Reviews and author responses:

This paper proposes a method that minimizes the worst-case risk among a set of some generalized weighted empirical risks. The generalized weighted empirical risks are defined using doubly non-negative (i.e., non-negative, positive (semi-)definite) matrices that is close to the matrix corresponding to the original sample weights. The proposed method boils down to solving a mini-max problem involving the matrix, but surprisingly the gradient of the optimal objective of the inner maximization admits a closed-form solution, allowing first order optimization for the outer minimization. The paper also provides experiments using four benchmark datasets and shows the effectiveness of the proposed method.  # Summary of my review
The paper presents an interesting framework with an elegant solution. I have a few concerns about the motivation and the experiments, but the technical part of the work is insightful and the techniques used in the paper may have other important applications. I tend to vote for acceptance.


# Originality
The idea of the adversarial learning with doubly non-negative matrices seems new as far as I can see. The methods and the theoretical results also seem original. However, I am not an expert of this domain, and there may be some similar work that I am not aware of.


# Quality
The theoretical results and the derived methods presented in the paper are highly non-trivial to me. The results and proofs are quite technical, but the paper presents them very gently and clearly.
My concerns are
* Despite its mathematical elegance, it is hard for me to interpret the practical meaning of taking the adversarial matrix from doubly non-negative matrices. It would be nice if the authors could provide some intuitive explanations.
* In the experiments, the authors only consider the perturbation to the output variable of the training samples. It is possible to consider other kinds of perturbations?


# Clarity
The writing is overall very clear and easy to follow except for the technical details. The only think that was unclear to me was how to set \Omega_{i, j} when i \neq 0 and j \neq 0. I think the authors should explain this in the rebuttal.


# Significance
The technical contributions are significant. I personally would not be able to obtain such elegant results if I did not see this paper.
On the other hand, the practical contribution remains unclear. How does adversarially learning with doubly non-negative matrices help learn better in practice? Also, is it really helpful to do adversarial robust learning in the first place? [14] claims that distributionally robust learning may not lead to robust solutions. My intuition is that adversarially weighting samples gives heavy weights to samples with large loss, which ends up being sensitive to outliers. It may work in the opposite way to the intention of making the solution robust. Does the proposed framework not have this issue?


# Update after the authors' response and discussions
After reading the other reviewers' comments and the feedback from the authors rethinking about the value of the paper, I lowered my score from 7 to 6. In a few words, it is technically interesting, but I am not convinced about its usefulness in the ways the authors claim.
I appreciate the technical contribution of the paper, but I am not sure that the motivation explained by the author naturally leads to the problem formulation of the paper, or that the problem solved in the paper really "robustifies" the estimator. According to the discussions with the authors, it seems that the connection between the min-max optimization over the NN-matrices and the robustness remains an open question, which means that the method is a heuristic in terms of its statistical effectiveness. The only evidence of the practical usefulness is the experimental result. If the authors support the usefulness of the approach only by experiments, there should be much more extensive empirical studies. For these reasons, I do not strongly recommend accepting the paper.
 The paper does not discuss the limitations or the societal impact. Perhaps, the authors should mention the scalability issue of the proposed method because it needs the inverse of a N-by-N matrix at every evaluation of the gradient.

 Thank you very much for your questions. Below is our reply.

> **(i) What problem does the paper focus on? What is the goal?**

+ **Goal**: to design a new estimator that has competitive performance (for example, low root mean square error on test data).

+ **Applications**: to improve the Nadaraya-Watson (NW) or local linear regression estimators (see Example 1.1 and 1.2). However, our approach extends to local learning algorithms, geographically weighted regression and many more (see Line 34--37). While they are very different problems, their mathematical formulas are similar.

+ **What is our approach?** We use robust optimization (or min-max formulation)

+ **Why is it non-trivial?** Because the weights are given through a kernel, which captures the relative physical/geographical relationship between training samples. In addition, we want to design a reweighting scheme that keeps this relative relationship coherently throughout the robustification procedure.
 
+ **Why do we need matrices?** If $\hat \Omega$ is constructed using the Gram matrix (as we suggested in the paper), then each component of the matrix will represent the relative relationship between each pair of samples. Thus, lifting to the matrix space helps us trace these pairwise relationships. (A side note: in our submission, we let $\hat \Omega$ to be an input parameter for generality purposes, and this generality may be confusing to the readers. For what we have in mind, $\hat \Omega$ should be the Gram matrix (see Line 122))


> **(ii) Why can we expect the proposed approach to solve the problem?**

Holistically speaking, the reweighting schemes will tend to reduce the dependence of the estimate on any particular sample point, thus leading to some “robustness” effect. 

For the specific case of the NW estimator with Gaussian kernel (where we want to estimate $\mathbb{E}[Y | Z = z\_0]$): the estimate depends more heavily on samples closer to $z\_0$. Our reweighting scheme reduces the weight of nearer points, and increases the weight of farther points, which means the estimate will depend more “uniformly” on the sample points.

Above are some hand-waving arguments to justify our approach. From a broad perspective, our method belongs to the class of robust (min-max) optimization estimators which dates back to the 1940s. Only very recently, it has been shown that min-max estimators are connected to variance regularization (arXiv:1610.02581), square-root LASSO regularization (arXiv:1610.05627), shrinkage regularization (arXiv:1805.07194) and many more. The connection between our robust estimator and some regularization scheme is an interesting question for future study; so far, the focus of this paper is on the design and computation of the new robustification schemes.


> **Further comments from the Reviewer:**

+ > Especially when the sample size is small, no method can perform well in general.

We agree with the Reviewer. This is why we have run the numerical experiments with different sample sizes $N$ to benchmark the performance of different estimators.

+ > So, it still makes sense to perform experiments with other types of perturbations in the class for sanity checks.

If the adversary can perturb in the set $\mathcal{U}_{\phi, \rho}$, then our estimator should provide good performance because our estimate already **optimizes** against these perturbations. In our experiments, we have shown that our robust estimator works well even with other perturbations (perturb in $y$ in the submission, and adding Gaussian noise in the covariate $z$ in the response). Coming back to the goal of this paper in the beginning of this reply, these results demonstrate that our estimator does deliver good performance under various perturbations/situations.

Thank you!


 I appreciate the authors' feedback. I believe that the discussion made my questions clearer. The following two points are unclear to me:
(i) What problem does the paper focus on? What is the goal?
(ii) Why can we expect the proposed approach to solve the problem?

Regarding (i), the authors raise several weak points with the standard weighted estimator as follows.
> Unfortunately, the predictive power of this type of models may severely deteriorate under low sample sizes or under covariate perturbations.

> The non-parametric statistical estimator obtained by solving Problem (1) is sensitive to the corruptions of the training data. Similar phenomenon is also observed in machine learning where the solution of the risk minimization problem (1) is not guaranteed to be robust or generalizable.

> The quality of the solution to Problem (1) also deteriorates if the training sample size is small.

> Moreover, our main goal in this paper is to robustify with respect to the weights in Equation (1).

If the proposed method can deal with all of those problems, that is amazing, but I am not convinced that is the case. Especially when the sample size is small, no method can perform well in general. Also, each statement is quite unclear: What kind of corruption and robustness?

Regarding (ii), the authors claim reweighting as a high-level approach has been successful in mitigating those problems:
> Reweighting the training samples has aroused as an effective mitigation strategy to these problems.

> Reweighting, obtained by modifying $\omega(\hat{z}_i)$, is arising as an attractive resolution to improve robustness and enhance the out-of-sample performance in the test data [23, 26, 32]. At the same time, reweighting schemes have shown to produce many favorable effects: reweighting can increase fairness [12, 15, 30], and can also effectively handle covariate shift [9, 14, 36].

However, "reweighting" is too general and vague to be grouped into one. The references mentioned in the paper use different reweighting methods, and they solve different problems. Weights designed for covariate shift do not provide fairness and vice versa. What is important is how the weights are designed.

Moreover, the authors do not motivate well the use of the matrix for representing the weights. The authors admit the most entries of the weights do not affect the nature of Problem (3). If so, why do we want to introduce the matrix in the first place? The paper says "it endows us with more flexibility to perturb the weights in a coherent manner," but it does not mean it leads to better performance.

In my understanding, the adversarial attack is the worst-case perturbation within the specified class in the inner minimization problem, when it is mathematically formulated. Since the min-max solution minimizes the worst-case risk, for any other perturbations within the class the risk will be smaller than that of the adversarial attack. So, it still makes sense to perform experiments with other types of perturbations in the class for sanity checks. However, the paper does not provide intuitive explanations or examples for what kind of perturbations are included in the class $\mathcal{U}_{\phi, \rho}$. Without such discussions provided, it is hard to say the proposed approach or the experiments are well-motivated.

 Thank you very much for your question. Please find below our replies.

**Q1**. We would like to clarify two terms: “perturbations” and “adversarial attacks”. The “adversarial attacks” is a subset of “perturbations”, and to create an attack, the adversary should be optimizing a certain objective function. Meanwhile, “perturbations” are more general, it may include just noisy observation (the noise does not need to be adversarial).

The claim in our paper that \"the predictive power of this type of models **may** severely deteriorate under low sample sizes or under covariate perturbations” can be easily demonstrated using a numerical experiment. Towards this end, we carried out the following experiment on the KIN dataset, where we perturb the training covariate by adding a Gaussian noise $\mathcal N(0, \sigma^2 I)$, where $I$ is the identity matrix. We set the nearest neighbor size to $N=50$, the perturbation level $\tau=N$. We report the average root mean square error (RMSE) over 10 independent replications, each replication contains 800 test samples in the below table.

| Average RMSE $\hspace{3 em}$ |  $\sigma=0 \hspace{4.2 em}$ | $\sigma=0.1 \hspace{3.8 em}$ |  $\sigma=0.5 \hspace{3.3 em}$ |  $\sigma=1 \hspace{4.3 em}$ |

| ------------------------------- | ------------------------ |--------------------------|-------------------------|--------------------------|

| $\rho=0 \hspace{6.3 em}$ | $1.2574 \pm 0.0241$ | $1.3164 \pm 0.0263$  | $1.5041 \pm 0.0266$ | $1.7396 \pm 0.0402$ |

| $\rho=0.1$ (NW-BuresW)    | $0.6448 \pm 0.0122$ | $0.6456 \pm 0.0123$  | $0.7625 \pm 0.0143$ |  $0.9140 \pm 0.0174$ |

| $\rho=0.1$ (NW-LogDet) $\hspace{0.2em}$| $0.6487 \pm 0.0125$ | $0.6489 \pm 0.0126$  | $0.7596 \pm 0.0144$ |  $0.9110 \pm 0.0178$ |

Errors at $\rho=0$ indicate the performance of the **vanilla** NW estimator. We can empirically observe that our adversarial reweighting schemes can alleviate the effect of covariate perturbations while the performances of the baseline (NW) are severely deteriorated.

The claim in our response that "we do not know how to benchmark the performance of our estimator w.r.t. adversarial attack of the covariate” comes from the fact that we can not find any paper in the literature that studies adversarial attack for the NW estimator.

**Q2**. How to set $\hat \Omega_{i,j}$ for $i \neq 0$ and $j \neq 0$

We agree with the reviewer that the choice of $\hat \Omega$ is not unique. We, unfortunately, do not have any further insights on how the choice of $\hat \Omega$ may affect the estimator.

As in the Problem (3), and the description for the nominal matrix of weights $\hat \Omega$ in line 103-106, the values of $\hat \Omega_{i, j}$ when $i \neq 0$ and $j \neq 0$ do not affect the objective and can be arbitrary. Following line 75-77 and line 121-124, we set the $\hat \Omega$ as the Gram matrix (See the function eval_Omega2 in our Matlab code, and the function nesterov_agd2 line 19 for more detail). Or, the $\hat \Omega_{i, j}$ when $i \neq 0$ and $j \neq 0$ is simply set to the kernel value between the $i^{th}$ and the $j^{th}$ data points for consistency. 

Thus, we would recommend to fill the matrix $\hat \Omega$ with the Gram matrix, which would be an intuitive and reasonable choice.


 I thank the authors for the response to my comments. It helps me understand the paper better, but two issues remain unclear to me.

1. What type of perturbations does the paper focuses on?

Does the following response mean the robustness that the proposed method offers is against perturbations to the weights?:
> our main goal in this paper is to robustify with respect to the weights $\omega(z)$ in Equation (1).

On the other hand, the paper appears to claim the proposed method is addressing performance deterioration caused by covariate perturbations.
> Unfortunately, the predictive power of this type of models may severely deteriorate under low sample sizes or under covariate perturbations. Reweighting the training samples has aroused as an effective mitigation strategy to these problems. In this paper, we propose a novel and coherent scheme for kernel-reweighted regression by reparametrizing the sample weights using a doubly non-negative matrix.

Doesn't that contradict the fact the authors "do not know how to benchmark the performance of our estimator w.r.t. adversarial attack of the covariate"?

Moreover, the experiments focuses on a perturbation to the response variable. All of these do not seem consistent with each other, and it confuses me.


2. How to set $\widehat{\Omega}_{i, j}$ for $i \neq 0$ and $j \neq 0$?

In my initial comments, I mistyped $\widehat{\Omega}$ as $\Omega$. I apologize for the typo. As far as I understand, these entries of $\widehat{\Omega}$ do affect the uncertainty set and thus need to be carefully designed.


I hope to get some clarifications for these issues from the authors before I fix the final score.

 We thank the review for your positive comments and valuable feedback on our work.

> I wish more interpretation was given to how to interpret these optimal re-weightings as compared to the original weighting matrix---right now they are simply stated as complicated mathematical expressions which are hard to parse.

For the original weighting matrix (the nominal matrix of weights) in line 103-105, the element $\hat{\Omega}_{0i}$ indicates the contribution of the sample-specific loss to the objective (line 17-18). As we can see in Example 1 for NW, the weight $\omega(z_i) = K(z_0, z_i)$ plays as the weighting role for the conditional expectation estimator.

In our formulation, the optimal estimator $\beta^{\*}$ and the optimal re-weighting matrix $\Omega^*$ constitute a saddle point, or a Nash equilibrium, of a two-player, zero-sum game (see equation (4)). Unfortunately, we do not have any analytical formula for the saddle point $\beta^{\*}$ and $\Omega^*$. In this paper, we need to use a gradient descent algorithm to find these two quantities, and hence we do not have any further information regarding the form of $\Omega^*$.

The only information we obtain from the Nash equilibrium is that $\Omega^*$ is the optimal weighting matrix that maximizes the loss with respect to the estimate $\beta^{\*}$.

> Why is optimizing with respect to a ball (w.r.t the given divergence) of radius ρ around the original weighting matrix the right notion of "robustifying" the estimator? For what sorts of data perturbations is the right notion of robustifying an estimator? I am not an expert in the field, and it might be standard to optimizing with respect to ball around the original matrix. However, it is not clear to me why. More context should be given as to what other notions of robustification have been studied in the literature, and why is the particular notion the authors choose to look at the right one. Indeed, it still remains unclear to me when one should choose to the log-determinant divergence vs. the Bures-Wasserstein divergence.

The min-max formulation in this paper is motivated by the “robustifying” mechanisms in adversarial training and robust optimization community (See arXiv:1706.06083, arXiv:1710.10571, arXiv:1908.08729). Thus we follow the convention of these (sub)fields and we call an estimator to be ``robust’’ if it is obtained by solving the min-max formulation (4). We agree that the term “robustifying” is ambiguous, and the readers may have different viewpoints on what is “robustifying” and what is not. We will elaborate on this term in the revision.

In this paper, we study the log-determinant divergence and the Bures-Wasserstein divergence as two alternatives to construct the set of possible weighting matrices. We have no normative justification on why one set is better than the other. However, from the numerical experiments, we observe that both methods have similar effectiveness, but the Bures-Wasserstein construction is more numerically stable, faster to solve and it also can handle the case where the nominal matrix is singular. We will report these findings more clearly in the numerical section.

> With respect to the experiments, the particular scheme use to perturb the training samples seems non-standard to me, i.e., y→κy where κ is sampled uniformly between 1.8 to 2.2. What form of data corruption seen in practice does this correspond to? Some standard notions of data corruption include "error-in-variables regression" and the "Huber contamination model". The experiments don't seem to correspond closely to either.

> to what sorts of natural data corruption seen in practice does their re-weighing scheme provide robustness to

We would like to emphasize that the data can be easily perturbed in reality. For example, to promote privacy when the response variable carries sensitive information, it is advisable to add a multiplicative noise to the data, see [A, B, C, D]. This corresponds to the setup that we test in the numerical experiments.

Moreover, please note that our proposed adversarial reweighting schemes are designed for a general loss function $\ell$. Therefore, it is straightforward for our proposed framework to use some loss functions which help to improve robustness and mitigate the effect of outliers such as the Huber loss and its variants.

**References**:

[A] Lin, Yan-Xia, and Phillip Wise. “Estimation of Regression Parameters from Noise Multiplied Data”. Journal of Privacy and Confidentiality 4 (2), 2013.

[B] Nayak, Tapan & Sinha, Bimal & Zayatz, Laura. “Statistical Properties of Multiplicative Noise Masking for Confidentiality Protection”. Journal of Official Statistics. 27, 2010. 

[C] Klein, Martin, Thomas Mathew, and Bimal Sinha. “Noise Multiplication for Statistical Disclosure Control of Extreme Values in Log-Normal Regression Samples”. Journal of Privacy and Confidentiality 6 (1), 2014.

[D] Ma Y., Lin YX., Krivitsky P.N., Wakefield B. “Quantifying the Protection Level of a Noise Candidate for Noise Multiplication Masking Scheme”. In: Domingo-Ferrer J., Montes F. (eds) Privacy in Statistical Databases. PSD 2018. Lecture Notes in Computer Science, vol 11126. Springer. 


 We thank the reviewer for your positive comments.

> Despite its mathematical elegance, it is hard for me to interpret the practical meaning of taking the adversarial matrix from doubly non-negative matrices. It would be nice if the authors could provide some intuitive explanations.

Please see line 17-18, in Problem (1), the weight function $\omega$ indicates the contribution of the sample-specific loss to the objective. Moreover, in line 56-64, we discuss the motivation to use the reweighting approach for the weighted regression problem. However, it becomes non-trivial for the reweighting approach when the weight function is tied to a kernel (see line 65-74). Our approach based on the adversarial reweighting schemes on doubly non-negative matrices can address this problem. At the same time, it also allows us to derive efficient algorithmic approaches based on first-order methods for the proposed framework.

> In the experiments, the authors only consider the perturbation to the output variable of the training samples. It is possible to consider other kinds of perturbations?

Firstly, we would like to note that our proposed adversarial reweighting schemes are for general loss $\ell$. Therefore, it is easy to plug in other loss functions which are robust to outliers such as the Huber loss and its variants. 

Secondly, the response generally summarizes the information of features, and the perturbation (i.e., multiplicative perturbation) is performed on nearest training samples w.r.t. a given test sample (since the nearest neighbors play the most important role with the highest weights on the loss in the objective) for adversarial perturbation purposes. 

Moreover, our main goal in this paper is to robustify with respect to the weights $\omega(z)$ in Equation (1). For example, to robustify w.r.t. the covariate $x$, we can add another layer of maximization in the variable $x$ for Equation (4), and existing results on robustifying with respect to $x$ can be applied to have a similar gradient descent algorithm.

In this paper, we consider nonparametric estimation methods (Nadaraya-Watson, Local Linear Regression). The literature on crafting attacks for nonparametric estimators at test time is scarce and non-trivial: there are attacks for k-NN estimators (arXiv:1706.03922, arXiv:1911.06591, arXiv:1906.03972, etc.) but we are not aware of any possible attack for NW and LLR estimators. We thus do not know how to benchmark the performance of our estimator w.r.t. adversarial attack of the covariate.

> The only think that was unclear to me was how to set \Omega_{i, j} when i \neq 0 and j \neq 0. I think the authors should explain this in the rebuttal.

As in the Problem (3), and the description for the nominal matrix of weights $\Omega$ in line 103-106, the values of $\Omega_{i, j}$ when $i \neq 0$ and $j \neq 0$ do not affect the objective and can be arbitrary. Following line 75-77 and line 121-124, we set the $\Omega$ as the Gram matrix (See the function eval_Omega2 in our Matlab code, and the function nesterov_agd2 line 19 for more detail). Or, the $\Omega_{i, j}$ when $i \neq 0$ and $j \neq 0$ is simply set to the kernel value between the $i^{th}$ and the $j^{th}$ data points for consistency. We will clarify it.

> the practical contribution remains unclear. How does adversarially learning with doubly non-negative matrices help learn better in practice? Also, is it really helpful to do adversarial robust learning in the first place? [14] claims that distributionally robust learning may not lead to robust solutions. My intuition is that adversarially weighting samples gives heavy weights to samples with large loss, which ends up being sensitive to outliers. It may work in the opposite way to the intention of making the solution robust. Does the proposed framework not have this issue?

Please see the motivation of our adversarial reweighting approach in line 56-64.

- The non-parametric statistical estimator obtained by solving Problem (1) is sensitive to the corruptions of the training data. Similar phenomenon is also observed in machine learning where the solution of the risk minimization problem (1) is not guaranteed to be robust or generalizable.

- The quality of the solution to Problem (1) also deteriorates if the training sample size $N$ is small.

- Reweighting, obtained by modifying $\omega(z_i)$, is an attractive resolution to improve robustness and enhance the out-of-sample performance in the test data. At the same time, reweighting schemes have shown to produce many favorable effects: reweighting can increase fairness, and can also effectively handle covariate shift.	
			
We would like to clarify that for the paper [14] (Hu et al’ 2018), if one simply replaces the empirical risk minimization (their Problem (1)) by the adversarial robust learning (their Problem (3)), then the distributionally robust learning may not lead to robust solutions. However, Hu et al [14] also proposed a carefully-designed distributionally robust learning in Problem (16) which is also a weighted loss function, and the weights are confined in some adversarial set. The approach in their Problem (16) is empirically effective for their applications.

As in line 35-37, the Problem (1) is the building block for the risk minimization with covariate shift [14, $\S$4] (or the Problem (16) in [14]).


 We thank the reviewer for your positive feedback.

> Q1: The proposed models (NW, LLR) are somewhat simple with few parameters. Would the method be applicable with more complex models provided by e.g. kernel methods (representer theorem allowing a parameterization of the model by n scalars, where n is the number of training samples)?

First, we would like to emphasize that kernel ridge regression differs from NW and LLR. NW and LLR use kernels to generate weights, while kernel ridge regression uses kernels for higher dimensional representation. The kernels used in these applications are different as we have explained in the introduction.

Second, our approach can be applied to the kernel ridge regression problem to ``robustify” the kernel matrix. Indeed, consider the dual form of the KRR problem as in equation (6.15) in [A], we can see that the objective function can be written using an inner product between the kernel matrix and the variables $\alpha$. This inner product form fits nicely with our framework (see equation (3)) and robustification can be carried out in a straightforward manner. We will add this observation to the revised version.

**Reference**:

[A] Mohri Mehryar, Afshin Rostamizadeh, and Ameet Talwalkar, Foundations of Machine Learning. The MIT Press, 2018.

> Q2: The problem tackled concerns empirical risk minimization without regularization, is there a reason for this choice, considering that one goal is to generalize better?

We consider empirical risk minimization for simplicity of exposure. Our proposed algorithmic approaches can be easily extended to the regularized objective function without any modification. Basically, the regularization is on the parameter $\beta$, and does not affect the adversarially reweighted loss function $F$ in line 135.

> Concerning the experimental baseline, I would have expected stronger baselines [...] such as kernel ridge regression (KRR) [...] or models based on the minimization of the Huber loss instead of the square loss [...] 

> Overall weakness: Not very strong baselines for conditional expectation estimation.

We would like to note that our approach is designed for a general loss. Therefore, we can straightforwardly use the Huber loss as the loss function $\ell$ without any modification. The squared loss function is currently used to make our approach consistent with other baselines, e.g., NW, LLR, NW-Metric, for a fair comparison. Additionally, as we note in the answer of **[Q2]**, our proposed approach is easily extended to the weighted regression with regularization.


 We thank the reviewer for your very positive comments about our work.

Our main goal in this paper is to robustify with respect to the weights $\omega(z)$ in Equation (1). To robustify w.r.t. the covariate $x$, we can add another layer of maximization in the variable $x$ for Equation (4), and existing results on robustifying w.r.t. $x$ can be applied to have a similar gradient descent algorithm.

In this paper, we consider nonparametric estimation methods (Nadaraya-Watson, Local Linear Regression). The literature on crafting attacks for nonparametric estimators at test time is scarce and non-trivial: there are attacks for $k$-NN estimators (arXiv:1706.03922, arXiv:1911.06591, arXiv:1906.03972, etc.), but we are not aware of any possible attack for NW and LLR estimators. We thus do not know how to benchmark the performance of our estimator w.r.t. adversarial attack of the covariate. We really appreciate it in case the reviewer can provide us with more guidance on this experimental setup.

Regarding the datasets for the experiments, we are following standard datasets in the literature, see for example, reference [20]. Unfortunately, these datasets are not rich enough (they don’t have enough distribution shifts in the data) to conduct the split as you suggested. We are eager to explore more datasets, and your recommendations of the datasets are warmly welcome.


The authors propose bring robustness to kernel weight regression (Eq. 1) through adversarial weight perturbation. The main difficulty lies in the tie between the weights and kernel, and therefore the need for the perturbation to be consistent with the kernel. The main idea here is to reparameterize the optimization in Eq. 1, to reach Eq. 3, where a doubly non-negative matrix is used to represent the weights. This reparameterization makes solving the inner maximization possible through a dual formulation. The authors studied the problem completely for two divergences and evaluated the method on various real/benchmark datasets.  I think the paper is in a good shape. Perhaps the only issue is why the authors perturbed only the response variable in the experimental results. What happens if the perturbation is applied on the features as well. More importantly, does the method resist such perturbations if they are adversarial, and not random? In addition, it would make more sense to evaluate the method based on some other datasets that exhibit larger natural distribution shift, i.e. through split of train/test according to a third variable (e.g. time in bank transactions, etc.)  The section is not provided in the paper. 

The paper proposes a novel way to address the problem of reweighting the empirical risk, which relies on a parameterization of the risk by a doubly non-negative weighting matrix.
The corresponding optimization problem is of "min-max" type, where the maximum is taken over some set of possible weights encoded by a divergence, here the log determinant and Bures-Wasserstein divergences.
The exploitation of dual techniques allows the problem to be tractable for both divergences, and experimental results emphasize on the efficiency of the approach in the context of perturbed training samples.  -Originality:
The problem tackled in this paper is well known, and the approach is novel.
Especially, the idea of treating the reweighting of the empirical risk in an adversarial manner is very original.
Related works are (up to my knowledge) adequately cited, and the contributions are clearly stated.

-Quality:
The submission is mathematically sound.
The difficulty brought in by the introduction of the max problem is adequately tackled using parametric duality, allowing for a practical computation of the gradient of the objective function and to first-order optimization methods.

Q1: The proposed models (NW, LLR) are somewhat simple with few parameters. Would the method be applicable with more complex models provided by e.g. kernel methods (representer theorem allowing a parameterization of the model by $n$ scalars, where $n$ is the number of training samples) ?

Q2: The problem tackled concerns empirical risk minimization without regularization, is there a reason for this choice, considering that one goal is to generalize better ?

The experimental section is furnished, and the code is available (matlab). Many aspects of the proposed approach are investigated, and the authors are honest about evaluating the strength of their method (especially about the hyperparameter $\rho$ induced by the size of the adversarial reweighting set, and the computational time).

Concerning the experimental baseline, I would have expected stronger baselines with a goal of being robust to the type of outliers introduced in the paper ($y \to \kappa y$ for some outlier level $\kappa$) such as kernel ridge regression (KRR) (the $\lambda$ parameter allowing for better generalization capabilities than plain empirical risk minimization), or models based on the minimization of the Huber loss instead of the square loss.
Especially, considering that the task at hand is conditional expectation estimation, a simple kernel ridge regression baseline (learned with regularization in RKHS norm) could bring more insights about the strength of the proposed approach.

-Clarity:
The submission is well-written and pleasant to read.

-Significance
The problem tackled is of prime interest, as the solution provided in the paper can help mitigating bias in the data, and achieve more fairness in machine learning algorithms.
The idea of introducing a min-max problem using a set of possible weights is fertile and could be adapted to algorithms other than the ones presented in the paper, thus leading to interesting potential future works built on the current submission.

-Overall strengh:
Original idea for reweighting the training risk, clever exploitation of dual techniques.
-Overall weakness:
Not very strong baselines for conditional expectation estimation. Yes

This paper discusses how to "robustify" a sample weighting scheme by finding a re-weighting that is minimax optimal for a pre-defined uncertainty set. The authors give expressions for the optimal re-weighting scheme for two different distance metrics: log-determinant divergence and Bures-Wasserstein divergence. Authors show the optimal re-weighting can be calculated via gradient updates. Experimental validation is provided where the labels are perturbed.  In terms of the motivation of this work, I believe optimal re-weighting of kernel matrices so that they have additional robustness properties to adversarial perturbation is likely to have an impact in the nonparametric regression literature. Relatively speaking, this paper is written in a pretty approachable way, and the results are clearly stated. 

Some points that would help me understand the context of this paper and the results they produce are as follows:

- I wish more interpretation was given to how to interpret these optimal re-weightings as compared to the original weighting matrix---right now they are simply stated as complicated mathematical expressions which are hard to parse. 
- Why is optimizing with respect to a ball (w.r.t the given divergence) of radius $\rho$ around the original weighting matrix the right notion of "robustifying" the estimator? For what sorts of data perturbations is the right notion of robustifying an estimator? I am not an expert in the field, and it might be standard to optimizing with respect to ball around the original matrix. However, it is not clear to me why. More context should be given as to what other notions of robustification have been studied in the literature, and why is the particular notion the authors choose to look at the right one. Indeed, it still remains unclear to me when one should choose to the log-determinant divergence vs. the Bures-Wasserstein divergence.
- With respect to the experiments, the particular scheme use to perturb the training samples seems non-standard to me, i.e., $y \to \kappa y$ where $\kappa$ is sampled uniformly between $1.8$ to $2.2$. What form of data corruption seen in practice does this correspond to? Some standard notions of data corruption include "error-in-variables regression" and the "Huber contamination model". The experiments don't seem to correspond closely to either.

I would be willing to increase my score if these points above were engaged with more meaningfully. In short, to what sorts of natural data corruption seen in practice does their re-weighing scheme provide robustness to, both theoretically and experimentally (details above). N/A

Meta-review: The paper is technically sound, however its motivation and practical impact are unclear: the reasons why the contribution is one method of choice to go for small sample situations are not spelled out clearly enough. This makes the paper a bit subpar wrt the expectations for a paper at NeurIPS.


Decision: accept