Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper focuses on the adversarial robustness of mixture of expert (MoE) models and proposes theoretical analysis focusing on the Lipschitz constant of an MoE model under some relaxed assumptions such as a smooth MoE and a linear model. 
The goal is to answer the interesting question of whether MoE models, with significantly higher number of parameters and roughly constant additional computation cost, have better adversarial robustness than their dense (single-model) counterparts.
The analysis shows that under certain conditions on the routing function and the structure of the data, MoE models can have a significantly lower Lipschitz constant. Under some ideal conditions, the lower bound on the Lipschitz constant established by Bubeck and Sellke [3] for single over-parameterized models can be reduced by a factor of $\sqrt{E}$ for a mixture of $E$ experts.
They experimentally evaluate the adversarial robustness of the recent Vision Transformers and Vision MoE models on a proprietary dataset (for pre-training) and the ImageNet dataset (for fine-tuning), and report interesting observations about the robustness of MoE models, the choice of experts, and redundancy among the experts.

strengths_and_weaknesses: ### Strengths
**S1:** The paper focuses on the theoretical and empirical analysis of the adversarial robustness of mixture of expert (MoE) models. This is an important problem since the robustness of neural networks to bounded adversarial perturbations is a desirable property. 

**S2:** The work seems to be relatively novel to my knowledge since existing works have not focussed much on the analysis of robustness of over-parameterized MoEs through the lens of Lipschitz constant. It is inspired by the recent work of Bubeck and Sellke [3], who establish a result on the lower bound of the Lipschitz constant of over-parameterized neural networks in terms of the number of parameters, samples, and the dimension.
 
**S3:** The presentation of technical details is fairly clear to follow, except in a few places where some clarifications would help (please see my questions). 

### Weaknesses

**W1:** Given that practical MoEs follow the sparse combination of experts as specified by equation (1), they are not continuous functions. Hence, small changes in their input can lead to large changes in their output. The analysis presented in the paper assumes a smooth weighted mixture of experts. It is not clear how this analysis can be extended to the sparse MoEs, or how one can draw insights from the analysis for the case of sparse MoEs.

**W2:** Some of the details about the experiments are not clear or missing (e.g., attack objectives, definition of metrics). This would be important to make the work more reproducible (please see my questions). 

**W3:** (Minor) The analysis could be supported with some intuitions and maybe a figure (a toy example) for illustration. That would convey the key ideas in a better way. 

questions: Please find a list of my questions and comments below. Major ones are listed first.

**1:** For the case $\mathbf{X}_1 \\,\bot\\, \mathbf{X}_2$, it is not clear to me how to go from 
$\\| (\mathbf{X}^T \mathbf{X})^\dagger \mathbf{X}^T \mathbf{y}  \\| \\,$ to $\\, \\| (\mathbf{X}_1^T \mathbf{X}_1)^\dagger \mathbf{X}_1^T \mathbf{y}_1 \\,+\\, (\mathbf{X}_2^T \mathbf{X}_2)^\dagger \mathbf{X}_2^T \mathbf{y}_2  \\| $. Maybe I missed some trick or detail.

**2:** Could the authors clarify the statement on lines 196 - 199 about the construction of the projection matrices and also specify their dimension?

**3:** Lines 242 - 243: it is mentioned that JFT-300M is a multi-label dataset and uses the sigmoid cross-entropy loss. By multi-label, does it mean each sample has 1 or more labels, or does it simply mean that there are multiple classes, with each sample assigned to a single class? Also, does sigmoid cross-entropy loss mean the same as binary cross-entropy loss (which would only be suitable for binary classification)? Please clarify how many classes are present in JFT-300M and provide some additional details on the dataset (that is reasonable to share).

**4:** Please provide a formal  definition of the following metrics (in the appendix if space is limited): Precision at 1, False discovery rate, and Intersection-over-union. 

**5:** In the results in Figure 1, the metrics reported are referred to as “classification error rate” and “false discovery rate”. It would be more accurate to call them “robust error rate” and “robust FDR” respectively because these are considering the worst-case adversarial input $x’$ for each clean input $(x, y)$. In other words, the metrics would be defined with a `max` over a $\ell_\infty$ norm ball. It is misleading to refer to them as a standard error rate or FDR both in the figures and the text. Same comment for Figure 3.

**6:** In figure 3, it is mentioned that dashed lines are used to indicate that the auxiliary losses are also being attacked. However, I do not see any curves with dashed lines.

**7:** Referring to figure 3 (rightmost figure) and lines 320 - 322, it is mentioned that for finetuning on ImageNet, all models with more than 2 experts achieve roughly the same accuracy (this should be robust accuracy). Can the authors provide some insight on why there is no notable improvement in the robustness (robust accuracy) as the number of experts increases?

**8:** In the interest of completeness and reproducibility, please define all the attack objectives including the ones against the auxiliary losses in an appendix. Please define the auxiliary losses used to balance the load among the experts. 

**9:** In section 4.4, it is mentioned that the result for the classification error on ImageNet has a similar trend. This result can be included in an appendix. 

**10:** In figures 1 and 3, it would be easier to see the contrast in metrics if the x-axis (perturbation size) is restricted to smaller value (say 0.01). The y-axis could be log-scaled if that helps.

**11:** There are a few places where it stated (for example) that the classification error is x and y respectively, but it is not clear which two models the values refer to. For example, on lines 260 - 262, the classification error of which two models are 19.3% and 17.8%? Same comment for the false discovery rate. 
 
**12:** In lines 85, 86 it is mentioned that only one expert is selected for each input, i.e. $K = 1$. Where is this assumption used?

**13:** In equations 3, 4, and 5, the loss function in the attack objectives should have a class label or target $y$ (currently missing). Also, a distinction should be made whether it is a untargeted or targeted attack. 

**14:** In Lemma 1 and other places, please clarify what type of norm is being used. Or state upfront that the norms are by default (e.g.) $\ell_2$. 

**15:** On line 186, I think it should be $\mathbf{X}^T = [\mathbf{X}_1^T ~\mathbf{X}_2^T]$ in order to have the right dimensions.

**16:** Nit: On line 142, it should be $\\{ \mathbf{s}_i \in \mathbb{R}^D \\}_\{i \in [E]\}$. It seems like the set of $\mathbf{s}_i$ lives in $\mathbb{R}^D$.

**17:** On line 168, I think it should be equation 7, not 6.

**18:** Nit: Hyphen missing in definitions 1 and 2. Should be “In-subspace distance” and “Cross-subspace distance”.

**19:** Lines 237 - 238: Please clarify that the 8.9 GFlops corresponds to the ViT-B/32, while the 17.9 GFlops corresponds to the ViT-B++/32 model. 

**20:** A table with the compute in GFlops and the (order of) number of parameters for the different architectures, as well as for different number of experts would be useful. Please have a forward reference to Table 1 in the Appendix.

**21:** Please clarify what is meant by a Token. 


limitations: The authors discuss some limitations of their work in section 5. One of them is the focus on linear models for simplicity of analysis. Additionally, the paper focuses on the analysis of smooth mixture of experts, whereas in practice sparse MoEs are not smooth or continuous. This should be discussed under limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper investigates the adversarial robustness of sparse Mixture of Experts models for image classification, in order to answer the question: do models with more parameters (but not necessarily more computational power) have better robustness? Bounds on the Lipschitz constant are derived to compare the sparse MoE models with their dense counterpart, to show that sparse MoE models are more robust. The claim is then verified with empirical experiments on large data sets.

strengths_and_weaknesses: ## Strengths
+ The topic of the paper is relevant, both from the perspective of robustness analysis (what makes a model robust?) and the MoE perspective (with trend towards large models, MoE is a viable alternative, so what other benefits we get besides computational cost?).
+ The paper has a very clear goal: are sparse MoE models more robust than their dense counterpart?
+ The paper is well organize and, generally, it was easy to read.
+ The empirical experimentation is performed on very large data sets and non-trivial models.
+ Originality: while there are papers investigating robustness of large models, they do not analyze MoE models.

## Weaknesses
- The mathematical expressions/descriptions are sometimes imprecise/incomplete (e.g. from the eqs in Sec 2.2, it would seem that the true label of the image is not used to generate the adversarial example). See more in the questions.
- The theoretical analysis (Sec 3) focus on linear (single layer) experts for a regression task and fixed routing. There's little discussion on how these results transfer to the much more complex setting of ViT-MoE (non-linear, multiple layers of MoE, non-fixed routing) for classification. 
- In the empirical evaluation, it is difficult to distinguish improvement (in particular for JFT-300M) and the significance of the results
- Part of the analysis is conducted in a private data set (JFT-300M), which makes impossible to reproduce those results.
- Limited discussion of related works

## Summary
Overall, the paper is clear, easy to follow and pursuits an original analysis of an interesting and relevant problem. However, it needs to be polished more as there are several issues that put in question the quality and significance of the results.
questions: * In Section 3.2, the setup of the analysis is a regression problem instead of a classification problem. How do the results presented are impacted by the actual setup of ViT-MoE, where we have a non-linear model with multiple MoE layers in a classification problem? The layers between the MoE layers could amplify the impact of the perturbations and reduce the "smoothness" gained by using MoE.
* In Definition 1, the statement is trivial. There's no restriction in $\epsilon_1$, so it could be any arbitrarily large value. The definition is just saying that there is a number larger than another number.
* In Section 4, the term "token" is used, but it is never described in the paper.
* From Figure 1, ViT++ and ViT-MoE are indistinguishable. Is the improvement in robustness coming from the MoE or just from the increased model size?
* Related to Figure 2: the figure has the rate of changes per layer. It would also be interesting to see if the attack always modifies the routing in all layers or if there is some pattern (e.g. for some inputs the change only happens in later layers; or, can the change in one layer have no impact on the next?)
* In Figure 3, again, how significant is the difference? How does MoE helps with robustness?
* Adaptive attacks: in Section 4.4, the auxiliary losses are included in the attack. What are those losses and how are they optimized? Does the attacker specifically tries to change the output of the routing? Can an attacker force all images to be routed to a single expert?
* Please expand the discussion of related works, for example: [1, 2].
* Improve the mathematical description in Section 2:
    - Add the dependency on the label $y$ in the formulation of the adversarial examples in eqs 3, 4, 5.
    - line 115: the clarification after "i.e." is actually confusing (is the 0/1 loss used for evaluation only or for generation of the examples in the preceding equation?)
    - Eq 5: in the current text the variable $t$ is defined as a real number. Replace with $\forall t \in \{0, \dots, \tau -1\}$

### References
[1]: Wu, B., Chen, J., Cai, D., He, X., & Gu, Q. (2021). Do Wider Neural Networks Really Help Adversarial Robustness? https://arxiv.org/abs/2010.01279  
[2]: Xu, K., Wang, C., Cheng, H., Kailkhura, B., Lin, X., & Goldhahn, R. (2021). Mixture of robust experts (more): A robust denoising method towards multiple perturbations. https://arxiv.org/abs/2104.10586
limitations: The main limitation of the work is how applicable their theoretical analysis is to the actual large models of interest. It is not clear how the limited setup of their analysis actually applies to more complex models as used in the empirical experiments.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the robustness of mixture-of-experts (MOE) models.  The authors study the Lipschitz properties of MOEs, and they study MOEs in a simple linear setting.  Ultimately, they seek to make connections to recent work that has studied how over-parameterization (OP) plays a role in robustness.  They also provide a set of experiments to empirical study this setting.
strengths_and_weaknesses: ### Strengths
---

**Similar linear setting.**  I liked the analysis that the authors performed for the case of linear MOEs.  Although the proofs seemed to follow standard techniques, I think that this part of the paper was a nice first step toward measuring the Lipschitzness of MOE models.

**New problem setting.** I have not seen many works that consider the problem setting of the robustness of MOEs.  If the robustness of MOEs is fundamentally different from that of other model classes, then this would represent a useful direction.


### Weaknesses
---

**Preliminaries** The authors repeatedly refer to the idea that MOEs have smaller computation cost.  I do not understand what this means.  Does it mean that the models are more computationally efficient at inference time?  Does it mean that they are less costly to train?  Without a description of how MOE models work and what their practical advantages are, I think that many readers will be left wondering the same thing.  For instance, they authors say that

> "Practical models also tend to this setup to match the FLOPs of dense models." (sic)

What does "FLOPS of dense models" mean?  Which dense models are the authors referring to?  What is even meant by a dense model -- e.g. convolutional neural networks, MLPs, etc?  I think this points to the need for a stronger preliminary section, where MOE and dense models are both introduced and defined.  Readers coming from the robustness literature may not be familiar with MOEs, so it's important to set those readers up so that they can understand the rest of the paper.

**Is this a problem?**  I'm having trouble discerning whether MOEs suffer from robustness issues.  The authors give some intuitive evidence for the lack of robustness of MOEs (e.g., that routing protocols can change given small perturbations), but there isn't any concrete  empirical evidence that MOEs are not robustness.  Therefore, I think it's not unreasonable to question why these results are meaningful.  

Let me explain a bit more.  In the standard robustness literature, it has been repeatedly demonstrated that CNNs and MLPs are highly vulnerable to attacks.  For example, a PGD attack can degrade performance on CIFAR10 for ResNet18 from ~94% to essentially 0% (i.e., worse than random guessing).  This empirical result spawned all of the research concerning the robustness of these models.

In this paper, we are considering a new model -- MOEs -- which the authors describe as being more computationally efficient and using a fundamentally different design.  Therefore, I think that before studying the theoretical aspects of the robustness of MOEs, it would be worthwhile to demonstrate whether or not MOEs actually lack robustness to small perturbations.  

**Related work.**  I think that the discussion of related work could be significantly improved.  At the very least, there should be a section called "Related Work" where the authors discuss past approaches for robustness and lines of work important to the development of MOE models.  Indeed, there seems to be previous work that studies the robustness of MOE models:

> Xu, Kaidi, et al. "Mixture of robust experts (more): A robust denoising method towards multiple perturbations." arXiv preprint arXiv:2104.10586 (2021).

The authors should cite works like these to distinguish their contribution from that of the literature.

**Claims in the introduction.**  Some of the claims made in the introduction seem to be inaccurate.  For instance, the claim that adversarial robustness only refers to studying *bounded* changes.  In fact, the field has evolved to study more general (non-norm-bounded) shifts in data; see e.g.,

> Wong, Eric, and J. Zico Kolter. "Learning perturbation sets for robust machine learning." arXiv preprint arXiv:2007.08450 (2020).

> Robey, Alexander, Hamed Hassani, and George J. Pappas. "Model-based robust deep learning: Generalizing to natural, out-of-distribution data." arXiv preprint arXiv:2005.10247 (2020).

> Hendrycks, Dan, et al. "The many faces of robustness: A critical analysis of out-of-distribution generalization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Furthermore, the authors claim that poor adversarial robustness is due to over-parameterization.  This also seems inaccurate, as poor robustness has also been attributed to adversarial subspaces

> Gilmer, Justin, et al. "Adversarial spheres." arXiv preprint arXiv:1801.02774 (2018).

and to the local linearity of DNNs with piece-wise linear activations

> Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).

among other causes.  These claims highlight the need for a more comprehensive literature review in my opinion.

**Connections to Bubeck and Selke [3].**  I don't understand the impact of the connections to [3].  Essentially, the argument seems to be that as MOEs have $E$ times more parameters, one can replace $P$ by $P\cdot E$ and then you get a constant factor in the bound.  The thing is, while we may consider taking the number of data points to infinity, we will never have very large number of experts in the same way from my understanding.  Sure, if you take $E\to\infty$, the bound gets small, but is this practically possible?

**Lack of a practical takeaway.**  One question I have about this work: What is are practical takeaways?  The goal of this paper seems to be to study the robustness of MOEs.  However, it's hard to draw any concrete conclusions based on my reading.  For example, does the theory inform the reader of any algorithm concerning how to improve MOE robustness?  Are there settings where MOEs are *significantly* more robust than ViT models?  The experiments seem rather non-conclusive to me in this regard, as the ViT models perform very similarly to the MOE models.  Indeed, in Figure 1, the curve are all almost overlapping.  The same is true in Figure 3.  So setting the theory aside, I'm not sure what the practical takeaways are from this paper.

**Proof of Lemma 1.**  I believe that there is an error in the proof of Lemma 1.  The proof begins by calculating the gradient of the mixture of experts:

$$ \nabla f(x) = \sum_{i=1}^E p_i(x) \cdot \nabla f_i(x) + \sum_{i=1}^E \nabla p_i(x) \cdot f_i(x)$$

We then take the norm of both sides, and apply the triangle inequality:

$$ || \nabla f(x) || \leq \sum_{i=1}^E p_i(x) \cdot ||\nabla f_i(x)|| + \left|\left| \sum_{i=1}^E \nabla p_i(x) \cdot f_i(x) \right|\right|. $$

Next we take the supremum over $x$ on both sides, yielding 

$$ \text{Lip}(f) \leq \sup_{x} \sum_{i=1}^E p_i(x) \cdot ||\nabla f_i(x)|| + \sup_{x}\left|\left| \sum_{i=1}^E p_i(x) [s_i - \bar{s}(x)] \cdot f_i(x) \right|\right|. $$

We can further upper bound the RHS by moving the supremum inside the sum, meaning that

$$ \text{Lip}(f) \leq  \sum_{i=1}^E \sup_{x} p_i(x) \cdot ||\nabla f_i(x)|| + \sup_{x}\left|\left| \sum_{i=1}^E p_i(x) [s_i - \bar{s}(x)] \cdot f_i(x) \right|\right|. $$

So far, this aligns with the proof given in the paper (although I've written out the steps in more detail here).  This is the point at which I feel problems start to arise.  For any index $i\in[E]$, consider the expression

$$ \sup_{x} p_i(x) \cdot ||\nabla f_i(x)||  $$

The authors assert that this expression is less than or equal to (or simply equal to -- the proof is ambiguous here) $p_i(x) \text{Lip}(f_i)$, which I believe is incorrect (although if I'm wrong, I'd welcome a discussion with the authors).  

In general, my intuition for this being wrong is that in general, for real-valued functions $q(x)$ and $g(x)$, it does **not** hold that $\sup_x(q(x) \cdot g(x)) = q(x) \cdot \sup_x(g(x))$ $\forall x$.  (In this particular case, we have $q(x) = p_i(x)$ and $g(x) = ||\nabla f_i(x)||$.)  To make this more concrete, let's consider a counterexample.  Let $f(x)$ be defined piecewise (here I've dropped the $i$ subscript): 

$$ f(x) = (-1/2)x + 1/2 \quad\text{for}\quad x<-1 $$
$$f(x) = -x \quad\text{for}\quad -1\leq x\leq 1$$
$$f(x) = (-1/2)x - 1/2 \quad\text{for}\quad x>1$$

The Lipschitz constant of this function is clearly $\text{Lip}(f)=1$.  Now let 

$$ p(x) = \epsilon \quad\forall x : |x| \leq 1$$
$$ p(x) = 1-\epsilon \quad\forall x : |x| > 1$$

where again, I've dropped the $i$ index for simplicity.  Then we have that 

$$ p(x) ||\nabla f(x)|| = \epsilon \quad \text{for}\quad |x| \leq 1$$
$$ p(x) ||\nabla f(x) || = (1-\epsilon)/2 \quad\text{for}\quad |x| >1 $$

and therefore $\sup_x p(x) ||\nabla f(x) || = (1-\epsilon) / 2$ provided that $\epsilon < 1/3$.  On the other hand, we have that

$$ p(x) \text{Lip}(f) = \epsilon \quad\text{for}\quad |x|\leq 1 $$
$$ p(x) \text{Lip}(f) = 1-\epsilon \quad\text{for}\quad |x| > 1 $$

Therefore, we find that 

$$ \epsilon = p(x) \text{Lip}(f) < \sup_x p(x) ||\nabla f(x)|| = (1-\epsilon)/2 $$

if $\epsilon < 1/3$ and $p(x) = \epsilon$ (since $p(x)$ is a function of $x$, which is a free variable, we can pick $p(x)$ to be whatever we like from the above definition).  Therefore, it does not hold that $\sup_x p(x) ||\nabla f(x)|| \leq p(x) \text{Lip}(f)$, breaking the proof of Lemma 1.

As several results and much of the discussion in the paper follow from this Lemma, I think it's important that this concern be addressed.

**Grammar.**  The grammar and structure of the paper could also be revised.  There are several typos as well, which should be addressed.

### Final thoughts
---

Overall, while I think that this paper goes in an interesting direction, I also believe that it would need to be fundamentally rewritten and expanded to be considered for acceptance.  The motivation for this problem is shaky, given that it is not clear whether MOE models lack robustness in the first place.  The literature review is sparse, and some of the proofs appear to be incorrect.  Furthermore, the experiments are not convincing, as the difference in robustness between dense and non-dense models doesn't seem to be large.  For these reasons, I cannot recommend accept at this time, especially given the possibility that the proof of Lemma 1 is incorrect.

questions: * The authors say that the $\ell_\infty$ and $\ell_2$ norms are "popular choices in practice."  What does "in practice" mean here?  I am not aware of any settings "in practice" that actually use $\ell_p$ robustness.  I think that in the broader context of the literature, $\ell_p$ robustness is seen as a stepping stone toward understanding more practically relevant security concerns, such as distribution shifts in autonomous vehicles tasks or medical imaging.

* In the sentence:

> "The compute needed to evaluate an image on the dense ViT models are 8.9 and 17.9 respectively."

What are the units on 8.9 and 17.9?
limitations: I did not see a discussion of this.  At the very least, the authors could comment on the limitations of their analysis.  Furthermore, they could comment on the potential for nefarious actors to use adversarial technology.  Although in principle I agree that this paper is theoretical in nature and that there aren't any concrete concerns from a societal point of view.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper gives a theoretical analysis to the adversarial robustness of MoEs, it shows MoEs achieve better robustness than the dense models through lower bounding the Lipschitz constant of MoEs by that of the dense models theoretically, and then shows superior robustness of MoEs compared to the dense models experimentally.
strengths_and_weaknesses: Strengths:
- This paper lower bounds the Lipschitz constant of MoEs by that of the dense models, showing that MoEs achieve better robustness than the dense models.
- This paper does some experiments to verify the superior robustness of MoEs compared to the dense models and uncovers some intriguing properties of adversarial attacks for MoEs.
- Although the linear regression model and the kernel regression is quite simple, this paper shows some interesting results about the robustness of MoEs.

Weaknesses:
- In the experiments, the paper just uses the PGD adversarial attacks, it is not convincing, as is said in [1], just use PGD attacks is not enough, I suggest to do further experiments on AutoAttack[1] and C&W[2].
- Lipschitz constant is a global property, but adversarial robustness is not a global property, it depends on the data distribution and other factors. So I think the Lipschitz constant is not sufficient to describe the robustness of a classifier.

[1] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.

[2] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy (SP), pages 39–57, 2017.
questions: As stated in the previous part, firstly, I think it is better to run the experiments on more attacks, just running it on the PGD attacks is not convincing; secondly, the theory in this paper lower bounds the Lipschitz constant of MoEs by that of the dense models, and then the authors claim that MoEs achieve better robustness than the dense counterparts theoretically, what I concern is that as a global property, can the Lipschitz constant fully describe the robustness of a model? Perhaps a smaller Lipschitz constant means the model will have better robustness in the worst case, i.e. the mass of the data is concentrated on the most steep part of the model.
limitations: The authors addressed the limitations of this work in the conclusion part.
review_title: null

Meta-review: metareview: The paper studies the adversarial robustness of Mixture of Expert models. The authors have addressed the concerns from the reviewers and the reviewers adjusted their scores. However, the reviewer still has some concern that the paper fails to present a strong and convincing argument in towards its goals. 

Please revise the paper as suggested by the reviewers in the detailed reviews. Please also complete the pending experiments and add to the next version. 

confidence: Less certain
Decision: Accept