Title: value: Data Debugging is NP-hard for Classifiers Trained with SGD
Authors: value: Zizheng Guo
Pengyu Chen
Yanzhang Fu
Dongjing Miao
Venue: NeurIPS 2024
Keywords: value: data debugging
machine learning interpretations
computational complexity

Abstract: value: Data debugging is to find a subset of the training data such that the model obtained by retraining on the subset has a better accuracy.
A bunch of heuristic approaches are proposed, however, none of them are guaranteed to solve this problem effectively.
This leaves an open issue whether there exists an efficient algorithm to find the subset such that the model obtained by retraining on it has a better accuracy.
To answer this open question and provide theoretical basis for further study on developing better algorithms for data debugging, we investigate the computational complexity of the problem named \textsc{Debuggable}.
Given a machine learning model $\mathcal{M}$ obtained by training on dataset $D$ and a test instance $(\mathbf{x}_\text{test},y_\text{test})$ where $\mathcal{M}(\mathbf{x}_\text{test})\neq y_\text{test}$, \textsc{Debuggable} is to determine whether there exists a subset $D^\prime$ of $D$ such that the model $\mathcal{M}^\prime$ obtained by retraining on $D^\prime$ satisfies $\mathcal{M}^\prime(\mathbf{x}_\text{test})=y_\text{test}$.
To cover a wide range of commonly used models, we take SGD-trained linear classifier as the model and derive the following main results.
(1) If the loss function and the dimension of the model are not fixed, \textsc{Debuggable} is NP-complete regardless of the training order in which all the training samples are processed during SGD.
(2) For hinge-like loss functions, a comprehensive analysis on the computational complexity of \textsc{Debuggable} is provided;
(3) If the loss function is a linear function, \textsc{Debuggable} can be solved in linear time, that is, data debugging can be solved easily in this case.
These results not only highlight the limitations of current approaches but also offer new insights into data debugging.


Review discussion:
The paper analyzes the computational complexity of linear classifiers with both fixed and non-fixed loss functions.
First, it shows that in the general case of non-fixed loss functions and dimensions, the problem of debugging is NP-hard.
However, for hinge-loss-like functions, depending on the dimension of the features and the sign of the intercept, the problem could be either NP-hard or solvable in linear time.
This result also implies that it is not accurate to estimate the impact of a subset of training data by summing up the scores of each training sample in the subset if we assign each sample point a scoring number.
- The model studied in the paper is limited to linear classifiers, which is very restrictive.
- Most of the manuscript is devoted to proving the theorems rather than discussing and interpreting the implications of the results.
- The setting involves debugging for any possible test point, which is far from practical.
The definition of DEBUGGABLE-LIN seems very pessimistic as it tries to find a subset for any possible test point.
However, in practice and in learning theory, one is interested in testing a point that comes from the same distribution as the training data or at least a distribution similar to the training data. Considering this practical scenario, would the problem still be NP-hard?
The paper does not have any potential negative societal impact.
No ethics review needed.
We appreciate the viewer for providing the valuable feedback.
-The model studied in the paper is limited to linear classifiers, which is very restrictive.
This paper studies the complexity **lower bound** of the data debugging problem. To this end, the studied model should be as simple as possible.
Therefore, studying the problem with linear classifiers becomes a natural choice, as they are simple and basic parts of modern model architectures.
This paper shows a quite surprising result that data debugging is NP-hard for even linear classifiers, as long as they are trained with SGD. Figuring out the cause of the hardness is helpful for understanding the relationship between training data and model predictions.
Results in this paper imply that the hardness seems unlikely to root from the model structure.
Instead, it seems more likely to stem from the way the model is trained (e.g., SGD in this paper).
-Most of the manuscript is devoted to proving the theorems rather than discussing and interpreting the implications of the results.
We appreciate the reviewer for the fruitful comment. We apologize for not clarifying the implications of our results. This paper mainly focus on the computational complexity of data debugging problem. Prior to our work, there have been studies that try to debug the training data of machine learning models.
However, most of them are empirical and propose mainly heuristic debugging algorithms.
For example, one popular approach is to first score each training sample, remove the ones with lower scores and then re-train the model. The approach is iteratively repeated until the behavior of the model is corrected. Existing works mainly focus on the ways to speed up this process (e.g., how to efficiently estimate the scores, how to update the parameters directly so as to avoid retraining the model, etc.). Since debugging the model usually requires deleting **multiple** training samples, it remains open whether one can both efficiently 1. estimate the score of each training sample, and 2. choose the subset of the training data $T^\prime$ such that the prediction of a model will be corrected if it is retrained on $T^\prime$.
Furthermore, it is still unclear whether some prediction of a model be efficiently corrected by retraining on some proper subset of the training data. To address this problem, it is critical to analyze its computational complexity.
To the best of our knowledge, we are the first to formalize the problem and have made the initial attempts to understand its computational complexity.
Our results suggest that, the complexity of the problem is related to \emph{how the model is actually trained}. Even under extremely simple settings such as linear models trained by SGD with the hinge loss, data debugging can still be NP-hard. This implies that a ``universal debugger'' seems unlikely to exist, which takes the training data $T$ and the test instance as the input and outputs the desired $T^\prime \subseteq T$. Developing fast data debugging algorithms needs further inspecting the training data. There might also be efficient algorithms with theoretical guarantees under certain assumptions on the training data.
-The setting involves debugging for any possible test point, which is far from practical.
We appreciate the reviewer for providing this valuable feedback. In Section 4, we showed that the problem remains NP-hard when all the training data is "good", i.e., $\forall (x,y)\in T, y_\text{test}(\alpha y \eta \otimes x)^\top x_\text{test}>0$.
Therefore under a practical setting where $\alpha>0$ and the learning rate $\eta_i$ is positive, the test instance $(x_\text{test},y_\text{test})$ and training data $(x,y)$ are similar in the sense of $yy_\text{test}x^\top x_\text{test}>0$.
-Reply to the question:
The problem remains NP-hard when the test instance $(x_\text{test},y_\text{test})$ and training data $(x,y)$ are similar in the sense of $yy_\text{test}x^\top x_\text{test}>0$.
The paper is well written with the key problem being well motivated. The proofs are succinct and easy to follow.
My key concern with the paper is that the constructions underlying the hardness results are immensely contrived and far removed from how classifiers are actually trained in practice. This casts severe doubts as to whether the key results of the paper implies anything significant about the hardness of data debugging as is performed in practice. To elucidate a few instances:
1. **Theorem 3.1**: The reduction to Monotone 1-in-3 SAT in Theorem 1 hinges on an adversarially constructed loss function (Line 186), one that is very far removed from any loss function I’ve seen used either in the learning theory literature or in practice.
Beyond this, it also requires a specific parameter initialization and learning rate, both of which, to my knowledge, are far removed from the typical random initialization schemes and learning rate scaling rules used in practice and analyzed in theory.
For instance, in Line 198, the first $m$ co-ordinates of the learning rate $\eta$ is set as $5$, the next $n$ are set as $\frac{1}{6N}$ while the next $m$ are set as $2000N$. This seems very very removed from any learning rate schedule either used in practice or analyzed in theory. This makes me severely doubt whether the result has any meaningful implication on the inherent hardness of the data debugging problem.\
In addition, the training set and test data point are also constructed adversarially (the latter is perhaps this is to be expected I wouldn’t perceive that in isolation as a major weakness)
2. **GTA Algorithm**: The correctness of the GTA algorithm is proved only for the linear case (which is honestly quite straightforward) and hinge-like losses for $\beta \geq 0$ and dimension $d=1$ (which is of limited interest as most statistical learning problems are high dimensional). This is particularly concerning as the paper does not perform any empirical evaluation of GTA.
3. **Theorem 4.3** While the analysis for the hinge loss is certainly more interesting than Theorem 3.1, the result suffers from a key weakness that the data ordering is adversarially chosen (In particular, the positioning of $(x_c, y_c), (x_b, y_b), (x_a, y_a)$ is crucial to the reduction). It is well known in the theory of optimization that adversarial data orderings lead to provably worse convergence in practice [Safran and Shamir COLT 2020; Das, Scholkopf and Muehlebach NeurIPS 2022].
In fact, adversarial data ordering is even the basis of an attack on deep neural networks [see Shumailov et. al. “Manipulating SGD with Data Ordering Attacks” NeurIPS 2021].\
The adversarial data ordering considered in the result differs both from the practical implementation of SGD which samples the data points in each epoch as per some random permutation [see Ahn and Sra NeurIPS 2020 and references therein] or the canonical version of SGD considered in theory where data indices are sampled uniformly with replacement [Bubeck 2015]. To the best of my understanding, the result does not hold for either of these commonly considered variants of SGD.
Furthermore, the training data and test data is again, adversarially chosen.
4. **No Analysis for Cross Entropy** : The paper does not contain any analysis for the binary entropy loss which is what is commonly used to train classifiers, further limiting the scope of the results.
5. **Complete Lack of Empirical Evaluation**: While this wouldn’t be a weakness by itself, the fact that the theoretical results in the paper have limited applicability (as argued above) makes me quite concerned about the absence of empirical evaluation on real world settings, or, for a start, even toy-like settings where the data is drawn from plausible distributions, the learning rates and parameters are initialized as one would normally expect them to be, and SGD is run either with replacement or without replacement and not with an adversarial data ordering.
While the paper studies a question which I found interesting, I believe the limited applicability of the theoretical results, the absence of experimental evaluation as well as the limited technical novelty of the proofs (the proofs, although crisply written, are based on straightforward complexity theoretic reductions and do not unearth any nontrivial mathematical insights regarding SGD or linear classifiers) makes me confidently feel that the paper is currently not ready for acceptance at NeurIPS.
1. Can you show that Theorem 3.1 continues to hold under standard distributional initialization schemes for the learning rate and the initial parameter (e.g. normal or Xavier initialization.
2. Can you construct a class of distributions such that, if your training samples are drawn i.i.d from it, your hardness results continue to hold with high probability (or for starters, even in expectation)
3. Are your results amenable to smoothed analysis, i.e., if the data points, learning rate and initialization in your construction is perturbed by some oblivious noise, would your hardness result continue to hold?
4. Can you comment on the applicability of your results for the cross entropy loss?
Mentioned in Section C as per the checklist.
Also refer to comments above in Weakness Section.
No ethics review needed.
We appreciate the viewer for providing the valuable feedback.
===REPLY TO THE WEAKNESSES===
1. Theorem 3.1 implies the intractability of a model-agnostic ``universal data debugger'', where the loss function, the learning rate, the model dimension, etc. are given as the input.  The majority of previous studies have focused on developing that debugger.
While a few have highlighted the limitations of existing approaches, none have provided a theoretical analysis. Our work addresses this gap by demonstrating, in Theorem 3.1, that the development of such universal data debuggers is inherently challenging.
Properties of the loss function, the learning rate and other factors related to model training should be taken into consideration when developing fast data debuggers.
Our primary aim was to establish a solid theoretical foundation that can guide the development of debugging tools under specific, well-defined conditions. We believe that by doing so, we can provide a more precise and actionable framework for future research and applications. Theorem 3.1 serves as the cornerstone of our theoretical framework, therefore it is not expected to encompass all aspects of the problem domain or to provide a comprehensive solution.
2. We would like to clarify that the GTA algorithm is not presented as one of the primary conclusions of our work.
Instead, its role is to serve as an illustrative example that demonstrates that the problem can be simple under highly simplified conditions. By doing so, we aim to provide a stark contrast that highlights the inherent complexity of the problem when it is not oversimplified.
3. We appreciate the reviewer for this valuable feedback. The definition of the studied problem can be slightly modified as follows:
Given the training data $T$, loss function $\mathcal{L}$, initial parameter $w^{(0)}$, learning rate $\eta$, threshold $\varepsilon$ and instance $(x_\text{test},y_\text{test})$, is there a $T^\prime \subseteq T$ and a proper training order such that $SGD_\Lambda (\mathcal{L},\boldsymbol{\eta},\varepsilon,T^\prime, x_\text{test})=y_\text{test}$?
The modified problem is still worth studying, and the proof of Theorem 4.3 can also proof the NP-hardness of the modified data debugging problem, since any training order not considered in the proof will not correct the model behavior.
4.We acknowledge that the breadth and depth of a single article are inherently limited, and as such, our current work is focused on establishing a solid foundation. The formalization of the problem and the initial complexity analysis are significant steps in advancing our understanding of the field.
However, we recognize that there is much more to explore, and we have intentionally left the analysis of the Cross-Entropy Loss for future research.
5. Our article aims to elucidate the inherent difficulty of the problem at hand, a perspective that complements the empirical conclusions drawn by other studies, such as in [1].
===REPLY TO THE QUESTIONS===
Reply to question 1,2,4:  A single article cannot encompass the vast array of aspects that a complex problem may present.
To the best of our knowledge, we are the first to \emph{identify and formally define} the data debugging problem and give the initial complexity results.
Our primary objective in this paper has been to rigorously define the problem at hand and to derive significant preliminary theoretical conclusions. The remaining work, which we recognize to be substantial, will be the focus of our future research efforts.
Reply to queation 3: Smooth analysis is a technique used to understand the difficulty of algorithms rather than the inherent difficulty of the problems they solve. Our work aims to identify and address this gap by developing new algorithms or improving existing ones to better cope with the challenges posed by the problem domain. The application of smooth analysis in this context is to evaluate and enhance the algorithms' performance, not to analyze the problem's inherent difficulty.
References:
[1]Pang Wei Koh, Kai-Siang Ang, Hubert Hua Kian Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In Neural Information Processing Systems, 2019.
I thank the authors for their response. Having read their response carefully, I regret to inform that it fails to adequately address my key concerns regarding how the constructions underlying the proof of the hardness results are contrived and far removed from practice.
While I agree with the authors in principle that a single article cannot be expected to encompass every problem in a field, I believe the article in its current form does not contribute much beyond formalizing the problem and making some rudimentary complexity theoretic investigations under  setups that diverges significantly from practice (a concern that has also been reflected in other reviews). To this end, I believe the current scope and technical contributions of this paper is quite limited, and unfit for acceptance.
I appreciate the modification suggested by the authors for Theorem 4.3.
However, the one line sketch alone is insufficient for me to judge the correctness and technical value of the modification.
I continue to retain my concerns regarding the lack of *any* empirical evaluation (even on toy data). Given the significant differences between the problem settings considered this paper and the practical settings under which data debugging is performed (e.g. contrived loss functions and other such concerns elucidated in my review and also mentioned by other reviewers), I respectfully disagree with the statement that the article elucidates the inherent difficulty of data debugging (as is performed in practice)
At its current state, I believe the article is unfit for publication at NeurIPS. I encourage the authors to either expand upon their theoretical results (by considering more commonly used loss functions, analyzing the hardness of the problem under distributional assumptions etc.) or complement the existing results with *some* empirics (at least on toy data), redo the proof of Theorem 4.3 with the proposed modification and resubmit.
For the above reasons, I have decided to keep my score same.
- The proofs look correct
- The proofs are creative and clever
- Generally speaking, understanding the relationship between training data and model predictions is important
Clarity and exposition:
- Many of the proofs have "magic constants" which makes them harder to understand. The authors may consider generalizing some of the results.
- The proofs look correct, but are not optimized for being easily read and understood in terms of logical flow or notation.
- Similarly, it is hard to extract the intuition from the proofs.
- The implications of results for the wider community are somewhat muddled: while the computational hardness of exactly finding "bad" training points according to a particular definition is interesting, it's unclear what the relationship is to scoring based methods is since any selection algorithm could be identified with a scoring method (output 1 on selected points, and -1 on the rest), and "CSP-solvers" and "random algorithms" are mentioned without any particular explanation/exploration.
Significance:
- The loss function used to prove Theorem 3.1 is rather pathological. The derivative is zero at most places, but has a few intervals with derivatives with wildly varying orders of magnitude: N, 1, and 1/N. Reading through the proof, this pathological loss function seems critical and not easily removed. I would find it much more significant if Theorem 3.1 could be proved for convex loss functions, or similar.
- The assumption of an adversarial training order in Theorem 4.3 seems unreasonably restrictive. The user is changing the dataset by removing points, why can't they change the training order too? Again, this piece seems critical to the proof, since without the adversarial training order, I think the constructed data debugging task would be trivially solvable by taking a gradient descent step on (x_c,y_c) first. Can the theorem be extended to the user choosing the training order?
Furthermore, the practical problem of data selection/cleaning is with regards to the presence/absence of datapoints, not the training order which is an optimization consideration. Can the theorem be proved not for a specific optimization algorithm like SGD, but for the minimizer(s) of the loss instead? A unique minimizer (for the hinge-loss and convex losses more generally) could be guaranteed by adding a small strictly convex regularization term.
- Is there a cleaner/clearer way to describe and prove the results?
- For convex loss functions and without a fixed training order, is the data debugging task NP-hard?
- For convex loss functions and assuming an oracle to compute the loss minimizer, is the data debugging task NP-hard?
- Is there a typo in the definition of the hinge loss on line 133? I think the conditional logical expression should have a minus sign in front of the beta.
Yes
No ethics review needed.
We appreciate the viewer for providing the valuable feedback.
===REPLY TO THE WEAKNESSES===
Relationship of scoring based methods to data debugging: since it is reported that score-based selection approaches (i.e., score each training sample in the first phase and delete training samples greedily with lower score in the second phase) for data debugging is not accurate[1], it is important to figure out the complexity of data debugging. The hardness results in this paper suggest that, there is unlikely an approach that in the same time 1) estimate the score of each training sample efficiently, and 2) efficiently select the proper subset $T^\prime \subseteq T$ based on the estimated scores such that the model retrained on $T^\prime$ will be corrected. Marking 1 on selected points, and -1 on the rest will indeed make 2) very fast.
However, the process of marking 1 and -1 itself will, according to our results, unlikely to be efficient.
Pathological loss functions: A single article cannot encompass the vast array of aspects that a complex problem may present. Our primary objective in this paper has been to rigorously define the problem at hand and to derive significant preliminary theoretical conclusions. The remaining work, which we recognize to be substantial, will be the focus of our future research efforts.
===REPLY TO THE QUESTIONS===
-Is there a cleaner/clearer way to describe and prove the results?
We apologize for not clarifying the intuition behind the proof. The proof of Theorem 3. 1 is illustrated in the attatched pdf of the general rebuttal. In a MONOTONE 1-IN-3 SAT instance, the truth values of each clause and variable are stored in a parameter. The process of gradient descent converging precisely simulates the process of determining the truth values of the clauses. The loss function has different gradients across various training samples, leading to varying update magnitudes for each parameter. The gradient on the loss function is determined by the current training sample and the corresponding parameter,  more precisely, $yw^\top x$. For training samples encoding variables (e. g. , var(1)), after one training iteration, the corresponding $w_i$ will increase, causing $yw^\top x$ to jump to the gradient-free gray area in the direction indicated by the arrow. This avoids further updates to that parameter. For training samples encoding clauses (e.
g. , clause(1)), $w_i$ will be updated in the direction of the arrow from one of the green area after the first round of iteration. During this process, the parameters encoding the truth values of clauses will be 'unlocked,' allowing them to be updated in the second round of iteration. The purpose of locking these parameters in the first round of iteration is to prevent premature updates of clause truth values before the parameters of variable information is fully updated. In the second round of iteration, parameters encoding the truth values of variables will no longer be updated; only the parameters corresponding to true clauses will be updated. After the updates, for any training sample $(x,y)$, $yw^\top x$ will fall within the gray area, indicating that the parameters will no longer be updated and the model has converged.
-For convex loss functions and without a fixed training order, is the data debugging task NP-hard?
We appreciate the reviewer for the insightful feedback. We acknowledge that the assumption of an adversarial training order is restrictive.
However, we also find that it is more valuable to study the following slightly modified data debugging problem:
Given the training data $T$, loss function $\mathcal{L}$, initial parameter $\mathbf{w}^{(0)}$, learning rate $\boldsymbol{\eta}$, threshold $\varepsilon$ and instance $(x_\text{test},y_\text{test})$, is there a $T^\prime \subseteq T$ and a proper training order such that $SGD_\Lambda (\mathcal{L},\boldsymbol{\eta},\varepsilon,T^\prime, x_\text{test})=y_\text{test}$?
Here the training order assumption is no longer needed, as the users can pick a training order whatever they want. The proof of Theorem 4.3 can also proof the NP-hardness of the modified data debugging problem, since any training order not considered in the proof will not correct the model behavior.
-For convex loss functions and assuming an oracle to compute the loss minimizer, is the data debugging task NP-hard?
It is worth noting that the real-world optimizers are usually NOT oracles, and taking them as oracles may oversimplify the problem. It is more valuable to understand \textbf{the cause} and \textbf{how to avoid} the hardness, which is helpful in designing efficient and easy-to-debug optimizers for training ML models.
-Is there a typo in the definition of the hinge loss on line 133? I think the conditional logical expression should have a minus sign in front of the beta.
Yes. We thank the reviewer for this helpful comment. The hinge loss should be $\mathcal{L}_\texttt{hinge}(y\mathbf{w}^\top\mathbf{x})=-\alpha(y\mathbf{w}^\top\mathbf{x}-\beta)$ when $y\mathbf{w}^\top\mathbf{x}< \beta$.
References:
[1]Pang Wei Koh, Kai-Siang Ang, Hubert Hua Kian Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In Neural Information Processing Systems, 2019.
Dear Authors,
Thank you for your response.
Weakness 1: I'm not disagreeing with the hardness results, I'm disagreeing with the attention and discussion given to scoring based methods which seem orthogonal to the proved results.
Weakness 2: I agree about the importance of preliminary work, however, I find the loss of the construction in the proof of Theorem 3.1 to be very different in nature from any loss function (including non-convex loss functions) that appears in the literature. Is there any existing loss function (ramp-loss, hinge loss, logistic loss, exponential loss, quadratic loss) where the hardness result can be proved? Along similar lines, I suspect that the constructions are not robust to small perturbations of the initial weight or data points which is unfortunate. Practitioners almost always choose the initial weight randomly or as 0.
Question 1: Thank you for the explanation. It would be nice if you could incorporate this into the next version of this paper.
Question 2: Is that statement true? As stated in my original review, "I think the constructed data debugging task would be trivially solvable by taking a gradient descent step on (x_c,y_c) first".
Question 3: The linear model with hinge loss or linear loss is a convex loss and there are a variety of convex solvers that can efficiently find a minimizer to arbitrary precision, which is essentially an optimization oracle.
As such, I still have major concerns with this paper.
Thank you,
Reviewer
1.High-level ideas of the proof of Theorem 3.1
The proof of Theorem 3. 1 is illustrated in the attatched pdf file. In a MONOTONE 1-IN-3 SAT instance, the truth values of each clause and variable are stored in a parameter. The process of gradient descent converging precisely simulates the process of determining the truth values of the clauses. The loss function has different gradients across various training samples, leading to varying update magnitudes for each parameter. The gradient on the loss function is determined by the current training sample and the corresponding parameter,  more precisely, $yw^\top x$. For training samples encoding variables (e. g. , var(1)), after one training iteration, the corresponding $w_i$ will increase, causing $yw^\top x$ to jump to the gradient-free gray area in the direction indicated by the arrow. This avoids further updates to that parameter. For training samples encoding clauses (e. g.
, clause(1)), $w_i$ will be updated in the direction of the arrow from one of the green area after the first round of iteration. During this process, the parameters encoding the truth values of clauses will be 'unlocked,' allowing them to be updated in the second round of iteration. The purpose of locking these parameters in the first round of iteration is to prevent premature updates of clause truth values before the parameters of variable information is fully updated. In the second round of iteration, parameters encoding the truth values of variables will no longer be updated; only the parameters corresponding to true clauses will be updated. After the updates, for any training sample $(x,y)$, $yw^\top x$ will fall within the gray area, indicating that the parameters will no longer be updated and the model has converged.
2.Implications of the results
To the best of our knowledge, we are the first to formalize the problem and have made the initial attempts to understand its computational complexity.
Our results suggest that, the complexity of the problem is related to \emph{how the model is actually trained}. Even under extremely simple settings such as linear models trained by SGD with the hinge loss, data debugging can still be NP-hard. This implies that a ``universal debugger'' seems unlikely to exist, which takes the training data $T$ and the test instance as the input and outputs the desired $T^\prime \subseteq T$. Developing fast data debugging algorithms needs further inspecting the training data. There might also be efficient algorithms with theoretical guarantees under certain assumptions on the training data.

Meta-review:
role: PC
data: comment: value: While all reviewers highlighted the importance of the proposed problem, all reviewers also highlighted concerns about the positioning/interpretation and the settings/results. On the positioning side, there are concerns that not enough careful exposition is devoted to the interpretation of the results. More critically, the results use either a pathological loss function and a specific and non-standard weight initialization, or require a fixed (adversarial) data ordering. Given these concerns, I recommend rejection.

Decision: Reject