Title: Stabilizing Linear Passive-Aggressive Online Learning with Weighted Reservoir Sampling
Venue: NeurIPS 2024
Accepted: yes
Keywords: online learning

passive aggressive

weighted reservoir sampling

stability
Abstract: Online learning methods, like the seminal Passive-Aggressive (PA) classifier, are still highly effective for high-dimensional streaming data, out-of-core processing, and other throughput-sensitive applications. Many such algorithms rely on fast adaptation to individual errors as a key to their convergence. While such algorithms enjoy low theoretical regret, in real-world deployment they can be sensitive to individual outliers that cause the algorithm to over-correct. When such outliers occur at the end of the data stream, this can cause the final solution to have unexpectedly low accuracy. We design a weighted reservoir sampling (WRS) approach to obtain a stable ensemble model from the sequence of solutions without requiring additional passes over the data, hold-out sets, or a growing amount of memory. Our key insight is that good solutions tend to be error-free for more iterations than bad solutions, and thus, the number of passive rounds provides an estimate of a solution's relative quality. Our reservoir thus contains $K$ previous intermediate weight vectors with high survival times. We demonstrate our WRS approach on the Passive-Aggressive Classifier (PAC) and First-Order Sparse Online Learning (FSOL), where our method consistently and significantly outperforms the unmodified approach. We show that the risk of the ensemble classifier is bounded with respect to the regret of the underlying online learning method.

Stabilizing Linear Passive-Aggressive Online Learning with Weighted Reservoir Sampling

Abstract

Online learning methods, like the seminal Passive-Aggressive (PA) classifier, are still highly effective for high-dimensional streaming data, out-of-core processing, and other throughput-sensitive applications. Many such algorithms rely on fast adaptation to individual errors as a key to their convergence. While such algorithms enjoy low theoretical regret, in real-world deployment they can be sensitive to individual outliers that cause the algorithm to over-correct. When such outliers occur at the end of the data stream, this can cause the final solution to have unexpectedly low accuracy. We design a weighted reservoir sampling (WRS) approach to obtain a stable ensemble model from the sequence of solutions without requiring additional passes over the data, hold-out sets, or a growing amount of memory. Our key insight is that good solutions tend to be error-free for more iterations than bad solutions, and thus, the number of passive rounds provides an estimate of a solution's relative quality. Our reservoir thus contains K previous intermediate weight vectors with high survival times. We demonstrate our WRS approach on the Passive-Aggressive Classifier (PAC) and First-Order Sparse Online Learning (FSOL), where our method consistently and significantly outperforms the unmodified approach. We show that the risk of the ensemble classifier is bounded with respect to the regret of the underlying online learning method.

1 Introduction

Online learning algorithms are especially attractive when working with high-volume and highdimensional streaming data, out-of-core processing, and other throughput-sensitive applications [1]. For example, the seminal Vowpal Wabbit uses importance-weighted online learning algorithms [2] to reach high quality solutions quickly, with an optional second pass using LBFGS to refine the solution. The MOA library still uses the Pegasos algorithm as its linear classifier [3]. Most relevantly, online learning algorithms are particularly appealing for binary classification tasks, such as web spam classification [1]. Such algorithms often enjoy fast theoretical convergence rates due to their fast adaptation to errors on individual data points, as opposed to batch or offline learning.

However, in real-world deployment, online algorithms can be very sensitive to noisy observations in the data stream and over-correct, resulting in out-of-sample performance dropping precipitously between timesteps. Indeed, in many cases (see Figure 1), an online learning algorithm might

James Holt

Avazu (App)

News20

0.9-

1.0

0.8-

0.8

U0.7

3

0.6

PAC

0.6

PAC

0.5

Oracle

Oracle

PAC-WRS

PAC-WRS

0.4

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0

5000

10000

1e7

0.9-

0.9

0.8-

0.8

0.7

0

0.7

S

F

0.6

FSOL

0.6

FSOL

0.5

Oracle

0.5

Oracle

FSOL-WRS

FSOL-WRS

0.4

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0

5000

10000

1e7

achieve over 90% test accuracy after a given timestep, but then see its test accuracy drop by 20 -30% after over-correcting on the next observation in the data stream. In many real-world settings, it may be infeasible computationally or memory-wise to maintain a hold-out evaluation set to select the highest-performance solutions learned by our online algorithm. It may also be practically infeasible to train our classifier over multiple passes of a given dataset or when online algorithms are used for 'any-time' ready predictions.

In this paper, we introduce a weighted reservoir sampling (WRS) [4] based approach that dramatically mitigates the aforementioned accuracy fluctuations. Our proposed method, WRS-Augmented Training (WAT), neither requires a hold-out evaluation set nor additional passes over the training data. Most importantly, WAT can be used to stabilize any passive-aggressive online learning algorithm. We demonstrate the promise of our WAT method on the Passive Aggressive Classifier (PAC) [5] and First-Order Sparse Online Learning (FSOL) [1] methods - creating two new methods PAC-WRS and FSOL-WRS. Strikingly, across 16 benchmark datasets, WAT is able to mitigate test accuracy fluctuations in all 16 datasets on FSOL and 14 datasets on PAC. To analyze the theoretical effectiveness of our method, we situate our approach in the online-to-batch conversion literature, enabling us to obtain generalization bounds on i.i.d. data streams.

2 Review of related work

Our work is motivated in part by a real-world need in malware detection [6], in which large datasets make online methods particularly attractive [7, 8], and a naturally noisy labeling process inhibits standard passive-aggressive methods [9-12]. We show results for the EMBER malware benchmark in Appendix C.

Online learning. In online learning for linear binary classification, one maintains a solution vector w t ∈ R D , where D is the dimensionality of our data points (usually quite large), and t represents the timestep. At each timestep t , we observe a single observation ( x t , y t ) from a high-throughput data stream, with feature vector x t ∈ R D and class label y t ∈ { +1 , -1 } . Given ( x t , y t ) , an online learning algorithm will make a minor (potentially no effect) update to w t to output w t +1 , before receiving the next ( x t +1 , y t +1 ) in the data stream. The classification rule using w t on any test point x ∗ is simply ˆ y ∗ = sign ( w ⊤ t x ∗ ) . The goal is that as t → ∞ , the sequence of w t will enjoy

low cumulative loss. Towards this end, many online learning algorithms with various update rules have been proposed in the literature, including the Passive-Aggressive Classifier (PAC) [5], the Adaptive Regularization of Weight Vectors (AROW) methods [13] and the Adaptive Subgradient method (ADAGRAD) [14]. Some online learning algorithms have also been designed to specifically learn sparse solutions for w t (proportion of zero entries), such as Truncated Gradient method [15], Stochastic MIrror Descent Algorithm made Sparse method (SMIDAS) [16], Regularized Dual Averaging (RDA) method [17], and First-Order and Second-Order Sparse Online Learning methods (FSOL and SSOL) [1]. In general, these sparsity-inducing methods are powered by some combination of solution truncation and L1 norm minimization.

Passive-aggressive online learning. Within the family of online learning algorithms, a passiveaggressive algorithm is one whose update rule makes no update to w t if ( x t , y t ) is classified correctly with sufficient margin. That is, we passively leave w t +1 = w t . If a margin error occurs, we aggressively update w t +1 such that the error is fully correct (though a regularization penalty C will tamper the degree of aggressiveness). Usually, correct classification with sufficient margin is defined using the hinge loss ℓ -the algorithm remains passive at timestep t if ℓ ( w t ; ( x t , y t )) = max ( 1 -y t w ⊤ t x t , 0 ) = 0 . Next, we introduce the two passive-aggressive algorithms that we will use to test our WRS-Augmented Training method.

Passive-Aggressive Classifier (PAC). Introduced by Crammer et al., PAC is actually a family of three algorithms: PA, PA-I, and PA-II [5]. The base PA algorithm update rule seeks to solve the following constrained optimization problem:

Because of the hard constraint of forcing w t +1 to satisfy ℓ ( w ; ( x t , y t )) = 0 , the optimization is particularly vulnerable to noisy data. As such, Crammer et al. introduce a new constrained optimization function with a slackness hyperparameter C err to allow for some residual hinge loss and induce less aggressive, but presumably more stable updates:

where setting m = 1 corresponds to PA-I and m = 2 to PA-II. From initial testing, PA-I and PA-II performed very similarly, with a slight edge to PA-II. As such, for this paper, we will focus on PA-II, which performs the following closed-form update when in aggressive mode [5]: w t +1 = w t + ℓ ( w t ;( x t ,y t )) ∥ x t ∥ 2 + 1 2 Cerr y t x t . For the remainder of this paper, 'PAC" will refer to PA-II.

First-Order Sparse Online Learning (FSOL). Introduced by Zhao et al. [1], FSOL is a passiveaggressive algorithm which attempts to find sparse solutions for w t . Governed by a learning rate η and a sparsity parameter λ , FSOL keeps track of two vectors θ t , w t ∈ R D and performs the following update rules when in aggressive mode [1]:

where λ t = ηλ and [ v ] + takes the maximum of each element in v and 0 . Zhao et al. note that the above update rules are identical to that of Xiao's RDA method with soft 1 -norm regularization [17, 1].

Weighted Reservoir Sampling (WRS). Suppose we have a collection of items V = { v 1 , . . . , v T } , with corresponding nonnegative weights w 1 , . . . , w T . Our goal is to collect a sizeK weighted random sample from V in one pass (imagine this process is indexed by time), where the population size T = | V | is potentially unknown. Introduced by Efraimidis and Spirakis [4], weighted random sampling with a reservoir , which we shorten to weighted reservoir sampling (WRS), is an algorithm that allows us to collect such a sizeK weighted random sample under the aforementioned conditions. Specifically, as we are making our one pass through the items in V , at each timestep t , we maintain and update a reservoir -atemporary storage unit with a maximum capacity of K items, with each item in the reservoir a potential candidate for our final sizeK sample. At time T , the K items that are currently in the reservoir will constitute our sample of sizeK . We invite the interested reader to look at Algorithm A in [4] for specific details.

Online-to-batch conversion. Online learning algorithms such as PAC generally do not impose any restrictions on the distribution of the training data sequence. Their regret bounds aim to control

the cumulative loss M T of the algorithm over any sequence of data, compared with a minimal-loss fixed model ˆ w : reg T := ∑ T t =1 ℓ ( w t ; z t ) -∑ T t =1 ℓ ( ˆ w ; z t ) = M T -∑ T t =1 ℓ ( ˆ w ; z t ) . When using a fixed model to classify unseen data, we need to impose an i.i.d. assumption on the data stream in order to measure the risk , or expected generalization error, of the model. Note that the distribution D itself is arbitrary and can still permit outliers or mixtures. Then the population risk is defined as R D ( w ) := E z ∼D [ ℓ ( w ; z )] .

To theoretically describe our algorithm, we leverage work on online-to-batch conversion, which takes an online algorithm with known regret bounds on an i.i.d. sequence of data and extracts a stable final model with low risk. For example, in the online perceptron algorithm, earlier work studied the pocket approach, which selects the longest-surviving model in the sequence as the final model [18, 19]. Other well-known approaches use the uniform average of the whole model sequence or the best-performing model over a validation set [20].

As will be seen, our method generalizes these approaches to utilize multiple long-survival models as an ensemble model. Furthermore, we will introduce novel improvements, including a limitedsize reservoir with probabilistic sampling. The risk bounds for our WAT model leverage improved techniques from [21, 22]. Our experimental results also demonstrate that our novel conversion technique outperforms prior baselines (see Appendix D).

3 Our method: WRS-Augmented Training (WAT)

In one extreme, if a candidate solution w t from a passive-aggressive algorithm had perfect classification with sufficient margin on any given data point, then the subsequent number of passive steps taken after time t (i.e., number of timesteps that our algorithm is in passive mode before going aggressive again) would be infinite. In the other extreme, if a candidate solution w t had extremely low performance, then our passive-aggressive algorithm is likely to go aggressive very soon after time t , implying a very small subsequent number of passive steps after time t . In short, our key insight is that high-performing solutions w t tend to be error-free for more iterations than low-performing solutions. As such, the subsequent number of passive steps taken after the formation of w t provides an estimate of w t 's relative quality (i.e., test accuracy).

However, we do not want to take the intermediate solution w t that had the most passive updates as this, too, can be noise (and luck) sensitive. Ideally, we would like to sample from the merging distribution of w t as they occur, and take an average of those solutions to obtain a singular, highly robust, solution vector that performs well with little variance. But, we do not wish to store all w t due to intractability.

Putting these thoughts together, our WRS-Augmented Training (WAT) method functions as follows. Given a base passive-aggressive algorithm (e.g., PAC or FSOL), we will run said algorithm through our data stream { ( x t , y t ) } ⊤ t =1 , as normal, but keep a sizeK reservoir of promising candidate solutions. The reservoir approach allows us to run through our data stream and collect a weighted random sample of candidate solutions of sizeK , weighted by their subsequent number of passive steps and without storing all intermediate solutions.

Naturally, this setup is suited for Efraimidis and Spirakis's WRS algorithm. Procedurally, every timestep that our algorithm goes aggressive, we obtain a new active candidate solution. Right before we apply our aggressive mode update rule, we will add the outgoing candidate solution to our sizeK reservoir (and remove a current resident of the reservoir, if necessary) following the steps of the WRS algorithm. At any timestep t , we can form an ensemble solution w WRS by taking an average of the candidate solutions currently in our reservoir. Hopefully, at any timestep t , w WRS will have more stable test performance than the current active candidate solution w t .

3.1 WAT variants

We will experiment with a few additional variants of the WAT method. First, instead of weighting using the subsequent number of passive steps (which we denote as standard weights ), what if we weight using the exponentiated subsequent number of passive steps (which we denote as exponential weights )? The idea with exponential weights is that it is closer in practice to deterministically picking the candidate solutions with the largest number of subsequent passive steps, while still maintaining some stochasticity - i.e., a 'greedier" policy. Second, when constructing our ensemble solution

Algorithm 1 WRS-Augmented Training (WAT)

Input: w - initial solution vector, { ( x t , y t ) } T t =1 - data stream, K - reservoir size, WS - weighting scheme ('Standard" or 'Exponential"), AS - averaging scheme ('Simple Average" or 'Weighted Average"), VZ - voting-based zeroing (True or False), and other base-method-specific hyperparameters H (e.g., for PAC or FSOL)

w WRS, should we take a simple average of the residents in our reservoir or a weighted average ? Third, there are reasonable concerns that constructing w WRS via averaging might negate the sparsity advantages of a method like FSOL, due to different candidates in the reservoir containing 0 s in different entries. However, if the majority of candidate solutions in the reservoir contain 0 s at a given entry, what if we tried zeroing out said entry in w WRS, as it is likely to be uninformative? We denote this add-on as voting-based zeroing .

We present the WAT method in full detail in Algorithm 1. To clarify, the elements in the data structures R , b , and k (all of which are sizeK arrays) are paired with each other, so that when we add/remove an element in R , the corresponding elements in b and k are removed as well. To be fully clear, R contains candidate solution vectors R [1] , . . . , R [ K ] , with each R [ k ] ∈ R D . The vector b contains scalar values b [1] , . . . , b [ K ] , and likewise for the vector k .

3.2 Theoretical Analysis

One of the goals of our method is to choose a set of effective models as the algorithm runs live, without the need for expensive evaluation on a validation set. In this section we first provide validity

arguments for using observed survival as a proxy to select high-accuracy models. These are finitesample bounds based on reasoning about the reservoir R . Note that we are interested in studying generalization to unseen data, a complementary setting to prior work which shows convergence in terms of training set error for the perceptron model [19]. After establishing validity, we will turn to learning bounds for our ensemble model risk in the i.i.d. setting. Proofs are deferred to Appendix A.

̸

Validity analysis. We assume a given dataset z T 1 = { z t } T t =1 is an i.i.d. sequence sampled from a generating distribution D . We first suppose that ℓ ( w ; z ) is the 0-1 loss, that is ℓ ( w ; ( x, y )) = ✶ (ˆ y = y ) . Then the risk R D ( w ) is the probability that a random z ∼ D is misclassified by w . At any time t , we have an online model w t , for which we define the survival time s w t to be the number of subsequent correct classifications, stopping when w t misclassifies a sample. As w t does not change until an error occurs, at any finite time we only collect K updated models { w ( j ) } K j =1 into R .

Our first result bounds the difference in risk among two models in R : (1) the minimal-risk hypothesis ˜ w = arg min R D ( w ( j ) ) which we do not know, and (2) the longest-surviving hypothesis w s = arg min s w ( j ) which we observe. We also denote s ˜ w and s w s to be their respective survival times.

Theorem 1. Let w (1) , . . . , w ( k ) ∈ R be the updated outputs of an online PA algorithm on inputs Z T 1 ∼ D . Also define r m as the minimal achievable risk of any model, such that r m ≤ R D ( ˜ w ) almost surely. Then

This immediately leads to

While this bound explains the use of top-1 survival in the worst case, we further justify the use of an ensemble (or reservoir) of top-surviving models. Specifically, given the use of topB models, there is always a certain probability that the top-( B +1) can lower the risk. Since B = 1 is a base case, this implies that any value of B is worthy of consideration, until this probability decays to 0.

To simplify the notation, we sort the models in R by decreasing survival time s ′ 1 , . . . , s ′ k , with corresponding re-indexed weights w ′ 1 , . . . , w ′ k and risks R ′ 1 , . . . , R ′ k . We also add an assumption, that there is a partition of R = R g ∪ R b such that R g = { w ∈ R : r m ≤ R D ( w ) ≤ r m + ε } and R b = { w ∈ R : r m + ε < R D ( w ) ≤ r m +3 ε } , and that |R b | ≥ B . That is, there is a set of good and bad models in terms of risk. In the Appendix we show that the assumption is readily satisfied.

Theorem 2. Let ¯ R D ( w B ) be the averaged risk of the topB surviving models, and let ¯ R D ( w B +1 ) be the averaged risk including the next highest survival model. Then ¯ R D ( w B +1 ) ≤ ¯ R D ( w B ) with probability at least |R g | ( |R b | B ) r 2 m ( r m +3 ε ) |R b |-B (1 -r m -3 ε ) B .

Finally note that we can define an averaged model (weighted or unweighted) ¯ w B . For any convex ℓ h ( w ; z ) with risk R h D , such as the hinge loss, Jensen's inequality gives R h D ( ¯ w B +1 ) ≤ ¯ R D ( w B +1 ) .

Learning bounds. Now we turn to generalization bounds of our method, when run to a fixed stopping time T . We assume more generally that ℓ ( w ; z ) is convex, such as the hinge loss. Furthermore we suppose that the loss of any point in the training set is bounded by C . This is a safe assumption for many passive-aggressive algorithms, where the input data is normalized and the update steps are not too large.

Theorem 3. Suppose ℓ is convex and bounded from above by C . By time T , suppose the reservoir contains K T models, with survival time of each at least s T . Let M wrs = ∑ T t =1 π t ℓ ( w t -1 ; z t ) be the cumulative loss of the WAT reservoir sequence, with π t formally defined in the proof. Then w.p. 1 -δ ,

The result shows that the risk of the ensemble model is stochastically bounded by the cumulative loss of the online procedure. By applying known regret bounds for the underlying online algorithms (e.g. PAC or FSOL), we can further bound the risk in terms of the original learner, and subsequently

the optimal risk. In particular, we can 'abstract out' the actual online learning method in the proof, as a generic regret bound of form M T T ≤ 1 T ∑ T t =1 ℓ ( w ; z t ) + r ( T ) T , for any w . Depending on the algorithm often r ( T ) = O ( √ T ) or even O (1) .

With this in hand, we can show that the risk actually approaches the minimal risk. One additional assumption is required: M wrs / ∑ T t =1 π t ≤ 1 T ∑ t ℓ ( w t -1 ; z t ) = M T T (the WRS sequence has lower cumulative loss than the original learner). By definition, the reservoir contains the models which have longest survivals, and hence lowest regret density. So this statement is readily satisfied, as long as the distributions of non-zero losses are not different among the two sequences. In fact, under certain conditions, the inequality is likely strict which further improves the bound. Altogether we obtain:

Theorem 4. Given a PA algorithm, let M T be its cumulative loss, and r ( T ) be the algorithm-specific excess regret. Then with probability 1 -δ , the deviation in risk of our WRS model w wrs from the optimal model w ∗ is bounded as

4 Numerical experiments

We combine WAT with base PAC and FSOL, forming two new methods PAC-WRS and FSOL-WRS 1 . We evaluate their performances across 16 binary classification datasets listed in Table 1. Please see Appendix B for more dataset details. We are interested in three metrics: 1) Final test accuracy: proportion of test set data points correctly-classified by solution obtained after making one pass through the training data. 2) Final sparsity: proportion of zeroes in our classification solution obtained after making one pass through the training data. 3) Relative oracle performance (ROP): let p t, base be the test accuracy of our base model (either PAC or FSOL) at time t and p t be the test accuracy of our model of interest (either base PAC, base FSOL, PAC-WRS, or FSOL-WRS). Then, define p t, oracle = max base p t, base to be the cumulative maximum test accuracy of the base method at time t . In other words, p t, oracle represents the highest performance we could obtain if we had an oracle telling us which candidate solution vector we encountered was best. Then, define ROP = 1 T ∑ ⊤ t =1 ( p t, oracle -p t ) . Intuitively, if a method has very stable test accuracy over time with minimal fluctuations, then ROP should be close to 0 , or negative (i.e. achieving a higher test accuracy than the oracle). In contrast, a large, positive ROP suggests fluctuations in test accuracy.

Because we are interested in the stability of WAT's test-set accuracy over the entirety of the training run, it is most illustrative to look at figures when possible. Representative figures will be shown in the

Avazu (App)

MNIST8 (4+9)

News20

0.03

0.075

0.10

0.02

0.050

0.01

0.05

0.025

0.00

IHII

0.000

1III

-0.01

0.00

K=1

K=4

K=16

K=1

K=4

K=16

K=1

K=4

K=16

SimpleAverage

WeightedAverage

PAC

SimpleAverage+Voting-BasedZeroing

WeightedAverage+Voting-BasedZeroing

PAC (Mean)

main paper, with all figures for all datasets in the appendix. When running any algorithm (PAC, FSOL, PAC-WRS, or FSOL-WRS) on any dataset, we perform a random 70/30 train-test split. We begin by tuning the C err , η , and λ hyperparameters for base PAC and FSOL, with details located in Appendix B. For PAC-WRS and FSOL-WRS, we use the hyperparameters for the corresponding base models, but try all possible WAT variants of weighting scheme (standard or exponential), averaging scheme (simple vs. weighted), voting-based zeroing (True or False), and reservoir size K ∈ { 1 , 4 , 16 , 64 } . We perform five trials for each PAC-WRS and FSOL-WRS variant with randomized shuffles of the training and test data, running through the training data only once for each trial. All candidate solutions are initialized as the 0 vector. Please see more experimental details in Appendix B.

4.1 Stabilizing test performance

Figure 1 shows visually how PAC-WRS and FSOL-WRS are highly-effective at stabilizing base PAC and FSOL's wildly-fluctuating test accuracy on Avazu (App) and News20. Corresponding figures for all 16 datasets and PAC-WRS/FSOL-WRS variants can be found in Appendix G.2. From Figure 2, we see that PAC-WRS is highly-effective at reducing ROP compared to base PAC, and that the larger the reservoir size K , the more stable the resultant test accuracy for w WRS becomes. Furthermore, looking more carefully at MNIST8 (4+9), we observe that many PAC-WRS variants were able to achieve negative ROP values, suggesting that w WRS could achieve higher test accuracies than even the oracle. Corresponding figures for all 16 datasets can be found in Appendix G.1.

From Table 2, we observe that FSOL-WRS with standard weights and simple averaging successfully stabilized test accuracy in all 16 tested datasets compared to base FSOL , as measured via ROP.

PAC-WRS with exponential weights and simple averaging successfully stabilized test accuracy in 14 of 16 tested datasets compared to base PAC. Following the best practices in [38, 39], we perform Wilcoxon signed-rank tests for sta-

tistical significance on the differences in ROP between PAC/FSOL-WRS versus base PAC and FSOL, taking into account performance on all 16 datasets. At a significance level of α = 0 . 05 , we find that both PAC/FSOL-WRS achieve statistically-significant reductions in ROP compared to their corresponding base models when equipped with standard weights ( p < 0 . 0386 in each case, see Table 6 in Appendix F).

These results also suggest that standard weights are preferable to exponential weights. This makes sense because using exponential weights may be too 'greedy," causing the algorithm to overly trust in the number of passive steps as an indicator of test performance, polluting the reservoir with poor 'lucky" candidate solutions, and refusing to remove them later.

Avazu (Site)

KDD2010(Algebra)

SST-2

0.80

0.875

0.850

0.84

0.78

0.825

0.82

0.76

0.800

0.74

0.80

0.775

0.72

0.750

0.78

FSOL

K=1

K=4

K=16K=64

FSOLK=1

K=4

K=16K=64

FSOLK=1

K=4

K=16K=64

SimpleAverage

WeightedAverage

Oracle

FSOL

SimpleAverage+Voting-Based Zeroing

WeightedAverage+Voting-BasedZeroing

Oracle (Mean)

A note on final test accuracy. From Figure 3, we see that the final test accuracies achieved by FSOL-WRS are not only higher on average than base FSOL, but also have significantly lower variance. On these datasets, there is no significant difference between FSOL-WRS variants with or without voting-based zeroing. As expected, final test accuracy seems to be higher for larger values of reservoir size K . However, Wilcoxon signed-rank tests to compare PAC-WRS and FSOL-WRS's improvements in final test accuracy over base PAC and FSOL indicate that FSOL-WRS with standard weights yields a statistically-significant improvement in final test accuracy compared to base FSOL, but PAC-WRS does not compared to base PAC (see Table 7 in Appendix F). One hypothesis for this discrepancy is that it is relatively unlikely for any given training data point to cause a massive drop in test accuracy. Furthermore, subsequent data points will usually help the base model self-correct (see Appendix G.2). Thus, this explains why the base method's mean final test accuracy after a particular fixed timestep (e.g., N train) will not differ significantly from that of the WRS-augmented method, especially not across only 5 trials.

Nonetheless, there are enough data points in the training stream that these massive fluctuations in test accuracy could still happen thousands of times throughout the training process, corroborating what we saw in Figure 1. If such a drop in test accuracy were to occur at an unlucky timestep when the model training is stopped, the consequences could be unacceptable. Furthermore, in continuously updated 'any-time' environments, it is hard to assess if one of these drops has occurred. As such, WAT is valuable as a simple and effective way of preventing such fluctuations in test accuracy.

Avazu (Site)

KDD2010(Algebra)

SST-2

0.9995

II1

0.7

0.982

I1

0.9990

0.981

0.6

I1

0.9985

0.980

一

0.9980

0.5

0.979

0.9975

亚亚

王王

0.978

0.4

亚亚

FSOL

K=1

K=4

FSOL

K=1

K=4

FSOL

K=1

K=4

SimpleAverage

WeightedAverage

FSOL

SimpleAverage+Voting-Based Zeroing

WeightedAverage+Voting-BasedZeroing

Preservation of sparsity. Finally, we aim to retain the favorable weight sparsity from FSOL. From Figure 4, we observe that even with reservoir size K = 64 , the final sparsities achieved by FSOL-WRS are quite similar to that of base FSOL. Furthermore, we observe that with voting-based zeroing, FSOLWRS often achieves even higher sparsity than the base model. Please see Appendix G.1 for additional

errorbar-type plots and Appendix G.2 for sparsity over timestep. One explanation for why even FSOL-WRS variants without voting-based zeroing can achieve similar sparsity to base FSOL is that the final reservoir will likely contain candidate solutions from significantly earlier timesteps, when weights were sparser due to fewer aggressive updates. Thus, WAT can maintain sparsity despite the use of weight averaging.

4.2 Ablations and extensions

TopK ablation. A natural question that one might ask is - instead of probabilistically sampling candidate solutions, what if we deterministically picked the topK candidate solutions with the largest subsequent numbers of passive steps and took their simple and/or weighted average? Setting K = 64 , we see from Table 3 that the best top64 PAC/FSOL variants are not as effective as stabilizing test accuracy as the best PAC/FSOL-WRS variants, as shown in Table 2 and measured via ROP. Furthermore, Wilcoxon signed-rank test p -values from Table 4 also suggest that this top64 ablation cannot produce as statistically-significant increases in test accuracy stability compared to WAT, as measured via ROP. Like WAT, topK is also ineffective at producing statistically-significant increases in final test accuracy on PAC. On FSOL, topK can produce statistically significant increases in final test accuracy, but these p -values are over an order of magnitude larger than FSOL-WRS's (see Table 7 in Appendix F). As such, WAT is still the best for stabilizing test accuracy.

Additional ablations and comparisons. For the interested reader, in Appendix D we include empirical performance comparisons of WRS-Augmented Training against two traditional ensembling mechanisms: moving average (e.g., averaging the most-recently-observed K = 64 weight vectors at each timestep) and exponential average (e.g., forming an ensemble vector ¯ w t = γ w t +(1 -γ ) ¯ w t -1 at each timestep), where w t is the base algorithm's candidate solution at timestep t . In short, especially in more real-world, large-scale settings where evaluation and checking are prohibitively expensive, WRS-Augmented Training is the fastest, most accurate, and most reliable method compared to all the aforementioned baselines.

Modifying WAT for non-passive aggressive methods. While the main theoretical and empirical results in this paper were primarily oriented towards passive-aggressive base models, in Appendix E, we include empirical simulations of applying a modified form of WRS-Augmented Training on top of three non-passive-aggressive online learning methods: Stochastic Gradient Descent with Momentum [40], ADAGRAD [14], and Truncated Gradient Descent [15]. In general, our modified WRS-Augmented Training effectively mitigates test accuracy when it exists, and does minimal harm when it does not.

5 Conclusion, limitations, and future work

In this paper, we introduced WRS-Augmented Training (WAT), a procedure that can be used to stabilize any passive-aggressive online learning algorithm, neither requiring a hold-out evaluation set nor additional passes over the training data. We applied WAT to base PAC and FSOL, demonstrating across 16 datasets that WAT is highly effective at mitigating the massive fluctuations in test accuracy between timesteps that affect many online learning algorithms. WAT runs at minimal cost, with only a fixed K multiple on memory for multiple weight vectors to be saved.

One limitation of this work is that WAT implicitly assumes that the training and test data come from fixed distributions that do not change over time. However, some data distributions will evolve over time. As such, a candidate solution that entered the reservoir early on due to having a large number of subsequent passive steps might not actually retain its performance as the data distribution evolves over time. While they may eventually be replaced, non-IID adaptions may be useful in the future.

References

Peilin Zhao, Dayong Wang, Pengcheng Wu, and Steven CH Hoi. A Unified Framework for Sparse Online Learning. ACM Transactions on Knowledge Discovery from Data (TKDD) , 14(5):1-20, 2020.

Nikos Karampatziakis and John Langford. Online Importance Weight Aware Updates, 2011.

Albert Bifet, Geoff Holmes, Bernhard Pfahringer, Philipp Kranen, Hardy Kremer, Timm Jansen, and Thomas Seidl. Moa: Massive online analysis, a framework for stream classification and clustering. In Proceedings of the first workshop on applications of pattern analysis , pages 44-50. PMLR, 2010.

Pavlos S Efraimidis and Paul G Spirakis. Weighted Random Sampling with a Reservoir. Information Processing Letters , 97(5):181-185, 2006.

Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. Online Passive-Aggressive Algorithms. Journal of Machine Learning Research , 7(19):551-585, 2006.

Edward Raff, Bobby Filar, and James Holt. Getting Passive Aggressive About False Positives: Patching Deployed Malware Detectors. In 2020 International Conference on Data Mining Workshops (ICDMW) , pages 506-515. IEEE, nov 2020.

H. S. Anderson and P. Roth. EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. ArXiv e-prints , April 2018.

Richard Harang and Ethan M. Rudd. Sorel-20m: A large scale benchmark dataset for malicious pe detection. In Proceedings of the Conference on Applied Machine Learning for Information Security , 2021.

Robert J Joyce, Edward Raff, Charles Nicholas, and James Holt. MalDICT: Benchmark Datasets on Malware Behaviors, Platforms, Exploitation, and Packers. Proceedings of the Conference on Applied Machine Learning in Information Security , 2023.

Robert J Joyce, Edward Raff, and Charles Nicholas. A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels. In Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (AISec '21) . Association for Computing Machinery, 2021.

Robert J Joyce, Edward Raff, and Charles Nicholas. Rank-1 Similarity Matrix Decomposition For Modeling Changes in Antivirus Consensus Through Time. In Proceedings of the Conference on Applied Machine Learning for Information Security , 2021.

Robert J. Joyce, Tirth Patel, Charles Nicholas, and Edward Raff. A VScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , AISec '23, page 185-196, New York, NY, USA, 2023. Association for Computing Machinery.

Koby Crammer, Alex Kulesza, and Mark Dredze. Adaptive Regularization of Weight Vectors. Machine learning , 91:155-187, 2013.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research , 12(7), 2011.

John Langford, Lihong Li, and Tong Zhang. Sparse Online Learning via Truncated Gradient. Journal of Machine Learning Research , 10(3), 2009.

Shai Shalev-Shwartz and Ambuj Tewari. Stochastic Methods for L1 Regularized Loss Minimization. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 929-936, 2009.

Lin Xiao. Dual Averaging Method for Regularized Stochastic Learning and Online Optimization. Advances in Neural Information Processing Systems , 22, 2009.

Yoav Freund and Robert E Schapire. Large Margin Classification Using the Perceptron Algorithm. In Proceedings of the eleventh annual conference on Computational learning theory , pages 209-217, 1998.

Marco Muselli. On convergence properties of pocket algorithm. IEEE Transactions on Neural Networks , 8(3):623-629, 1997.

Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the Generalization Ability of On-line Learning Algorithms. IEEE Transactions on Information Theory , 50(9):2050-2057, 2004.

Nicolo Cesa-Bianchi and Claudio Gentile. Improved Risk Tail Bounds for On-line Algorithms. IEEE Transactions on Information Theory , 54(1):386-390, 2008.

Ofer Dekel. From Online to Batch Learning with Cutoff-Averaging. Advances in Neural Information Processing Systems , 21, 2008.

Steve Wang and Will Cukierski. Click-Through Rate Prediction. Kaggle. https://kaggle.com/competitions/avazu-ctr-prediction, 2014.

Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Dexter. UCI Machine Learning Repository, 2008. DOI: https://doi.org/10.24432/C5P898.

Isabelle Guyon, Steve Gunn, Asa Ben-Hur, , and Gideon Dror. Dorothea. UCI Machine Learning Repository, 2008. DOI: https://doi.org/10.24432/C5NK6X.

J. Stamper, A. Niculescu-Mizil, S. Ritter, G.J. Gordon, and K.R. Koedinger. Algebra I 20082009. Challenge Data Set from KDD Cup 2010 Educational Data Mining Challenge. Find it at http://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp, 2010.

Jingwei Liang and Clarice Poon. Screening for Sparse Online Learning. arXiv preprint arXiv:2101.06982 , 2021.

Gaëlle Loosli, Stéphane Canu, and Léon Bottou. Training Invariant Support Vector Machines using Selective Sampling. In Léon Bottou, Olivier Chapelle, Dennis DeCoste, and Jason Weston, editors, Large Scale Kernel Machines , pages 301-320. MIT Press, Cambridge, MA., 2007.

S Sathiya Keerthi, Dennis DeCoste, and Thorsten Joachims. A Modified Finite Newton Method for Fast Solution of Large Scale Linear SVMs. Journal of Machine Learning Research , 6(3), 2005.

Ken Lang. Newsweeder: Learning to Filter NetNews. In Proceedings of the Twelfth International Conference on Machine Learning , pages 331-339, 1995.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature Selection: A Data Perspective. ACM Computing Surveys (CSUR) , 50(6):94, 2018.

David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. Rcv1: A New Benchmark Collection for Text Categorization Research. Journal of machine learning research , 5(Apr):361397, 2004.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1631-1642, 2013.

Justin Ma, Lawrence K Saul, Stefan Savage, and Geoffrey M Voelker. Identifying Suspicious urls: An Application of Large-Scale Online Learning. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 681-688, 2009.

John C Platt. Fast Training of Support Vector Machines Using Sequential Minimal Optimization. 1998.

Steve Webb, James Caverlee, and Calton Pu. Introducing the Webb Spam Corpus: Using Email Spam to Identify Web Spam Automatically. In CEAS , 2006.

Alessio Benavoli, Giorgio Corani, and Francesca Mangili. Should We Really Use Post-Hoc Tests Based on Mean-Ranks? Journal of Machine Learning Research , 17(5):1-10, 2016.

Janez Demšar. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research , 7(1):1-30, 2006.

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , pages 1139-1147. PMLR, 2013.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

A Proofs

A.1 Validity analysis

Proof of Theorem 1 . The main statement involves two independent bounds. See the proofs for Theorem 5 and Theorem 6.

Theorem 5. Let w (1) , . . . , w ( k ) ∈ R be the updated outputs of an online PA algorithm on inputs z T 1 ∼ D . Define w s and ˜ w as previously. Also define r m as the minimal achievable risk of any model, such that r m ≤ R D ( ˜ w ) almost surely.

Then

Proof. First note that while R D ( w ) is a constant value for any fixed w , R D ( w s ) and R D ( ˜ w ) are random variables induced by the distribution of Z T 1 . The sequence of observed samples affects both the algorithm's output w as well as the subsequent survival of w .

Next, we define the event of interest A = { R D ( w s ) > R D ( ˜ w ) + ε } . For a given realization of Z T 1 let the set R ε = { w ∈ R : R D ( w ) > R D ( ˜ w ) + ε } . Then

We aim to split this into separate events and apply a union bound, but the contents and size of R depend on Z T 1 .

Therefore we enumerate the models within each R ε in the order they are received: { w (1) ε , w (2) ε , . . . } , up to w ( K ) ε , enabling the index i to refer to the corresponding model w ( i ) ε in any realization of Z T 1 . Then we can split the event that a hypothesis in the reservoir has longer survival into the union of events on each ordered hypothesis.

We have

Applying the union bound, Lemma 1, and then Lemma 2,

To further clarify, the probabilities are computed over the probability space of Z T 1 ∼ D , and ˜ w depends on the realization of Z T 1 . Lemma 1 allows us to reason about the dependent model instances as independent geometric variables, but we need to extract ˜ w into a instance-independent state. Thus we need to condition on the weight values, and then bound the probability uniformly across the expectation, using the premise that the weight risks are ε -distant.

Theorem 6. (Alternative bound) Let the same conditions hold as in Theorem 5. Then

Proof. As before, let A = { R D ( w s ) > R D ( ˜ w ) + ε } . Then

̸

The event { w s = ˜ w } c implies ˜ w had or was tied for the highest survival, and we make the simplification that ties are resolved in favor of ˜ w . Then

where in the third step, we apply Lemma 3. In the last step, we use the premise to suppose that there is at least one hypothesis which is ε -worse than ˜ w .

Considering the first-term only,

Remark. Note that this gives the worst-case bound presuming no knowledge over the distribution of risks among the K hypotheses. In such a scenario, the worst case is that all hypotheses achieve R D ( ˜ w ) except for one which is R D ( ˜ w ) + ε . This is unrealistically adversarial, but the bound can easily be tightened by adding assumptions that the risks R D ( w ( j ) ) are well-behaved and applying those in the last step.

Lemma 1. Let R D ( w ) be the misclassification risk. Suppose an online learner outputs hypothesis w t at time t after observing samples Z t 1 . Let s t be the subsequent survival time of w t , conditional on the observed sequence Z t 1 . Also let S ( w t ) be the survival of w t over the distribution of sample sequences from D . Then S ( w t ) has a Geometric distribution with parameter R ( w t ) . In addition s t d = S ( w ) .

Proof. For the first statement, observe that when sampling z independently from D ,

̸

R ( w t ) = E D [ ✶ ( w t ( x ) = y )] = P ( x,y ) ∼D ( w t ( x ) = y ) Since each i.i.d. sample has a probability R ( w t ) to be misclassified, the number of trials until the first misclassification is a Geometric distribution with parameter R ( w t ) .

̸

For the second, we need to consider a sequence of correct/incorrect classifications by w t as a Bernoulli process of r.v. W i with parameter R ( w t ) , with a stopping time τ indicated by the first misclassification. τ is a random variable and we see that S t = τ -t . Similarly S ( w t ) can be defined on another Bernoulli process starting from t = 0 . From the memoryless property, these have the same distribution.

Lemma 2. Given random variables X and Y which are independent Geometric with parameters p and q respectively (e.g. PMF of X is P ( X = k ) = (1 -p ) k -1 p , then

and

Proof. For the first statement,

Lemma 3. Let Y 1 , . . . , Y t be independent Geometric random variables with respective parameters q 1 , . . . , q t . Let X ∼ Geom( p ) be independent. Then

For the second, note that

Proof.

by independence. Substitute the PMF and CMFs of the geometric distribution to finish.

Proof of Theorem 2 .

Proof. Note that

We now consider a subset of this event. We will compute the probability that R ′ 1 , . . . , R ′ B ∈ R b and that R ′ B +1 ∈ R g . By definition, this latter event falls into A .

For this to occur, note from Lemma 1 that each survival is drawn from a Geometric distribution. We want the chance that the B largest survivals are from set R b and then the next largest is from R g . For additional simplicity, we further minimize the probability of this event by assigning all entries in R g risk r m and all entries in R b risk r m +3 ε .

Conditioning on s ′ B +1 = x , all entries from R g are below x except for one, which is equal to x . There are |R g | ways to pick this largest one.

Then |R| B of the models in R b are also below x , and B are above x . There are ( |R b | B ) ways to choose such B .

Finally this probability is summed over all values of x . This is

As before, to get a simpler expression, we just consider x = 1 , yielding

We also discuss the additional assumption of this Theorem to show that it is readily satisfied.

Assumption. There is a partition of R = R g ∪ R b such that R g = { w ∈ R : r m ≤ R D ( w ) ≤ r m + ε } and R b = { w ∈ R : r m + ε < R D ( w ) ≤ r m +3 ε } , and that |R b | ≥ B . That is, there is a set of good and bad models in terms of risk.

We can set ε > 0 to be any small number that satisfies a partition within the reservoir separating "good" from "bad" models - e.g. ε = (max k R ′ k -r m ) / 4 works. For demonstration, suppose the

underlying risks of the models in the reservoir range from 0.1 to 0.3, with 0.1 being the best risk r m . Then ε = 0 . 05 . R g then contains the models with risks in [0 . 1 , 0 . 15] and R b contains those with risks in (0 . 15 , 0 . 3] . The only time this isn't satisfied for any ε is if one of the partitions always ends up empty, which only happens if all the K models have the same risk, a highly unlikely scenario where it is obvious that an ensemble isn't needed.

A.2 Learning bounds

Our bound follows from the following lemma from [22]. This itself is based on martingale inequalities from [41] and developments in [21].

Lemma 4 (Dekel) . Let L 1 , . . . , L T be a sequence of real-valued random variables and let Z 1 , . . . , Z T be a sequence of random variables such that L t = E [ L t | Z 1 , . . . , Z t ] . Also assume that L t ∈ [0 , C ] for all t . Define U t = E [ L t | Z 1 , . . . , Z t -1 ] , and let ˜ L t = ∑ t i =1 L i . Then for any T ≥ 4 and δ ∈ (0 , 1) ,

From this we obtain our learning bound. The proof follows analogously to the one in [22], with modifications to handle our specific algorithm.

Proof of Theorem 3 .

Proof. In our method we have a reservoir R containing K models w with associated weights b w . We focus on the case where the weight is equal to the passive steps survived by the model. To simplify this proof we consider a slightly modified algorithm, where all models with survival exceeding a threshold s are included in the reservoir.

We rewrite the weighted average of the reservoir models as a function of the entire training sequence of length T by defining π t to be 1 if w t is in R and 0 otherwise. Furthermore on the first instance that w t joins R , the π t is instead set to s . This makes the weighting deterministic with respect to the information at time t -1 , at the cost of the reservoir size not being explicitly defined. While this is analogous to a top-k averaging scheme, the full WRS scheme will follow from the same reasoning, with a more complicated definition of π t . This is because all decision-making is performed using information that has already been seen.

Then we write w wrs = 1 ∑ T t =1 π t ∑ T t =1 π t w t , and our goal is to control R D ( w wrs ) . Furthermore, define the cumulative loss as M wrs = ∑ T t =1 π t ℓ ( w t -1 , z t ) . (We note that this term is a little different from the observed cumulative WRS loss because of the requirement of reweighting on the s -th survival.)

In the lemma, let L t = π t ℓ ( w t -1 , z t ) and let U t = E [ π t ℓ ( w t -1 , z t ) | z t -1 1 ] . Noting that L t is deterministic when conditioning on z t 1 , the assumptions are met. Then with probability 1 -δ , we

Now ∑ T i =1 U i = ∑ T t =1 π t E [ ℓ ( w t -1 , z t ) | z t -1 1 ] = ∑ T t =1 π t R D ( w t -1 ) , where the first equality uses the fact that π t is determined by the filtration F t -1 , and the second similarly uses the conditioning information to fix w t -1 and the i.i.d. assumption over z .

Finally, we have R D ( w wrs ) = R D ( 1 ∑ T t =1 π t ∑ T t =1 π t w t ) ≤ 1 ∑ T t =1 π t ∑ T t =1 π t R D ( w t ) from Jensen's inequality.

Dividing the previous statement by ∑ T t =1 π t , we put the pieces together to conclude

Finally, note that ∑ t π t is the total time steps involved with the reservoir weights, which is at least K T s T with K T weights and s T the minimal survival time of the reservoir. Substituting completes the proof.

Proof of Theorem 4 .

Proof. Combining the penultimate form of Theorem 3 (with ∑ t π t instead of K T s T ) and the deduction that M wrs / ∑ t π t ≤ M T /T we have

Next, we turn our attention to the generic base-model online regret bound. From this we get M T /T ≤ 1 T ∑ t ℓ ( w ∗ ; z t ) + r ( T ) /T .

We need to bound the cumulative loss of w ∗ , that is 1 T ∑ t ℓ ( w ∗ ; z t ) , in terms of the risk: R D ( w ∗ ) . Using an application of Hoeffding bound, we get

which holds with probability 1 -δ .

Combining the undesired tail events under the union bound, we get

with probability 1 -2 δ . As before, substitute in K T s T for ∑ t π t to complete.

Remark. The use of the online regret bound also indicates that the base models w t are improving over time. This means that their risks are decreasing, and hence the observed survival times are increasing. Thus even in the error term without T in the denominator, s T is an increasing function of T so we expect the error terms to decay in T .

B Additional details on methods and experimental setups

B.1 Datasets

Of the 16 datasets used in our study (see Table 1), 14 are directly from existing repositories and/or literature, while 2 of them - Newsgroups (Binary, CS) and SST-2 - were modified from pre-existing assets. First, Newsgroups (binary, CS) was formed by taking the original multi-class Newsgroups dataset hosted in [31] and combining the labels 'Computers" and 'Science" as a +1 class, and all other labels as the -1 class. Second, SST-2 is a sentiment classification dataset originally containing text excerpts that we transformed into a linear classification dataset by using the CountVectorizer from [31].

B.2 Base PAC and FSOL hyperparameter tuning

We start our set of experiments by tuning the C err hyperparameter for base PAC, testing values in the set 10 -3 , 10 -2 , . . . , 10 2 , 10 3 . For base FSOL, following Zhao et al.'s approach, we test η values in 2 -3 , 2 -2 , . . . , 2 8 , 2 9 and λ values in 0 , 10 -3 , 10 -2 , . . . , 10 2 , 10 3 . For each dataset, we perform five runs of our base PAC and FSOL variants with randomized shuffles of the training and test data, running through the training data only once for each run. For each dataset, we pick the C err , η , and λ values corresponding to the base PAC and FSOL variants, within the top 2.5% in terms of final test accuracy, that had the highest ROP, with all metrics averaged across the five runs. This experimental choice was made to strike a balance between simulating as difficult and risky conditions as possible and still choosing useful base model variants.

B.3 Metrics logging

Given that some of our datasets contain up to 36 million training points, it would be computationally and storage-wise unfeasible to record metrics at each timestep. As such, for each dataset, we record metrics at ≈ 200 evenly-spaced timesteps (exact number depends on divisibility and integer division of N train by 200 ) throughout the training stream, in addition to the initial and final timesteps. As such ROP was also approximated by proxying oracle accuracy by taking the cumulative maximum base model test accuracy across timesteps where metrics were recorded. As shown in the over-time figures in Appendix G.2, such a reduced resolution still tells us a very clear picture, while cutting down computation and storage requirements by many orders of magnitude.

B.4 Compute requirements and code availability

All experiments were run on a Linux computing cluster with 32 nodes, each with 40 Intel Xeon E5-2650 CPU cores and a total of 500 GB of RAM per node, managed using SLURM. Nonetheless, no experiments require multiprocessing or multiple cores. Depending on dataset size, some trials could take less than a minute to run, while the largest datasets would take a couple hours at max. However, larger datasets like Criteo will require 32 GB of RAM to load the dataset into memory.

Our source code for reproducing all experiments can be found at https://github.com/FutureComputing4AI/Weighted-Reservoir-Sampling-Augmented-

Training/tree/main. All experiments were run on CPU.

C Applications to Malware Classification

Given that the original motivation for this work was a real-world need in malware detection, we present empirical results of PAC, PAC-WRS, FSOL, and FSOL-WRS on the EMBER malware classification dataset [7]. The EMBER dataset is comprised of features extracted from 1.1M benign and malicious Windows portable executable files and thus can be considered very close to a real-world test case of our WRS-Augmented Training method on a real-world deployment setting.

In Figure 5, we observe that both base PAC and FSOL experience significant and frequent test accuracy fluctuations throughout the training process. However, both PAC-WRS and FSOL-WRS very successfully mitigate these test accuracy fluctuations and even outperform the oracle model, all without the use of a separate validation set.

PAC + PAC-WRS

FSOL+FSOL-WRS

2

0.95

0.95

Test Accurac

0.90

0.90

0.85

PAC

0.85

FSOL

PAC-WRS

FSOL-WRS

Oracle

Oracle

0.80

0.80

0

100000

200000

300000

400000

0

100000

200000

300000

400000

Timestep

Timestep

Indeed, WRS-Augmented Training is a promising tool towards stabilizing test accuracy performance in real-world deployment settings like malware detection.

D Comparisons to other ensembling and averaging schemes

One natural question a reader may ask is: how does WRS-Augmented Training compare to traditional weight-averaging schemes such as moving average (e.g., averaging the most-recently-observed K = 64 weight vectors at each timestep) or exponential average (e.g., forming an ensemble vector ¯ w t = γ w t +(1 -γ ) ¯ w t -1 at each timestep), where w t is the base algorithm's candidate solution at timestep t .

From Figures 6 and 7, we observe that the exponential average scheme is consistently ineffective at mitigating the test accuracy instabilities of the base models PAC and FSOL. The stability of the exponential average scheme is usually not much better than that of the base model, which makes sense because it still puts the majority of the weighting on the most recent candidate solution. From these figures, we also observe that the K = 64 moving average also has very mixed effectiveness. Overall, we see that WRS-Augmented Training is more preferable to the K = 64 moving average. This makes sense because with WRS-Augmented Training, we are much more selective about the quality of the candidate solutions that we retain in our reservoir, compared to moving average, which necessarily by definition must include poor-performing solutions as they appear.

Third, from Table 5, we see that for datasets with dimension D > 100 K , the moving average method can be significantly computationally slower per iteration than WRS-Augmented Training. For example, with PAC as the base model, the moving average method was, on average, 6 . 579 x slower per iteration than WRS-Augmented Training on KDD2010 (Algebra) and 10 . 050 x slower per iteration than WRS-Augmented Training on URL. Similar trends hold when using FSOL as the base algorithm. These results make sense because with WRS-Augmented Training, we do not always add candidate solutions to our reservoir for averaging, while with the moving average method, we must always add new candidate solutions into our set. These insertion and deletion costs will accrue over time. On smaller datasets, the moving average is faster or slower depending on the dataset, but runtime is dominated by IO and all methods finish within minutes. However, in more real-world, large-scale settings where evaluation and checking are prohibitively expensive, WRS-Augmented Training is the fastest, most accurate, and most reliable method compared to all the baselines.

Avazu(App)

Avazu (Site)

Criteo

0.9

0.84

1.0

0.82

0.82

0.9

0.8

0.80

0.80

0.8 -

0.7

0.78

0.78

0.7

0.76

0.76

0.6

0.6

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

0

1

2

3

0

100

200

300

400

1e7

1e7

1e7

MNIST8 (4+9)

News20

0.92

1.0

0.950

0.95

0.9

0.925

0.90

0.90

0.8

0.900

0.88

0.875

0.85

0.7

0.86

0.850

0.80

0.6

0

200

400

600

800

0

2

4

6

0.000.250.500.75

1.00

0

5000

10000

1e6

1e6

Newsgroups(Binary,CS)

PCMAC

RCV1

0.9

1.0

0.9

1.00

0.8

0.95

0.9

0.8

0.7

0.90

0.8

0.6

0

5000

10000

0.7

0

500

1000

0.85

0

200000

400000

0

20000

40000

SST-2

URL

W8A

Webspam

0.9

0.95

1.00

1.00

0.8

0.90

0.7

0.95

0.95

0.85

0.6

0.90

0.90

0.80

0

20000

40000

0.0

0.5

1.0

1.5

0

10000200003000040000

0

100000

200000

1e6

WRS-Augmented Training

Moving Avg.

PAC

WRS-Augmented Training (For Emphasis)

Expo.Avg.

Avazu(App)

Avazu (Site)

Criteo

0.9

0.84

0.82

0.82

0.9

0.8

0.80

0.80

0.8

0.7

0.78

0.78

0.7

0.76

0.76

0.6

0.6

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

0

1

2

3

0

100

200

300

400

1e7

1e7

1e7

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.90

0.95

0.95

0.9

0.85

0.90

0.8

0.90

0.80

0.85

0.7

0.85

+

0

200

400

600

800

0.75

0

2

4

6

0.80

0.000.250.500.751.00

0.6

0

5000

10000

1e6

1e6

Newsgroups(Binary,CS)

PCMAC

RCV1

0.9

0.90

1.00

1.0

0.8

0.85

0.95

0.9

0.80

0.7

0.90

0.75

0.8

0.6

0.70

0.85

0

5000

10000

0

500

1000

0

200000

400000

0

20000

40000

SST-2

URL

W8A

Webspam

1.0

0.95

1.000

0.8

0.975

0.9

0.90

0.8

0.950

0.7

0.925

0.7

0.85

0.6

0

20000

40000

0.900

0.0

0.5

1.0

1.5

0.6

0

10000200003000040000

0.80

0

100000

200000

1e6

WRS-AugmentedTraining

Moving Avg.

FSOL

WRS-Augmented Training (For Emphasis)

Expo. Avg.

E Modified WRS-Augmented Training on non-passive aggressive online learning methods

In this section, we explore applying a modified form of WRS-Augmented Training (WAT) on top of three non-passive-aggressive online learning methods: Stochastic Gradient Descent with Momentum [40], ADAGRAD [14], and Truncated Gradient Descent [15], all using default hyperparameters. We emphasize that these three methods are not passive-aggressive as they always update their weight vectors at each time step, even if the data point was correctly classified. As such, the original WAT is not directly applicable.

However, we can modify WAT as follows. First, we define a pseudo-passive step as one where the current solution candidate made no classification error (i.e., no more concept of margin). Second, at a time step when the current solution candidate does make an error, we will sample the last weight vector before a mistake was made into our reservoir with probability proportional to the number of pseudo-passive steps, before resetting our counter. The rest of WAT operates as normal under this pseudo-passive step weighting.

From Figures 8 - 10, we observe that all three non-passive-aggressive online algorithms, in particular Stochastic Gradient Descent with Momentum (SGD+M) and Truncated Gradient Descent (TGD), are susceptible to experiencing concerning fluctuations in test accuracy. Consistently, across the algorithms, we observe that when such fluctuations are present, modified WAT effectively mitigates such fluctuations very well. On the other hand, when there is little fluctuation in the base model (e.g., see ADAGRAD), applying modified WAT will do little to no harm. We emphasize that WAT was not designed for non-passive-aggressive algorithms, but is still demonstrably useful and adaptable to a wider class of base models, which can constitute fruitful future work.

Avazu (App)

Avazu (Site)

Criteo

0.88

0.80

0.80

0.9

0.86

0.75

0.75

0.8

0.84

0.70

0.70

0.82

0.65

0.65

0.7

0.80

0.0

0.2

0.4

0.6

0.8

1.0

0.60

0.0

0.5

1.0

1.5

0.60

0

1

2

3

0.6

0

100

200

300

400

1e7

1e7

1e7

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.95

0.95

0.90

0.85

0.90

0.9

0.85

0.8

0.80

0.80

0.85

0.7

0.75

0.75

0.6

0

200

400

600

800

0

2

4

6

0.00

0.25

0.50

0.75

1.00

0

5000

10000

1e6

1e6

Newsgroups(Binary,CS)

PCMAC

RCV1

0.9

0.97

0.95

0.8

0.8

0.96

0.95

060

0.7

0.7

0.94

0.6

0.6

0.93

0.85

Q

2500 5000 7500 1000012500

0

250 500 75010001250

010000Q0000G000040000G00000°

01000020000300004000050000

SST-2

URL

W8A

Webspam

0.98

0.98

0.8

0.96

0.96

0.90

0.94

0.94

0.7

0.85

0.92

0.92

0.6

0

100002000030000 40000

0.90

0.0

0.5

1.0

1.5

0.90

0

10000200003000040000

0.80

05000010000Q5000Q0000Q50000

1e6

WRS-Augmented Training

WRS-Augmented Training (For Emphasis)

SGD+Momentum

Avazu(App)

Avazu (Site)

Criteo

0.78

0.80

0.95

0.882

0.77

0.90

0.78

0.85

0.881

0.76

0.76

0.80

0.880

0.75

0.75

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

1

2

3

0

100

200

300

400

1e7

1e7

1e7

MNIST8 (4+9)

News20

0.95

0.88

0.945

0.9

0.90

0.87

0.940

0.8

0.85

0.80

0.86

0.935

0.7

0.75

0.85

0.930

0.6

0

200

400

600

800

0

2

4

6

0.000.250.500.75

1.00

0

5000

10000

1e6

1e6

Newsgroups(Binary,CS)

PCMAC

RCV1

0.9

0.9

0.97

0.95

0.8

0.96

0.8

0.7

0.7

0.95

0.90

0.94

0.6

0.6

0.93

0.85

0

5000

10000

0

500

1000

200000

400000

0

20000

40000

SST-2

URL

W8A

Webspam

0.985

0.900

0.8

0.98

0.980

0.875

0.7

0.97

0.975

0.850

0.825

0.6

0.96

0.970

0.800

0

20000

40000

0.0

0.5

1.0

1.5

0

10000200003000040000

0

100000

200000

1e6

WRS-AugmentedTraining

WRS-Augmented Training (For Emphasis)

AdaGrad

Avazu(App)

Avazu (Site)

Criteo

0.88

0.80

0.9

0.770

0.87

0.78

0.765

0.8

0.760

0.86

0.7

0.76

0.755

0.85

0.750

0.6

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

0

1

2

0

100

200

300

400

1e7

1e7

1e7

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.9

0.86

0.8

0.9

0.8

0.84

0.8

0.7

0.7

0.82

0.7

0.6

0.80

0.6

0.6

0

200

400

600

800

0

2

4

6

0.000.250.500.751.00

0

5000

10000

1e6

1e6

Newsgroups(Binary,CS)

PCMAC

RCV1

0.92

0.8

0.8

0.90

0.90

0.85

0.7

0.7

0.88

0.86

0.80

0.6

0.6

0

5000

10000

0

500

1000

0

200000

400000

0

20000

40000

SST-2

URL

W8A

Webspam

0.9

0.98

0.70

0.8

0.65

0.8

0.97

0.7

0.60

0.7

0.96

0.55

0.6

0.6

0.95

0.50

0

20000

40000

0.0

0.5

1.0

1.5

0

10000200003000040000

0

100000

200000

1e6

WRS-Augmented Training

WRS-Augmented Training(For Emphasis)

Truncated Gradient Descent(TGD)

F Tables of p -values for Wilcoxon Signed-Rank Tests

Below, we provide full tables of p -values computed using Wilcoxon Signed-Rank Tests towards probing the statistical significance of the differences in ROP and final test accuracy between PAC/FSOLWRS and their base model counterparts.

Table 6: Wilcoxon signed-rank test p -values testing whether differences in relative oracle performance between K = 64 PAC/FSOL-WRS variants and base PAC/FSOL methods are statistically significant.

G Additional results figures for PAC-WRS and FSOL-WRS

G.1 Uncertainty-quantified aggregate metrics across runs

Below, we provide figures with error bars (minimum and maximum across 5 trials) showing the relative oracle performances (ROP), final test accuracies, and final sparsities of PAC-WRS and FSOL-WRS alongside the base PAC and FSOL models, across all 16 datasets. The main takeaways are that a) PAC-WRS and FSOL-WRS overall incur substantially-lower ROP than their corresponding base models; b) PAC-WRS and FSOL-WRS overall achieve comparable, if not improved, final test accuracy compared to their corresponding base models, consistently with lower variance, too; c) the final sparsities of PAC-WRS and FSOL-WRS are overall comparable, if not higher than those of their corresponding base models.

G.1.1 FSOL and FSOL-WRS

Avazu(App)

Avazu (Site)

Criteo

0.20

0.005

-------.

0.008

-------

0.20

0.15

0.004

0.006

0.003

0.15

0.10

0.004

0.002

0.002

0.10-

0.05

0.001

0.000

0.05

0.000

0.00

-0.002

0.00

K=1

K=1K=4 K=16 K=64

K=1K=4K=16K=64

K=1

K=4 K=16K=64

MNIST8 (4+9)

News20

0.06

0.06

------

0.4

0.04

0.04

0.3

0.02

0.02

0.2

0.2

0.00-

0.00

0.1

0.0

亚重亚重

-0.02

0.0

K=1K=4 K=16 K=64

K=1

K=1

K=1

Newsgroups(Binary,CS)

PCMAC

RCV1

0.20

0.15

0.08

0.15

0.02

0.06

1

0.10

0.10

0.04

0.01

0.05

0.05

0.02

0.00

0.00

0.00

K=1

K=1K=4K=16K=64

K=1K=4K=16K=64

K=1

SST-2

URL

W8A

Webspam

0.08

0.125

0.04

0.06

0.100

1

0.03

0.06

0.04

0.075

0.02-

0.04

0.02-

0.050

0.00

0.025

0.01

0.02

-0.02

II

0.000

0.00

0.00

K=1 K=4 K=16 K=64

K=1K=4 K=16 K=64

K=1

K=1 K=4 K=16 K=64

SimpleAverage

WeightedAverage

FSOL

SimpleAverage+Voting-Based Zeroing

WeightedAverage+Voting-BasedZeroing

FSOL (Mean)

Avazu (App)

Avazu(Site)

Criteo

0.88

0.80

0.795

0.9

0.86

0.78

0.790

0.76

0.785

0.8

0.84

0.74

0.780-

0.72

0.775

0.7

FSOL K=1 K=4 K=16K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.875

0.94

0.9

0.850

0.94

0.92

0.8

0.825

班

0.7

0.92

0.800

0.90

0.775

0.6

0.90

0.750

0.5

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.90

HHII"

0.85

0.97

0.96

0.80

0.85

0.75

0.96

0.95

0.80

0.70

0.95

0.65

0.94

0.75

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

SST-2

URL

W8A

Webspam

0.9725

0.92

0.84

0.9

0.9720

0.90-

0.82

0.8

0.9715

0.88

0.80

0.9710

0.86

0.7

0.9705

0.78

0.84

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Simple Average

WeightedAverage

Oracle

FSOL

Simple Average + Voting-Based Zeroing

Weighted Average +Voting-Based Zeroing

Oracle(Mean)

Avazu (App)

1e-5+9.999e-1

Avazu (Site)

Criteo

8

0.982

0.50

0.94

6

0.981

0.45

0.980

0.92

0.40

0.979

0.90

0.978

0.35

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOLK=1K=4 K=16 K=64

MNIST8 (4+9)

News20

0.9995

II

0.90

0.985

THTH

0.980

0.99995

0.9990

0.85

0.975

0.9985

0.970

0.99990

0.80

0.9980

0.965

0.99985

0.9975

0.75

亚亚

0.960

II

11

FSOL K=1K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.8

0.75

0.40

0.40

Iili

0.35

0.7

0.70-

0.35

0.30 -

0.65-

TT

0.25

0.6

0.30

0.60

0.20

0.5

0.25

0.55

0.15

0.4

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16K=64

0.7

SST-2

9.5

1e-5+9.999e-1

URL

W8A

Webspam

1.04

0'6

1.02

0.85

0.6

8.5

1.00

0.80

0.5

8.0

0.98

0.75

7.5

0.96

0.4

0.70

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Simple Average

WeightedAverage

FSOL

Simple Average + Voting-Based Zeroing

Weighted Average+Voting-Based Zeroing

Avazu (App)

Avazu (Site)

Criteo

0.10

0.005

0.014

0.125

0.08

0.004

0.012

0.100

0.06

0.003

0.010

0.075

0.04

I1

0.002

0.008

0.050

0.02-

0.001

0.006

0.025

0.00

··

·

++++

K=1

K=1K=4K=16K=64

K=1

K=1

MNIST8 (4+9)

News20

0.04

0.20

0.08

0.4

0.02

0.15

0.06

0.10

0.04

0.2

0.00

0.05

0.02

0.0

HII1

0.00

IIII

K=1K=4K=16K=64

K=1

K=4K=16K=64

K=1

K=4K=16K=64

K=1

K=4K=16K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.08

0.08

0.04

0.06

----

0.06

0.06

0.03

0.04

111

0.02

0.04

0.04

0.02

11

0.02

0.00-

T1

0.02

0.01

0.00

K=1

K=1K=4K=16K=64

K=1

K=1

SST-2

URL

W8A

Webspam

0.10

0.04

F---------

0.075

0.08

0.08

0.03

0.050

0.06

0.025

0.06

0.02

i

0.000

0.04

0.01

0.04

0.02

-0.025

11

0.02

0.00

··

0.00-

K=1

K=1

K=1

K=1K=4 K=16 K=64

SimpleAverage

WeightedAverage

FSOL

SimpleAverage+Voting-Based Zeroing

Weighted Average+Voting-BasedZeroing

FSOL (Mean)

Avazu(App)

Avazu (Site)

Criteo

0.795

0.88

0.80

0.95

0.790

0.87

0.78

T

0.90

0.86

0.76

0.785

Ⅱ

0.74

0.780

0.85

0.85

0.72

0.775

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

MNIST8 (4+9)

News20

0.96

0.87

班班

0.90

0.92

0.94

0.86

0.85

0.92

0.85

0.80

0.90

0.90

0.84

0.75

0.88

0.83

0.88

0.70

0.86

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.875

0.90

96'0

0.96

0.850

0.88

0.94

0.825

0.86

0.94

0.92

0.800

0.84

0.90

0.92

0.775

0.82

0.88

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

SST-2

URL

W8A

Webspam

0.85

0.975

0.9725

0.92

0.950

0.9720

0.90-

0.80

0.925

0.9715

0.88

0.86

0.900

0.9710

0.75

0.84

0.875

0.9705

0.82

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Simple Average

WeightedAverage

Oracle

一

FSOL

Simple Average +Voting-Based Zeroing

Weighted Average +Voting-Based Zeroing

Oracle(Mean)

Avazu(App)

10

1e-5+9.999e-1

Avazu (Site)

Criteo

0.96

0.8

8

0.990

0.7

0.94

6

0.985

0.6

0.92

0.5

0.980

0.4

0.90

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOLK=1K=4 K=16 K=64

MNIST8 (4+9)

News20

T

1.0000

1.00

II

0.99

0.99995

0.95

0.9995

0.98

0.90

0.99990

0.97

0.9990

0.85

0.99985

0.80

0.96

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.75

0.4

0.9

王

0.7

0.70

0.6

0.8

0.3

0.5

0.65

0.7

0.4

0.60

0.2

0.6

0.3

0.55

0.5

FSOL K=1K=4 K=16 K=64

FSOLK=1K=4K=16 K=64

FSOL K=1K=4 K=16 K=64

FSOL K=1K=4 K=16 K=64

SST-2

URL

W8A

Webspam

1e-5+9.999e-1

0.65

亚亚亚亚

TTTT

T亚亚

1.04

0.95

0.60

9.5

1.02

0.90

0.55

1.00

0.85

0.50

9.0

0.98

0.80

0.45

0.96

0.75

0.40

8.5

1

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

FSOL K=1 K=4 K=16 K=64

SimpleAverage

WeightedAverage

FSOL

Simple Average +Voting-Based Zeroing

Weighted Average +Voting-Based Zeroing

G.1.2 PAC and PAC-WRS

Avazu (App)

Avazu (Site)

Criteo

0.08

0.0020

0.004

0.20

0.06

-

0.0015

0.15

0.003

0.04

0.0010

11

0.10

0.02

IIII

0.0005

1l1

H

0.002

0.05

0.00

K=1

K=1K=4K=16 K=64

K=1K=4K=16 K=64

K=1

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.125

0.02

0.03

0.125

0.100

0.100

0.02

0.075

0.01

0.075

0.050

0.01

0.050

0.000

0.025

0.00

立宜京室

-0.01

0.00

0.025

0.000

K=1K=4K=16K=64

K=1

K=1

K=1

Newsgroups (Binary, CS)

PCMAC

RCV1

0.150

0.10

0.125

0.125

0.08

0.010

0.100

0.100

0.06

0.005

0.075

0.050

0.075

0.04

0.000

l

THIITHI

0.050

0.025

0.025

0.02

-0.005

0.000

IIli

重重亚亚亚里亚亚

K=1

K=4K=16K=64

K=1

K=1

K=1

SST-2

URL

W8A

Webspam

0.075

0.006

0.0100

0.03

0.0075

0.050

0.02

0.004

0.0050

0.025

0.000

0.002

0.0025

0.01

E

-0.025七

0.000

亚亚

0.0000

K=1

K=1

K=1

K=1K=4K=16 K=64

SimpleAverage

WeightedAverage

PAC

Simple Average +Voting-Based Zeroing

----Weighted Average +Voting-Based Zeroing

PAC (Mean)

Avazu (App)

Avazu(Site)

Criteo

0.808

0.88

0.796

0.96

HIT

0.86

0.807

0.794

0.94

0.84

0.792

0.92

0.806

0.790

0.90

0.82

0.805

0.788

0.88

PAC

K=1K=4 K=16 K=64

PACK=1K=4 K=16 K=64

PAC

K=1K=4 K=16K=64

PAC

K=1K=4 K=16 K=64

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.94

0.88

HH

亚亚亚亚

0.94

HITI

0.950

0.925

0.93

0.87

0.93

0.900

0.92

0.86

0.92

0.875

0.91

0.850

0.91

0.85

0.90七

0.825

PACK=1K=4 K=16 K=64

PACK=1K=4 K=16 K=64

PACK=1K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.92

0.975

0.85

0.90

0.950

亚亚亚亚

0.95

0.80

0.88

0.925

0.90

0.86-

0.75

0.900

0.85

0.84

0.875

0.70

PACK=1K=4 K=16K=64

PAC K=1 K=4 K=16 K=64

PAC

K=1

PAC

K=1K=4 K=16 K=64

SST-2

URL

W8A

Webspam

0.86

0.985

1

0.92

0.98

0.84

0.980

0.97

0.91

0.82

0.96

0.90

0.975

0.80

0.95

0.89

PAC K=1 K=4 K=16 K=64

PAC

K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

PAC

K=1 K=4 K=16 K=64

SimpleAverage

WeightedAverage

Oracle

PAC

Simple Average +Voting-Based Zeroing

Weighted Average +Voting-Based Zeroing

Oracle (Mean)

Avazu (App)

Avazu (Site)

Criteo

0.984

0.984

1

0.675

0.983

0.982

0.342

0.650

0.982

0.980

0.340

0.625

0.981

0.978

0.600

0.980

0.976

0.338

0.575

PAC

K=1K=4 K=16 K=64

PAC

K=1

K=4 K=16K=64

PAC

K=1

PAC

K=1

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.35

0.8

0.050

T!

0.70

0.30

0.7

0.65

0.25

0.045

0.60

0.6

0.55

0.20

0.040

0.15

0.5

0.035

0.50

PACK=1K=4 K=16K=64

PAC

K=1K=4 K=16 K=64

PAC

K=1

PAC

K=1K=4 K=16K=64

Newsgroups(Binary,CS)

PCMAC

RCV1

0.30

0.34

0.25

0.25

0.6

0.32

0.20

0.20

0.30

0.4

0.15

0.28

0.15

0.10

0.05

0.26

0.10

PAC

K=1

PAC

K=1

PAC

K=1

PAC

K=1

K=4 K=16K=64

SST-2

URL

W8A

Webspam

0.20

----.

==

0.94 -

0.003

0.52

0.18-

0.51

0.002

0.16

0.92

0.50

+++e

0.14

H

0.90

0.001

0.49

0.48

0.12

0.000

PAC

K=1

PAC

K=1

PAC

K=1

PAC

K=1K=4 K=16 K=64

SimpleAverage

Weighted Average

PAC

Simple Average+ Voting-Based Zeroing

Weighted Average + Voting-Based Zeroing

Avazu (App)

Avazu (Site)

Criteo

0.008

0.08

0.003

0.10

0.06

0.006

0.08

0.002

0.04

0.004

0.06

0.02

0.001

0.002

0.04

0.00

+**

0.000

0.02

K=1

K=1

K=1

K=1

MNIST8 (4+9)

News20

0.025

0.020

0.0050

III

0.04

0.08

0.0025

0.06

0.015

0.02

0.0000

0.04

0.010

-0.0025

0.005

-0.0050

旺

0.00

I1

0.02

0.00

K=1K=4K=16K=64

K=1

K=1

K=4K=16K=64

K=1

K=4K=16K=64

Newsgroups(Binary,CS)

PCMAC

RCV1

0.10

0.08

0.020

0.03

0.08

0.06

0.015

0.02

0.06

0.010

0.04

0.04

0.005

0.01

0.02

0.02

1l1

0.000

11

0.00七

I1

K=1

K=1

K=1

K=4K=16K=64

K=1

K=4K=16K=64

SST-2

URL

W8A

Webspam

0.08

0.06

0.020

I

0.015

1

0.020

0.04

0.015

11

0.010

0.015

0.00

0.02

0.010

T

0.005

0.010

1

-0.02

I1

0.005

II

0.000

0.005

K=1

K=1

K=1K=4 K=16 K=64

K=1

SimpleAverage

WeightedAverage

PAC

SimpleAverage+Voting-BasedZeroing

Weighted Average+Voting-Based Zeroing

PAC (Mean)

Avazu(App)

Avazu (Site)

Criteo

0.88

0.808

0.7950

0.96

0.807

！

0.86

0.7925

0.94

0.806

0.7900

0.92

0.84

0.805

0.7875

0.90

0.7850

0.82

0.804

0.88

PAC K=1 K=4 K=16K=64

PAC K=1 K=4K=16 K=64

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16K=64

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.96

0.94

HTE

0.880

TT

0.94

0.95

0.93

0.875

TT

0.93

0.94

0.92

0.92

0.93

0.870

0.91

0.91

0.92

0.90

0.90

0.865

0.89

0.91-

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.97

0.85

0.900

0.96

1

0.875

0.94

0.96

0.80

0.850

0.92

0.95

0.75

0.90

0.825

0.88

0.94

0.70

0.800

PACK=1K=4 K=16K=64

PAC K=1 K=4 K=16 K=64

PACK=1K=4 K=16 K=64

PACK=1K=4K=16 K=64

SST-2

URL

W8A

Webspam

0.925

1T

0.985

0.850

0.980

0.98

班年

0.920

1

0.825

0.915

0.975

0.97

0.800

0.970

0.96

0.910-

0.905

0.775

0.965

iIi

0.750

0.960

0.95

006°0

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

PAC K=1 K=4 K=16 K=64

SimpleAverage

WeightedAverage

Oracle

PAC

Simple Average +Voting-Based Zeroing

Weighted Average +Voting-Based Zeroing

Oracle (Mean)

Avazu(App)

Avazu(Site)

Criteo

I11

0.9770

0.675

0.995

0.9765

IIII

0.344

0.650

0.990

0.342

0.9760

0.625

0.985

0.9755

0.340

0.600

0.980

0.338

0.575

0.9750

车

PAC

K=1K=4 K=16 K=64

PACK=1K=4 K=16K=64

PAC

K=1

PAC

K=1K=4 K=16 K=64

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.40

0.8

0.150

0.575

0.35 -

0.125

0.30

0.7

0.100

0.550

0.525

0.25

0.6

0.075

0.500

0.20

0.15

0.5

”

0.050

0.475

PACK=1K=4 K=16 K=64

PACK=1K=4 K=16 K=64

PAC

K=1K=4K=16 K=64

PAC

K=1K=4 K=16 K=64

Newsgroups (Binary,CS)

PCMAC

RCV1

0.6

0.25

0.150

0.6

0.5

0.125

0.20

0.4

0.100

0.4

0.15

0.075

0.3

0.10

亚亚

PAC

K=1 K=4 K=16 K=64

PAC

K=1

K=4 K=16K=64

PAC

K=1

PAC

K=1K=4 K=16K=64

SST-2

URL

W8A

Webspam

0.25

1.000

0.8

0.52

0.975

0.6

0.51

0.20

0.950

0.4

0.50

0.925

0.49

0.15

0.2

0.900

0.48

王

0.0

PAC

K=1 K=4 K=16 K=64

PAC

K=1 K=4 K=16 K=64

PAC

K=1 K=4 K=16 K=64

PACK=1K=4 K=16 K=64

SimpleAverage

Weighted Average

一

PAC

Simple Average + Voting-Based Zeroing

Weighted Average + Voting-Based Zeroing

G.2 Test accuracies and sparsities over time

Below, we provide figures showing test accuracies and sparsities over time on individual runs of PAC-WRS and FSOL-WRS, compared against their base model counterparts, on all 16 datasets. The main takeaways are that a) test accuracies over time on PAC-WRS and FSOL-WRS are overall much stabler, if not also higher, than their base model counterparts (and much closer to that of the oracle); b) applying WAT to PAC and FSOL does not significantly impact sparsity - in some cases, especially with voting-based zeroing, sparsity actually increases compared to the base model!

G.2.1 FSOL and FSOL-WRS

Avazu (App)

Avazu (Site)

Criteo

0.90

0.85

0.80

0.80

60

0.80

0.75

0.75 -

0.8 -

0.75

0.70

0.70 -

0.70

0.65

0.65 -

0.7

0.65

0.60

0.60

0.60

0.6

0

3

6

9

0.0

0.5

1.0

1.5

0

1

2

3

0

150

300

1e6

1e7

1e7

KDD2010(Algebra)

MNIST8(4+9)

News20

06°0

0.9

0.85

0.9

0.9

0.80

0.8

0.75

0.8

0.8

0.7

0.70

0.7

0.7

0.65

0.6

0.60

0.6

0.6

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups(Binary,Cs)

PCMAC

RCV1

0.90

0.90

1.0

1.0

0.85

0.85-

0.9

0.9

0.80

0.80

0.75

0.75

0.8

0.8

0.70

0.70

0.7

0.7

0.65

0.65-

0.60

0.60

0.6

0.6

0

4000

8000

12000

0

400

800

1200

0

150000

300000

450000

0

15000

30000

45000

SST-2

URL

W8A

Webspam

1.0

1.0

0.85

0.9

0.80

0.9

0.9

0.75

0.8

0.8

0.8

0.70

0.7

0.7

0.7

0.65

0.60

0.6

0.6

0.6

0

15000

0000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

FSOL-WRS(SimpleAverage)

FSOL-WRS(WeightedAverage)

FSOL

FSOL-WRS(Simple Average +Voting-Based Zeroing)

FSOL-WRS(Weighted Average+Voting-Based Zeroing)

Oracle

Avazu (App)

1e-5+9.999e-1

Avazu (Site)

Criteo

10

1.000

1.0

1.00

0.995

0.8-

0.98

6

066'0

0.96

0.6 -

0.985

t60

2

0.980

0.4 -

0.92

0

3

6

9

0.0

0.5

1.0

1.5

0

1

2

0

150

300

1e6

1e7

1e7

+1

MNIST8 (4+9)

News20

0.00000

1.0000

1.00

1.00

-0.00002

0.9995

0.95

660

0.00004

0.9990

0.90

-0.00006

0.9985

0.85

0.98

-0.00008

08660

0.80

0.97

-0.00010 -

-0.00012

0.9975

0.75 -

0.96

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups (Binary,CS)

PCMAC

RCV1

1.0

1.0

1.0

1.0

0.9

0.9

0.8

0.8

0.8

0.8

0.6-

0.7

0.6

0.7

0.4

0.6

0.5

0.4

0.6

0.2

0.4

0

4000

8000

12000

0

400

800

1200

0

150000300000450000

0

15000

30000

45000

URL

SST-2

1e-5+9.999e-1

W8A

Webspam

1.0

10.0

1.000

1.00

0.9

9.5

0.95

0.8

9.0

6660

0.90

0.7

8.5

0.998

0.85

0.6

8.0

0.80

0.5

7.5

0.997

0.75 -

0.4七

0

15000

30000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

FSOL-WRS (Simple Average)

FSOL-WRS(WeightedAverage)

FSOL

FSOL-WRS(Simple Average+Voting-Based Zeroing)

FSOL-WRS(Weighted Average+Voting-Based Zeroing)

Oracle

Avazu (App)

Avazu (Site)

Criteo

0.90

0.85

08'0

0.80

60

0.80

0.75

0.75

0.8 -

0.75

0.70

0.70 -

0.70

0.65

0.65 -

0.7

0.65

0.60

0.60

0.60

0.6

0

3

6

9

0.0

0.5

1.0

1.5

0

1

2

3

0

150

300

1e6

1e7

1e7

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.90

60

0.85 -

0.9

0.9

0.80

80

0.75

0.8

0.8

0.70

0.7

0.7

0.7

0.65

0.6

0.60

0.6

0.6

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups(Binary,Cs)

PCMAC

RCV1

0.90

060

1.0

1.0

0.85

0.85-

0.9

60

0.80

0.80

0.75

0.75

0.8

0.8

0.70

0.70

0.7

0.7

0.65

0.65-

0.60

0.60

0.6

0

4000

8000

12000

0

400

800

1200

0

150000

300000

450000

0

15000

30000

45000

SST-2

URL

W8A

Webspam

0.90

1.0

1.0

0.85

0.9

0.9

0.9

0.80

0.75

0.8

0.8

0.8

0.70

0.7

0.7

0.7

0.65

0.60

0.6

0.6

0

15000

30000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

FSOL-WRS(SimpleAverage)

FSOL-WRS(WeightedAverage)

FSOL

FSOL-WRS(Simple Average +Voting-Based Zeroing)

FSOL-WRS(Weighted Average+Voting-Based Zeroing)

Oracle

Avazu (App)

1e-5+9.999e-1

Avazu (Site)

Criteo

10

1.000

1.0

1.00

9

8

0.995

80

0.98

7

0.990

0.96

6

0.985

0.6

60

5

0.980

0.4 -

0.92

3

6

9

0.0

0.5

1.0

1.5

0

1

2

0

150

300

1e6

1e7

1e7

+1

KDD2010(Algebra)

MNIST8 (4+9)

News20

0.00000

1.00000

1.00

1.00

-0.00002

1.

0.99975

0.95

660

-0.00004

0.99950

-0.00006

0.99925

0.90

86°0

-0.00008

0.99900

0.85

L60

-0.00010

0.99875

-0.00012

0.80-

0.96

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups (Binary, CS)

PCMAC

RCV1

1.0

1.0

1.0

1.0

0.9

60

0.8

0.8

0.8

0.8

0.6

0.7

0.6

0.7

0.4

0.6

0.4

0.6

0.2

0.5

0

4000

8000

12000

0

400

800

1200

0

150000300000450000

0

15000

30000

45000

URL

SST-2

1e-5+9.999e-1

W8A

Webspam

1.0

10.0

1.000

1.00

0.9

0.95

0.8

9.5

W

6660

0.90

0.7

0.85

9.0

0.998 -

0.6

0.80

0.5

8.5

0.997

0.75

0.4

0

15000

30000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

FSOL-WRS (Simple Average)

FSOL-WRS(Weighted Average)

FSOL

FSOL-WRS(SimpleAverage+Voting-BasedZeroing)

----FSOL-WRS(Weighted Average+Voting-Based Zeroing)

Oracle

G.2.2 PAC and PAC-WRS

Avazu(App)

Avazu (Site)

Criteo

06°0

1.0

0.85

0.80

0.80

0.9

0.80

0.75

0.75

0.75

0.70

0.70

80

0.70

0.65

0.65

0.7

0.65

0.60

0

3

6

9

0.60

0.0

0.5

1.0

1.5

0.60

0

1

2

3

0.6

0

150

300

1e6

1e7

1e7

KDD2010(Algebra)

MNIST8 (4+9)

News20

1.0

060

0.9

0.85

0.9

0.9

0.80

0.8

0.75

0.8

0.8

0.7

0.70

0.7

0.7

0.65

0.6

0

250

500

750

0.60

0

2

4

6

0.6

400000

800000

0.6

0

4000

8000

12000

1e6

Newsgroups(Binary,Cs)

PCMAC

RCV1

060

1.0

1.0

0.85

0.9

0.9

60

0.80

0.8-

0.75

A

0.8

0.8

0.70

0.7

0.7

0.7

0.65

0.60

0.6

0.6

0

4000

8000

12000

0

400

800

1200

0

150000

300000

450000

0

15000

30000

45000

SST-2

URL

W8A

Webspam

0.90

1.0

1.0

0.85

0.9

0.80

0.9

0.9

0.8

0.75

0.8

0.8

0Z0

0.7

0.7

0.7

0.65

0.60

0.6

0.6

0.6

0

15000

30000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

PAC-WRS(SimpleAverage)

PAC-WRS(WeightedAverage)

PAC

PAC-WRS(WeightedAverage+Voting-BasedZeroing)

Oracle

Avazu(App)

Avazu (Site)

Criteo

1.000

1.000

1.0

1.0

0.995

0.995

0.9

0.8

0660

0.990

0.8

0.985

0.6

0.985

0.7

0860

0.980

0.975

0.4

0.6

0

E

6

6

0.0

0.5

1.0

1.5

2

3

0

150

300

1e6

1e7

1e7

MNIST8 (4+9)

News20

1.0

1.0

1.0

1.0

0.8

0.9

0.8

60

0.8

0.6

0.8

0.6

0.7

0.4

0.7

0.4

0.6-

0.6

0.2

0.2

0.5

0.5

0.0

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups (Binary, CS)

PCMAC

RCV1

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6-

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.4

0.2

0

4000

8000

12000

0

400

800

1200

0

150000

300000

450000

0

15000

30000

45000

SST-2

URL

W8A

Webspam

1.0

1.00

1.0

1.0

0.8

0.98

0.8

0.9

0.6

0.96

0.6

0.8

0.94

0.4

0.7

0.4

0.92

0.2

0.90

0.2

0.6

1

0.0

0.5

0

15000

30000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

PAC-WRS (Simple Average)

PAC-WRS(WeightedAverage)

PAC

----PAC-WRS(Weighted Average+Voting-Based Zeroing)

Oracle

Avazu (App)

Avazu (Site)

Criteo

06°0

1.0

0.85

0.80

0.80

60

0.80

0.75

0.75

0.75

0.70

0.70

0.8

0.70

0.65

0.65

0.7

0.65

0.60

0

3

6

9

0.60

0.0

0.5

1.0

1.5

0.60

0

1

2

3

0.6

0

150

300

1e6

1e7

1e7

MNIST8 (4+9)

News20

060

1.0

0.9

0.85

0.9

0.9

08'0

0.8

0.75

0.8

0.8

0.7

0.70

0.7

0.7

0.65

0.6

0.60

0.6

0.6-

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups(Binary,Cs)

PCMAC

RCV1

060

1.0

1.0

0.85

60

0.9 -

0.9

0.80

0.8

0.75

0.8

0.8

0.70

0.7

0.7

0.7

0.65

0.60

0.6

0.6

0

4000

8000

12000

0

400

800

1200

0

150000

300000

450000

0

15000

30000

45000

SST-2

URL

W8A

Webspam

0.90

1.0

1.0

0.85

0.9

0.80-

0.9

0.9 -

0.8

0.75

0.8

0.8

040

0.7

0.7

0.7

0.65

0.60

0

15000

30000

45000

0.6

0.0

0.5

1.0

1.5

0.6

0

15000

30000

0.6

0

80000

160000

240000

1e6

PAC-WRS(SimpleAverage)

PAC-WRS(WeightedAverage)

PAC

PAC-WRS(WeightedAverage+Voting-BasedZeroing)

Oracle

Avazu(App)

Avazu (Site)

Criteo

1.000

1.000

1.0

1.0

0.995

0.995

0.9

0.8

0660

0.990

0.8

0.985

0.6

0.985

0.7

0.980

0.980

0.975

0.4

0.6

0

3

6

9

0.0

0.5

1.0

1.5

1

2

3

0

150

300

1e6

1e7

1e7

MNIST8 (4+9)

News20

1.0

1.0

1.0

1.0

0.8

0.9

0.8

60

0.8

0.6

0.8

0.6

0.7

0.4

0.7

0.4

0.6

0.6

0.2

0.2

0.5

0.5

0.0

0

250

500

750

0

2

4

6

0

400000

800000

0

4000

8000

12000

1e6

Newsgroups (Binary, CS)

PCMAC

RCV1

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6-

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.4

0.2

0

4000

8000

12000

0

400

800

1200

0

150000

300000

450000

0

15000

30000

45000

SST-2

URL

W8A

Webspam

1.0

1.00

1.0

1.0

0.8

0.98

0.8

0.9

0.6

0.96

0.6

0.8

0.94

0.4

0.7

0.4

0.92

0.2

0.6

0.2

0.90

0.0

0.5

0

15000

30000

45000

0.0

0.5

1.0

1.5

0

15000

30000

0

80000

160000

240000

1e6

PAC-WRS (Simple Average)

PAC-WRS(WeightedAverage)

PAC

----PAC-WRS(Weighted Average+Voting-Based Zeroing)

Oracle

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please see our Abstract and Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please see Section 3.2 and Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please see Section 4 and Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please see our GitHub at https://github.com/FutureComputing4AI/WeightedReservoir-Sampling-Augmented-Training/tree/main, with a detailed README.md file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see Algorithm 1, Section 4, and Appendix B, as well as our GitHub at https://github.com/FutureComputing4AI/Weighted-Reservoir-SamplingAugmented-Training/tree/main.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please see error bars and significance tests in Section 4 and Appendix G.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read the NeurIPS Code of Ethics and have made sure that our research conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper introduces a foundational research method that does not have any direct paths to negative societal impacts, aside from the general interactions of machine learning with society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not involve any data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We enumerate and cite all datasets in Table 1. Aside from our own preprocessed versions of Newsgroups (Binary, CS) and SST-2, the 14 other datasets are all publicly available.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Please see the README.md file in our accompanying GitHub https://github.com/FutureComputing4AI/Weighted-Reservoir-Sampling-AugmentedTraining/tree/main.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or work with human subjects was performed for this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not have any study participants and/or crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors are interested in sequential learning algorithms, within an IID data setup. That is, the ultimate objective is expected loss for a single algorithm output, not the regret incurred by a sequence of candidates. More specifically, they are interested in linear binary classification, and their overarching goal is to design an algorithm which is less sensitive to idiosynchratic data points, especially those near the end of the learning process.\n\nBroadly speaking, their proposal is a procedure to construct a weighted sum of \"good candidates\" for the final algorithm output, ideally being far less sensitive to outliers that can take traditional online learning algorithms astray. Their procedure \"wraps\" around a base update (here, PAC or FSOL), and selects candidates for inclusion in a \"reservoir\" in a stochastic fashion, and the core mechanism underlying their weighting is to use the number of \"passive steps\" as an indicator of quality. Passive steps are those steps, given a certain candidate, where the 0-1 loss is zero (i.e., correct classification) and no updates are made.\n\nThe underlying idea itself is heuristic, but the authors provide theoretical analysis in the form of risk bounds for candidates selected based on the number of cumulative passive steps. Essentially, when the minimum achievable risk is small, the passive step-based choice is not likely to be much worse than the best choice among the stocked candidates. They also conduct a series of empirical tests, which suggest that their proposed approach can be applied in a straightforward way to achieve better accuracy and stability when compared with the base procedure their algorithm wraps around, on a variety of datasets.\n\nSTRENGTHS:\nThe authors present a procedure which is intuitive, can be given some theoretical grounding, and appears to work well in practice. The paper is well-structured, and the writing (aside from some technical points to be mentioned later) is very clear. While the basic idea of taking a weighted sum of good candidates to \"smooth out\" the performance of sequential learning algorithms is a well-studied tactic, the authors consider a rather unique approach applying weighted reservoir sampling to passive-aggressive base algorithms.\n\nWEAKNESSES:\nWhile I highlighted writing clarity as one of the paper's strengths, I found the methodology presented here rather difficult to understand. The WRS approach is positioned as a central part of the authors' strategy, but what part of Algorithm 1 is specific to WRS? If we remove the uniformly sampled random variable and just determine inclusion in the $K$-sized set of candidates by the number of passive steps, would things really change all that much for the worse?\n\nIn addition, the use of formal notation in the paper is in my opinion *quite* sloppy in parts. The technical material feels rushed, and makes it difficult for readers/reviewers to effectively parse the procedure being proposed. I will provide some concrete examples in the next field.\n\nFinally, considering the fact that there is a *massive* literature on taking averages of candidates from iterative learning procedures, I think many readers will find it troubling that the empirical investigations are completely inward-facing, i.e., they simply evaluate the base algorithms and the proposed wrapper under different settings, and do not treat other alternative averaging strategies which are far more generally applicable (e.g., averaging $K$ most recent candidates, downweighting over time, using so-called [\"anytime\" sub-routines](https://arxiv.org/abs/1903.00974), and so forth).\n\nQUESTIONS:\nI do not have any critical questions for the authors. For the most part, I get their idea and I understand the investigation that they have carried out. Below is a handful of small points I tripped up on while reading the paper.\n\n- Abstract, *\"Our reservoir thus contains $K$ previous intermediate...\"*: here $K$ really has no meaning at all. Why not just say \"a subset of previous intermediate weight vectors...\" or something?\n\n- First sentence of 2.1: If my reading is correct, the first time \"linear binary classification\" is mentioned is here at the start of 2.1. I think most readers, having read up to this point (including the abstract!), will be shocked that the scenario considered by the authors is so narrow. This should be established earlier, and more clearly.\n\n- The symbol $T$ is used for iteration numbers as well as for transposing the weight vector in the main equation presented in 2.1 (for $\\\\hat{y}^{\\\\ast}$); this should be avoided if possible.\n\n- Reading through 2.3, I found the WRS exposition to be quite mysterious. What in particular about the problem being studied here makes WRS a natural fit? What parts of Algorithm 1 use the original WRS as-is, and which parts are original? It's not clear to me.\n\n- 3.2, *\"A given online learning algorithm outputs model $\\\\mathbf{w}\\_{t}$ after seeing $z\\_{t}$, with loss $\\\\ell(\\\\mathbf{w}\\_{t},z\\_{t})$\"*: I found this sentence clunky. Shouldn't the *output* be $\\\\mathbf{w}\\_{t+1}$, considering we have seen the data and loss for step $t$ already?\n\n- 3.2, third paragraph: thus far the \"model\" has been characterized by $\\\\mathbf{w}_{t}$ or simply $\\\\mathbf{w}$, but suddenly here $h$ appears. Plus, the $x$ here (not bold) is inconsistent with previous data notation.\n\n- 3.2, fourth paragraph: how are the *\"updated models\"* $\\\\mathbf{w}^{(j)}$ defined? What is $\\\\mathcal{J}$? This is all totally unclear.\n\n- Algorithm 1, line 15: this critical definition of $k^{\\\\ast}$ includes $w\\_{t}$, an undefined quantity. The only place I can find it is earlier in 2.3, regarding \"weights\" used in WRS. Are these supposed to be pre-fixed?\n\n- Algorithm 1, line 16: here $T$ is used as a \"threshold\" of sorts, but notation clashes with $T$ used for the number of iteration in the for-loop.\n\n- Algorithm 1, line 19: to update \"accordingly\" means what? Are the elements supposed to be paired up with elements of $\\\\mathcal{R}$, so that when an element is removed from $\\\\mathcal{R}$, the corresponding elements in $\\\\mathbfit{b}$ and $\\\\mathbfit{k}$ are removed as well? This is unclear. I can infer it from line 29 with some confidence, but readers shouldn't have to infer such things.\n\n- Algorithm 1, lines 27 and 29: the use of $\\\\mathbf{w}\\_{r}$ is bad notation; withing Algorithm 1, $\\\\mathcal{R}$ is said to be a set of promising solutions, but here it magically transforms into a set of the *indices* of promising solutions. This is even more problematic because $\\\\mathbf{w}$ with no index subscript is all we have within Algorithm 1, so $\\\\mathbf{w}\\_{r}$ is effectively undefined in this context. Note it also completely clashes with $\\\\mathbf{w}\\_{r} \\\\in \\\\mathcal{R}$ used in line 32.\n\n- Algorithm 1: when WS is \"Standard\", then $b^{\\\\ast}$ is updated, but under AS as \"Simple Average,\" $b^{\\\\ast}$ plays no subsequent role; is this correct? If so, considering the fact that \"Simple Average\" performs very similarly to \"Weighted Average,\" one wonders if the passive-step count based approach is actually meaningful at all. Note also that the critical $k^{\\\\ast}$ check in Algorithm 1 is unclear due to the undefined $w\\_{t}$.\n\nLIMITATIONS:\nLimitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-18"}

comments: SUMMARY:
Passive-aggressive algorithm is a seminal method in online learning. However, it may be unstable when outliers arise. This paper uses weighted reservoir sampling to stabilize the linear passive-aggressive online learning. The key idea is that the subsequent number of passive steps can reflect the generalization error. Then the weighted reservoir sampling is used to sample the models with low generalization error. The final model is the ensemble of the sampled models.

STRENGTHS:
This paper is clear and easy to follow. The proposed method makes a good use of the characteristic of the passive-aggressive algorithm. The experiments show the superiority of the proposed method.

WEAKNESSES:
The theoretical analysis assumes i.i.d. condition, which is not consistent with the motivation that individual outliers exist and do harm to PA classifier.

QUESTIONS:
I wonder whether the online gradient descent with momentum can perform well, because it can also be seen as a special ensemble method. I suggest authors add this method for comparison.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

comments: SUMMARY:
This paper resolves the outlier sensitivity problem in online learning. The proposed approach (WAT) can stabilize passive-aggressive online learning algorithms and does not introduce common overheads like hold-out evaluation sets or additional passes.

STRENGTHS:
1. The proposed WAT shows a significant reduction in test accuracy fluctuations.

2. WAT maintains, and in some cases, enhances the sparsity of solutions in FSOL, which is crucial for high-dimensional data.

3. The implementation of WAT does not require additional passes over the training data or hold-out evaluation sets.

WEAKNESSES:
1. While FSOL-WRS shows statistically significant improvements in final test accuracy, PAC-WRS does not consistently show the same level of improvement.

2. Although minimal, there is still an additional memory and computational overhead associated with managing the reservoir of candidate solutions.

QUESTIONS:
1. Can the authors provide more detailed explanations on how intermediate weight vectors are selected and maintained in the reservoir, and discuss the impact of different reservoir sizes on the algorithm's performance?

2. I think more compared baselines are needed, such as ADAGRAD or Truncated Gradient.

LIMITATIONS:
This paper comprehensively discusses its limitations regarding the requirements for base and the assumptions on data distributions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm for a binary online learning problem where the streaming data consists of i.i.d. samples. The authors propose a variant of the seminal algorithm of Passive-Aggressive classifier. The PAC algorithm only updates its current model when a missclassification occurs. The main idea of the paper is to augment this algorithm with a weighted reservoir sampling of previous models,  where the weight of each model is proportional to how much the model “lived” before an update occurs. At each time step $t$, the weighted ensemble solution from the reservoir is used for prediction. \n\nThe paper presents theoretical results for the risk of their algorithm, and conduct experiments to show the superiority of the methods compared against 2 online algorithm baselines.\n\nSTRENGTHS:\nThe idea of using reservoir sampling for a online algorithm is novel, and I have not seen it before. \n\nThe experimental section seems throughout, and a lot of datasets are used for the comparison with the baselines.\n\nWEAKNESSES:\nI believe that the theory is the weaker part of this paper. Also, the comparison with the previous baselines is not very well motivated.\n\nI am confused by the theoretical setting of this work. In online learning, it is often assumed that the data is generated by an adversary and it does not come from the same distribution. For example, the analysis of [1,5] (baselines used and described by this work) never uses this assumption, and they upper bound measures of error related to regret (also this inconsistency is in line 3-4 of abstract). In this context, there is no distribution, so the sentence at line 64 is false “The goal is that as t goes to infinity, w_t will generalize well to out-of-sample examples” (there is no distribution to generalize w.r.t., and the examples are not samples from a distribution).  I think this is the case for the analysis of the online algorithms described in Lines 65-74 other than [1,5] (I did not check)\n\nTo this end, the PAC (and FSOL) algorithm is trying to solve a different problem. The intuition that a solution is more promising if it is correct for a longer period of time only applies if the data is i.i.d., which is a different problem from the one of the paper introducing PAC. The same concern applies to the experimental section, where due to the random split (and thus shuffling) of the data, it can be considered as i.i.d., which is a different setting than the online learning setting.\n\nThe theoretical results (Thm 1 and Thm 2) are very hard to interpret, and it is unclear whether they provide any meaningful contribution. \n\nIf the data is i.i.d., a desirable property is that the gap between R_D(w_T) and R_D(w^*) goes to zero as T goes to infinity, where $w_T$ is the model of the algorithm at time $T$ (i.e., once I observe a large amount of data, my error converge to the error of $w*$), most likely with rate 1/sqrt(T). Alternatively, if I do not assume that the data is i.i.d. and I focus on a online setting, I would like to see an upper bound to the regret of order O(sqrt(T)). [This could possibly also imply the former with an online to batch conversion].\n\nAdditionally, both Thm 1 and Thm 2 are very hard to read. The definition of $r_m$ is unclear (isn’t the minimum risk achievable by any model w always less than R_{D}(w^*)?).  What is $r_m^{K}$ and $r_m^{K+1}?$.\n\nTheorem 2 depends on assumptions on the models stored by the algorithm that are unclear whether they are ever satisfied. Theorem 2 has dependencies on those assumptions that make it unclear whether the result is ever meaningful (e.g., is the probability of Thm2 ever $\\geq 1/2$)? Also, what is $\\epsilon$ in Thm 2 and lines 185-189 ( is it arbitrarily chosen?).\n\nI found lines 168 confusing. It seems that R(h) is defined but never used in the main paper, only R_{D}(w). Also it is unclear to me on what loss is used, since l(w,z) is previously defined as the hinge loss, but l(h,z) is the 0-1 loss (81).\n\nSuggestions: I would formally define concepts outside of related work (e.g, sparsity in line 73-74).\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe paper discusses limitations in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper seeks to address the issue of instability in passive-aggressive (PA) online learning algorithms, which are highly sensitive to individual data points, particularly outliers. These instabilities can lead to significant fluctuations in the model's accuracy, especially when an outlier is encountered late in the data stream. To mitigate this, the authors propose a novel approach called Weighted Reservoir Sampling (WRS), which augments the standard PA learning process by maintaining a reservoir of high-quality weight vectors that are less prone to overfitting due to outliers. This method is tested on two PA algorithms—Passive-Aggressive Classifier (PAC) and First-Order Sparse Online Learning (FSOL)—across several datasets, showing that WRS significantly stabilizes accuracy without the need for additional data passes or hold-out sets.\n\nSTRENGTHS:\nOriginality:\nThe introduction of Weighted Reservoir Sampling (WRS) as a method to stabilize PA algorithms is a novel contribution. The idea of leveraging the number of passive updates as a proxy for the quality of solutions is innovative, especially in the context of online learning.\n\nQuality:\nThe paper is methodologically rigorous, providing a thorough theoretical analysis to support the proposed WRS method. The empirical validation across multiple datasets strengthens the claim that WRS enhances the stability of PA learning algorithms.\n\nClarity:\nThe paper is well-written and clearly explains the problem, the proposed solution, and the results. The step-by-step presentation of the WRS method, along with detailed explanations of the experiments, makes the paper accessible to readers familiar with online learning.\nThe authors also use lots of wrap tables, making best use of the space. \n\nSignificance:\nThe significance of the work lies in its potential to improve the robustness of PA algorithms in real-world applications, where data streams often contain noisy or outlier data points. This method could be particularly valuable in fields like online advertising, real-time recommendation systems, and other high-dimensional streaming data applications.\n\nWEAKNESSES:\nLimited Applicability:\nWhile the WRS method is shown to work well with PAC and FSOL, its applicability to other online learning algorithms, especially those that are not passive-aggressive, is not explored. This limits the generality of the method.\n\nAssumption on Data Distribution:\nThe method assumes that the data distribution remains relatively stable over time. However, in real-world scenarios where data distributions might shift (concept drift), the effectiveness of WRS may diminish. This limitation is acknowledged but not deeply explored in the paper.\n\nQUESTIONS:\nMaintaining a reservoir of high-quality weight vectors adds computational overhead, particularly in terms of memory usage and processing time. While the paper claims this overhead is minimal, I would like to see if there are more detailed analysis or comparisons with baseline methods in terms of computational efficiency.\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-08-11"}