Title: Provably Faster Algorithms for Bilevel Optimization via Without-Replacement Sampling
Venue: NeurIPS 2024
Accepted: yes
Keywords: sampling

bilevel optimization
Abstract: Bilevel Optimization has experienced significant advancements recently with the introduction of new efficient algorithms. Mirroring the success in single-level optimization, stochastic gradient-based algorithms are widely used in bilevel optimization. However, a common limitation in these algorithms is the presumption of independent sampling, which can lead to increased computational costs due to the unique hyper-gradient structure in bilevel problems. To address this challenge, we study the example-selection strategy for bilevel optimization in this work. More specifically, we introduce a without-replacement sampling based algorithm which achieves a faster convergence rate compared to its counterparts that rely on independent sampling. Beyond the standard bilevel optimization formulation, we extend our discussion to conditional bilevel optimization and also two special cases: minimax and compositional optimization. Finally, we validate our algorithms over both synthetic and real-world applications. Numerical results clearly showcase the superiority of our algorithms.

Provably Faster Algorithms for Bilevel Optimization via Without-Replacement Sampling

Abstract

Bilevel Optimization has experienced significant advancements recently with the introduction of new efficient algorithms. Mirroring the success in single-level optimization, stochastic gradient-based algorithms are widely used in bilevel optimization. However, a common limitation in these algorithms is the presumption of independent sampling, which can lead to increased computational costs due to the complicated hyper-gradient formulation of bilevel problems. To address this challenge, we study the example-selection strategy for bilevel optimization in this work. More specifically, we introduce a without-replacement sampling based algorithm which achieves a faster convergence rate compared to its counterparts that rely on independent sampling. Beyond the standard bilevel optimization formulation, we extend our discussion to conditional bilevel optimization and also two special cases: minimax and compositional optimization. Finally, we validate our algorithms over both synthetic and real-world applications. Numerical results clearly showcase the superiority of our algorithms.

1 Introduction

Bilevel optimization [51, 47] has received a lot of interest recently due to its wide-ranging applicability in machine learning tasks, including hyper-parameter optimization [39], meta-learning [57] and personalized federated learning [46]. A bilevel optimization problem is a type of nested two-level problems as follows:

which includes an outer problem f ( x, y ) and a inner problem g ( x, y ) . The outer problem f ( x, y ) relies on the solution y x of the inner problem g ( x, y ) . Eq. (1) can be solved through gradient descent: x t +1 = x t -η ∇ h ( x t ) , with η be the stepsize and ∇ h ( x t ) be gradient at the state x t . Specially, the gradient ∇ h ( x ) [17] is a function of the inner problem minimizer y x , first order derivatives ∇ x f ( x, y ) , ∇ y f ( x, y ) , ∇ y g ( x, y ) and second order derivatives ∇ xy g ( x, y ) , ∇ y 2 g ( x, y ) . In particular, y x can be solved though gradient descent: y t +1 = y t -γ ∇ y g ( x, y t ) with γ be the stepsize and ∇ y g ( x t , y t ) be gradient at the iterate ( x t , y t ) . In the standard setup, the outer and inner problems are defined over two datasets D u = { ξ i , i ∈ [ m ] } and D l = { ζ j , i ∈ [ n ] } , respectively:

Naturally, we sample from the datasets to estimate the first and second-order derivatives in the hyper-gradient formulation. And to guarantee convergence, previous approaches require the examples

be mutually independent [25] even at the same state ( x, y ) . For example, two different examples ζ 1 and ζ 2 are sampled to estimate ∇ y g ( x, y ) and ∇ y 2 g ( x, y ) , respectively. This leads to extra computational cost in practice. Indeed, two backward passes are required to evaluate ∇ x f ( x, y ) and ∇ y f ( x, y ) , while five backward passes are needed to evaluate ∇ y g ( x, y ) , ∇ y 2 g ( x, y ) and ∇ xy g ( x, y ) for a given state of ( x, y ) . In contrast, one backward pass are needed to evaluate the former and two backward passes for the latter if not sampling independently for each property. This leads to notable computational cost difference for the current large-scale machine learning models [4]. Therefore, it is beneficial to explore other example-selection strategies beyond independent sampling. More specifically, we aim to answer the following question in this work: Can we solve the bilevel optimization problem without using independent sampling?

Although example-selection is under-explored for bilevel optimization, it has been well studied in the single level optimization literature [2, 42, 44]. For single level problems: min x ∈ R p f ( x ) =

1 n ∑ n i =1 f ( x ; ξ i ) , we iteratively perform the stochastic gradient step x t +1 = x t -η ∇ f ( x t , ξ t ) , where the sample gradient is used to estimate the full gradient and the example ξ t is sampled according to some order. More specifically, the sampling schemes are roughly divided into two categories: withreplacement sampling and without-replacement sampling. Traditional stochastic gradient descent typically employs with-replacement sampling to ensure an unbiased estimation of the full gradient in each iteration. In contrast, the without-replacement sampling has biased estimation at each iteration. Despite this, when averaging the gradients of consecutive examples, without-replacement sampling can approximate the full gradient more efficiently [34]. In fact, any permutation-based withoutreplacement sampling has that the gradient estimation error contracts at rate of O ( K -2 ) [34] where K is the number of steps, while stochastic gradient descent only has a rate of O ( K -1 ) . Inspired by the properties of without-replacement sampling demonstrated in the single-level optimization, we design the algorithm named WiOR-BO (and its variants) which performs without-replacement sampling for bilevel optimization. Naturally, WiOR-BO does not require the independent sampling as in previous methods. In fact, our algorithm enjoys favorable convergence rate. As shown in Table 1, WiOR-BO converges faster compared with their counterparts ( e.g. StocBiO [25]) using independent sampling. Actually, WiOR-BO gets comparable convergence rate with methods using more complicated variance reduction techniques ( e.g. MRBO [52]). In Table 2, we perform a more comprehensive comparison with other variance reduction techniques and in Section C of Appendix, we further compare WiOR-BO with these methods.

The contributions of our work can be summarized as:

We propose WiOR-BO which performs without-sampling to solve the bilevel optimization problem and converges to an ϵ stationary point with a rate of O( ϵ -3 ). This rate improves over the O( ϵ -4 ) rate of its counterparts ( e.g. stocBiO [25]) using independent sampling.

We propose WiOR-CBO for the conditional bilevel optimization problems and show that our algorithm converges to an ϵ stationary point with a rate of O( ϵ -4 ). This rate improves over the O( ϵ -6 ) rate of its counterparts ( e.g. DL-SGD [23]) using independent-sampling.

Wecustomize our algorithms to the special cases of minimax and compositional optimization problems and demonstrate similar favorable convergence rates.

Notations. ∇ ( ∇ x ) denotes full gradient (partial gradient, over variable x ), higher order derivatives follow similar rules. [ n ] represents sequence of integers from 1 to n . O ( · ) is the big O notation, and we hide logarithmic terms.

2 Related Works

Bilevel Optimization has its roots tracing back to the 1960s, beginning with the regularization method proposed by [51], and then followed by many research works [14, 47, 43, 55]. In the field of machine learning, similar implicit differentiation techniques were used to solve Hyper-parameter Optimization [28, 6, 12]. Exact solutions for Bilevel Optimization solve the inner problem for each outer variable but are inefficient. More efficient algorithms solve the inner problem with a fixed number of steps, and use the 'back-propagation through time' technique to compute the hypergradient [13, 35, 16, 40, 45]. Recently, single loop algorithms [17, 22, 25, 26, 52, 9, 30, 24] are introduced to perform the outer and inner updates alternatively. Besides, other aspects of the bilevel optimization are also investigated: [23] studied a type of conditional bilevel optimization where the inner problem depends on the example of the outer problem; [53, 56] studied the Hessian-free bilevel optimization and proposed an algorithm with optimal convergence rate; [32] proposed an efficient single loop algorithm for the general non-convex bilevel optimization; [54, 31] studies the bilevel optimization in the federated learning setting.

Sample-selection in stochastic optimization has been extensively studied for the case of single level optimization problems [2, 3, 19, 38]. Beyond the with-replacement sampling adopted by SGD, various without-replacement strategies were also studied in the literature. The random-reshuffling strategy shuffles the data samples at the start of each training epoch, and its property was first studied in [42] via the non-commutative arithmetic-geometric mean conjecture and followed by [20, 19]. [44, 36] studied the shuffling-once strategy, where the data samples are only shuffled at the start of the training once; [8] investigated data echoing where one data sample is repeatedly used. Quasi-Monte Carlo sampling is also applied in stochastic gradient descent [5, 34]. Recently, [34, 33] proposed to use average gradient error to study the convergence of various example order strategies and then proposed a herding-based sampling strategy [50] to proactively minimize the average gradient error.

3 Without-Replacement Sampling in Bilevel Optimization

In this work, we focus on the following finite-sum bilevel optimization problem:

where both the outer and the inner problems are defined over a finite dataset, and we denote them as D u = { ξ i , i ∈ [ m ] } and D l = { ζ j , j ∈ [ n ] } , respectively. Under mild assumptions, the hypergradient of h ( x ) is:

Our objective is to minimize h ( x ) by exploiting Eq. (3) with examples from D u and D l at each step. More specifically, for a given example order π denoting the following example-selection sequence as { ξ π t ∈ D u } and { ζ π t ∈ D l } for t ∈ [ T ] , where T denotes the length of the sequence and can be arbitrarily large. We then perform the following update steps iteratively for t ∈ [ T ] :

where Eq. (4) adopts the single-loop [30] update. More specifically, Eq. (4a) performs gradient descent to estimate the minimizer y x ; Eq. (4b) performs gradient descent to estimate u x defined in Eq. (3); and Eq. (4c) perform the gradient descent step given the estimation of y x and u x .

Algorithm 1 Without-Replacement Bilevel Optimization (WiOR-BO)

1: Input: Initial states x 0 I , y 0 I and u 0 I ; learning rates { γ r i , ρ r i , η r i } , i ∈ [ I ] , r ∈ [ R ] , I := lcm ( m,n ) denotes the least common multiple of m and n .

2: for epochs r = 1 to R do

3: Randomly sample I/m permutations of outer dataset and concatenate them to have { ξ π i , i ∈ [ I ] } , sample I/n permutations of inner dataset and concatenate them to have { ζ π i , i ∈ [ I ] } ;

5: for i = 0 to I -1 do

4: Set y r 0 = y r -1 I , u r 0 = P ι ( u r -1 I ) and x r 0 = x r -1 I

6: y r i +1 = y r i -γ r i ∇ y g ( x r i , y r i ; ζ π i ) ; u r i +1 = u r i -ρ r i ( ∇ y 2 g ( x r i , y r i ; ζ π i ) u r i -∇ y f ( x r i , y r i ; ξ π i )) ;

7: x r i +1 = x r i -η r i ( ∇ x f ( x r i , y r i ; ξ π i ) -∇ xy g ( x r i , y r i ; ζ π i ) u r i ) ;

8: end for

9: end for

Note that at each step t , we use a pair of examples { ξ π t , ζ π t } to estimate the involved properties and require only three backward passes, i.e. ∇ f ( x t , y t ; ξ π t ) , ∇ g ( x t , y t ; ζ π t ) and ∇ ( ∇ y g ( x t , y t ; ζ π t )) . In contrast, if we independently sample examples for each property, seven backward passes are needed. For large-scale machine learning models where back-propagation is expensive, our method can lead to significant computation saving in practice.

Equation (4) is independent of the specific order of samples. It can accommodate both withreplacement and without-replacement sampling strategies. In particular, we consider two most widely-used without-replacement sampling methods: random-reshuffling and shuffle-once . For random-reshuffling, we set T = R × lcm ( m,n ) , where lcm ( m,n ) denotes the least common multiple of m and n and R is a constant. More specifically, we generate R × lcm ( m,n ) /m random permutations of examples in D u then concatenate them to get { ξ π t ∈ D u } , and we follow similar procedure to generate { ζ π t ∈ D u } . In algorithm 1, we show our WiOR-BO algorithm for solving Eq. (2) with random-reshuffling sampling. Note that in Line 4 of Algorithm 1, P ι ( · ) is the projection operation to an ι -size ball and we perform projection to make u t be bounded during updates. Shuffleonce is another widely used without-replacement sampling strategy, where a single permutation is generated and reused for both D u and D l , and we can modify Line 3 of Algorithm 1 to incorporate shuffle-once. Finally, compared to independent sampling-based algorithm such as stocBiO [25], we require extra space to generate and store the orders of examples, but this cost is negligible compared to memory and computational cost of training large scale models.

3.1 Conditional Bilevel Optimization

In Eq. (2), we assume the outer dataset D u and the inner dataset D l are independent with each other. However, it is also common in practice that the inner problem not only relies on the outer variable x , but also the current outer example. This type of problems is known as conditional (contextual) bilevel optimization problems [23] in the literature and we consider the finite setting as follows:

Note that in Eq. (5), the outer problem is defined on a finite dataset D u = { ξ i , i ∈ [ m ] } , and for each example ξ i ∈ D u , we have a inner problem g ( ξ i ) ( x, y ) defined on a dataset D l,ξ i = { ζ ξ i ,j , j ∈ [ n ξ i ] } . The hyper-gradient of Eq. (5) is:

The key difference of Eq. (6) with Eq. (3) is that y ξ i x and u ξ i x are defined for each example ξ i ∈ D u . To minimize h ( x ) in Eq. (5), we assume a sample order π denoting the following example selection sequence as { ξ π t ∈ D u , t ∈ [ T ] } , where T denotes the example sequence length and can be arbitrarily

large. We then perform the following update steps iteratively:

where ˆ y t and ˆ u t are estimations of y ξ π t x t u ξ π t x t . We get them by performing the following rules T l steps over the sample order { ζ π ξ π t ,t l ∈ D l,ξ π t , t l ∈ [ T l ] }} :

where both the outer variable state x t and sample ξ π t are fixed. We then set ˆ y t = y T l and ˆ u t = u T l in Eq. (7). Note that Eq. (7) and Eq. (8) perform a double-loop update rule [23]. Since the inner problem relies on the example of the outer problem, we compute y ξ i x and u ξ i x for each new state x and sample ξ i instead of reusing them as in the single loop update of Eq. (4).

Note that an important feature of our algorithm is that we select samples based on an example order π at each step instead of sampling independently. Similar to the unconditional bilevel optimization, various example orders can be incorporated. For random-reshuffling, we set T = Rm with R be a constant. Then we generate R random permutations of examples in D u and concatenate them to get the example order { ξ π t ∈ D u , t ∈ [ T ] } . Similarly, we concatenate S random permutations of the inner dataset D l,ξ t to get { ζ π ξ π t ,t l ∈ D l,ξ π t , t l ∈ [ T l ] }} with a sequence length T l = S × n ξ i . We summarize this in Algorithm 2 and denote it as WiOR-CBO . We slightly abuse the notation to replace n ξ i with n i in the inner loop update for better clarity.

3.2 MiniMax and Compositional Optimization Problems

Our discussion of without-replacement sampling for bilevel optimization can be customized for two important nested optimization problems: compositional optimization and minimax optimization problems, as they can be viewed as a special type of bilevel optimization problem. Suppose we let the outer problem only rely on y , set the inner problem as g ( x, y ) = 1 2 ∥ y -r ( x ) ∥ 2 . Consider the finite case of f ( y ) and r ( x ) , we have the following compositional optimization problem:

Then we can derive the hyper-gradient as: ∇ h ( x ) = ∇ x r ( x ) u x , where u x = ∇ y f ( r ( x )) , meanwhile, we have ∇ y g ( x, y ) = y -r ( x ) . Then we can instantiate Algorithm 1 to get an algorithm for compositional optimization problem using without-replacement sampling (Algorithm 3 of Appendix). Similar to the SCGD algorithm [49], we track the exponential moving average of the inner state

y , besides we also track the exponential moving average of inner gradient (the update rule of u in Algorithm 3).

Next, similar to the unconditional case, the conditional compositional optimization problem can be specialized from the conditional bilevel optimization problem Eq. (5) to have:

and we can instantiate Algorithm 2 in the special case of Eq. (10) to get Algorithm 5 (in the Appendix).

The minimax optimization can also be viewed as a special type of bilevel optimization. More specifically, if we have g ( x, y ) = -f ( x, y ) in Eq. (2), then we get the following minimax problem:

In the special case of Eq. (11), the hyper-gradient ∇ h ( x ) in Eq. (3) degenerates to ∇ h ( x ) = ∇ x f ( x, y x ) due to the fact that ∇ y f ( x, y x ) = -∇ y g ( x, y x ) = 0 by the optimality condition. Meanwhile, we have ∇ y g ( x, y ) = -∇ y f ( x, y ) . As a result, we can simplify Algorithm 1 to get an alternative update algorithm for minimax optimization using without-replacement sampling (Algorithm 4 in the Appendix). Note that due to the linearity of minimax problem w.r.t. examples, the conditional case for minimax optimization degenerates to the unconditional case. [11, 7] also considers the without-replacement sampling for the minimax problem, however, they focus on the strongly-convex-strongly-concave and two-sided Polyak-Łojasiewicz settings, in contrast, our WiOR-MiniMax can be applied to nonconvex-strongly-concave setting.

4 Theoretical Analysis

In this section, we provide theoretical analysis of our algorithms. We first state some assumptions:

Assumption 4.1. Function f ( x, y ; ξ ) , ξ ∈ D u , is possibly non-convex, L -smooth and has C f -bounded gradient w.r.t. the variable y .

Assumption 4.2. Function g ( x, y ; ζ ) , ζ ∈ D l , is µ -strongly convex w.r.t y for any given x , and L -smooth. Furthermore, ∇ xy g ( x, y ; ζ ) and ∇ y 2 g ( x, y ; ζ ) are Lipschitz continuous with constants L xy and L y 2 respectively.

Assumption 4.3. For all examples ξ i ∈ D u and states ( x, y ) ∈ R p + d , there exists an upper bound A such that the sample gradient errors satisfy ∥∇ f ( x, y ; ξ i ) -∇ f ( x, y ) ∥ ≤ A .

Assumption 4.4. For all examples ζ j ∈ D l and states ( x, y ) ∈ R p + d , there exists an upper bound A such that the sample gradient errors satisfy ∥∇ g ( x, y ; ζ i ) -∇ g ( x, y ) ∥ ≤ A and ∥∇ 2 g ( x, y ; ζ i ) -∇ 2 g ( x, y ) ∥ ≤ A .

As stated in The assumption 4.1 and 4.2, we consider the non-convex-strongly-convex bilevel optimization problems, this class of problems is widely studied in bilevel optimization literature [17, 25].Assumption 4.3 and 4.4 guarantee that the example gradients are good estimation of full gradient and is widely used in the analysis of single-level finite-sum optimization problems [36]. Note that as the hyper-gradient (Eq. 3) involves second order derivatives, Assumption 4.4 also requires that the second order example gradients have bounded bias.

We next make the following assumptions about the example selection sequence. We assume { ξ π t ∈ D u , t ∈ [ T ] } and { ζ π t ∈ D l , t ∈ [ T ] } satisfy:

Assumption 4.5. Given sequences of samples: { ξ π t ∈ D u , t ∈ [ T ] } and { ζ π t ∈ D l , t ∈ [ T ] } , we assume there exists some constants α , C such that for any step t and any interval k > 0 , we have:

and similar bounds hold for ∇ g and ∇ 2 g (See Appendix B.1 for the full version of this assumption.)

Assumption 4.5 measures how well the sample gradients approximate the full gradient over an interval on average. Similar assumptions are used to study the sample order in single level optimization problems [34]. If samples are selected independently at each step, we get α = 1 and C = A based on Assumption 4.3 and 4.4. For any permutation-based order ( e.g. random-reshuffling and shuffle-once), we can show the assumption holds with α = 2 and C = max( m,n ) × A . Note that the C depends on the number of samples m ( n ), this dependence can be reduced to (max( m,n )) 0 . 5 for sufficiently small learning rates [34, 37]. More recently, [33] proposed a herding-based algorithm which explicitly optimizes the gradient error in Assumption 4.5 and shows that we can reach α = 2 and C = O ( A ) .

Bilevel Optimization . We are ready to show the convergence of Algorithm 1. Firstly, we denote the following potential function G t := G t = h ( x t ) + ϕ y ∥ y t -y x t ∥ 2 + ϕ u ∥ u t -u x t ∥ 2 , where ϕ y and ϕ u are constants that relates to the smoothness parameters of h ( x ) . Then we have:

Theorem 4.6. Suppose Assumptions 4.1-4.4 are satisfied, and Assumption 4.5 is satisfied with some α and C for the example order we use in Algorithm 1. We choose learning rates η t = η , γ t = c 1 η , ρ t = c 2 η and denote T = R × I be the total number of steps. Then for any pair of values (E, k) which has T = E × k and η ≤ 1 k ˜ L 0 , we have:

where c 1 , c 2 , ˜ L 0 , ˜ L are constants related to the smoothness parameters of h ( x ) , ∆ is the initial sub-optimality, R and I are defined in Algorithm 1.

To reach an

ϵ

stationary point, we choose

η

=

1

˜

k

L

0

4 ˜

L

0

ϵ

2

( 4 ˜ L 2 C 2 ϵ 2 ) 1 /α , which means we need to choose the number of epochs R = T I = 4 ˜ L 0 (4 ˜ L 2 C 2 ) 1 /α ∆ Iϵ 2+2 /α . Specially, if we choose examples independently at each step, we have C = A and α = 1 , then we need T = O ( ϵ -4 ) steps to reach an ϵ -stationary point. This recovers the rate of stocBiO algorithm [25] and SOBA [9]. Next, if we use some permutation-based without-replacement sampling strategy witch has C = (max( m,n )) q × A ( q ∈ [0 , 1] ) and α = 2 , then we need T = O ((max( m,n )) q ϵ -3 ) . In particular, by adopting a herding-based permutation as in [33], we get q = 0 , and our WiOR-BO strictly outperforms the independent sampling-based stocBiO and SOBA algorithms.

Conditional Bilevel Optimization . Next, we study the convergence rate of Algorithm 2. Besides Assumptions 4.3 and 4.4, we need the bias of sample hyper-gradient be bounded too:

Assumption 4.7. For all examples ξ i ∈ D u and states x ∈ R p , there exists an upper bound A such that the sample gradient errors satisfy ∥∇ h ( x ; ξ i ) -∇ h ( x ) ∥ ≤ A .

We also augment Assumption 4.5 to include the case of ∇ h ( x ) (Seem Appendix B.1 for more details.). Then we are ready to show the convergence rate, we use the potential function G t = h ( x t ) and have:

Theorem 4.8. Suppose Assumptions 4.1-4.4 and 4.7 are satisfied, and Assumption 4.5 is satisfied with some α and C for the example order we use in Algorithm 2. We choose learning rates η t = η = 1 8 k ¯ L , γ t = γ = 1 256 kLκ , ρ t = ρ = 1 512 kLκ and denote T = R × m be the total number of outer steps and T l = S × max( n i ) be the maximum inner steps. Then for any pair of values (E, k) which has T = E × k and T l = E l × k , we have:

where C u , C y , C ′ , ˜ L are constants related to the smoothness parameters of h ( x ) , ∆ h , ∆ u , ∆ y are the initial sub-optimality, R and S are defined in Algorithm 2.

To reach an ϵ stationary point, we choose: E = 128 ¯ L ∆ ϵ 2 , and , k = ( 2( C ′ ) 2 C 2 ϵ 2 ) 1 /α and E l =

O ( log ( ϵ -1 )) . Then the sample complexity of Algorithm 2 is EE l k 2 = O ( C 4 /α ϵ -(2+4 /α ) ) , where we omit the logarithmic term E l . Specially, if we choose examples independently at each step, then we have sample complexity O ( ϵ -6 ) steps for an ϵ -stationary point. This recovers the rate of DL-SGD algorithm [23]. Next, if we use some permutation-based without-replacement sampling strategy,

, and for

(

E,k

)

, we choose:

E

=

∆

and

k

=

then we have sample complexity O ((max( m,n )) 2 q ϵ -4 ) , which can match the state-of-art algorithm RT-MLMC [23] if we choose an example order with q = 0 [33].

MiniMax and Compositional Optimization . As special cases of Bilevel Optimization, we can also show that Algorithm 3, Algorithm 4 and Algorithm 5 converge under similar rate. Please see Corollary B.21 and Corollary B.20 for more details.

5 Applications and Numerical Experiments

In this section, we verify the effectiveness of the proposed algorithm through a synthetic invariant risk minimization task and two real world bilevel tasks: Hyper-Data Cleaning and Hyper-representation Learning. The code is written in Pytorch. Our experiments were conducted on servers equipped with 8 NVIDIA A5000 GPUs. Experiments are averaged over five independent runs, and the standard deviation is indicated by the shaded area in the plots. 2.2

5.1 Invariant Risk-Minimization

In this section, we consider a synthetic invariant risk minimization task [21]. More specifically, we perform logistic regression over a set of examples { ( c i , b i ) , i ∈ [ M ] } with c be the input and b be the target. However, we does not have access to c but a set of noisy observations { c j,i , j ∈ n i } for each c i . To learn the coefficient x between c and b , we optimize the following objective:

WiR-CBO

WiOR-CBO(SO)

WiOR-CBO(RR)

2.0

Validation Loss

1.8

1.6

1.4

0

600

1200

1800

2400

NumberofHyper-lteration

This is a conditional bilevel optimization problem and we can solve it using our WiOR-CBO. We generate synthetic data to test the effects of different sample order (sample generation process is described in Appendix A). More specifically, we compare independent-sampling (WiR-CBO), shuffleonce (WiOR-CBO (SO)) and random-reshuffling (WiOR-CBO (RR)). As shown in Figure 1, the two without-replacement sampling methods outperforms the independent sampling one.

5.2 Hyper-Data Cleaning

In the Hyper-Data Cleaning task, we are given noisy training dataset whose labels are corrupted by noise, meanwhile, we have a validation set whose labels are not corrupted. Our target is then using the clean validation samples to identify corrupted samples in the training set. More specifically, we learn optimal weights for training samples such that a model learned over the weighted training set performs well on the validation set. We can formulate this task as a bilevel optimization problem (Eq. (2)). A mathematical formulation of the task is included in Appendix A.

Dataset and Baselines. We construct datasets based on MNIST [29]. For the training set, we randomly sample 40000 images from the original training dataset and then randomly perturb a fraction of labels of samples. For the validation set, we randomly select 5000 clean images from the original training dataset. In our experiments, we test our WiOR-BO algorithm (Algorithm 1), with the shuffle-once (WiOR-BO-SO) and random-reshuffling (WiOR-BO-RR) sampling; additionally, we also consider the following non-adaptive bilevel algorithms as baselines: reverse [16], stocBiO [25], BSA [17], AID-CG [18], MRBO [52], VRBO [52] and WiR-BO (the variant of our WiOR-BO using independent sampling, which is similar to the single loop algorithms such as SOBA [9], AmIGO [1] and FLSA [30]). We perform grid search for hyper-parameters and report the best results.

We summarize the results in Figure 2. In the figure, we compare the validation loss and F1 score w.r.t. both Hyper-iterations and running time. As shown in the figure, the two variants of our algorithm WiOR-BO-SO and WiOR-BO-RR get similar performance and they both outperform other baselines. In particular, note that WiR-BO differs with WiOR-BO-SO (RR) only over the example order, but the

0.85

0.8

0.55

0.55

Validation Loss

Validation Loss

Validation Loss

score

0.7

040

0.25

0.40

reverse

stocBiO

BSA

AID-CG

VRBO

MRBO

WiR-BO

WiOR-BO (SO)

0.6

0.5

reverse

stocBiO

BSA

AID-CG

MRBO

VRBO

WiR-BO

WiOR-BO (SO)

0.40

0.25

reverse

stocBiO

BSA

AID-CG

VRBO

MRBO

WR-BO

WOR-BO (SO)

0.55

reverse

stocBiO

BSA

AID-CG

VRBO

MRBO

WR-BO

WiOR-BO (SO)

WiOR-BO (RR)

WiOR-BO (RR)

WIOR-BO (RR)

WiOR-BO (RR)

0.10

0

500

Number of Hyper-lteration

1000

1500

0.4

0

500

Number of Hyper-lterations

1000

1500

0.10

0

10

Runinng Time (s)

20

30

0.40

0

10

Runinng Time (s)

20

WIOR-CBO (RR)

DL-SGD

RT-MLMC

(05) 080-H0M

0.25

DL-SGD

RT-MLMC

WIOR-CBO (SO)

WIOR-CBO (RR)

1.4

WIOR-CBO (RR)

DL-SGD

RT-MLMC

WIOR-CBO (SO)

1.1

WIOR-CBO (RR)

095-10

RT-MLMC

WiOR-CBO (SO)

SS07

Validation Loss

Validation Loss

0.20

1.2

0.9

0.4

0.15

1.0

0.7

0.3

0

15

Running Time (s)

30

0.10

0

15

Running Time (s)

30

0.8

0

15

Running Time (s)

30

0.5

0

15

Running Time (s)

30

DL-SGD

DL-SGD

DL-SGD

81

RT-MLMC

RT-MLMC

RT-MLMC

84

WIOR-CBO (SO)

WiOR-CBO (RR)

WiOR-CBO (RR)

WiOR-CBO (SO)

68

WiOR-CBO (SO)

WiOR-CBO (RR)

Test Accuracy

Test Accuracy

94

Test Accuracy

Test Accuracy

76

62

8

26

56

71

DL-SGD

RT-MLMC

(05) 08-0

75

0

15

Running Time (s)

30

90

0

15

Running Time (s)

30

50

0

15

Running Time (s)

30

66

0

15

Running Time (s)

30

WIOR-CBO (RR)

5.3 Hyper-Representation Learning

In this section, we consider Hyper-representation learning task. In this task, we learn a hyperrepresentation of the data such that a linear classifier can be learned quickly with a small number of samples. We consider the Omniglot [27] and MiniImageNet [41] data sets. This task can be viewed as a conditional bilevel optimization problem (Eq. (2)) and a mathematical formulation of the task is included in Appendix A.

Dataset and Baselines. The details of the datasets are included in Appendix A and we consider N-way-K-shot classification task following [48]. In our experiments, we test our WiOR-CBO (Algorithm 2), using the shuffle-once (WiOR-BO-SO) and random-reshuffling (WiOR-BO-RR) sampling. Besides, we compare with the following baselines: DL-SGD [23] and RT-MLMC [23]. Note that our WiOR-CBO can also use independent sampling, and it has similar performance as DL-SGD. We perform grid search of hyper-parameters for each method and report the best results.

We summarize the experimental results for the Omniglot dataset in Figure 3 and we defer the results for MiniImageNet to the Appendix A. As shown in the figure, our WiOR-CBO SO/RR outperforms DL-SGD and is comparable to the state-of-the-art algorithm RT-MLMC.

6 Conclusion

In this work, we investigated example-selection of bilevel optimization. Beyond the classical independent sampling, we assumed an example-order based on without-replacement sampling, such as random-reshuffling and shuffle-once. We proposed the WiOR-BO algorithm for bilevel optimization and show the algorithm converges to an ϵ -stationary point with rate O ( ϵ -3 ) . After that, we also discussed the conditional bilevel optimization problems and introduced an algorithm with convergence rate of O ( ϵ -4 ) . As special cases of bilevel optimization, we studied the minimax and

compositional optimization problems. Finally, we validated the efficacy of our algorithms through one synthetic and two real-world tasks; the numerical results show the superiority of our algorithms.

Acknowledgments

This work was partially supported by NSF IIS 2347592, 2347604, 2348159, 2348169, DBI 2405416, CCF 2348306, CNS 2347617.

References

M. Arbel and J. Mairal. Amortized implicit differentiation for stochastic bilevel optimization. arXiv preprint arXiv:2111.14580 , 2021.

L. Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade: Second Edition , pages 421-436. Springer, 2012.

L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM review , 60(2):223-311, 2018.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners, 2020.

A. Buchholz, F. Wenzel, and S. Mandt. Quasi-monte carlo variational inference. In International Conference on Machine Learning , pages 668-677. PMLR, 2018.

D. Chen and M. T. Hagan. Optimal use of regularization and cross-validation in neural network modeling. In IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH36339) , volume 2, pages 1275-1280. IEEE, 1999.

H. Cho and C. Yun. Sgda with shuffling: faster convergence for nonconvex-p {\ L } minimax optimization. arXiv preprint arXiv:2210.05995 , 2022.

D. Choi, A. Passos, C. J. Shallue, and G. E. Dahl. Faster neural network training with data echoing. arXiv preprint arXiv:1907.05550 , 2019.

M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. arXiv preprint arXiv:2201.13409 , 2022.

M. Dagréou, T. Moreau, S. Vaiter, and P. Ablin. A lower bound and a near-optimal algorithm for bilevel empirical risk minimization. In S. Dasgupta, S. Mandt, and Y. Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 82-90. PMLR, 02-04 May 2024.

A. Das, B. Schölkopf, and M. Muehlebach. Sampling without replacement leads to faster rates in finite-sum minimax optimization, 2022.

C. B. Do, C.-S. Foo, and A. Y. Ng. Efficient multiple hyperparameter learning for log-linear models. In NIPS , volume 2007, pages 377-384. Citeseer, 2007.

J. Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics , pages 318-326. PMLR, 2012.

M. C. Ferris and O. L. Mangasarian. Finite perturbation of convex programs. Applied Mathematics and Optimization , 23(1):263-273, 1991.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1126-1135. JMLR. org, 2017.

L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1165-1173. JMLR. org, 2017.

S. Ghadimi and M. Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

R. Grazzi, L. Franceschi, M. Pontil, and S. Salzo. On the iteration complexity of hypergradient computation. In International Conference on Machine Learning , pages 3748-3758. PMLR, 2020.

M. Gürbüzbalaban, A. Ozdaglar, and P. A. Parrilo. Why random reshuffling beats stochastic gradient descent. Mathematical Programming , 186:49-84, 2021.

J. Haochen and S. Sra. Random shuffling beats SGD after finite epochs. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2624-2633. PMLR, 09-15 Jun 2019.

L. He and S. P. Kasiviswanathan. Debiasing conditional stochastic optimization. arXiv preprint arXiv:2304.10613 , 2023.

M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05170 , 2020.

Y. Hu, J. Wang, Y. Xie, A. Krause, and D. Kuhn. Contextual stochastic bilevel optimization, 2023.

F. Huang, J. Li, S. Gao, and H. Huang. Enhanced bilevel optimization via bregman distance. Advances in Neural Information Processing Systems , 35:28928-28939, 2022.

K. Ji, J. Yang, and Y . Liang. Provably faster algorithms for bilevel optimization and applications to meta-learning. arXiv preprint arXiv:2010.07962 , 2020.

P. Khanduri, S. Zeng, M. Hong, H.-T. Wai, Z. Wang, and Z. Yang. A near-optimal algorithm for stochastic bilevel optimization via double-momentum. arXiv preprint arXiv:2102.07367 , 2021.

B. Lake, R. Salakhutdinov, J. Gross, and J. Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society , volume 33, 2011.

J. Larsen, L. K. Hansen, C. Svarer, and M. Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop , pages 62-71. IEEE, 1996.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

J. Li, B. Gu, and H. Huang. A fully single loop algorithm for bilevel optimization without hessian inverse. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7426-7434, 2022.

J. Li, F. Huang, and H. Huang. Communication-efficient federated bilevel optimization with global and local lower level problems. Advances in Neural Information Processing Systems , 36, 2024.

R. Liu, Z. Liu, W. Yao, S. Zeng, and J. Zhang. Moreau envelope for nonconvex bi-level optimization: A single-loop and hessian-free solution strategy, 2024.

Y. Lu, W. Guo, and C. M. De Sa. Grab: Finding provably better data permutations than random reshuffling. Advances in Neural Information Processing Systems , 35:8969-8981, 2022.

Y. Lu, S. Y. Meng, and C. De Sa. A general analysis of example-selection for stochastic gradient descent. In International Conference on Learning Representations (ICLR) , volume 10, 2022.

D. Maclaurin, D. Duvenaud, and R. Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning , pages 21132122, 2015.

K. Mishchenko, A. Khaled, and P. Richtárik. Random reshuffling: Simple analysis with vast improvements. Advances in Neural Information Processing Systems , 33:17309-17320, 2020.

A. Mohtashami, S. Stich, and M. Jaggi. Characterizing & finding good data orderings for fast convergence of sequential gradient methods, 2022.

L. M. Nguyen, Q. Tran-Dinh, D. T. Phan, P. H. Nguyen, and M. Van Dijk. A unified convergence analysis for shuffling-type gradient methods. The Journal of Machine Learning Research , 22(1):9397-9440, 2021.

T. Okuno, A. Takeda, and A. Kawana. Hyperparameter learning via bilevel nonsmooth optimization. arXiv preprint arXiv:1806.01520 , 2018.

F. Pedregosa. Hyperparameter optimization with approximate gradient. arXiv preprint arXiv:1602.02355 , 2016.

S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International conference on learning representations , 2017.

B. Recht and C. Re. Toward a noncommutative arithmetic-geometric mean inequality: Conjectures, case-studies, and consequences. In S. Mannor, N. Srebro, and R. C. Williamson, editors, Proceedings of the 25th Annual Conference on Learning Theory , volume 23 of Proceedings of Machine Learning Research , pages 11.1-11.24, Edinburgh, Scotland, 25-27 Jun 2012. PMLR.

S. Sabach and S. Shtern. A first order method for solving convex bilevel optimization problems. SIAM Journal on Optimization , 27(2):640-660, 2017.

I. Safran and O. Shamir. How good is sgd with random shuffling? In Conference on Learning Theory , pages 3250-3284. PMLR, 2020.

A. Shaban, C.-A. Cheng, N. Hatch, and B. Boots. Truncated back-propagation for bilevel optimization. arXiv preprint arXiv:1810.10667 , 2018.

A. Shamsian, A. Navon, E. Fetaya, and G. Chechik. Personalized federated learning using hypernetworks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9489-9502. PMLR, 18-24 Jul 2021.

M. Solodov. An explicit descent method for bilevel convex optimization. Journal of Convex Analysis , 14(2):227, 2007.

O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems , pages 3630-3638, 2016.

M. Wang, E. X. Fang, and H. Liu. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming , 161(1-2):419-449, May 2016.

M. Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1121-1128, 2009.

R. A. Willoughby. Solutions of ill-posed problems (an tikhonov and vy arsenin). SIAM Review , 21(2):266, 1979.

J. Yang, K. Ji, and Y . Liang. Provably faster algorithms for bilevel optimization. arXiv preprint arXiv:2106.04692 , 2021.

Y. Yang, P. Xiao, and K. Ji. Achieving O ( ϵ -1 . 5 ) complexity in hessian/jacobian-free stochastic bilevel optimization, 2023.

Y. Yang, P. Xiao, and K. Ji. Simfbo: Towards simple, flexible and communication-efficient federated bilevel learning. Advances in Neural Information Processing Systems , 36, 2024.

M. Ye, B. Liu, S. Wright, P. Stone, and Q. Liu. Bome! bilevel optimization made easy: A simple first-order approach, 2022.

P. Yu, J. Li, and H. Huang. Dropout enhanced bilevel training. In The Twelfth International Conference on Learning Representations , 2023.

L. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning , pages 7693-7702. PMLR, 2019.

A More Details for Numerical Experiments

In this section, we introduce more details of the experiments.

A.1 Invariant Risk-Minimization

The data generation process is as follows: we first randomly sample a ground truth x ∗ , then we randomly generate m = 1000 input variables c , and calculate the ground truth label b = cx , then for each c i , we generate a set of n = 100 noisy observations by adding Guassian noise of scale 0.1. We also add a L 2 regularization term of strength 0.1 for the outer problem. During trainging, we use both the inner and outer learing rates of 0.001.

A.2 Hyper-Data Cleaning

The formulation of the problem is as follows:

In the above formulation, we have a pair of (noisy) training set { ξ tr n } N ( tr ) n =1 and validation set { ξ val n } N ( val ) n =1 , and x n , n ∈ [ N ( tr ) ] are weights for training samples, y is the parameter of a model, and we denote the model by Θ . Note that y x is the model learned over the weighted training set. We fit a model with 3 fully connected layers for the MNIST dataset. We also use L 2 regularization with coefficient 10 -3 to satisfy the strong convexity condition. In the Experiments, we choose inner learning rate ( γ , ρ ) as 0.1 and outer learning rate η 1000.

2.4

DL-SGD

RT-MLMC

1.8

RT-MLMC

DL-SGD

DL-SGD

RT-MLMC

095·10

RT-MLMC

WiOR-CBO (RR)

(05) 080-H0M

WiOR-CBO (RR)

WiOR-CBO (SO)

3.3

WIOR-CBO (RR)

WIOR-CBO (SO)

3.0

WiOR-CBO (SO)

WIOR-CBO (RR)

Validation Loss

Validation Loss

Validation Loss

2.1

2.8

1.8

1.4

1.5

0

150

Running Time (s)

300

1.2

0

150

Running Time (s)

300

2.9

0

150

Running Time (s)

300

2.6

0

150

Running Time (s)

300

32

DL-SGD

WiOR-CBO (RR)

RT-MLMC

WiOR-CBO (SO)

45

DL-SGD

RT-MLMC

WiOR-CBO (RR)

WiOR-CBO (SO)

DL-SGD

WiOR-CBO (SO)

RT-MLMC

WiOR-CBO (RR)

14

RT-MLMC

DL-SGD

Validation Loss

Validation Loss

Validation Loss

30

3

8

28

39

12

26

0

150

Running Time (s)

300

36

0

150

Running Time (s)

300

150

Running Time (s)

300

11

0

150

Running Time (s)

300

A.3 Hyper-Representation Learning

In the above formulation, we have N tasks and each task T n is defined by a pair of training set { ξ tr i } N tr n i =1 and validation set { ξ val i } N val n i =1 . Θ defines the model, x is the parameter of the backbone

model and y is the parameter of the linear classifier. In summary, the lower level problem is to learn the optimal linear classifier y given the backbone x , and the upper level problem is to learn the optimal backbone parameter x .

The Omniglot dataset includes 1623 characters from 50 different alphabets and each character consists of 20 samples. We follow the experimental protocols of [48] to divide the alphabets to train/validation/test with 33/5/12, respectively. We perform N -wayK -shot classification, more specifically, for each task, we randomly sample N characters and for each character, we sample K samples for training and 15 samples for validation. We augment the characters by performing rotation operations (multipliers of 90 degrees). We use a 4-layer convolutional neural network where each convolutional layer has 64 filters of 3 × 3 [15]. For the MiniImageNet, it has 64 training classes and 16 validation classes. Similar to Omniglot, we also perform the N -wayK -shot classification. We use a 4-layer convolutional neural network where each convolutional layer has 64 filters of 3 × 3 [15] for experiments. For the experiments, we use inner learning rates 0.4 and outer learning rates 0.1 for Omniglot related experiments and inner learning rates 0.01 and outer learning rates 0.05 for MiniImageNet-related experiments. We perform 4 inner gradient descent steps and set K m ax = 6 for the RT-MLMC method. The experimental results for the MiniImageNet dataset is shown in Figure 4.

B Proof for Theorems

In this section, we provide proof for the convergence results in the main text. First, we restate all assumptions needed in our proof below:

Assumption B.1 (Assumption 4.1) . The function f ( x, y ) is possibly non-convex and uniformly L -smooth, i.e. for any x 1 , x 2 ∈ X , y 1 , y 2 ∈ R d , any ξ ∈ D u . Denote z 1 = ( x 1 , y 1 ) , z 2 = ( x 2 , y 2 ) , then we have:

or equivalently: ||∇ f ( z 1 ; ξ ) -∇ f ( z 2 ; ξ ) || ≤ L || z 1 -z 2 || . We also assume for any x ∈ X and any y ∈ R d , and we denote z = ( x, y ) , then we have ||∇ y f ( z ) || ≤ C f .

Assumption B.2 (Assumption 4.2) . Function g ( x, y ) is uniformly µ -strongly convex w.r.t y for any given x and uniformly L -smooth, i.e. for any y 1 , y 2 ∈ R d and ζ ∈ D l , we have:

and we have:

equivalently: ||∇ g ( z 1 ; ζ ) - ∇ g ( z 2 ; ζ ) || ≤ L || z 1 -z 2 || . Furthermore, we have ∇ xy g ( x, y ) and ∇ y 2 g ( x, y ) are Lipschitz continuous with constant L xy and L y 2 respectively, i.e. we have: ||∇ xy g ( z 1 ; ζ ) -∇ xy g ( z 2 ; ζ ) || ≤ L xy || z 1 -z 2 || and ||∇ y 2 g ( z 1 ; ζ ) -∇ y 2 g ( z 2 ; ζ ) || ≤ L y 2 || z 1 -z 2 || .

Assumption B.3 (Assumption 4.3) . For all examples ξ i ∈ D u and states ( x, y ) ∈ R p + d , there exists an outer bound A such that the sample gradient errors satisfy ∥∇ f ( x, y ; ξ i ) -∇ f ( x, y ) ∥ ≤ A .

Assumption B.4 (Assumption 4.4) . For all examples ζ j ∈ D l and states ( x, y ) ∈ R p + d , there exists an outer bound A such that the sample gradient errors satisfy ∥∇ g ( x, y ; ζ i ) -∇ g ( x, y ) ∥ ≤ A and ∥∇ 2 g ( x, y ; ζ i ) -∇ 2 g ( x, y ) ∥ ≤ A .

Assumption B.5 (Assumption 4.7) . For all examples ξ i ∈ D u and states x ∈ R p , there exists an upper bound A such that the sample gradient errors satisfy ∥∇ h ( x ; ξ i ) -∇ h ( x ) ∥ ≤ A .

Assumption B.6. Given two sequences { ξ π t ∈ D u , t ∈ [ T ] } and { ζ π t ∈ D l , t ∈ [ T ] } , we assume there exists some constants α , C such that for any step t and any k > 0 , we have: ∥ ∥ 1 k ∑ t + k -1 τ = t ∇ f ( x t , y t ; ξ π τ ) -∇ f ( x t , y t ) ∥ ∥ 2 ≤ k -α C 2 , ∥ ∥ 1 k ∑ t + k -1 τ = t ∇ g ( x t , y t ; ζ π τ ) - ∇ g ( x t , y t ) ∥ ∥ 2 ≤ k -α C 2 , ∥ ∥ 1 k ∑ t + k -1 τ = t ∇ 2 g ( x t , y t ; ζ π τ ) - ∇ 2 g ( x t , y t ) ∥ ∥ 2 ≤ k -α C 2 , ∥ ∥ 1 k ∑ t + k -1 τ = t ∇ h ( x t ; ξ π τ ) - ∇ h ( x t ) ∥ ∥ 2 ≤ k -α C 2 ( Only for the conditional bilevel optimization case ) .

Algorithm 3 Without-Replacement Compositional Optimization (WiOR-Comp)

1: Input: Initial states x 0 I , y 0 I and u 0 I ; learning rates { γ r i , ρ r i , η r i } , i ∈ [ I ] , r ∈ [ R ] , I = lcm ( m,n ) .

2: for epochs r = 1 to R do

3: Generate sample order sequence { ξ π i } and { ζ π i } for i ∈ [ I ] as Line 3 of Algorithm 1;

5: for i = 0 to I -1 do

4: Set y r 0 = y r -1 I , u r 0 = P ι ( u r -1 I ) and x r 0 = x r -I

6: y r i +1 = (1 -γ r i ) y r i + γ r i r ( x r i , ζ π i ) ; u r i +1 = (1 -ρ r i ) u r i + ρ r i ∇ y f ( y r i ; ξ π i ) ;

7: x r i +1 = x r i -η r i ∇ r ( x r i ; ζ π i ) u r i ;

8: end for

9: end for

Algorithm 4 Without-Replacement MiniMax Opt. (WiOR-MiniMax)

1: Input: Initial states x 0 m , y 0 m ; learning rates { γ r i , η r i } , i ∈ [ m ] , r ∈ [ R ] .

2: for epochs r = 1 to R do

3: sample a permutation of outer dataset to have { ξ π i , i ∈ [ m ] } ;

4: Set y r 0 = y r -1 m and x r 0 = x r -1 m

5: for i = 0 to m -1 do

6: y r i +1 = y r i + γ r i ∇ y f ( x r i , y r i ; ξ π i ) ;

7: x r i +1 = x r i -η r i ∇ x f ( x r i , y r i ; ξ π i ) ;

8: end for

9: end for

B.1 Proof for Bilevel Optimization Problems

Suppose we define:

Then we have ∇ h ( x ) = Φ( x, y x ) . Furthermore, we have the following proposition:

Proposition B.7. Suppose Assumptions 4.1 and 4.2 hold, the following statements hold:

y x is Lipschitz continuous in x with constant ρ = κ , where κ = L µ is the condition number of g ( x, y ) .

h ( x ) is smooth in x with constant ¯ L i.e., for any given x 1 , x 2 ∈ X , we have ∥∇ h ( x 2 ) -∇ h ( x 1 ) ∥ ≤ ¯ L ∥ x 2 -x 1 ∥ where ¯ L = O ( κ 3 ) .

This is a standard results in bilevel optimization and we omit the proof here.

The main technique we use to analysis the convergence of Algorithm 1 is aggregated analysis , in other words, we perform analysis over a block of length k instead of the more common per step analysis in stochastic bilevel optimization. Note that this technique is essential in analyzing without-replacement sampling for single-level optimization problems, and we extend them to the bilevel setting. More specifically, suppose the total number of training steps are T := R × I , where R and I are denoted in Algorithm 1, then we consider a block of training steps [ t, t + k ] for t ∈ [ T -k ] . Furthermore, we denote: ¯ η t = ∑ t + k -1 τ = t η τ , and ¯ ν t = 1 ¯ η t ∑ t + k -1 τ = t η τ ν τ ; ¯ γ t = ∑ t + k -1 τ = t γ τ and ¯ w t = 1 ¯ γ t ∑ t + k -1 τ = t γ τ w τ ; ¯ ρ t = ∑ t + k -1 τ = t ρ τ , and ¯ q t = 1 ¯ ρ t ∑ t + k -1 τ = t ρ τ q τ , where ν t = ∇ x f ( x t , y t ; ξ t ) - ∇ xy g ( x t , y t ; ; ζ t ) u t , w t = ∇ y g ( x t , y t ; ζ t ) and q t = ∇ y 2 g ( x t , y t ; ζ t ) u t -∇ y f ( x t , y t ; ξ t ) . In particular, we denote r x ( u ) as:

by assumption it is L smooth and µ strongly convex. It is straightforward to see that u x = arg min u ∈ R d r x ( u ) , where u x is defined in Eq. (3). Finally, in the proof, we consider the general case of different orders for all involved example derivatives. In other words, we have { ξ t, 1 } , { ξ t, 2 } ,

1

Algorithm 5 Without-Replacement Conditional Compositional Optimization (WiOR-CComp)

3: Randomly sample a permutation of outer dataset to have { ξ π i , i ∈ [ m ] } and set x r 0 = x r -1 m or x r = 1

7: Sample a permutation of inner dataset to have { ζ π ξ i ,j , i ∈ [ n i ] } , set y s 0 = y s -1 n i and u s 0 = P ϵ ( u s -1 n ) or y s 0 = y 0 and u s 0 = u 0 if s = 1 .

1: Input: Initial state x 0 , learning rates { η r i } , i ∈ [ m ] , r ∈ [ R ] . 2: for epochs r = 1 to R do 0 if . 4: for i = 0 to m -1 do 5: Initial states y 0 and u 0 , learning rates { γ s j , ρ s j } , j ∈ [ n i ] , s ∈ [ S ] . 6: for s = 1 to S do i 8: for j = 0 to n i -1 do 9: y s j +1 = (1 -γ s j ) y s j + γ s j r ( x r i ; ζ π ξ i ,j ) ; 10: u s j +1 = (1 -ρ s j ) u s j + ρ s j ∇ f ( y s j ; ξ π i ) ; 11: end for 12: end for 13: x r i +1 = x r i -η r i ∇ r ξ π i ( x r i ) u S n i ; 14: end for 15: end for

{ ζ t, 1 } , { ζ t, 2 } and { ζ t, 3 } for ∇ y f ( x, y ) , ∇ x f ( x, y ) , ∇ y g ( x, y ) , ∇ y 2 g ( x, y ) , ∇ xy g ( x, y ) respectively. The two orders used by Algorithm 1 is a special case of the proof and is practically appealing as they can reuse computation as we discussed in the introduction.

τ -1 2

For the first and the third terms above, we have:

and

Combine Eq (13) and (14) with Eq. (12) and denote ˜ L 2 1 = ( L 2 + 2 L 2 xy C 2 f µ 2 ) to have:

Next, for the term ∥ ∑ τ -1 ¯ τ = t ρ ¯ τ ( ∇ r x t ( u t ) -q ¯ τ ) ∥ 2 , we have:

The first and the third terms can be bounded similarly as in Eq. (13) and Eq. (14), and denote ˜ L 2 2 = ( L 2 + 2 L 2 y 2 C 2 f µ 2 ) , then we have:

This completes the proof.

Lemma B.9. For all t ≥ 0 and any constant m> 0 , suppose ¯ η t < 1 2 ¯ L , the iterates generated satisfy:

Proof. By the smoothness of f , we have:

where equality ( a ) uses ⟨ a, b ⟩ = 1 2 [ ∥ a ∥ 2 + ∥ b ∥ 2 - ∥ a -b ∥ 2 ] ; (b) follows the assumption that ¯ η t < 1 / 2 ¯ L . Next, for the third term, by using Eq. (15), we have:

where ˜ L 2 1 = ( L 2 + 2 L 2 xy C 2 f µ 2 ) . This completes the proof.

Lemma B.10. When ¯ γ t < 1 4 L , the error of the inner variable y t are bounded through Eq. (19) .

Proof. then by proposition B.19 and choose ¯ γ t < 1 4 L , we have:

Furthermore, by the generalized triangle inequality, we have:

where the second inequality is due to ¯ γ t < 1 4 L < 1 µ and uses Eq. (17). This completes the proof.

Lemma B.11. When ¯ ρ t < 1 4 L , the error of variable u t are bounded through Eq. (20) .

Proof. suppose we choose ¯ ρ t < 1 4 L , then we have:

Furthermore, by the generalized triangle inequality, we have:

where in the last inequality, we use the condition that ¯ ρ t < 1 4 L and Eq. (16). This completes the proof.

Lemma B.12. Suppose ¯ η t ≤ min ( 1 96 ˜ L 2 1 , 1 32 L 2 c 2 1 , 1 128 L 2 , 1 64 L 2 c 2 2 , 1 96 ˜ L 2 2 c 2 2 ) , then the drift of z τ = ( x τ , y τ ) and u τ can be bounded as in Eq. (25) and Eq. (24) .

Proof. We first have:

For the term ∥ z t -z τ ∥ 2 , we have ∥ z t -z τ ∥ 2 = ∥ x t -x τ ∥ 2 + ∥ y t -y τ ∥ 2 . Combine Eq. (21) with Eq. (15) and (17), sum τ over [ t, t + k -1] and use Proposition B.18 to have:

By the fact that τ ≤ t + k , we have:

For ∥ u t -u τ ∥ 2 , we combine Eq. (21) with Eq. (16) and sum τ over [ t, t + k -1] and use the fact that τ ≤ t + k to have:

Suppose we have γ τ = c 1 η τ = 20 κ ˜ L 3 µ η τ , ρ τ = c 2 η τ = 40 κ ˆ Lη τ and set:

Then we combine Eq. (22) and Eq. (23) to have:

and

This completes the proof.

Suppose we define the potential function G ( t ) :

Theorem B.13. Suppose Assumptions 4.1-4.4 are satisfied, and Assumption 4.5 is satisfied with some α and C for the example order we use in Algorithm 1. We choose learning rates η t = η , γ = c 1 η , ρ = c 2 η and denote T = R × I be the total number of steps. Then for any pair of values (E, k) which has T = E × k and η ≤ 1 k ˜ L 0 , we have:

where c 1 , c 2 , ˜ L 0 , ˜ L are constants related to the smoothness parameters of h ( x ) , ∆ is the initial sub-optimality, R and I are defined in Algorithm 1.

Proof. First we combine Lemma B.9-Lemma B.11, and use the condition that γ t = 20 κ ˜ L 3 µ η t , ρ t = 40 κ ˆ Lη t , and by the assumption about gradient error:

Suppose we denote ˜ L 2 4 = ( 960 κ 2 ˜ L 2 2 +3 ˜ L 2 1 +80 κ 2 ˜ L 2 3 +640 κ 2 L 2 +4 L 2 ) , and set:

and we bound the terms ∥ u x t -u t ∥ 2 and ∥ z t -z τ ∥ 2 through Lemma B.12 to have:

Next suppose we set ˜ L 4 ¯ η t < 1 , then we have:

For ease of notation we denote ˜ L 2 = ( 24+ 48 C 2 f µ 2 +8 c 2 1 +24 c 2 2 + 48 c 2 2 C 2 f µ 2 +3+ 6 C 2 f µ 2 + 80 ˜ L 2 3 µ 2 +960 κ 2 + 1920 κ 2 C 2 f µ 2 ) , then we have:

Sum the above inequality over E phases to have:

we define ∆ h = h ( x 0 ) -h ∗ as the initial sub-optimality of the function, ∆ y = ∥ y 0 -y x 0 ∥ 2 as the initial sub-optimality of the inner variable estimation, ∆ u = ∥ u 0 -u x 0 ∥ 2 as the initial sub-optimality of the hyper-gradient estimation. Then suppose we choose η t = η be some constant, then we have:

Divide by kEη/ 2 on both sides to have:

Suppose we choose the largest η = 1 k ˜ L 0 such that the conditions of Lemma B.9-Lemma B.12 are satisfied, then we have:

Then to reach an ϵ -stationary point, we need to have:

in other words T = Ek ≥ 4 ˜ L 0 (4 ˜ L 2 C 2 ) 1 /α ∆ 2+2 /α

ϵ . This completes the proof.

B.2 Proof for conditional bilevel optimization problems

In this section we study the convergence rate of Algorithm 2.

Lemma B.14. For all t ≥ 0 and any constant k > 0 , suppose ¯ η t < 1 4 ¯ L , the iterates generated satisfy:

Proof. By the smoothness of f , follow similar derivation as in Eq. (18), we have:

In particular, for the term ∥ ∑ τ -1 ¯ τ = t η ¯ τ ( ν ¯ τ -∇ h ( x t )) ∥ 2 , we have:

For the third term, we follow similar derivation as in Eq. (12)-Eq. (15), we have:

where ˜ L 2 1 = ( L 2 + 2 L 2 xy C 2 f µ 2 ) . Combine Eq. (27) and Eq. (28), we have:

For the term ∑ t + k -1 τ = t η τ ∥ ∥ x t -x τ ∥ ∥ 2 , by Eq. (27) and Eq. (28), we have:

Multiply η τ on both sides and sum over [ t, t + k -1] , we have:

By the condition that ¯ η t < 1 4 ¯ L , we have:

Combine Eq. (30) with Eq. (29), then we have:

where the second inequality uses the conditions that ¯ η t < 1 4 ¯ L . By Lemma B.16,we have:

where ∆ u and ∆ y denotes the upper bounds of the initial estimation errors of y x and u x . This completes the proof.

Lemma B.15. Suppose we have ¯ γ l < 1 2 L and ¯ ρ l < 1 2 L , then ∑ l + k -1 l ′ = l γ l ′ ∥ y l ′ -y l ∥ 2 and ∑ l + k -1 l ′ = l ρ l ′ ∥ u l -u l ′ ∥ 2 can be bounded as in Eq. (33) and Eq. (34)

Proof. We first bound ∑ l + k -1 l ′ = l γ l ′ ∥ y l ′ -y l ∥ 2 , in fact, we have:

Multiple γ l ′ on both sides, and then sum l ′ in [ l, l + k -1] , we have:

Since we have ¯ γ l < 1 2 L , we have:

Next for ∑ l + k -1 l ′ = l ρ l ′ ∥ u l -u l ′ ∥ 2 , we have:

Multiple ρ l ′ on both sides, and then sum l ′ in [ l, l + k -1] , we have:

Since we have ¯ ρ l < 1 2 L , we have:

Lemma B.16. When ¯ γ t < 1 128 Lκ and ¯ ρ t < 1 256 Lκ , the error of the inner variable y l and the variable u l are bounded through Eq. (35) and Eq. (36) .

Proof. Since the outer iteration t is fixed for each inner loop, we omit t and ξ t in the notation for clarity. Furthermore, we assume performing the updates to y and u separately in Algorithm 2. Although the alternative updates in Algorithm 2 is appealing in practice as we can reuse ∇ y g for

∇ y 2 g , update y first and u next can avoid treating complicated higher order terms. Follow similar derivation of Lemma B.10, if ¯ γ t < 1 4 L , we have:

Combine with Eq. (33) and use the condition that ¯ γ l < 1 128 Lκ < 1 4 L , we have:

Suppose we perform E l rounds, then by telescoping, we have:

Next, follow similar derivations as in Lemma B.11 and choose ¯ ρ t < 1 4 L , then we have:

Combine with Eq. (34) and use the condition that ¯ ρ l < 1 256 Lκ < 1 4 L , we have:

we perform E l rounds, then by telescoping, we have:

This completes the proof.

Theorem B.17. Suppose Assumptions 4.1-4.4 and 4.7 are satisfied, and Assumption 4.5 is satisfied with some α and C for the example order we use in Algorithm 2. We choose learning rates η t = η = 1 8 k ¯ L , γ t = γ = 1 256 kLκ , ρ t = ρ = 1 512 kLκ and denote T = R × m be the total number of outer steps and T l = S × max( n i ) be the maximum inner steps. Then for any pair of values (E, k) which has T = E × k and T l = E l × k , we have:

where C u , C y , C ′ , ˜ L are constants related to the smoothness parameters of h ( x ) , ∆ h , ∆ u , ∆ y are the initial sub-optimality, R and S are defined in Algorithm 2.

Proof. First, by Lemma B.14, we have:

Sum the above inequality over E phases to have:

Divide by kEη/ 8 on both sides to have:

By the condition η < 1 4 k ¯ L , γ < 1 128 kLκ and ρ < 1 256 kLκ , we choose η = 1 8 k ¯ L , γ = 1 256 kLκ and ρ = 1 512 kLκ , then we have:

Then to reach an ϵ -stationary point, we need to have:

and E l = O ( log ( ϵ -1 )) . This completes the proof.

Proposition B.18. Given Assumptions 4.3-4.4, Assumption 4.7 and Assumption 4.5 hold, then we have: ∥ ∑ τ -1 ¯ τ = t η ¯ τ ( ∇ y g ( x t , y x t ) - ∇ y g ( x ¯ τ , y ¯ τ ; ζ ¯ τ, 1 ) ) ∥ 2 ≤ η 2 k 2 -α C 2 for any τ ≤ t + k -1 . The same upper bound is achieved for other properties: ∇ h ( x ) , ∇ x f ( x, y ) , ∇ y g ( x, y ) , ∇ y 2 g ( x, y ) , ∇ xy g ( x, y )

Proof. This proposition can be adapted from Lemma 2 of [34].

Proposition B.19. Suppose we have function g ( y ) , which is L-smooth and µ -strongly-convex, then suppose γ < 1 2 L , the progress made by one step of gradient descent is:

where y ∗ is the minimum of g ( y ) and we have update rule y t +1 = y t -γω t .

Proof. First, by the strong convexity of of function g ( y ) , we have:

Then by L -smoothness, we have:

Combining the 37 with 38, we have:

By definition of y ∗ , we have g ( y ∗ ) ≥ g ( y t +1 ) . Thus, we obtain

Considering the outer bound of the second term ⟨∇ y g ( y t ) -w t , y ∗ -y t ⟩ , we have

Combining with Eq. 39:

By y t +1 = y t -γω t , we have:

Then since we choose γ < 1 4 L < 1 µ , we obtain:

This completes the proof.

Corollary B.20. For Algorithm 3 and Algorithm 4, suppose Assumptions (the bounded gradient assumption is not required for the minimax problem) and conditions are satisfied as in Theorem 4.6, to reach an ϵ -stationary point, we need O ( ϵ -4 ) examples if they are sampled independently, while O ((max( m,n )) p ϵ -3 ) if some random-permutation-based without-replacement sampling is used.

Corollary B.21. For Algorithm 5, suppose Assumptions and conditions are satisfied as in Theorem 4.8, to reach an ϵ -stationary point, we need O ( ϵ -6 ) examples if they sampled independently, while O ((max( m,n )) 2 p ϵ -4 ) if examples are sampled following some random-permutation-based withoutreplacement sampling.

It is straightforward to derive the above corollaries from Theorem 4.6 and Theorem 4.8, we omit the proof here.

C Comparison of WiOR-BO with Other Acceleration Methods of Bilevel Optimization

In this section, we compare our algorithms with other variance reduction based algorithms for bilevel optimization. For the unconditional case, WiOR-BO obtain the same O ( ϵ -3 ) rate as STORM-based

algorithm MRBO [52], while is slower than SVRG-type algorithms which have O ( n q ϵ -2 ) such as VRBO [52], SABA [9] and SRBA [10]. This relationship is similar to that for the single level optimization problems. However, as a SGD-type algorithm, WiOR-BO is much simpler to implement in practice compared to STORM-based and ARVG-based algorithms. More specifically, WIORBO has fewer hyper-parameters (learning rates for inner and outer problems) to tune compared to MRBO, which has six independent hyper-parameters, and the optimal theoretical convergence rate is achieved only if the complicated conditions among hyper-parameters are satisfied in Theorem 1 of [52] are satisfied, and this requires significant effort in practice. Next, SRBO/SABA evaluates the full hyper-gradient at the start of each outer loop, where we need to evaluate the first and second order derivatives over all samples in one step, which is very expensive for modern ML models (such as the transformer model with billions of parameters). In contrast, our WiOR-BO never evaluates the full gradient. Note that the inner loop length I = lcm ( m,n ) is analogous to the concept of "epoch" in single level optimization: where we go over the data samples following a given order, but at each step we evaluate gradient over a mini-batch of samples. The practical advantage of our WiOR-BO is further verified by the superior performance over MRBO and VRBO in the Hyper-Data Cleaning Task.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide detailed discussion of our proposed algorithms in the section 'Without Replacement Sampling in Bilevel Optimization', and followed by theoretical and numerical studies in the subsequent sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Our theoretical analysis is based on a set of assumptions and we discuss their limitations at the start of the section 'Theoretical Analysis'.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions and Proofs are included in the 'Theoretical Analysis' section in the main text and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: we clearly discuss the experimental settings in detail in section 'applications and numerical experiments'

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: the datasets used in experiments are publicly available and we include the code implementation in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most of the experimental details can be found in Section 5, while the rest is included in the Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The numerical experiments in the paper are averaged over 5 independent runs and we report the mean and standard deviation (shadowed area).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We state the compute resource to run our experiments at the start of Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow all of NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work related to the theoretical aspect of Machine Learning. The topic of the paper, bilevel optimization, has many applications in machine learning. So there are some potential societal consequences of our work when our algorithms are applied to various machine learning tasks, but none which we feel is directly related to our algorithm.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the datasets used in our experiments are publicly available

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes WiOR-(C)BO for without-replacement sampling algorithms for (conditional) bilevel optimization. The authors prove the convergence rate for both algorithms under regular assumptions, which are improved upon existing works on the complexity of $\\epsilon$.\n\nSTRENGTHS:\n1. The assumptions are standard and the convergence rates are better than existing results.\n2. The empirical behaviors are better than compared algorithms.\n\nWEAKNESSES:\n1. The improvement from order $\\mathcal{O}(\\epsilon^{-4})$ to $\\mathcal{O}(\\max\\\\{m,n\\\\}^p\\epsilon^{-3})$ is not as great as stated to be $\\mathcal{O}(\\epsilon^{-3})$. If we can easily ignore the existence of $m$ and $n$, why not calculate the true gradients and Hessian matrices (instead of stochastic estimates) all time and reach a $\\mathcal{O}(\\max\\\\{m,n\\\\}\\epsilon^{-2})$ rate of SOBA? Please clarify the influence of term $\\\\{m,n\\\\}^p$ so that the degree of improvement is clearer. If $p$ can be forced to 0 without additional assumptions, the result will be better.\n2. The $p$ values in both $\\mathcal{O}(\\max\\\\{m,n\\\\}^p\\epsilon^{-3})$ for bilevel optimization and $\\mathcal{O}(\\max\\\\{m,n\\\\}^{2p}\\epsilon^{-4})$ for conditional bilevel optimization are not clear. Please clarify how $p$ is determined. By the way, the authors should have used another character since $p$ has already been used as the dimension of $x$.\n3. The motivation of using without-replacement sampling is not solid. In the introduction section, the author stated that using without-replacement sampling achieves more accurate gradient estimations than with-replacement sampling. However, SOBA implemented by with-replacement sampling can achieve $\\mathcal{O}(\\epsilon^{-2})$ complexity which is better than $\\mathcal{O}(\\epsilon^{-3})$. Could you give some explanations on this?\n4. The novelty in algorithmic design is limited. It looks quite like a direct combination of SOBA and natural without-replacement sampling strategies.\n\nQUESTIONS:\n1. The paper regards the special cases as one of the main contributions. Can the special cases be fully covered by the general cases? Are convergence results or algorithms for these cases different from what we could achieve by applying the general case?\n2. How is $p$ (or $q$) in the convergence rate determined? If it can be forced to 0 by using certain sampling strategy, why not directly use that strategy? Will it contradict with other assumptions?\n3. Why should we use without-replacement sampling rather than with-replacement sampling? I'm asking because I believe SOBA with moving average and with-replacement sampling has a better convergence rate of $\\mathcal{O}(\\epsilon^{-2})$ under regular assumptions. Please correct me if I'm wrong.\n4. Can you compare the emprical performance with SOBA?\n\nLIMITATIONS:\nThe limitations are stated as assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

comments: SUMMARY:
This paper introduces new algorithms that leverage without-replacement sampling to enhance bilevel optimization. The main contributions are:
1. Introducing more practical bilevel optimization algorithms using without-replacement sampling.
2. Providing comprehensive theoretical analysis demonstrating improved convergence rates and efficiency.
Novel Algorithms: Introduces simpler, more practical bilevel optimization algorithms using without-replacement sampling.
3. Validating the effectiveness through applications in hyper-data cleaning and hyper-representation learning.

STRENGTHS:
The paper's strengths can be summarized as follows:

Originality: The paper introduces a novel application of without-replacement sampling for bilevel optimization, presenting a fresh and practical approach in contrast to traditional methods.

Quality: The research exhibits high quality through rigorous theoretical analysis and comprehensive comparisons with existing methods, showcasing superior performance and efficiency.

Clarity: The paper is well-structured and clearly written, effectively communicating complex concepts and providing detailed explanations and results.

Significance: The proposed methods significantly enhance process efficiency.

WEAKNESSES:
1. Why are the upper bounds the same in Assumptions 4.3, 4.4, and 4.7?

QUESTIONS:
Line 71: Please use $O(\cdot)$ to denote the big O notation and $\tilde{O}(\cdot)$ to indicate that logarithmic terms are hidden.
LInes 131, 536, etc.: Some formulas exceed the line width.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper improves the computational inefficiencies in bilevel optimization algorithms that rely on independent sampling. It proposes a novel without-replacement sampling-based algorithm, which achieves a faster convergence rate than existing methods.

STRENGTHS:
This paper develops independent sampling for bilevel optimization, which save the large number of backward passes and improve the convergence rate. Experimental results show the good results compared to other bilevel baselines.

WEAKNESSES:
- The paper claimed that "Compared to independent sampling-based algorithm such as stocBiO, we require extra space to generate and store the orders of examples, but this cost is negligible compared to memory and computational cost of training large scale models." Is there any empirical result to show the memory costs?

- Projection is applied to $u_i^r$, but I found other bilevel papers (such as StocBio, BSA) don't use that. Is there any intuition or explanation about that?

- In Algorithm 2, the generation of $\\{\zeta_{\xi_t^{\pi}, t_l}^{\pi} \in \mathcal{D}_{l, \xi_t^{\pi}}. t_l\in [T_l]\\}$ is not very clear. It seems that there are $S\times n_i$ samples for inner loops given a outer sample $\xi_i$. How do you execute the permutation for the inner dataset.

- In experiments, what is the best hyper-parameters for each baselines, such inner and outer learning rates?

- What are the details of data sampling for other baselines? For other baselines, they usually need the independent training and validation set for upper and lower-level update, how did you construct data for them?

- Typos: Line 107, $w_x$ should be $u_x$.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates stochastic bilevel optimization. One common practice herein is that the data samples used to calculate all the derivatives are mutually independent, which may however introduce more computational cost compared to the case where the samples are reused. Motivated by this, the authors explore stochastic bilevel optimization without using independent sampling, particularly under the without-replacement sampling scheme. Based on previous single-loop algorithm, two with-replacement bilevel optimization algorithms are proposed for standard bilevel optimization and conditional bilevel optimization, respectively, which can also be applied to several special cases. The authors show that faster convergence rates can be achieved by the proposed algorithms compared to the counterparts with independent sampling. Experiments on multiple tasks are conducted to further justify the performance of the proposed algorithms.\n\nSTRENGTHS:\n1. The idea of investigating bilevel optimization without independent sampling is interesting and novel.\n\n2. Algorithms and theoretical results are provided for various setups with a faster convergence rate.\n\n3. The experimental results also clearly show the superior performance of the proposed algorithms.\n\nWEAKNESSES:\n1. It is not clear what the key technical contributions in the theoretical analysis are.\n\n2. As the authors claim the extension to the special cases of minimax and compositional optimization problems as a contribution, it is better to compare with the baselines in these problems.\n\n3. The presentation can be further improved. For examples, the figures in the experiments are not clear because of the small font size.\n\nQUESTIONS:\n1. How was the bias due to the without-replacement sampling handled in the analysis? Using window averaging? In this case, did you need to store the gradients of previous samples?\n\n2. When you compared the convergence rate of algorithm 1 with baselines, e.g., stocBiO, how did you justify that the faster convergence is because of the sampling but not the single-loop update? Given that stocBiO is not a single-loop algorithm.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new algorithm called WiOR-BO, which leverages without-replacement sampling to achieve faster convergence rates compared to traditional independent sampling methods. The algorithm is applied to standard bilevel optimization, conditional bilevel optimization, and specific cases such as minimax and compositional optimization problems. The authors provide a theoretical analysis demonstrating that WiOR-BO achieves a significantly improved convergence rate over existing methods. Empirical validation on synthetic and real-world tasks, including hyper-data cleaning and hyper-representation learning, shows the superior performance and computational efficiency of the proposed algorithms.\n\nSTRENGTHS:\nBy employing without-replacement sampling, the algorithm effectively reduces the number of necessary backward passes, leading to substantial computational savings. This is especially beneficial in large-scale machine learning models where back-propagation can be expensive. The algorithm's design allows for simultaneous updates of upper and lower-level parameters, improving convergence rates and computational efficiency compared to traditional methods. Additionally, the theoretical guarantees provided on convergence offer a robust foundation for its application, making the algorithm a valuable tool for tasks such as hyperparameter optimization and data cleaning. Its ability to utilize a finite subset of data without replacement ensures reduced variance in gradient estimates, enhancing the stability and reliability of the optimization process.\n\nWEAKNESSES:\nThe authors have selected learning rates and other tuning parameters to optimize performance, but they do not report any sensitivity analysis regarding these choices. This lack of analysis leaves open questions about the robustness of the algorithm's performance under different parameter settings. Additionally, there is no provided methodology or algorithm for efficiently selecting these parameters without the risk of overfitting to the validation set, which could undermine the generalizability of the results.\n\nFurthermore, the proposed algorithms are designed under the assumption of a static dataset, which restricts their applicability in scenarios where data arrives sequentially, such as in dynamic or online learning environments. This limitation could be mitigated by extending the algorithms to handle streaming data or datasets that evolve over time, making the methods more versatile and applicable to real-world situations where data distribution may change. Addressing these issues would enhance the robustness and broader applicability of the proposed solutions.\n\nQUESTIONS:\n1. Comparative Benchmarks: The paper provides empirical validation, but could the authors include more comprehensive comparisons with state-of-the-art methods, especially in dynamic environments? How does the proposed method perform in cases where data distributions shift over time?\n2. Computational Efficiency: While the method reduces the number of backward passes, how does it scale with very large datasets or high-dimensional data?\n3. Hyperparameter Tuning: Could the authors elaborate on the process of selecting hyperparameters, such as learning rates and other tuning parameters? Specifically, are there any strategies employed to avoid overfitting during hyperparameter tuning? Additionally, how sensitive is the performance of the algorithm to these hyperparameter choices, and have the authors conducted any sensitivity analyses to assess this? Addressing these points could provide more clarity on the robustness and practical implementation of the proposed algorithms.\n\nLIMITATIONS:\nAre there any specific limitations that the authors anticipate in applying this method to different domains or tasks? What future directions do the authors envision for improving the algorithm, particularly concerning hyperparameter tunning, adaptivity and scalability?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-30"}