Title: A representation-learning game for classes of prediction tasks
Venue: NeurIPS 2023
Accepted: no
Keywords: representation learning

dimensionality-reduction

regret

minimax solution

mixed strategies

multiplicative weights update
Abstract: We introduce a formulation for learning dimensionality-reducing representations of unlabeled feature vectors, when a prior knowledge on future prediction tasks is available. The formulation is based on a three-player game, in which the first player chooses a representation, the second player then adversarially chooses a prediction task, and the third player predicts the response based on the represented features. The first and third player aim is to minimize, and the second player to maximize, the regret: The minimal prediction loss using the representation compared to the same loss using the original features. Our first contribution is theoretical and addresses the mean squared error loss function, and the case in which the representation, the response to predict and the predictors are all linear functions. We establish the optimal representation in pure strategies, which shows the effectiveness of the prior knowledge, and the optimal regret in mixed strategies, which shows the usefulness of randomizing the representation. We prove that optimal randomization requires a precisely characterized finite number of representations, which is smaller than the dimension of the feature vector, and potentially much smaller. Our second contribution is an efficient gradient-based iterative algorithm that approximates the optimal mixed representation for a general loss function, and general classes of representations, response functions and predictors.

Abstract

We introduce a formulation for learning dimensionality-reducing representations of unlabeled feature vectors, when a prior knowledge on future prediction tasks is available. The formulation is based on a three-player game, in which the first player chooses a representation, the second player then adversarially chooses a prediction task, and the third player predicts the response based on the represented features. The first and third player aim is to minimize, and the second player to maximize, the regret : The minimal prediction loss using the representation compared to the same loss using the original features. Our first contribution is theoretical and addresses the mean squared error loss function, and the case in which the representation, the response to predict and the predictors are all linear functions. We establish the optimal representation in pure strategies, which shows the effectiveness of the prior knowledge, and the optimal regret in mixed strategies, which shows the usefulness of randomizing the representation. We prove that optimal randomization requires a precisely characterized finite number of representations, which is smaller than the dimension of the feature vector, and potentially much smaller. Our second contribution is an efficient gradient-based iterative algorithm that approximates the optimal mixed representation for a general loss function, and general classes of representations, response functions and predictors.

1 Introduction

A common practice in modern data-science is to collect as much data as possible, even without an exact knowledge of a subsequent prediction task it will be used for. The data collected is an unlabeled set of feature vectors { x i } ⊂ R d . Then, when a specific prediction task becomes of interest, responses y i ∈ Y are collected, and a learning algorithm is trained on the pairs { ( x i , y i ) } . Modern sources, such as high-definition images or genomic sequences, have high dimension d , and this raises the question of dimensionality-reduction , either for a better generalization [1], for storage/communication savings [2-4], or for interpretability [5]. The goal is thus to find a representation z = R ( x ) ∈ R r , where d glyph[greatermuch] r , that preserves the relevant part of the features, without a full knowledge of their utility for future prediction tasks. In this paper, we propose an unsupervised-learning game-theoretic framework for this goal, whose central aspect is an assumption of prior knowledge on the class of future prediction tasks. Our contributions are a theoretical solution in a fully linear setting, under the mean squared error (MSE) loss, and an algorithm for the general setting.

Popular approaches to dimensionality reduction are oblivious to prior knowledge on the prediction 32 task. Most prominently, principal component analysi s (PCA) [6-9], and non-linear extensions such 33 as kernel PCA [10] and auto-encoders (AE) [11-13, 1], aim that the representation z will maximally 34 preserve the variation in x . Nonetheless, prior knowledge may indicate that the highly varying di35 rections in the feature space are irrelevant for future prediction tasks. From the supervised learn36

ing perspective, it is well established that efficient representations are inherent to efficient learning 37 [14, 15]. In this respect, the information bottleneck (IB) principle [16-19] was used to postulate 38 that efficient supervised learning learns representations which are both low-complexity and relevant 39 [20-24] (this spurred a debate, e.g., [25, 26]). The original IB formulation is based on the mutual in40 formation functional [27], which is difficult to estimate (especially in high dimensions), and ignores 41 complexity constraints on the representation or prediction [28, 29]; see a review in Appendix B. Us42 ing the notion of usable information , introduced in [28], optimal representations for the supervised 43 learning setting were explored in [29] via a two-player game between Alice, which selects a pre44 diction problem of y given x , and Bob, which then selects the representation z . Alice then uses an 45 empirical risk minimizer with the standard goal of minimizing the expected risk. It was established 46 in [29] that ideal generalization is obtained for representations that optimize the decodable IB. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

In this paper, we build upon [29], and propose a three-player game for unsupervised representationlearning, chosen without a specific prediction problem (Section 2). First, the representation player reduce x ∈ R d to a representation z ∈ R r , where r < d . Second, the response function player chooses a response (label) y rule f for x , from a given known class of (random) response functions F . The choice of this class manifests the prior knowledge available on the type of prediction problems that the representations will be used for. Third, the predictor player optimally predicts y from z . The value of the game is determined by the regret : The prediction loss based on the representation z compared to prediction loss based on x . The first and last player cooperate in order to minimize the regret, whereas the response function player aims to maximize it. In other words, the representation is chosen to minimize the worst-case prediction loss for any response in F . The output of this game is the representation chosen by the first player. In order to focus on the representation aspect we side-step the generalization problem, and assume that sufficient labeled data will be provided to the predictor later on in order to accurately estimate the prediction rule.

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

This formulation directly addresses the relevance of a 'direction' in the feature space to the prediction tasks in F , rather than its variability, as in standard unsupervised learning (e.g., PCA and AE). Compared to [29], the representation is chosen based only on the class of possible response functions, rather than a specific one. Such knowledge on F may stem from various considerations: Domain specific, imposed by privacy or fairness constraints, or stem from transfer or continual learning setting; see Appendix A for an extended discussion. Technically, the game in [29] replaces the order of the first (representation) and second (response) players. From a different perspective, our method is a self-supervised learning method , for which the prior knowledge on F serves as a 'self-defined signal' for choosing an optimal representation, without any labeled data (see, e.g., [30] and [31] for recent surveys). In addition, our game formulation naturally leads to a mixed strategies solution [32], that is, allowing the representation player to randomized the representation rule, in order to mix up the adversarial response player. This randomization is an inherent aspect of the IB formulation, but its usage there is not rigorously justified. By contrast, for standard unsupervised learning, mixed representation does not improve the regret (see Proposition 14 in Appendix E.1 for the PCA setting). In Appendix B we provide a thorough discussion of related work.

Contributions 76

· Theoretical: We address the fundamental setting in which the representation, the response, and 77 the prediction are linear functions, under the MSE loss function (Section 3). The prior knowledge 78 on F is represented by a symmetric matrix S that determines the principal directions of the function 79 in the feature space. We establish the optimal representation and regret in pure strategies, which 80 shows the utility of the prior information, and in mixed strategies, which shows that randomizing the 81 representation yields strictly lower regret. We prove that randomizing between merely glyph[lscript] ∗ different 82 representation rules suffices, where r +1 ≤ glyph[lscript] ∗ ≤ d is a precisely characterized effective dimension . 83

· Algorithmic: We develop an iterative gradients-based algorithm that approximates the optimal 84 mixed representation (Section 4) for general representations/response/predictors and loss functions. 85 The algorithm is greedy, and alternates between finding a new representation rule and an adversarial 86 function. We empirically verify that the output mixed representation has close-to-optimal regret 87 in the linear MSE setting. To optimize the weights of the representation, we essentially solve a 88 minimax two-player games, and to this end, we utilize the classic multiplicative weights update 89

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

2 Problem formulation

We use mostly conventional notation that is detailed in Appendix C. Specifically, the eigenvalues of a positive-semidefinite matrix S are denoted as λ max ( S ) ≡ λ 1 ( S ) ≥ ··· ≥ λ d ( S ) = λ min ( S ) and v i ( S ) denotes an eigenvector corresponding to λ i ( S ) such that V = V ( S ) := [ v 1 ( S ) , v 2 ( S ) , · · · , v d ( S )] ∈ R d × d and S = V ( S )Λ( S ) V glyph[latticetop] ( S ) is an eigenvalue decomposition. For a matrix W ∈ R d × d we let W i : j := [ w i , . . . , w j ] ∈ R ( j -i +1) × d denote the matrix comprised of the columns indexed by { i, . . . , j } . We denote the probability law of a random variable x as L ( x ) .

Let x ∈ X be a random feature vector, where P x := L ( x ) is known. Let y ∈ Y be a corresponding response drawn according to a probability kernel y ∼ f ( · | x = x ) , where for brevity, we will refer to f as the response function . We assume f ∈ F for some known class F . Let z := R ( x ) ∈ R r be an r -dimensional representation of x where R : X → R r is chosen from a class R of representation functions, and let Q : R r → Y be a prediction rule from a class Q , with the loss function loss : Y × Y → R + . The regret of the representation R for the response function f is

The minimax regret in mixed strategies is defined via the worst case response function in F as

where P ( R ) is a set of probability measures on the possible set of representations R . The minimax regret in pure strategies restricts P ( R ) to degenerated measures (deterministic), and so the expectation in (2) is removed. Our main goal is to determine the optimal representation strategy, either in pure R ∗ ∈ R or mixed strategies L ( R ∗ ) ∈ P ( R ) . To this end, we will also utilize the maximin version of (2). Specifically, let P ( F ) denote a set of probability measures supported on F , and assume that for any R ∈ R , there exists a measure in P ( R ) that puts all its mass on R . Then, the minimax theorem [32, Chapter 2.4] [36] implies that

The right-hand side of (3) is the maximin regret in mixed strategies, and the maximizing probability law L ( f ∗ ) is known as the least favorable prior . In general, regret mix ( R , F | P x ) ≤ regret pure ( R , F | P x ) , and the inequality can be strict. We mention that the use of expectation in the definition of the mixed regret over the randomized representation, implies that the empirical performance of a system based on this randomized representation achieves the mixed minimax regret value in the limit of large number of repeating representation games. The size of the dataset for each of these games should be large enough to allow for accurate learning of f to be used by the predictor. By contrast, the pure minimax regret guarantee is valid for a single representation, and thus more conservative from this aspect.

3 The linear setting under MSE loss

In this section, we focus on linear classes and the MSE loss function. The response function class is characterized by a quadratic constraint, to wit, the class F is specified by a matrix S ∈ S d ++ that represents the relative importance of each direction in the feature space in determining y .

Definition 1 (The linear MSE setting) . Assume that X = R d , that Y = R and the loss function is the MSE, loss ( y 1 , y 2 ) = | y 1 -y 2 | 2 . Assume that E [ x ] = 0 and let Σ x := E [ xx T ] ∈ S d ++ be its invertible covariance matrix. The classes of representations, response functions, and predictors are all linear, that is: (1) The representation is z = R ( x ) = R glyph[latticetop] x for R ∈ R := R d × r where d > r ; (2) The response function is F ∈ F ⊂ R d , and y = f glyph[latticetop] x + n ∈ R , where n ∈ R is a heteroscedastic noise that satisfies E [ n | x ] = 0 , and given some specified S ∈ S d ++

where ‖ f ‖ S := ‖ S -1 / 2 f ‖ 2 = ( f glyph[latticetop] S -1 f ) 1 / 2 is the Mahalanobis norm; (3) The predictor is Q ( z ) = q glyph[latticetop] z ∈ R for q ∈ R r . Since the regret will depend on P x only via Σ x , we will abbreviate the notation of the pure (resp. mixed) minimax regret to regret pure ( F | Σ x ) (resp. regret mix ( F | Σ x ) ).

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

In Appendix E.1 we show that standard PCA can be similarly formulated, by assuming that F is a singleton containing the noiseless identity function, so that y = x surely holds, and ˆ x = Q ( z ) ∈ R d . Proposition 14 therein shows that the pure and mixed minimax representations are both R = V 1: r (Σ x ) , and so randomization is not unnecessary. We begin with the pure minimax regret.

Theorem 2. For the linear MSE setting (Definition 1)

A minimax representation matrix is

and the worst case response function is

The optimal representation thus whitens the feature vector x , and then projects it on the top r eigenvectors of the adjusted covariance matrix Σ 1 / 2 x S Σ 1 / 2 x , which reflects the prior knowledge that f ∈ F S . The proof is deferred to Appendix E.2, and its outline is as follows: Plugging the optimal predictor into the regret results a quadratic form in f ∈ R d , determined by a matrix which depends on the subspace spanned by the representation R . The worst-case f is the determined via the Rayleigh quotient theorem [37, Theorem 4.2.2], and the optimal R is found via the CourantFischer variational characterization [37, Theorem 4.2.6] (see Appendix D for a summary of useful mathematical results). We next consider the mixed minimax regret.

Theorem 3. For the linear MSE setting (Definition 1)

where λ i ≡ λ i ( S 1 / 2 Σ x S 1 / 2 ) and glyph[lscript] ∗ is any member of

(with λ d +1 ≡ 0 ). Furthermore:

· The covariance matrix of the least favorable prior of f : Let Λ glyph[lscript] := diag( λ 1 , . . . , λ glyph[lscript] ∗ , 0 , · · · , 0) , and let V ≡ V ( S 1 / 2 Σ x S 1 / 2 ) . Then, the covariance matrix of the least favorable prior of f is

· The probability law of the minimax representation: Let A ∈ { 0 , 1 } glyph[lscript] ∗ × ( glyph[lscript] ∗ r ) be a matrix whose 154 columns are the members of the set 155

(in an arbitrary order). Let b = ( b 1 , . . . , b glyph[lscript] ∗ ) glyph[latticetop] be such that 156

Then, there exists a solution p ∈ [0 , 1] ( glyph[lscript] ∗ r ) with support size at most glyph[lscript] ∗ +1 to Ap = b . For j ∈ [ ( glyph[lscript] ∗ r ) ] , 157 let I j := { i ∈ [ glyph[lscript] ∗ ]: A ij = 0 } be the zero indices on the j th column of A , and let V I j denote the r 158 columns of V whose index is in I j . A minimax representation is 159

with probability p j , for j ∈ [ ( glyph[lscript] ∗ r ) ] . 160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

Interestingly, while the eigenvalues λ i (Σ 1 / 2 x S Σ 1 / 2 x ) = λ i ( S 1 / 2 Σ x S 1 / 2 ) are equal, the pure minimax regret utilizes the eigenvectors of Σ 1 / 2 x S Σ 1 / 2 x whereas the mixed minimax regret utilizes those of S 1 / 2 Σ x S 1 / 2 , which are possibly different. The proof of Theorem 3 is also in Appendix E.2, and is substantially more complicated and longer than for the pure regret. We use a two-step indirect approach, since it seems challenging to directly maximize over L ( R ) . First, we solve the maximin problem (3), and find the least favorable prior L ( f ∗ ) . Second, we propose a probability law for the representation L ( R ) , and show that its regret equals the maximin value, and thus also the minimax. With more detail, in the first step, we show that the regret only depends on L ( f ) via Σ f = E [ ff glyph[latticetop] ] , and we explicitly construct a probability law that is both fully supported on F S and has this covariance matrix. This reduces the problem from optimizing L ( f ) to optimizing Σ f , whose solution (Lemma 16) leads to the least favorable Σ ∗ f , and then to the maximin value. In the second step, we explicitly construct a representation that achieves the maximin regret. Concretely, we construct representation matrices that use r of the glyph[lscript] ∗ principal components of Σ 1 / 2 x S Σ 1 / 2 x , where glyph[lscript] ∗ > r . The defining property of glyph[lscript] ∗ (9) established in the maximin solution is utilized to find weights on the ( glyph[lscript] ∗ r ) possible representations, that achieves the maximin solution, and thus also the minimax. The proof uses Carathéodory's theorem (see Appendix D) which also establishes that the optimal { p j } is supported on at most glyph[lscript] ∗ +1 matrices, much less than ( glyph[lscript] ∗ r ) . We next make a few comments:

Computing the mixed minimax probability: This requires solving A glyph[latticetop] p = b for a probability vector p , which is a linear-program feasibility problem that is routinely solved [38]. For illustration, if r = 1 then A ∈ { 0 , 1 } glyph[lscript] ∗ × glyph[lscript] ∗ is a square all ones matrix, except for a zero diagonal, and p j = 1 -( glyph[lscript] ∗ -1) λ -1 j / ( ∑ glyph[lscript] ∗ i =1 λ -1 i ) for j ∈ [ glyph[lscript] ∗ ] . Similarly, the case glyph[lscript] ∗ = r + 1 is solved by setting p j = ( λ -1 j ) / ( ∑ glyph[lscript] ∗ j ′ =1 λ -1 j ′ ) on the glyph[lscript] ∗ standard basis vectors. Nonetheless, the dimension of p is ( glyph[lscript] ∗ r ) and thus increases fast as Θ(( glyph[lscript] ∗ ) r ) , and this approach may be intractable. However, in this case the algorithm we present in Section 4 can be used. As we empirically show, it approximately achieves the optimal regret, and the number of atoms is not much larger than glyph[lscript] ∗ +1 .

Required randomness: The regret formulation (2) assumes that the actual realization of the representation rule is known to the predictor. Formally, this can be conveyed to the predictor using an small header of less than log 2 ( glyph[lscript] ∗ + 1) ≤ log( d + 1) bits. Practically, this is unnecessary and an efficient predictor can be learned from a labeled data set ( z , y ) .

The rank of Σ ∗ f : The rank of the covariance matrix of the least favorable prior is an effective dimension , satisfying (see (8))

By convention, { λ -1 i } i ∈ [ d ] is a monotonic non-decreasing sequence, and so is the partial Cesàro mean ψ ( glyph[lscript] ) := 1 glyph[lscript] ∑ glyph[lscript] i =1 λ -1 i . For example, if λ i = i -α with α > 0 then ψ ( glyph[lscript] ) = Θ( glyph[lscript] α ) . If, e.g., ψ ( glyph[lscript] ) = glyph[lscript] α , then it is easily derived that glyph[lscript] ∗ ≈ min { α +1 α r, d } . So, if α ≥ r d -r is large enough and the decay rate of { λ i } is fast enough then glyph[lscript] ∗ < d , and otherwise glyph[lscript] ∗ = d . As the decay rate of { λ i } becomes faster, the rank of Σ ∗ f decreases to r . Importantly, glyph[lscript] ∗ ≥ r +1 always holds, and so the optimal mixed representation is not deterministic even if S 1 / 2 Σ x S 1 / 2 has less than r significant eigenvalues (which can be represented by a single matrix R ∈ R d × r ). Hence, the mixed minimax regret is always strictly lower than the pure minimax regret. Thus, even when S = I d , and no valuable prior knowledge is known on the response function, the mixed minimax representation is different from the standard PCA solution of top r eigenvectors of Σ x .

Uniqueness of the optimal representation: Since one can always post-multiply R glyph[latticetop] x by some invertible matrix, and then pre-multiply z = R glyph[latticetop] x by its inverse, the following simple observation holds: When R and Q are not further restricted, then if R is a minimax representation, and W ( R ) ∈ R r × r is an invertible matrix, then R · W ( R ) is also a minimax representation.

Example 4. Assume S = I d , and denote, for brevity, V ≡ V (Σ x ) := [ v 1 , . . . , v d ] and Λ ≡ Λ(Σ x ) := diag( λ 1 , . . . , λ d ) . The optimal minimax representation in pure strategies (Theorem 2) is

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

1

1.5

2

2.5

3

3.5

4

0

1

2

3

4

5

6

7

8

9

10 -3

30

35

40

45

50

1

1.5

2

2.5

3

3.5

4

0

0.2

0.4

0.6

0.8

1

1.2

1.4

10 -4

38

40

42

44

46

48

50

then 211

212

213

214

which is comprised of the top r eigenvectors of Σ x , scaled so that v glyph[latticetop] i x has unit variance. By Comment 4 above, V 1: r is also an optimal minimax representation. The worst case response is f = v r +1 (Σ x ) and, as expected, since R uses the first r principal directions

The minimax regret in mixed strategies (Theorem 3) is different, and given by 215

where glyph[lscript] ∗ is determined by the decay rate of the eigenvalues of Σ x (see (9)). The least favorable covariance matrix is given by (Theorem 3)

Intuitively, the least favorable Σ ∗ f equalizes the first glyph[lscript] ∗ eigenvalues of Σ x Σ ∗ f (and nulls the other d -glyph[lscript] ∗ ) so that the representation is indifferent to these glyph[lscript] ∗ directions. As evident from the regret, the 'equalization' of the i th eigenvalue adds a term of λ -1 i to the denominator, and if λ i is too small then v i is not chosen for the representation, as agrees with Comment 3 above (a fast decay of { λ i } reduces glyph[lscript] ∗ away from d ). The mixed minimax representation sets

with probability p j , where I j ≡ { i j, 1 , . . . , i j,r } (the derivation is similar to (15)). Thus, the optimal representation chooses a random subset of r vectors from { v 1 , . . . , v glyph[lscript] ∗ } . See the left panel of Figure 1 for a numerical example.

Example 5. To demonstrate the effect of prior knowledge on the response function, we assume Σ x = diag( σ 2 1 , . . . , σ 2 d ) and S = diag( s 1 , . . . , s d ) , where σ 2 1 ≥ σ 2 2 ≥ · · · ≥ σ 2 d (but { s i } i ∈ [ d ] are not necessarily ordered). Letting f = ( f 1 , . . . , f d ) , the class of response functions is F S := { f ∈ R d : ∑ d i =1 ( f 2 i /s i ) ≤ 1 } , and so coordinates i ∈ [ d ] with a large s i have large influence on the response. Let ( i (1) , . . . , i ( d ) ) be a permutation of [ d ] so that σ 2 i ( j ) s i ( j ) it the j th largest value of ( σ 2 i s i ) i ∈ [ d ] . The pure minimax regret is (Theorem 2)

The optimal representation is R = [ e i (1) , e i (2) , . . . , e i ( r ) ] , that is, uses the most influential coordinates, according to { s i } , which may be different from the r principal directions of Σ x . For the minimax regret in mixed strategies, Theorem 3 results

for glyph[lscript] ∗ ∈ [ d ] \ [ r ] satisfying (9), and the covariance matrix of the least favorable prior is given by 235

That is, up to a scale factor ( ∑ glyph[lscript] ∗ i =1 s -1 i σ -2 i ) -1 , the matrix is diagonal so that the k th term on the diagonal is Σ ∗ f ( k, k ) = σ -2 k if k = i j for some j ∈ [ glyph[lscript] ∗ ] and Σ ∗ f ( k, k ) = 0 otherwise. As in Example 4, Σ ∗ f equalizes the first glyph[lscript] ∗ eigenvalues of Σ x Σ f (and nulls the other d -glyph[lscript] ∗ ). However, it does so in a manner that chooses them according to their influence on f glyph[latticetop] x . The random minimax representation in mixed strategies is

with probability p j . Again, all the first glyph[lscript] ∗ coordinates are used, and not just the top r . See the right panel of Figure 1 for a numerical example. We finally remark that, naturally, in the non-diagonal case, the minimax regret will also depend on the relative alignment between S and Σ x .

4 An iterative algorithm for general classes and loss functions

In this section, we develop an iterative algorithm for finding the optimal representation in mixed strategies, i.e., solving (2) for general classes and loss functions. Since optimizing general probability measures over R is formidable, we restrict the optimization to finite mixed representations, i.e., assume that R = R ( j ) ∈ R with probability p ( j ) , where j ∈ [ m ] (which suffices for the linear MSE setting of Section 3, but possibly sub-optimal in general). Furthermore, the algorithm's operation will require randomization also for the response player, and so we set f = f ( i ) ∈ F with probability o ( i ) where i ∈ [ m ] , and m = m 0 + m for some m 0 ≥ 0 . The resulting optimization problem then becomes

under the constraints p ( j ) ≥ 0 and ∑ j p ( j ) = 1 , and o ( i ) ≥ 0 and ∑ i o ( i ) = 1 . Note that the prediction rule Q ( j,i ) is determined based on both R ( j ) and f ( i ) , and that the ultimate goal of solving (24) is just to extract the optimal R .

Ahigh level description of the algorithm is to gradually add more representations to the support size of R up to m , where next k will denote the current number of representations, k ∈ [ m ] . Initialization requires an representation R (1) , as well as a set of functions { f ( i ) } i ∈ m 0 , so that the final support size of f will be m = m 0 + m . Finding this initial representation and the set of functions is based on the specific loss function and a possible set of representation/predictors. At iteration k ∈ [ m ] , the main loop of the algorithm has two phases. In the first phase, a new adversarial function is added to the set of functions, as the worse function for the current random representation. In the second phase, a new representation atom is added to the set of possible representations. This representation is determined based on the given set of functions. Concretely, the two phases operate as follows:

· Phase 1 - Given k representations { R ( j ) } j ∈ ( k ) with weights { p ( j ) } j ∈ [ k ] , the algorithm determines the function f ( m 0 + k ) as the worst function for this random representation (optimal adversarial action of the response function player). Specifically,

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

is solved, and f ( m 0 + k ) is set to be the maximizer. This simplifies (24) in the sense that m is replaced 268 by k , the random representation R is kept fixed, and f ∈ F is optimized as a pure strategy (the 269 previous functions { f ( i ) } i ∈ [ m 0 + k -1] are ignored). 270

Phase 2 - Adding a representation atom: Given fixed { f ( j ) } j ∈ [ m 0 + k ] and { R ( j ) } j ∈ [ k ] , a new 271 representation R ( k +1) is found as the most incrementally valuable representation atom. Specifically, 272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

is solved, the solution R ( k +1) is added to the set of representations, and the weights are updated to the optimal { p ( j 1 ) } j 1 ∈ [ k +1] . Compared to (24), here the response functions and current k representations are kept fixed, and only their weights { p ( j 1 ) } { o ( j 2 ) } are optimized, along with R ( k +1) .

The procedure is described in Algorithm 1, where, following the main loop, m ∗ = arg min k ∈ [ m ] reg k representation atoms are chosen and the output is { R ( j ) , p ( j ) } j ∈ [ m ∗ ] . Algorithm 1 relies on solvers for the Phase 1 (26) and Phase 2 (27) problems. In Appendix G we propose two algorithms for these problems, which are based on gradient steps for updating the adversarial response and the new representation, and on the MWU algorithm [33] ( follow-the-regularized-leader [35]) for updating the weights. In short, the Phase 1 algorithm updates the response function f via a projected gradient step of the expected loss, and then adjusts the predictors { Q ( j ) } to the updated response function f and the current representations { R ( j ) } j ∈ [ k ] . The Phase 2 algorithm only updates the new representation R ( k +1) via projected gradient steps, while keeping { R ( j ) } j ∈ [ k ] fixed. Given the representations { R ( j ) } j ∈ [ k +1] and the functions { f ( i ) } i ∈ [ m 0 + k ] , a predictor Q ( j,i ) is then fitted to each representation-function pair, which also determines the loss for this pair. The weights { p ( j ) } j ∈ [ k +1] and { o ( i ) } i ∈ [ m 0 + k ] are updated towards the equilibrium of the two-player game determined by the loss of the predictors { Q ( j,i ) } j ∈ [ k +1] ,i ∈ [ m 0 + k ] via the MWU algorithm.

Algorithm 1 Solver of (24): An iterative algorithm for learning mixed representations.

1: input P x , R , F , Q , d, r, m, m 0 glyph[triangleright] Feature distribution, classes, dimensions and parameters

2: input R (1) , { f ( j ) } j ∈ [ m 0 ]

3: begin

4: for k = 1 to m do

5: phase 1: f ( m 0 + k ) is set by a solver of (26) and

glyph[triangleright] Solved using Algorithm 2

6: phase 2: R ( k +1) , { p ( j ) k } j ∈ [ k +1] is set by a solver of (27) glyph[triangleright] Solved using Algorithm 3; step can be removed if k = m

7: end for

8: set m ∗ = arg min k ∈ [ m ] reg k

9: return { R ( j ) } j ∈ [ m ∗ ] and p m ∗ = { p ( j ) k } j ∈ [ m ∗ ]

We next outline two examples, where full details can be found in Appendix H.

Example 6. We validate that efficiency of Algorithm 1 in the linear MSE setting (Section 3), for which a closed-form solution exists. We ran Algorithm 1 on randomly drawn diagonal Σ x , and computed the ratio between the regret obtained by the algorithm to the theoretical value. The left panel of Figure 2 shows that the ratio is between 1 . 15 -1 . 2 in a wide range of d values. We mention again that Algorithm 1 is useful even for this setting since finding an ( glyph[lscript] ∗ + 1) -sparse solution to Ap = b is computationally difficult when ( glyph[lscript] ∗ r ) is very large. For example, in the largest dimension of the experiment, the potential number of representation matrices is ( d r ) = ( 19 5 ) = 11 , 628 .

Our next example pertains to a logistic regression setting, under the cross-entropy loss function. Definition 7 (The linear cross-entropy setting) . Assume that X = R d , that Y = {± 1 } and that E [ x ] = 0 . Assume that the class of representation is linear z = R ( x ) = R glyph[latticetop] x for some R ∈ R :=

glyph[triangleright] Initial representation and initial function (set)

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

1.30

0.14

1.25

AccuracyRatio

0.12

1.20

Regret

0.10

1.15

0.08

中

m=1

m=4

1.10

0.06

m=7

m=10

7

8

9

10

11

12

13

1415161718

319

5

6

78910111213141516171819

120

d

d

R d × r where d > r . Assume that a response function and a prediction rule determine the probability that y = 1 via logistic regression modeling, as f ( y = ± 1 | x ) = 1 / [1 + exp( ∓ f glyph[latticetop] x )] . Assume the cross-entropy loss function, where given that the prediction that y = 1 with probability q results the loss loss ( y, q ) := -1 2 (1 + y ) log q -1 2 (1 -y ) log(1 -q ) . The set of predictor functions is Q := { Q ( z ) = 1 / [1 + exp( -q glyph[latticetop] z )] , q ∈ R r } . As for the linear case, we assume that f ∈ F S for some S ∈ S d ++ . It is not difficult to show that the regret is then given by the expected binary Kullback-Leibler (KL) divergence

Example 8. We ran Algorithm 1 on empirical distributions of features drawn from an isotropic normal distribution, in the linear cross-entropy setting. Algorithm 1 is suitable in this setting since gradients of the regret have closed-form (see Appendix H). The right panel of Figure 2 shows the reduced regret obtained by increasing the support size m of the random representation, and thus the effectiveness of mixed representations.

We refer the reader to Appendix I for additional experiments with Algorithm 1.

5 Conclusion

We proposed a game-theoretic formulation for learning representations of unlabeled features when prior knowledge (or assumptions) on the class of future prediction tasks is available. We focused on the fundamental of linear MSE setting, and derived the optimal solution. Beyond the lower regret that is directly obtained from utilizing the prior knowledge, our results also revealed the importance of using randomized representations. We have then proposed an iterative algorithm suitable for general classes of functions and losses, and exemplified its effectiveness.

Wenext discuss limitations and potential future research: (1) We have focused on the elementary and simplified class F S = { f : ‖ f ‖ S ≤ 1 } , mainly for theoretical investigations. A natural refinement to non-linear functions is the general class F S x := E [ ‖∇ x f ( x ) ‖ 2 S x ] ≤ 1 , where { S x } x ∈ R d is now locally specified (somewhat similarly to the regularization term used in contractive AE [39], though for different reasons). (2) Since the proposed iterative algorithm includes optimization over three players, it is of interest to develop version of the algorithm with lower computational optimization cost. (3) We have assumed that F S is given in advance, and a natural follow-up goal is to efficiently learn S from previous experience, e.g., improving S from one episode to another in a meta-learning setup [40]. (4) It is interesting to evaluate the effectiveness of the learned representation in our formulation, as an initialization for further optimization when labeled data is collected. One may postulate that since our learned representation is uniformly good for all response functions in the class, it may serve as a universal initialization for such training.

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

Broader impact

The research described in this paper is foundational, and does not aim for any specific application. Nonetheless, the learned representation is based on a prior assumption on the class of response functions, and the choice of this prior may have positive or negative impacts: For example, a risk of this choice of prior is that the represented features completely ignore a viable feature for making future predictions. A benefit that can stem from choosing a proper prior is that the representation will null the effect of features that lead to unfair advantages for some particular group, in future predictions. Anyhow, the results presented in the paper are indifferent to such future utilization, and any usage of these results should take into account the aforementioned possible implications.

References

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning . MIT press, 2016.

John N. Tsitsiklis. Decentralized detection. 1989.

XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. On surrogate loss functions and f -divergences. The Annals of Statistics , 37(2):876 - 904, 2009. doi: 10.1214/ 08-AOS595. URL https://doi.org/10.1214/08-AOS595 .

John Duchi, Khashayar Khosravi, and Feng Ruan. Multiclass classification, information, divergence and surrogate risk. Annals of Statistics , 46(6B):3246-3275, 2018. ISSN 00905364. doi: 10.1214/17-AOS1657.

Robert E. Schapire and Yoav Freund. Boosting: Foundations and Algorithms . The MIT Press, 2012. ISBN 0262017180.

Karl Pearson. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science , 2(11):559-572, 1901.

Ian Jolliffe. Principal component analysis. Encyclopedia of statistics in behavioral science , 2005.

John P. Cunningham and Zoubin Ghahramani. Linear dimensionality reduction: Survey, insights, and generalizations. The Journal of Machine Learning Research , 16(1):2859-2900, 2015.

Iain M. Johnstone and Debashis Paul. PCA in high dimensions: An orientation. Proceedings of the IEEE , 106(8):1277-1292, 2018.

Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural computation , 10(5):1299-1319, 1998.

Mark A. Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal , 37(2):233-243, 1991.

Geoffrey E. Hinton and Ruslan R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science , 313(5786):504-507, 2006.

Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. Unsupervised learning of hierarchical representations with convolutional deep belief networks. Communications of the ACM , 54(10):95-103, 2011.

Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research , 11(12), 2010.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8): 1798-1828, 2013.

Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057 , 2000.

Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for 378 Gaussian variables. Advances in Neural Information Processing Systems , 16, 2003. 379

380

381

NoamSlonim, Nir Friedman, and Naftali Tishby. Multivariate information bottleneck. Neural computation , 18(8):1739-1789, 2006.

Peter Harremoës and Naftali Tishby. The information bottleneck revisited or how to choose 382 a good distortion measure. In 2007 IEEE International Symposium on Information Theory , 383 pages 566-570. IEEE, 2007. 384

Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. 385 In 2015 ieee information theory workshop , pages 1-5. IEEE, 2015. 386

387

388

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017.

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

Ravid Shwartz-Ziv. Information flow in deep neural networks. arXiv preprint arXiv:2202.06749 , 2022.

Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research , 19(1):1947-1980, 2018.

Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence , 40(12):2897-2905, 2018.

Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D. Tracey, and David D. Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment , 2019(12):124020, 2019.

Bernhard C. Geiger. On information plane analyses of neural network classifiers- A review. IEEE Transactions on Neural Networks and Learning Systems , 2021.

T. M. Cover and J. A. Thomas. Elements of Information Theory . Wiley-Interscience, 2006. ISBN 0471241954.

Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689 , 2020.

Yann Dubois, Douwe Kiela, David J. Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. Advances in Neural Information Processing Systems , 33:18674-18690, 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress-self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355 , 2023.

Guillermo Owen. Game theory . Emerald Group Publishing, 2013. 412

Yoav Freund and Robert E. Schapire. Adaptive game playing using multiplicative weights. 413 Games and Economic Behavior , 29(1-2):79-103, 1999. 414

415

416

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends R © in Machine Learning , 4(2):107-194, 2012.

417

418

Elad Hazan. Introduction to online convex optimization. Foundations and Trends R © in Optimization , 2(3-4):157-325, 2016.

Maurice Sion. On general minimax theorems. 1958. 419

Roger A. Horn and Charles R. Johnson. Matrix analysis . Cambridge university press, 2012. 420

Dimitris Bertsimas and John N. Tsitsiklis. Introduction to linear optimization , volume 6. 421 Athena scientific Belmont, MA, 1997. 422

Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contrac423 tive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th 424 international conference on international conference on machine learning , pages 833-840, 425 2011. 426

427

428

429

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(9):5149-5169, 2021.

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79: 151-175, 2010.

Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning , pages 3987-3995. PMLR, 2017.

Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10628 , 2017.

Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 , 2019.

Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11254-11263, 2019.

Stephen Boyd, Stephen P. Boyd, and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science , 411(29-30):2696-2711, 2010.

XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory , 56(11):5847-5861, 2010.

Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, and George Tucker. On variational lower bounds of mutual information. In NeurIPS Workshop on Bayesian Deep Learning , 2018.

Tailin Wu, Ian Fischer, Isaac L. Chuang, and Max Tegmark. Learnability for the information bottleneck. In Uncertainty in Artificial Intelligence , pages 1050-1060. PMLR, 2020.

David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics , pages 875-884. PMLR, 2020.

Matthew Chalk, Olivier Marre, and Gasper Tkacik. Relevant sparse codes with variational information bottleneck. Advances in Neural Information Processing Systems , 29, 2016.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning , pages 531-540. PMLR, 2018.

Behrooz Razeghi, Flavio P. Calmon, Deniz Gunduz, and Slava Voloshynovskiy. Bottlenecks CLUB: Unifying information-theoretic trade-offs among complexity, leakage, and utility. arXiv preprint arXiv:2207.04895 , 2022.

Matías Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of information complexity and randomization in representation learning. arXiv preprint arXiv:1802.05355 , 2018.

Borja Rodriguez Galvez. The information bottleneck: Connections to other problems, learn470 ing and exploration of the ib curve, 2019. 471

472

473

Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

Martin J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Karthik Sridharan and Sham M. Kakade. An information theoretic framework for multi-view learning. 2008.

Rana Ali Amjad and Bernhard C. Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE transactions on pattern analysis and machine intelligence , 42(9):2225-2239, 2019.

Artemy Kolchinsky, Brendan D. Tracey, and David H. Wolpert. Nonlinear information bottleneck. Entropy , 21(12):1181, 2019.

D. J. Strouse and David J. Schwab. The information bottleneck and geometric clustering. Neural computation , 31(3):596-612, 2019.

Ankit Pensia, Varun Jog, and Po-Ling Loh. Extracting robust and accurate features via a robust information bottleneck. IEEE Journal on Selected Areas in Information Theory , 1(1): 131-144, 2020.

Shahab Asoodeh and Flavio P Calmon. Bottleneck problems: An information and estimationtheoretic view. Entropy , 22(11):1325, 2020.

Vudtiwat Ngampruetikorn and David J. Schwab. Perturbation theory for the information bottleneck. Advances in Neural Information Processing Systems , 34:21008-21018, 2021.

Xi Yu, Shujian Yu, and José C Príncipe. Deep deterministic information bottleneck with matrix-based entropy functional. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3160-3164. IEEE, 2021.

Vudtiwat Ngampruetikorn and David J. Schwab. Information bottleneck theory of high-dimensional regression: Relevancy, efficiency and optimality. arXiv preprint arXiv:2208.03848 , 2022.

Deniz Gündüz, Zhijin Qin, Inaki Estella Aguerri, Harpreet S Dhillon, Zhaohui Yang, Aylin Yener, Kai Kit Wong, and Chan-Byoung Chae. Beyond transmitting bits: Context, semantics, and task-oriented communications. IEEE Journal on Selected Areas in Communications , 41 (1):5-41, 2022.

Vudtiwat Ngampruetikorn and David J. Schwab. Generalized information bottleneck for Gaussian variables. arXiv preprint arXiv:2303.17762 , 2023.

Vudtiwat Ngampruetikorn, William Bialek, and David Schwab. Information-bottleneck renormalization group for self-supervised representation learning. Bulletin of the American Physical Society , 65, 2020.

William B. Johnson. Extensions of Lipschitz mappings into a Hilbert space. Contemp. Math. , 26:189-206, 1984.

Santosh S. Vempala. The random projection method , volume 65. American Mathematical Soc., 2005.

Michael W. Mahoney et al. Randomized algorithms for matrices and data. Foundations and Trends R © in Machine Learning , 3(2):123-224, 2011.

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

David P. Woodruff et al. Sketching as a tool for numerical linear algebra. Foundations and Trends R © in Theoretical Computer Science , 10(1-2):1-157, 2014.

Fan Yang, Sifan Liu, Edgar Dobriban, and David P. Woodruff. How to reduce dimension with PCAandrandom projections? IEEE Transactions on Information Theory , 67(12):8154-8189, 2021.

John F. Nash Jr. Equilibrium points in n-person games. Proceedings of the national academy of sciences , 36(1):48-49, 1950.

Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning , pages 224-232. PMLR, 2017.

Paulina Grnarova, Kfir Y. Levy, Aurelien Lucchi, Thomas Hofmann, and Andreas Krause. An online learning approach to generative adversarial networks. arXiv preprint arXiv:1706.03269 , 2017.

Ilya O. Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, and Bernhard Schölkopf. Adagan: Boosting generative models. Advances in neural information processing systems , 30, 2017.

Max Welling, Richard Zemel, and Geoffrey E. Hinton. Self supervised boosting. Advances in neural information processing systems , 15, 2002.

Abraham Wald. Contributions to the theory of statistical estimation and testing hypotheses. The Annals of Mathematical Statistics , 10(4):299-326, 1939.

Larry Wasserman. All of statistics: A concise course in statistical inference , volume 26. Springer, 2004.

Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence. Annals of Statistics , pages 1564-1599, 1999.

Peter D. Grünwald and A. Philip Dawid. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. The Annals of Statistics , 32(4):1367-1433, 2004.

David Haussler and Manfred Opper. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Statistics , 25(6):2451-2492, 1997.

Farzan Farnia and David Tse. A minimax approach to supervised learning. Advances in Neural Information Processing Systems , 29, 2016.

Jorge Silva and Felipe Tobar. On the interplay between information loss and operation loss in representations for classification. In International Conference on Artificial Intelligence and Statistics , pages 4853-4871. PMLR, 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine , 35(1):53-65, 2018.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization , volume 28. Princeton university press, 2009.

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances in neural information processing systems , 29, 2016.

Constantinos Daskalakis, Alan Deckelbaum, and Anthony Kim. Near-optimal no-regret algorithms for zero-sum games. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms , pages 235-254. SIAM, 2011.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems , 26, 2013.

James P. Bailey and Georgios Piliouras. Multiplicative weights update in zero-sum games. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 321-338, 2018.

Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger B. Grosse. Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization. In International Conference on Artificial Intelligence and Statistics , pages 7659-7679. PMLR, 2022.

Florian Schäfer and Anima Anandkumar. Competitive gradient descent. Advances in Neural Information Processing Systems , 32, 2019.

Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of GANs. Advances in neural information processing systems , 30, 2017.

Alistair Letcher, David Balduzzi, Sébastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. Differentiable game mechanics. The Journal of Machine Learning Research , 20(1):3032-3071, 2019.

Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1802-1811. PMLR, 2019.

Guodong Zhang and Yuanhao Wang. On the suboptimality of negative momentum for minimax optimization. In International Conference on Artificial Intelligence and Statistics , pages 2098-2106. PMLR, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Dimitri Bertsekas, Angelia Nedic, and Asuman Ozdaglar. Convex analysis and optimization , 596 volume 1. Athena Scientific, 2003. 597

Ky Fan. On a theorem of Weyl concerning eigenvalues of linear transformations i. Proceed598 ings of the National Academy of Sciences , 35(11):652-655, 1949. 599

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends and formulates learning representation procedure as a three-player game when the downstream task is known. Unlike conventional unsupervised learning algorithms, the proposed framework helps the learning process in the feature space be directed. They set up a general and basic mathematic form (linear-MSE) to study, which proves the utility of prior information. And, they extend results via studying mixed strategies, which shows the importance to use randomized representations. They propose gradient-based iterative algorithms to optimize objectives and validate the effectiveness of their framework with examples (i.e., linear-MSE and linear-CE).\n\nSTRENGTHS:\nThis paper presents several strengths, including a clear and well-written introduction that guides the reader to understand the problem and the motivations behind it. The findings from this paper is interesting: it shows the benefits to use randomized representation. Besides, the authors provide concrete steps to introduce iterative algorithm for their general cases to establish it, although the takeaway conclusion is not always straightforward to see, such as in the figure for Example 5.\n\nWEAKNESSES:\nHere are some potential improvements to the original content:\n\n1. The paper could benefit from a more concrete approach to evaluating the effectiveness of the proposed framework. While using regret as a metric is a relative approach, it may be more informative to show performance using exact MSE. For example, the paper could compare results between a conventional representation (without prior knowledge about downstream task) and a representation found by the proposed framework, with prior knowledge about the downstream task. This could help demonstrate any advantages of the proposed framework more clearly.\n\n2. To strengthen the evidence presented in the paper, it would be beneficial to include experiments with real-world datasets. Providing results with tabular datasets could help establish stronger findings.\n\nQUESTIONS:\n1. It would be beneficial to observe the learning curve of three players during algorithm optimization of the regret value in order to better understand their learning behaviors.\n2. The authors introduce the concept of \"self-defined signal\" as another perspective, but further clarification is needed to fully comprehend this idea.\n3. Although recent language models do not require prior knowledge of future prediction tasks, they have demonstrated impressive generalization performance. It would be interesting to explore the potential of incorporating the proposed framework into the training procedure.\n\nLIMITATIONS:\nThe paper presents a foundational approach that can be applied to various scenarios. The limitations and wider implications discussed in the paper are well-defined. Finding proper prior knowledge to use and further compute the representation should be studied and explored.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of dimensionality reduction given prior information about downstream tasks via game theoretic optimization point of view. They define a three player game.  One player chooses the parameters of the representation function that reduces the dimensionality of the data. One player chooses the downstream task, constrained to respect a task distribution prior. The last player chooses the parameters of the model learned based on the dimensionality reduced inputs and the task. The authors theoretically analyze the case of linear dimensionality reduction and linear prediction functions under the MSE loss function. They then propose a greedy algorithm for the general case, which they evaluate empirically.\n\nSTRENGTHS:\nThe framework and proposed solution are clearly explained. To the best of my knowledge, while adversarial learning between two or more agents is a well explored tool for improving generalization and robustness, the specific framework proposed by the authors is novel. In addition, even in the seemingly easy case of linear models and MSE loss, Theorem 3 is quite non trivial. The iterative greedy algorithm for the general case may also be of interest to practitioners.\n\nWEAKNESSES:\nI would have liked to see more motivation for the specific framework proposed. More specifically:\n\n * It is not clear why the game is not a a two player game instead of a three player game. I do not understand why we need two separate players for controlling $R$ and $Q$. In fact the paper mostly ignores the $Q$ player by focusing on the regret of the $R$ player, implicitly assuming that $Q$ best responds by finding the optimal model. Is there a specific motivation for having the $Q$ player?\n*  If we were to collapse the $R$ and $Q$ player, the setting is closer to adversarial learning between two players, the one that chooses the model and the one that (with some constraints) is allowed to change a given training dataset in an adversarial way. The main difference is that in this work the adversary controls a response function $f$ instead of manipulating a training set. It is not clear to me if/when this difference is significant, especially in practical settings where the input feature and response function distributions are discrete. \n* It is not very clear to me if/how the randomized representation function $R$ in the mixed strategy setting can be used in practice. While it makes sense from a game theoretic point of view to extend to mixed strategies, I think connecting this back to the representation learning use case will help make the paper more compelling.\n\nQUESTIONS:\nI would suggest to the authors to clarify the points mentioned in the Weaknesses section.\n\nLIMITATIONS:\nLimitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the important problem of representation learning, where the goal is to learn a representation that works well for all problems in a known problem class. It presents theoretical results for the special case where both the representation function and prediction function class are linear, and also provide an extension of the algorithm to the general function class setting.\n\nSTRENGTHS:\nThe problem of representation learning is obviously important, and the paper did a good job describing the problem setting, motivation and results.\n\nWEAKNESSES:\n1. The discussion regarding mixed representation in the paragraph on line 61 seems not practically relevant. In most cases, it makes sense to assume that there exists a representation z that captures all necessary information from x that predict any function f in F, which implies that there is a pure-strategy Nash Equilibrium. \n\n2. An important piece of prior work is missing. [3] studies the exact same problem setting that is studied in this paper. Their theoretical result seems to subsume the theoretical result in this paper, as they allow arbitrary nonlinear representation and prediction functions. In addition, there is another line of work that assumes the tasks come in an i.i.d fashion, and uses uniform convergence analysis to bound the regret on a new task [1,2]. At the moment, it is unclear how the current paper compare with these prior works mentioned above. In particular, what are the differences between Theorem 2 in the current paper v.s. Theorem 2 and 3 in [3] when specialized to the linear setting.\n\n[1] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:\n149–198, 2000.\n\n[2] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask\nrepresentation learning. The Journal of Machine Learning Research, 2016.\n\n[3] Tripuraneni, Nilesh, Michael Jordan, and Chi Jin. \"On the theory of transfer learning: The importance of task diversity.\" Advances in neural information processing systems， 2020.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies the representation learning from the multi-layer gaming perceptive. The authors propose an min-max optimization problem for learning a good representation with the prior knowledge of the latter prediction tasks. The authors provide some theoretical analyses and examples in the linear case. For the general tasks, some iterative algorithms are proposed and the numerical results are provided to demonstrate the performance.

STRENGTHS:
- This paper is well written and easy to follow. The theoretical results are provided with several examples to demonstrate the intuition behind the theories.
- Numerical results demonstrate the effectiveness of the proposed result.

WEAKNESSES:
- The connection between the linear case and the general, three-player game case is not enough. In the linear case setting, it's easy to show a close form solution for this min-max problem. However, in the general setting, the authors does not fully demonstrate why the iterative based algorithm can convergence.
- I wonder the time complexity of the proposed general case algorithm since it need to call Algorithm 2 and 3 in the nested loops.

QUESTIONS:
Please address my concerns in the weakness part, if possible.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a game theoretic framework for learning representations in a general setup where some knowledge/prior about the prediction task is available. The representation learning task is cast as a certain game --- which is worked out exactly in the fully linear setting under the squared loss (casting new light on PCA-type techniques), and for which a minimax solution-seeking iterative gradient-based method is provided for a more general case. Theory is accompanied by examples, both worked-out and experiments, that support the theory and offer new insights into well-known representation learning questions (such as the necessity of mixed rather than pure representations etc).\n\nSTRENGTHS:\nThe main strength of this paper is that the game theoretic framework proposed by the authors appears to shed light onto many important questions about representations (including the necessity and type of required randomization in the representation, how to form universal representation-based initializations, how to use priors, etc.) in a very compact setup. Secondly, the theory in the linear case is worked out quite precisely and nicely, thus shedding new (to me at least) light on PCA-based representation technique. Nice worked-out examples are given throughout the paper to support the general theory. As well as, finally, not only is an interesting general representation-building algorithm given, but it is also, through a few experiments, confirmed to be implementable, reasonable, and overall sufficiently practical.\n\nWEAKNESSES:\nA weakness of the proposed game-theoretic framework (which could alternately be viewed as its strength, paradoxically) is that it doesn't seem to allow us to run away from the intrinsic hardness (and hardness of analysis) of finding good representations --- instead, this difficulty simply translates into the difficulty of procuring convergence analyses/rates for methods that solve for generally nonconvex/nonconcave saddle points that result out of this formulation. Now, why this could in the future prove to be a strength is that conceivably, it might allow for some future work on connecting hardness of finding good representations (in settings such as the one discussed in this paper), to the hardness of certain saddle point problems. Still, such future use is not fully clear at this point, which is why at this point in time I listed it here. (Of course, I would welcome arguments from the authors as to how this weakness could be turned into a strength.) \n\nThat said, given the hardness of the problem and the overall amount of interesting contributions in this paper, I will not consider the above to be of too much impact for the purposes of rating this paper.\n\nQUESTIONS:\nFirst off, the discussion about the information bottleneck approaches, which is in part done in the intro and in part --- in the appendix, left me still wondering about the mentioned existing two-player game theoretic formulation of this problem, and how it morally compares to the current proposed approach. Would the authors be able to provide me with some more detailed insight on this? \n\nAdditionally, and perhaps relatedly, I am wondering why the authors insist on branding their new setting as a three-player game rather than a two player game --- I am not sure if I am missing any crucial semantics behind this naming choice, and would appreciate any insights here. (In principle, the third player isn't even explicitly featuring in the general proposed Algorithm 1 for this setting --- currently it seems to just implicitly fit Q.)\n\nI am also interested in hearing more about the connection between the stepwise fitting procedure, proposed by the authors, to boosting setups (which broadly speaking tend to also gradually add elements into the set to increase some notion of goodness). A sentence or two was mentioned to that extent in the appendix, but I would be interested in understanding any deeper existing underpinnings that led the authors to employ this particular setup here.\n\nFinally, from what was described in the paper, I was still left to wonder what kind of useful initialization the authors imagine for the initial representation and initial function in Algorithm 1. A couple of concrete thought experiments would suffice and be much appreciated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}