Title: Block Coordinate Descent Methods for Optimization under J-Orthogonality Constraints with Applications
Venue: NeurIPS 2024
Accepted: no
Keywords: Orthogonality Constraints

Nonconvex Optimization

Nonsmooth Composite Optimization

Block Coordinate Descent

Convergence Analysis
Abstract: The J-orthogonal matrix, also referred to as the hyperbolic orthogonal matrix, is a class of special orthogonal matrix in hyperbolic space, notable for its advantageous properties. These matrices are integral to optimization under J-orthogonal constraints, which have widespread applications in statistical learning and data science. However, addressing these problems is generally challenging due to their non-convex nature and the computational intensity of the constraints. Currently, algorithms for tackling these challenges are limited. This paper introduces \textbf{JOBCD}, a novel Block Coordinate Descent method designed to address optimizations with J-orthogonality constraints. We explore two specific variants of \textbf{JOBCD}: one based on a Gauss-Seidel strategy (\textbf{GS-JOBCD}), the other on a variance-reduced and Jacobi strategy (\textbf{VR-J-JOBCD}). Notably, leveraging the parallel framework of a Jacobi strategy, \textbf{VR-J-JOBCD} integrates variance reduction techniques to decrease oracle complexity in the minimization of finite-sum functions. For both \textbf{GS-JOBCD} and \textbf{VR-J-JOBCD}, we establish the oracle complexity under mild conditions and strong limit-point convergence results under the Kurdyka-Lojasiewicz inequality. To demonstrate the effectiveness of our method, we conduct experiments on hyperbolic eigenvalue problems, hyperbolic structural probe problems, and the ultrahyperbolic knowledge graph embedding problem. Extensive experiments using both real-world and synthetic data demonstrate that \textbf{JOBCD} consistently outperforms state-of-the-art solutions, by large margins.

Abstract

The J-orthogonal matrix, also referred to as the hyperbolic orthogonal matrix, is a class of special orthogonal matrix in hyperbolic space, notable for its advantageous properties. These matrices are integral to optimization under J-orthogonal constraints, which have widespread applications in statistical learning and data science. However, addressing these problems is generally challenging due to their non-convex nature and the computational intensity of the constraints. Currently, algorithms for tackling these challenges are limited. This paper introduces JOBCD , a novel Block Coordinate Descent method designed to address optimizations with J-orthogonality constraints. We explore two specific variants of JOBCD : one based on a Gauss-Seidel strategy ( GS-JOBCD ), the other on a variance-reduced and Jacobi strategy ( VR-J-JOBCD ). Notably, leveraging the parallel framework of a Jacobi strategy, VR-J-JOBCD integrates variance reduction techniques to decrease oracle complexity in the minimization of finite-sum functions. For both GS-JOBCD and VR-J-JOBCD , we establish the oracle complexity under mild conditions and strong limit-point convergence results under the Kurdyka-Lojasiewicz inequality. To demonstrate the effectiveness of our method, we conduct experiments on hyperbolic eigenvalue problems, hyperbolic structural probe problems, and the ultrahyperbolic knowledge graph embedding problem. Extensive experiments using both real-world and synthetic data demonstrate that JOBCD consistently outperforms state-of-the-art solutions, by large margins.

1 Introduction

Amatrix X ∈ R n × n is a J-orthogonal matrix if X T JX = J , where J = [ I p 0 0 -I n -p ] , and I p is a p × p identity matrix. Here, J ∈ R n × n is the signature matrix with signature ( p, n -p ) . In this paper, we mainly focus on the following optimization problem under J-orthogonality constraints:

Here, f ( X ) could have a finite-sum structure, each component function f i ( X ) is assumed to be differentiable, and N is the number of data points. For brevity, the J-orthogonality constraint X T JX = J in Problem (1) is rewritten as X ∈ J .

We impose the following assumptions on Problem (1) throughout this paper. ( A -i) For any matrices X and X + , we assume f i : R n × n ↦→ R is continuously differentiable for some symmetric positive semidefinite matrix H ∈ R nn × nn that:

for all i ∈ [ N ] , where ∥ H ∥ ≤ L f for some constant L f > 0 and ∥ X ∥ 2 H ≜ vec( X ) ⊤ H vec( X ) . 31 This further implies that: ∥∇ f i ( X ) - ∇ f i ( X + ) ∥ F ≤ L f ∥ X -X + ∥ F for all i ∈ [ N ] . Impor32 tantly, the function f ( X ) = 1 2 tr( X ⊤ CXD ) = 1 2 ∥ X ∥ 2 H with H = D ⊗ C satisfies the equality 33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

∀ X , X + , f ( X + ) = Q ( X + ; X ) in (2), where C ∈ R n × n and D ∈ R n × n are arbitrary symmetric matrices. ( A -ii) The function f i ( X ) is coercive for all i ∈ N , that is, lim ∥ X ∥ F →∞ f i ( X ) = ∞ , ∀ i . Problem (1) defines an optimization framework that is fundamental to a wide range of models in statistical learning and data science, including hyperbolic eigenvalue problem [6, 43, 40], hyperbolic structural probe problem [20, 7], and ultrahyperbolic knowledge graph embedding [48]. Additionally, it is closely related to machine learning in hyperbolic spaces, including Lorentz model learning [35, 50, 8] and ultrahyperbolic neural networks [27, 54, 42]. It also intersects with hyperbolic linear algebra [3, 21], addressing problems such as the indefinite least squares problem, hyperbolic QR factorization, and indefinite polar decomposition.

1.1 Related Work

▶ Block Coordinate Descent Methods . Block Coordinate Descent (BCD) is a well-established iterative algorithm that sequentially minimizes along block coordinate directions. Its simplicity and efficiency have led to its widespread adoption in structured convex applications [37]. Recently, BCD has gained traction in non-convex problems due to its robust optimality guarantees and/or excellent empirical performance in areas including optimal transport [22], matrix optimization [12], fractional minimization [52], deep neural networks [5, 53, 32], federated learning[47], black-box optimization [4], and optimization with orthogonality constraints [51, 14]. To our knowledge, this is the first application of BCD methods to optimization under J-orthoginality constraints, with a focus on analyzing their theoretical guarantees and empirical efficacy.

▶ Minimizing Smooth Functions under J-Orthogonality Constraints . The J-orthogonal matrix belongs to a subset of generalized orthogonal matrices [16, 36, 23]. However, projecting onto the J-orthogonality constraint poses challenges, complicating the extension of conventional optimization algorithms to address optimization problems under these constraints [1, 16]. This contrasts with computing orthogonal projections using methods such as polar or SVD decomposition, or approximating them via QR factorization. Existing methods for addressing Problem (1) can be categorized into three classes. ( i ) CS-Decomposition Based Methods. These approaches involve parameterizing four orthogonal matrices (as described in Proposition 2.2) and subsequently minimizing a smooth function over these matrices in an alternating fashion. The involvement of 3 × 3 block matrices makes the implementation of these methods very challenging. Consequently, the work of [48] focuses on optimizing a reduced subspace of the CS decomposition parameters, albeit at the expense of losing some degrees of freedom. ( ii ) Unconstrained Multiplier Correction Methods [31, 13, 14]. These methods leverage the symmetry and explicit closed-form expression of the Lagrangian multiplier at the first-order optimality condition. Consequently, they address an unconstrained problem, resulting in efficient first-order infeasible approaches. ( iii ) Alternating Direction Method of Multipliers [19]. This method reformulates the original problem into a bilinear constrained optimization problem by introducing auxiliary variables. It employs dual variables to handle bilinear constraints, iteratively optimizing primal variables while keeping other primal and dual variables fixed, and using a gradient ascent strategy to update the dual variables. This approach has become widely adopted for solving general nonconvex and nonsmooth composite optimization problems. Notably, all the aforementioned methods solely identify critical points of Problem (1).

▶ Finite-Sum Problems via Stochastic Gradient Descent . The finite-sum structure is prevalent in machine learning and statistical modeling, facilitating decomposition into smaller, more manageable components. This property is advantageous for developing efficient algorithms for large-scale problems, such as Stochastic Gradient Descent (SGD). Reducing variance is crucial in SGD because it can lead to more stable and faster convergence. Various techniques, such as mini-batch SGD, momentum methods, and variance reduction methods like SAGA [10], SVRG [25], SARAH [34], SPIDER [11, 44], SNVRG [55], and PAGE [30], have been developed to address this issue. Additionally, SGD for minimizing composite functions has also been investigated by the authors [15, 24, 29].

1.2 Contributions

This paper makes the following contributions. ( i ) Algorithmically: We introduce the JOBCD algorithm, a novel Block Coordinate Descent method specifically designed to tackle optimizations constrained by J-orthogonality. We explore two specific variants of JOBCD , one based on a Gauss-Seidel strategy ( GS-JOBCD ), the other on a variance-reduced and Jacobi strategy ( VRJ-JOBCD ). Notably, VR-J-JOBCD incorporates a variance-reduction technique into a parallel framework to reduce oracle complexity in the minimization of finite-sum functions (See Section 2). ( ii ) Theoretically: We provide comprehensive optimality and convergence analyses for both

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

algorithms (see Sections 3 and 4). ( iii ) Empirically: Extensive experiments across hyperbolic eigenvalue problems, structural probe problems, and ultrahyperbolic knowledge graph embedding, using both real-world and synthetic data, consistently show the significant superiority of JOBCD over state-of-the-art solutions (see Section 5).

2 The Proposed JOBCD Algorithm

This section proposes JOBCD for solving optimization problems under J-orthogonality constraints in Problem (1), which is based on randomized block coordinate descent. Two variants of JOBCD are explored, one based on a Gauss-Seidel strategy ( GS-JOBCD ), the other on a variance-reduced and Jocobi strategy ( VR-J-JOBCD ).

Notations . We define [ n ] ≜ { 1 , 2 , . . . , n } . We denote Ω ≜ {B 1 , B 2 , . . . , B C 2 n } as all the possible combinations of the index vectors choosing 2 items from n without repetition. For any B ∈ Ω , we define U B ∈ R n × 2 as ( U B ) ji = 1 if B i = j, else 0 for all j and i , leading to U T B X = X ( B , :) ∈ R 2 × n . We denote J B ≜ { V | V T J BB V = J BB } , where J BB ∈ R 2 × 2 is the sub-matrix of J indexed by B . Further notations are provided in Appendix A.1.

2.1 Gauss-Seidel Block Coordinate Descent Algorithm

This subsection describes the proposed GS-JOBCD algorithm. We consider Problem (1) with N = 1 only, without utilizing its finite-sum structure.

GS-JOBCD is an iterative algorithm that, in each iteration t , randomly and uniformly (with replacement) selects a coordinate B from the set Ω and then solves a small-sized subproblem. The row index [ n ] of the decision variable X are separated to two sets B and B c , where B ∈ Ω with | B | = 2 is the working set and B c = [ n ] \ B . For simplicity, we use B instead of B t . Following [51], we consider the following block coordinate update rule: [ X t +1 ( B , :) = VX t ( B , :)] ⇔ [ X t +1 = X t + U B ( V -I ) U T B X t ] , where V ∈ R 2 × 2 is some suitable matrix.

The following lemma illustrates matrix selection for enforcing J-orthogonality constraints via the update rule X + ⇐X B ( V ) ≜ X + U B ( V -I ) U T B X , and presents associated properties.

Lemma 2.1. (Proof in Section C.1) For any B ∈ Ω , we define X + ≜ X B ( V ) ≜ X + U B ( V -I ) U T B X . We have: ( a ) If V ∈ J B and X ∈ J , then X + ∈ J . ( b ) ∥ X + -X ∥ 2 F ≤ ∥ X ∥ 2 F · ∥ V -I ∥ 2 F . ( c ) ∥ X + -X ∥ 2 H ≤ ∥ V -I ∥ 2 Q for all Q ≽ Q ≜ ( Z ⊤ ⊗ U B ) ⊤ H ( Z ⊤ ⊗ U B ) , Z ≜ U ⊤ B X ∈ R k × n .

▶ The Main Algorithm . Using the above update rule, we consider the following iterative procedure: X t +1 ⇐X t B ( ¯ V t ) , where ¯ V t ∈ arg min V f ( X t B ( V )) . However, the resulting subproblem could be still difficult to solve. This inspires us to use sequential majorization minimization [38, 33] to address it. This technique iteratively constructs a surrogate function that upper-bounds the objective function, allowing for effective optimization and gradual reduction of the objective function. We derive:

where step ① uses Inequality (2); step ② uses Claim ( c ) of Lemma 2.1, θ ≥ 0 and the fact that ⟨ U B ( V -I ) U T B X , ∇ f ( X ) ⟩ = ⟨ V -I , [ ∇ f ( X ) X T ] BB ⟩ , and the choice of Q ∈ R 4 × 4 that:

Therefore, the function G ( V ; X t , B t ) becomes a majorization function of f ( X ) at X t ∈ J for all B t ∈ Ω . We can consider the following optimization problem to find V t : V t ∈ arg min V G ( V ; X t , B t ) .

We summarize the proposed GS-JOBCD in Algorithm 1.

Although the J-orthogonality constraint typically has a sorted diagonal with diag( J ) ∈ {-1 , +1 } n , GS-JOBCD is also applicable to problems with more general constraints X T JX = J where diag( J ) ∈ {± 1 } n is unsorted.

▶ Solving the Small-Sized Subproblem . We now elaborate on how to find the global optimal solution of Problem (6). We notice that V ∈ J B ≜ { V | V T J BB V = J BB } , where J BB ∈ { ( 1 0 0 -1 ) , ( 1 0 0 1 ) , ( -1 0 0 -1 ) } . We now concentrate on the first case where J BB = ( 1 0 0 -1 ) . The following proposition provides a strategy to decompose any J-orthogonal matrix.

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

Algorithm 1: GS-JOBCD : Block Coordinate Descent Methods using a Gauss-Seidel Strategy for Solving Problem (1)

Init.: Set X 0 to satisfy J-orthogonality constraints (e.g., via Hyperbolic CS Decomposition). for t from 0 to T do

(S1) Choose a coordinate B t with | B t | = 2 from the set Ω randomly and uniformly (with replacement) for the t -th iteration. Denote B = B t .

(S2) Choose a matrix Q ∈ R 4 × 4 using Formula (4).

(S3) Solve the following small-size subproblem globally.

where P ≜ [ ∇ f ( X t )( X t ) T ] BB -mat( ˙ Q vec( I 2 )) , ˙ Q = Q + θ I and c ≜ f ( X t ) -⟨ I 2 , [ ∇ f ( X t )( X t ) T ] BB ⟩ + 1 ∥ I ∥ 2 ˙ is a constant.

2 Q (S4) X t +1 ( B , :) = V t X t ( B , :) end

Proposition 2.2. ( Hyperbolic CS Decomposition [41]) Let V be J-orthogonal with signature ( p, n -p ) . Assume that n -p ≤ p . Then there exist vectors ˙ c, ˙ s ∈ R n -p with ˙ c ⊙ ˙ c -˙ s ⊙ ˙ s = 1 , and orthogonal matrices U 1 , V 1 ∈ R p × p and U 2 , V 2 ∈ R ( n -p ) × ( n -p ) such that: V = [ U 1 0 0 U 2 ][ Diag(˙ c ) 0 Diag(˙ s ) 0 I p -( n -p ) 0 Diag(˙ s ) 0 Diag(˙ c ) ][ V T 1 0 0 V T 2 ] .

Applying Proposition 2.2 with n = 2 , p = 1 , and U 1 = U 2 = V 1 = V 2 = ± 1 , ˜ c 2 -˜ s 2 = 1 with ˜ c , ˜ s ∈ R , we parametrize V as: V = ( ± 1 0 0 ± 1 ) · ( ˜ c ˜ s ˜ s ˜ c ) · ( ± 1 0 0 ± 1 ) , where we denote ˜ s as sinh( µ ) , ˜ c as cosh( µ ) , and ˜ t as tanh( µ ) for some µ ∈ R , for simplicity of notation. It is not difficult to show that Problem (6) reduces to the following one-dimensional search problem:

We apply a breakpoint search method to solve Problem (7). For simplicity, we provide an analysis only for the first case. A detailed discussion of all four cases can be found in Appendix Section B.1. For the case where V = ( ˜ c ˜ s ˜ s ˜ c ) , Problem (7) reduces to the following problem:

where a = P 11 + P 22 , b = P 12 + P 21 , c = 1 2 ( ˙ Q 11 + ˙ Q 41 + ˙ Q 14 + ˙ Q 44 ) , d = 1 2 ( ˙ Q 21 + ˙ Q 31 + ˙ Q 12 + ˙ Q 42 + ˙ Q 13 + ˙ Q 43 + ˙ Q 24 + ˙ Q 34 ) , and e = 1 2 ( ˙ Q 22 + ˙ Q 32 + ˙ Q 23 + ˙ Q 33 ) . Then we perform a substitution to convert Problem (8) into an equivalent problem that depends on the trigonometric functions: ( i ) ˜ c 2 = 1 1 -˜ t 2 ; ( ii ) ˜ s 2 = ˜ t 2 1 -˜ t 2 ; ( iii ) ˜ t = ˜ s ˜ c . The following lemma provides a characterization of the global optimal solution for Problem (8).

Lemma 2.3. (Proof in Section C.2) We let ˘ F (˜ c, ˜ s ) ≜ a ˜ c + b ˜ s + c ˜ c 2 + d ˜ c ˜ s + e ˜ s 2 . The optimal solution ¯ µ to Problem (8) can be computed as: [cosh(˜ µ ) , sinh(˜ µ )] ∈ arg min [ c,s ] ˘ F ( c, s ) , s . t . [ c, s ] ∈ { [ 1 √ 1 -( ¯ t + ) 2 , ¯ t + √ 1 -( ¯ t + ) 2 ] , [ -1 √ 1 -( ¯ t -) 2 , -¯ t -√ 1 -( ¯ t -) 2 ] } , where ¯ t + ∈ arg min t p ( t ) ≜ a + bt √ 1 -t 2 + w + dt 1 -t 2 ; ¯ t -∈ arg min t ˜ p ( t ) ≜ -a -bt √ 1 -t 2 + w + dt 1 -t 2 . Here w = c + e .

We now describe how to find the optimal solution ¯ t + , where ¯ t + ∈ arg min t p ( t ) ≜ a + bt √ 1 -t 2 + w + dt 1 -t 2 ; this strategy can naturally be extended to find ¯ t -. Initially, we have the following first-order optimality conditions for the problem: 0 = ∇ p ( t ) =[ b (1 -t 2 ) + ( a + bt ) t ] √ 1 -t 2 + [ d (1 -t 2 ) + ( w + dt )(2 t )] ⇔ dt 2 +2 wt + d = -[ b + at ] √ 1 -t 2 . Squaring both sides yields the following quartic equation: c 4 t 4 + c 3 t 3 + c 2 t 2 + c 1 t + c 0 = 0 , where c 4 = d 2 + a 2 , c 3 = 4 wd +2 ab ,

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

c 2 = 4 w 2 +2 d 2 -a 2 + b 2 , c 1 = 4 wd -2 ab , c 0 = d 2 -b 2 . This equation can be solved analytically by Lodovico Ferrari's method [46], resulting in all its real roots { ¯ t 1 , ¯ t 2 , . . . , ¯ t j } with 1 ≤ j ≤ 4 .

For the second and third cases, Problem (6) essentially boils down to optimization under orthogonality constraints. The work of [51] derives a breakpoint search method for finding the optimal solution for Problem (6) with J BB ∈ { ( 1 0 0 1 ) , ( -1 0 0 -1 ) } using the Givens rotation and Jacobi reflection matrices.

2.2 Variance-Reduced Jacobi Block Coordinate Descent Algorithm

This subsection proposes the VR-J-JOBCD algorithm, a randomized block coordinate descent method derived from GS-JOBCD . Importantly, by leveraging the parallel framework of a Jacobi strategy [17, 9], VR-J-JOBCD integrates variance reduction techniques [39, 30, 18] to decrease oracle complexity in the minimization of finite-sum functions. This makes the algorithm effective for minimizing large-scale problems under J-orthogonality constraints.

Notations . We assume n is an even number in this paper. We create ( n/ 2) pairs by non-overlapping grouping of the numbers in any arbitrary combination, with each pair containing two distinct numbers from the set [ n ] . It is not hard to verify that such grouping yields C J = ( n !) / (2 n/ 2 n 2 !) possible combinations. The set of these combinations is denoted as Υ ≜ { ˜ B i } C J i =1 ≜ { ˜ B 1 , ˜ B 2 , . . . , ˜ B C J } 1 .

▶ Variance Reduction Strategy . We incorporate state-of-the-art variance reduction strategies from the literature [30, 5] into our algorithm to solve Problem (1). These methods iteratively generate a stochastic gradient estimator as follows:

Here, { S t + , S t ∗ } are uniform random minibatch samples with | S t + | = b , | S t ∗ | = b ′ , and ˜ G 0 = 1 b ∑ i ∈ S 0 + ∇ f i ( X 0 ) . We drop the superscript t for { S t + , S t ∗ } as t can be inferred from context. We only focus on the default setting that [30, 5]: b = N , b ′ = √ b and p = b ′ b + b ′ .

▶ Jacobi Block Coordinate Descent Method . The proposed algorithm is built upon the parallel framework of a Jacobi strategy. In each iteration t , we randomly and uniformly (with replacement) select a coordinate set B t ≜ { B t (1) , B t (2) , · · · , B t ( n 2 ) } from the set Υ with B t ∈ N n 2 × 2 and B t ( i ) ∈ N 2 .

For all t , we have: B t ( i ) ∩ B t ( j ) = ∅ and ∪ n/ 2 i =1 ( B t ( i ) ) = [ n ] . We drop the superscript t if t can be inferred from context.

The following lemma shows how to choose a suitable matrix Q so that the Jacobi strategy can be applied.

̸

Lemma 2.4. (Proof in Section C.3) We let B t ≜ { B t (1) , B t (2) , · · · , B t ( n 2 ) } ∈ Υ for all t . We let Q = ς I 4 , where ς is some suitable constant with ς ≤ L f . For any B t ( i ) and B t ( j ) with i = j , their corresponding objective functions as in Equation (3) are independent.

We consider the following block coordinate update rule in VR-J-JOBCD : X t +1 ⇐ ˜ X t B ( V : ) ≜ X t +[ ∑ n/ 2 i =1 U B ( i ) ( V i -I 2 ) U ⊤ B ( i ) ] X t . The following lemma provides properties of this rule.

Lemma 2.5. (Proof in Section C.4) We let B ∈ Υ , V i ∈ J B ( i ) , X ∈ J , and i ∈ [ n 2 ] . We define X + ≜ ˜ X B ( V : ) ≜ X + [ ∑ n/ 2 i =1 U B ( i ) ( V i -I 2 ) U ⊤ B ( i ) ] X . We have: ( a ) ∑ n 2 i =1 ∥ U B ( i ) ( V i -I 2 ) U ⊤ B ( i ) X ∥ 2 F = ∥ ∑ n 2 i =1 U B ( i ) ( V i -I 2 ) U ⊤ B ( i ) X ∥ 2 F . ( b ) ∥ X + -X ∥ 2 F ≤ ∥ X ∥ 2 F · ∑ n/ 2 i =1 ∥ V i -I 2 ∥ 2 F . ( c ) ∥ X + -X ∥ 2 H ≤ ∑ n/ 2 i =1 ∥ V i -I 2 ∥ 2 Q with Q = ς I 4 . ( d ) For all ˜ G ∈ R n × n , it follows that: 2 ∑ n/ 2 i =1 ⟨ V i -I 2 , [( ∇ f ( X ) -˜ G ) X ⊤ ] B ( i ) B ( i ) ⟩ ≤ ∥ X ∥ 2 F ∑ n/ 2 i =1 ∥ V i -I 2 ∥ 2 F + ∥ [ ∇ f ( X ) -˜ G ] ∥ 2 F .

▶ The Main Algorithm . Using the update rule above, we consider the following iterative procedure: X t +1 ⇐ ˜ X t B ( V : ) , where ¯ V t : ∈ arg min V : f ( ˜ X t B ( V : )) . We establish the majorization function for

f ( ˜ X t B ( V : )) , as follows: 201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

where step ① uses the results of telescoping Inequality (2) over i from 1 to N ; step ② uses X t +1 -X t = [ ∑ n/ 2 i =1 U B ( i ) ( V i -I 2 ) U ⊤ B ( i ) ] X t , Claim ( c ) of Lemma 2.5, θ ≥ 0 , and Q = ς I .

Instead of computing the exact Euclidean gradient ∇ f ( X t ) as GS-JOBCD , VR-J-JOBCD maintains and updates a recursive gradient estimator ˜ G t using a variance-reduced strategy as in Formula (9). We consider minimizing the following function instead of the one on the right-hand side of Inequality (10):

Here, T ( V : ; X t , B t ) can be termed as a stochastic majorization function of f ( ˜ X t B ( V : )) at the current solution X t . Therefore, we can consider the following optimization problem to find { V : } using: ¯ V t : ∈ arg min V : T ( V : ; X t , B t ) , which can be decomposed into ( n/ 2) independent subproblems and solved in parallel. It is important to note that each V i in Problem (12) is identical to Problem (6), which can be efficiently solved in O (1) using the breakpoint search method, as in GS-JOBCD .

We summarize the proposed VR-J-JOBCD in Algorithm 2. Notably, when N = 1 , VR-J-JOBCD simplifies to a direct Jacobi strategy for solving Problem (1), which we refer to as J-JOBCD .

Algorithm 2: VR-J-JOBCD : Block Coordinate Descent Methods using a variance-reduced and Jacobi strategy for Solving Problem 1

Init.: Set X 0 to satisfy J-orthogonality constraints (e.g., via Hyperbolic CS Decomposition). for t from 0 to T do

(S1) Choose a coordinate B t from the set Υ randomly and uniformly (with replacement) for the t -th iteration. Denote B = B t . In our implementation, we simply randomly permute the set { 1 , 2 , ..., n } and then output the grouping { [1 , 2] , [3 , 4] , [5 , 6] , · · · , , [ n -1 , n ] } .

(S2) Use a variance-reduced strategy (9) to obtain

˜

G

t

.

(S3) Solve small-sized subproblems in parallel with Q = ς I ∈ R 4 × 4 .

for i = 1 to n/ 2 in parallel do

where P i ≜ [ ∇ f ( X t )( X t ) T ] B ( i ) B ( i ) -mat( ¨ Q vec( I 2 )) -θ I 2 , ¨ Q = ( ζ + θ ) I . (S4) Update the solution X t +1 in parallel as follows:

end

3 Optimality Analysis

This section provides an optimality analysis for the proposed algorithms.

Initially, we define the first-order optimality condition for Problem (1). Since the matrix X T JX is symmetric, the Lagrangian multiplier Λ corresponding to the constraints X T JX = J is also a symmetric matrix. The Lagrangian function of problem (1) is L ( X , Λ) = f ( X ) -1 2 ⟨ Λ , X T JX -J ⟩ .

We obtain the following lemma for the first-order optimality condition for Problem (1).

Lemma 3.1. (Proof in Section D.1, First-Order Optimality Condition) We let J ≜ { X | X T JX = J } . We have ( a ) A solution ˇ X ∈ J is a critical point of problem (1) if and only if: 0 =

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

∇ J f ( ˇ X ) ≜ ∇ f ( ˇ X ) -J ˇ X [ ∇ f ( ˇ X )] ⊤ ˇ XJ . The associated Lagrangian multiplier can be computed as Λ = J ˇ X T ∇ f ( ˇ X ) . ( b ) The critical point condition is equivalent to the requirement that the matrix X ∇ f ( ˇ X ) T J is symmetric, which is expressed as XG T J = [ XG T J ] T .

Remarks . While our results in Lemma 3.1 show similarities to existing works focusing on problems under orthogonality constraints [45], this study marks the first investigation into the first-order optimality condition for optimization problems under J-orthogonality constraints.

The following definition is useful in our subsequent analysis of the proposed algorithms.

Definition 3.2. (Block Stationary Point, abbreviated as BS-point) Let θ > 0 . A solution ¨ X ∈ J is termed as a block stationary point if, for all B ∈ Ω ≜ {B 1 , B 2 , . . . , B C 2 n } , the following condition is satisfied: I 2 ∈ arg min V ∈J B G ( V ; ¨ X , B ) .

The following theorem shows the relation between critical points and BS-points.

Theorem 3.3. (Proof in Section D.2) Any BS-point is a critical point, while the reverse is not necessarily true.

4 Convergence Analysis

This section provides a convergence analysis for GS-JOBCD and VR-J-JOBCD .

For GS-JOBCD , the randomness of output ( V t , X t +1 ) for all t are influenced by the random variable ξ t ≜ ( B 1 ; B 2 ; · · · ; B t ) . For VR-J-JOBCD , the randomness of output ( ¯ V t : , X t +1 ) are influenced by the random variables ι t ≜ ( B 1 , S 1 + , S 1 ∗ ; B 2 , S 2 + , S 2 ∗ ; · · · ; B t , S t + , S t ∗ ) .

We denote ¯ X as the global optimal solution of Problem (1). To simplify notations, we define: u t = ∥ ˜ G t -∇ f ( X t ) ∥ 2 F , and ∆ i = f ( X i ) -f ( ¯ X ) .

We impose the following additional assumptions on the proposed algorithms.

Assumption 4.1. There exists constants { X , V } that: ∥ X t ∥ F ≤ X, and ∥ V t ∥ F ≤ V for all t .

Assumption 4.2. There exists a constant G that: ∥∇ f ( X t ) ∥ F ≤ G, and ∥ ˜ G t ∥ F ≤ G for all t .

Assumption 4.3. For any X ∈ R n × n , E i [ ∥∇ f i ( X t ) -∇ f ( X t ) ∥ 2 F ] ≤ σ 2 , where i is drawn uniformly at random from [ N ] .

Remarks . ( i ) Assumption 4.1 is satisfied as the function f i ( X ) is coercive for all i . ( ii ) Assumption 4.2 imposes a bound on the (stochastic) gradient, a fairly moderate condition frequently employed in nonconvex optimization [26]. ( iii ) Assumption 4.3 ensures that the variance of the stochastic gradient is bounded, which is a common requirement in stochastic optimization [30, 5].

4.1 Global Convergence

We define the ϵ -BS-point as follows.

Definition 4.4. ( ϵ -BS-point) Given any constant ϵ > 0 , a point ¨ X is called an ϵ -BS-point if: E ( ¨ X ) ≤ ϵ . Here, E ( X ) is defined as E ( X ) ≜ 1 C 2 n ∑ C 2 n i =1 dist( I 2 , arg min V G ( V ; X , B i )) 2 for For GS-JOBCD , while it is defined as E ( X ) ≜ 1 C J ∑ C J i =1 E ι t [dist( I 2 , arg min V : T ( V : ; X , ˜ B i )) 2 ] for VR-J-JOBCD , where the expectation is with respect to the randomness inherent in the algorithm [30].

We have the following useful lemma for VR-J-JOBCD .

Lemma 4.5. (Proof in Section E.1) Suppose Assumption 4.3 holds, then the variance E ι t [ u k ] of the gradient estimators { ˜ G t } of Algorithm 2 is bounded by: E ι t [ u t ] ≤ p ( N -b ) b ( N -1) σ 2 +(1 -p ) E ι t -1 [ u t -1 ]+ L 2 f X 2 (1 -p ) b ′ E ι t -1 [ ∑ n/ 2 i =1 ∥ V t -1 i -I 2 ∥ 2 F ]

The following two theorems establish the iteration complexity (or oracle complexity) for GS-JOBCD and VR-J-JOBCD .

Theorem 4.6. (Proof in Section E.2) GS-JOBCD finds an ϵ -BS-point of Problem (1) within O ( ∆ 0 N ϵ ) arithmetic operations.

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

Theorem 4.7. (Proof in Section E.3) Let b = N , b ′ = √ N , and p = b ′ b + b ′ . VR-J-JOBCD finds an ϵ -BS-point of Problem (1) within O ( nN + ∆ 0 √ N ϵ ) arithmetic operations.

Remark. Theorems 4.6 and 4.7 demonstrate that the arithmetic operation complexity of GS-JOBCD is linearly dependent on N , while VR-J-JOBCD is linearly dependent on √ N . Therefore, VR-JJOBCD reduces the iteration complexity significantly.

4.2 Strong Convergence under KL Assumption

We prove algorithms achieve strong convergence based on a non-convex analysis tool called KurdykaŁojasiewicz inequality[2].

We impose the following assumption on Problem (1).

Assumption 4.8. (Kurdyka-Łojasiewicz Property). Assume that f ◦ ( X ) = f ( X ) + I J ( X ) is a KL function. For all X ∈ dom f ◦ , there exists σ ∈ [0 , 1) , η ∈ (0 , + ∞ ] a neighborhood Υ of X and a concave and continuous function φ ( t ) = ct 1 -σ , c > 0 , t ∈ [0 , η ) such that for all X ′ ∈ Υ and satisfies f ◦ ( X ′ ) ∈ ( f ◦ ( X ) , f ◦ ( X ) + η ) , the following holds: dist( 0 , ∇ f ◦ ( X ′ )) φ ′ ( f ◦ ( X ′ ) -f ◦ ( X )) ≥ 1 .

We establish strong limit-point convergence for VR-J-JOBCD and GS-JOBCD .

Theorem 4.9. (Proof in Section E.5, a Finite Length Property). The sequence { X t } ∞ t =0 of GSJOBCD has finite length property that: ∀ t, ∑ t i =1 E ξ t [ ∥ X t +1 -X t ∥ F ] ≤ O ( φ (∆ 1 )) < + ∞ , where φ ( · ) is the desingularization function defined in Proposition 4.8.

Theorem 4.10. (Proof in Section E.4, a Finite Length Property). Choosing b = N , b ′ = √ N and p = b ′ b + b ′ , then the sequence { X t } ∞ t =0 of VR-J-JOBCD has finite length property that: ∀ t, ∑ t i =1 E ι t [ ∥ X t +1 -X t ∥ F ] ≤ O ( φ (∆ 1 ) N 1 / 4 ) < + ∞ , where φ ( · ) is the desingularization function defined in Assumption 4.8.

5 Applications and Numerical Experiments

This section demonstrates the effectiveness and efficiency of JOBCD on three optimization tasks: ( i ) the hyperbolic eigenvalue problem, ( ii ) structural probe problem, and ( iii ) Ultra-hyperbolic Knowledge Graph Embedding problem. We provide experiments for the last problem in Section F.2.

Application to the Hyperbolic Eigenvalue Problem (HEVP) . The hyperbolic eigenvalue problem refers to the generalized eigenvalue problem in hyperbolic spaces [40]. This problem is a fundamental component in machine learning models, such as Hyperbolic PCA [43, 6]. Given a data matrix D ∈ R m × n and a signature matrix J with signature ( p, n -p ) , HEVP can be formulated as the following optimization problem: min X -tr ( X T D T DX ) , s . t . X T JX = J .

Application to the Hyperbolic Structural Probe Problem (HSPP) . The Structure Probe (SP) is a metric learning model aimed at understanding the intrinsic semantic information of large language models [20] [7]. Given a data matrix D ∈ R m × n and its associated Euclidean distance metric matrix T ∈ R m × m , HSPP employs a smooth homeomorphic mapping function φ ( · ) to project the data D into ultra-hyperbolic space. Subsequently, it seeks an appropriate linear transformation X ∈ R n × n constrained within a specific structure X ∈ J , such that the resulting transformed data Q ≜ φ ( D ) X ∈ R m × n exhibits similarity to the original distance metric matrix T under the ultra-hyperbolic geodesic distance d α ( Q i : , Q j : ) , expressed as T i,j ≈ d α ( Q i : , Q j : ) for all i, j ∈ [ m ] , where Q i : is i -th row of the matrix Q ∈ R m × n . This can be formulated as the following optimization problem: min X 1 m 2 ∑ i,j ∈ m ( T i,j -d α ( Q i : , Q j : )) 2 , s . t . Q ≜ φ ( D ) X , X ∈ J . For more details on the functions φ ( · ) and d α ( · , · ) , please refer to Appendix Section F.1.

Datasets . To generate the matrix D ∈ R m × n , we use 8 real-world or synthetic data sets for both HEVP and HSPP tasks: 'Cifar', 'CnnCaltech', 'Gisette', 'Mnist', 'randn', 'Sector', 'TDT2', 'w1a'. We randomly extract a subset from the original data sets for the experiments.

Compared Methods . We compare GS-JOBCD and VR-J-JOBCD with 3 state-of-the-art optimization algorithms under J-orthogonality constraints. ( i ) The CS Decomposition Method ( CSDM ) [48]. ( ii ) Stardard ADMM ( ADMM ) [19]. UMCM : Unconstrained Multiplier Correction Method [31, 13].

314

315

316

317

318

319

320

321

322

323

324

325

dataname(m-n-p)

UMCM

ADMM

CSDM

GS-JOBCD

J-JOBCD

CSDM+GS-JOBCD

cifar(1000-100-50)

-1.05e+04(3.0e-09)

-1.05e+04(3.0e-09)

-5.28e+04(5.4e-09)

-1.03e+05(2.6e-08)

-1.11e+05(1.4e-07)

-1.24e+05(2.6e-08)(+)

CnnCal(2000-1000-500)

-5.89e+02(2.9e-08)

-5.89e+02(3.1e-10)

-1.11e+03(5.2e-10)

-1.07e+03(1.3e-09)

-9.16e+03(6.9e-08)

-1.15e+03(6.9e-10)(+)

gisette(3000-1000-500)

-3.22e+06(3.1e-10)

-3.22e+06(3.1e-10)

-8.53e+06(4.9e-10)

-9.49e+06(1.2e-09)

-1.36e+07(2.6e-08)

-9.65e+06(7.9e-10)(+)

mnist(1000-780-390)

-8.65e+04(4.1e-10)

-8.65e+04(4.1e-10)

-2.56e+05(5.6e-10)

-3.14e+05(1.2e-09)

-1.20e+06(4.1e-08)

-3.06e+05(7.6e-10)(+)

randn(10-10-5)

1.29e+02(9.7e-02)

1.29e+02(9.7e-02)

2.45e+02(2.3e-01)

-3.96e+01(9.7e-02)

-3.97e+02(9.7e-02)

1.55e+01(2.3e-01)(+)

randn(100-100-50)

-1.03e+04(3.0e-09)

-1.03e+04(2.5e-07)

-1.98e+04(4.4e-09)

-2.28e+04(5.6e-08)

-4.37e+04(2.6e-07)

-2.41e+04(4.2e-08)(+)

randn(1000-1000-500)

-1.16e+06(3.1e-10)

-1.16e+06(3.1e-10)

-1.93e+06(5.0e-10)

-1.22e+06(6.9e-10)

-1.04e+07(2.3e-07)

-1.95e+06(6.7e-10)(+)

sector(500-1000-500)

-3.61e+03(3.1e-10)

-3.61e+03(3.1e-10)

-7.90e+03(4.9e-10)

-9.24e+03(1.3e-09)

-1.06e+04(2.0e-08)

-8.51e+03(6.4e-10)(+)

TDT2(1000-1000-500)

-4.25e+06(3.1e-10)

-4.25e+06(3.1e-10)

-9.39e+06(4.8e-10)

-1.05e+07(1.1e-09)

-1.42e+07(2.1e-08)

-1.04e+07(6.5e-10)(+)

w1a(2470-290-145)

-3.02e+04(1.1e-04)

-3.02e+04(1.1e-04)

-5.72e+04(2.7e-05)

-9.21e+04(1.1e-04)

-9.32e+06(1.1e-04)

-7.94e+04(2.7e-05)(+)

(a) Gisette (3000-100-50)

(b) Sector (500-1000-500)

(c) wla (2470-290-145)

(a) Cifar (10000-50-45)

(b) Gisette (6000-50-45)

(c) News20 (7967-50-45)

Experiment Settings . All methods are implemented using Pytorch on an Intel 2.6 GHz processor with an A40 (48GB). For HSPP , we fix α to 1. Each method employs the same random J-orthogonal matrix. The built-in solver Admm is used to solve the unconstrained minimization problem in CSDM . We provide our code in the supplemental material.

Experiment Results. Table 1 and Figure 1 display the accuracy and computational efficiency for HEVP, while Figure 2 presents the results for HSPP, leading to the following observations: ( i ) GS-JOBCD and JJOBCD consistently deliver better performance than the other methods. ( ii ) Other methods frequently encounter poor local minima, whereas GS-JOBCD effectively escapes these minima and typically achieves lower objective values, aligning with our theory that our methods locate stronger stationary points. ( iii ) VR-J-JOBCD outperforms both J-JOBCD and CSDM when dealing with a large dataset characterized by an infinite-sum structure.

6 Conclusions

In this paper, we propose a new approach JOBCD, which is based on block coordinate descent, for 326 solving the optimization problem under J-orthogonality constraints. We discuss two specific variants 327 of JOBCD: one based on a Gauss-Seidel strategy (GS-JOBCD), the other on a variance-reduced 328 Jacobi strategy. Both algorithms capitalize on specific structural characteristics of the constraints to 329 converge to more favorable stationary solutions. Notably, VR-J-JOBCD incorporates a variance330 reduction technique into a parallel framework to reduce oracle complexity in the minimization of 331 finite-sum functions. For both GS-JOBCD and VR-J-JOBCD , we establish the oracle complexity 332 under mild conditions and strong limit-point convergence results under the Kurdyka-Lojasiewicz 333 inequality. Some experiments on the hyperbolic eigenvalue problem and structural probe problem 334 show the efficiency and efficacy of the proposed methods. 335

336

337

338

339

340

341

References

P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds . Princeton University Press, 2008.

Hédy Attouch, Jérôme Bolte, Patrick Redont, and Antoine Soubeyran. Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdyka-łojasiewicz inequality. Mathematics of operations research , 35(2):438-457, 2010.

Adam Bojanczyk, Nicholas J Higham, and Harikrishna Patel. Solving the indefinite least squares 342 problem by hyperbolic qr factorization. SIAM Journal on Matrix Analysis and Applications , 343 24(4):914-931, 2003. 344

345

346

347

HanQin Cai, Yuchen Lou, Daniel McKenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In International Conference on Machine Learning , pages 1193-1203. PMLR, 2021.

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

Xufeng Cai, Chaobing Song, Stephen Wright, and Jelena Diakonikolas. Cyclic block coordinate descent with variance reduction for composite nonconvex optimization. In International Conference on Machine Learning , pages 3469-3494. PMLR, 2023.

Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Re. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference on Machine Learning (ICML) , volume 139, pages 1419-1429, 2021.

Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Probing bert in hyperbolic spaces. ICLR , 2021.

Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. arXiv preprint arXiv:2105.14686 , 2021.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems , 32, 2019.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal nonconvex optimization via stochastic path-integrated differential estimator. Advances in neural information processing systems , 31, 2018.

Hamza Fawzi and Harry Goulbourne. Faster proximal algorithms for matrix optimization using jacobi-based eigenvalue methods. Advances in Neural Information Processing Systems , 34:11397-11408, 2021.

Bin Gao, Xin Liu, Xiaojun Chen, and Ya-xiang Yuan. A new first-order algorithmic framework for optimization problems with orthogonality constraints. SIAM Journal on Optimization , 28(1):302-332, 2018.

Bin Gao, Xin Liu, and Ya-xiang Yuan. Parallelizable algorithms for optimization problems with orthogonality constraints. SIAM Journal on Scientific Computing , 41(3):A1949-A1983, 2019.

Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming , 155(12):267-305, 2016.

Gene H Golub and Charles F Van Loan. Matrix computations . JHU press, 2013. 377

Eldon R Hansen. On cyclic jacobi methods. Journal of the Society for Industrial and Applied 378 Mathematics , 11(2):448-459, 1963. 379

Vjeran Hari and Erna Begovi´ c Kovaˇ c. On the convergence of complex jacobi methods. Linear 380 and multilinear algebra , 69(3):489-514, 2021. 381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

Bingsheng He and Xiaoming Yuan. On the O (1 /n ) convergence rate of the douglas-rachford alternating direction method. SIAM Journal on Numerical Analysis , 50(2):700-709, 2012.

John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 4129-4138, 2019.

Nicholas J Higham. J-orthogonal matrices: Properties and generation. SIAM review , 45(3):504519, 2003.

Minhui Huang, Shiqian Ma, and Lifeng Lai. A riemannian block coordinate descent method for computing the projection robust wasserstein distance. In International Conference on Machine Learning , pages 4446-4455. PMLR, 2021.

Bo Hui and Wei-Shinn Ku. Low-rank nonnegative tensor decomposition in hyperbolic space. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 646-654, 2022.

Sashank J Reddi, Suvrit Sra, Barnabas Poczos, and Alexander J Smola. Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization. Advances in neural information processing systems , 29, 2016.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations (ICLR) , 2015.

Marc Law. Ultrahyperbolic neural networks. Advances in Neural Information Processing Systems , 34:22058-22069, 2021.

Marc Law and Jos Stam. Ultrahyperbolic representation learning. Advances in neural information processing systems , 33:1668-1678, 2020.

Qunwei Li, Yi Zhou, Yingbin Liang, and Pramod K Varshney. Convergence analysis of proximal gradient with momentum for nonconvex optimization. In International Conference on Machine Learning , pages 2111-2119. PMLR, 2017.

Zhize Li, Hongyan Bao, Xiangliang Zhang, and Peter Richtárik. Page: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In International conference on machine learning , pages 6286-6295. PMLR, 2021.

Wei Liu, Yinyu Zhang, Hongqiao Yang, and Shuzhong Zhang. A class of smooth exact penalty function methods for optimization problems with orthogonality constraints. Optimization , 69(3):399-426, 2020.

Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 5085-5092, 2020.

Julien Mairal. Optimization with first-order surrogate functions. In International Conference on Machine Learning (ICML) , volume 28, pages 783-791, 2013.

Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáˇ c. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International conference on machine learning , pages 2613-2621. PMLR, 2017.

Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning , pages 3779-3788. PMLR, 2018.

Vedran Novakovi´ c and Sanja Singer. A kogbetliantz-type algorithm for the hyperbolic svd. 429 Numerical algorithms , 90(2):523-561, 2022. 430

431

432

433

Julie Nutini, Issam Laradji, and Mark Schmidt. Let's make block coordinate descent converge faster: faster greedy rules, message-passing, active-set complexity, and superlinear convergence. Journal of Machine Learning Research , 23(131):1-74, 2022.

434

435

436

437

438

Meisam Razaviyayn, Mingyi Hong, and Zhi-Quan Luo. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM Journal on Optimization , 23(2):1126-1153, 2013.

Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017.

Ivan Slapnicar and Ninoslav Truhar. Relative perturbation theory for hyperbolic eigenvalue 439 problem. Linear Algebra and its Applications , 309(1):57-72, 2000. 440

Michael Stewart and Paul Van Dooren. On the factorization of hyperbolic and unitary trans441 formations into rotations. SIAM Journal on Matrix Analysis and Applications , 27(3):876-890, 442 2005. 443

Puoya Tabaghi and Ivan Dokmani´ c. Hyperbolic distance matrices. In Proceedings of the 444 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 445 1728-1738, 2020. 446

Puoya Tabaghi, Michael Khanzadeh, Yusu Wang, and Sivash Mirarab. Principal component 447 analysis in space forms. ArXiv , abs/2301.02750, 2023. 448

Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: 449 Faster variance reduction algorithms. Advances in Neural Information Processing Systems , 32, 450 2019. 451

Zaiwen Wen and Wotao Yin. A feasible method for optimization with orthogonality constraints. 452 Mathematical Programming , 142:397 - 434, 2012. 453

WikiContributors. Quartic equation. https://en.wikipedia.org/wiki/Quartic_ 454 equation . 455

456

457

458

459

Ruiyuan Wu, Anna Scaglione, Hoi-To Wai, Nurullah Karakoc, Kari Hreinsson, and Wing-Kin Ma. Federated block coordinate descent scheme for learning global and personalized models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10355-10362, 2021.

460

461

462

Bo Xiong, Shichao Zhu, Mojtaba Nayyeri, Chengjin Xu, Shirui Pan, Chuan Zhou, and Steffen Staab. Ultrahyperbolic knowledge graph embeddings. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2130-2139, 2022.

Bo Xiong, Shichao Zhu, Nico Potyka, Shirui Pan, Chuan Zhou, and Steffen Staab. Semi463 riemannian graph convolutional networks. ArXiv , abs/2106.03134, 2021. 464

465

466

Tao Yu and Christopher M De Sa. Numerically accurate hyperbolic embeddings using tilingbased models. Advances in Neural Information Processing Systems , 32, 2019.

Ganzhao Yuan. A block coordinate descent method for nonsmooth composite optimization 467 under orthogonality constraints. ArXiv , abs/2304.03641, 2023. 468

469

470

Ganzhao Yuan. Coordinate descent methods for fractional minimization. In International Conference on Machine Learning , pages 40488-40518, 2023.

471

472

473

Jinshan Zeng, Tim Tsz-Kit Lau, Shaobo Lin, and Yuan Yao. Global convergence of block coordinate descent in deep learning. In International conference on machine learning , pages 7313-7323. PMLR, 2019.

Yiding Zhang, Xiao Wang, Chuan Shi, Nian Liu, and Guojie Song. Lorentzian graph convolu474 tional networks. In Proceedings of the Web Conference 2021 , pages 1249-1261, 2021. 475

Dongruo Zhou, Pan Xu, and Quanquan Gu. Stochastic nested variance reduction for nonconvex 476 optimization. The Journal of Machine Learning Research , 21(1):4130-4192, 2020. 477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

Appendix

The appendix is organized as follows.

Appendix A introduces some notations, technical preliminaries, and relevant lemmas.

Appendix B concludes some additional discussions.

Appendix C presents the proofs for Section 2.

Appendix D offers the proofs for Section 3.

Appendix E contains the proofs for Section 4.

Appendix F contains several extra experiments, extensions and discussions of the proposed methods.

A Notations, Technical Preliminaries, and Relevant Lemmas

A.1 Notations

In this paper, we denote the Lowercase boldface letters represent vectors, while uppercase letters represent real-valued matrices. We use the Matlab colon notation to denote indices that describe submatrices. The following notations are used throughout this paper.

N : Set of natural numbers

R : Set of real numbers

[ n ] : { 1 , 2 , ..., n }

∥ x ∥ : Euclidean norm: ∥ x ∥ = ∥ x ∥ 2 = √ ⟨ x , x ⟩

x i : the i -th element of vector x

X i,j or X ij : the ( i th , j th ) element of matrix X

vec( X ) : vec( X ) ∈ R nn × 1 , the vector formed by stacking the column vectors of X

mat( x ) ∈ R n × n , Convert x ∈ R nn × 1 into a matrix with mat(vec( X )) = X

X T : the transpose of the matrix X

sign( t ) : the signum function, sign( t ) = 1 if t ≥ 0 and sign( t ) = -1 otherwise

X ⊗ Y : Kronecker product of X and Y

det ( D ) : Determinant of a square matrix D ∈ R n × n D ∈ R n × n

C 2 n : the number of possible combinations choosing k items from n without repetition.

0 n,r : A zero matrix of size n × r ; the subscript is omitted sometimes

I r : I r ∈ R r × r , Identity matrix

X ⪰ 0 ( or ≻ 0 ) : the Matrix X is symmetric positive semidefinite (or definite)

Diag( x ) : Diagonal matrix with x as the main diagonal entries.

tr ( A ) : Sum of the elements on the main diagonal A : tr ( A ) = ∑ i A i,i

∥ X ∥ ∗ : Nuclear norm: sum of the singular values of matrix X

∥ X ∥ : Operator/Spectral norm: the largest singular value of X

∥ X ∥ F : Frobenius norm: ( ∑ ij X 2 ij ) 1 / 2

∇ f ( X ) : classical (limiting) Euclidean gradient of f ( X ) at X

∇ J f ( X ) : Riemannian gradient of f ( X ) at X

I ξ ( X ) : the indicator function of a set ξ with I ξ ( X ) = 0 if X ∈ ξ and otherwise + ∞

dist ( ξ, ξ ′ ) : the distance between two sets with dist ( ξ, ξ ′ ) ≜ inf X ∈ ξ, X ′ ∈ ξ ′ ∥ X -X ′ ∥ F

I ξ ( x ) : the indicator function of a set ξ with I ξ ( x ) = 0 if x ∈ ξ and otherwise + ∞ .

A.2 Relevant Lemmas 517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

Lemma A.1. (Lemma 6.6 of [51]) For any W ∈ R n × n , we have: ∑ C k n i =1 ∥ W ( B i , B i ) ∥ 2 F = k n C k n ∑ i W 2 ii + C k -2 n -2 ∑ i ∑ j,j = i W 2 ij . Here, the set {B 1 , B 2 , · · · , B C k n } represents all possible combinations of the index vectors choosing k items from n without repetition.

̸

Lemma A.2. We have S + be the set of | S + | = b samples from [ N ] , drawn with replacement and uniformly at random. Then, ∀ t, X t ∈ R n × n , we have:

Proof. The proof is exactly the same as in Lemma 2.8 of [5].

Lemma A.3. The tangent space T X J of manifold constructed by X ⊤ JX = J , with X ∈ R n × n , is :

where Y = t ˜ Y with t is a positive scalar approaching 0.

Proof. Assuming point X ∈ R n × n lies on manifold J , we have: h ( X ) = X ⊤ JX -J . Moving along Y ∈ R n × n in the tangent space of X , we obtain:

where step ① uses X ⊤ JX = J ; step ② uses Y = t ˜ Y .

Since t is a positive scalar approaching 0, we can ignore the higher-order term: t 2 ˜ Y ⊤ J ˜ Y . According to the properties of the tangent space of any manifold, we have: h ( X + Y ) = 0 , In other words, X ⊤ JY + Y ⊤ JX = 0 , i.e. we obtain the defining equation for the tangent space: T X J ≜ { Y ∈ R n × n | X ⊤ JY + Y ⊤ JX = 0 } .

B Additional Discussions

B.1 On the Global Optimal Solution for Problem (7)

In Section 2.1, we have demonstrated how to use the breakpoint search method to obtain an optimal solution for the case of V = ( ˜ c ˜ s ˜ s ˜ c ) of Problem (7). Since the structure of the other three cases V ∈ { ( ˜ c -˜ s -˜ s ˜ c ) , ( -˜ c -˜ s ˜ s ˜ c ) , ( ˜ c -˜ s ˜ s -˜ c ) } is exactly the same except for the coefficients of Problem (8), we will provide the corresponding coefficients in Problem (8): min ˜ c , ˜ s a ˜ c+ b ˜ s+ c ˜ c 2 + d ˜ c ˜ s + e ˜ s 2 , and omit the specific analysis process.

C Proofs for Section 2

C.1 Proof of Lemma 2.1

Proof. Defining J BB = J ( U B , U B ) , then we have: JU B = U B J BB , U ⊤ B J = J BB U ⊤ B , and U ⊤ B JU B = J BB .

Part (a) . For any V ∈ R 2 × 2 and B ∈ {B i } C 2 n i =1 , we have: 548

Part (b) . Using the update rule for X + = X + U B ( V -I 2 ) U ⊤ B X ∈ R n × n , we derive: 549

where step ① and step ② use the norm inequality that ∥ AX ∥ F ≤ ∥ A ∥ F · ∥ X ∥ F for any A and X ; 550 step ③ uses ∥ U B ∥ = ∥ U ⊤ B ∥ = 1 . 551

Part (c) . We define Z ≜ U ⊤ B X . We derive: 552

553

554

555

556

557

558

559

560

561

where step ① uses ∥ X ∥ 2 H = vec ( X ) ⊤ H vec ( X ) ; step ② uses ( Z ⊤ ⊗ R ) vec ( U ) = vec ( RUZ ) for all R , Z and U of suitable dimensions; step ③ uses the choice of Q ≽ Q ≜ ( Z ⊤ ⊗ U B ) ⊤ H ( Z ⊤ ⊗ U B ) .

C.2 Proof of Lemma 2.3

Proof. We denote w = c + e . According to the properties of trigonometric functions, we have: ( i ) ˜ c 2 = 1 1 -˜ t 2 ; ( ii ) ˜ s 2 = ˜ t 2 1 -˜ t 2 ; ( iii ) ˜ t = ˜ s ˜ c , leading to: ˜ c = ± 1 √ 1 -˜ t 2 , ˜ s = ± ˜ t √ 1 -˜ t 2 with | ˜ t | < 1 .

We discuss two cases for Problem (8).

Case ( a ) . ˜ c = 1 √ 1 -˜ t 2 , ˜ s = ˜ t √ 1 -˜ t 2 . Problem (8) is equivalent to the following problem: ¯ µ + = arg min µ a +˜ t b √ 1 -˜ t 2 + w +˜ t d 1 -˜ t 2 -e . Therefore, the optimal solution ¯ µ + can be computed as:

Case ( b ) . ˜ c = -1 √ 1 -˜ t 2 , ˜ s = -˜ t √ 1 -˜ t 2 . Problem (8) is equivalent to the following problem: ¯ µ -= 562 arg min µ -a -˜ t b √ 1 -˜ t 2 + w +˜ t d 1 -˜ t 2 -e . Therefore, the optimal solution ¯ µ -can be computed as: 563

We define the objective function as: ˘ F (˜ c, ˜ s ) ≜ a ˜ c + b ˜ s + c ˜ c 2 + d ˜ c ˜ s + e ˜ s 2 . In view of (14) and (15), 564 the optimal solution pair [ cosh (¯ µ, sinh (¯ µ )] for problem (8) can be computed as: 565

Importantly, it is not necessary to compute the values ¯ µ + for (14) and ¯ µ -for (15). 566 567

C.3 Proof of Lemma 2.4 568

Proof. The objective function for B t ( i ) as in Equation (3) is formulated as :

Part (1). For the part of 1 2 ∥ V i -I ∥ 2 Q + θ I , it is obviously irrelevant. 569

570

571

572

Part (2). For the part of ⟨ V i -I , [ ∇ f ( X t )( X t ) T ] B t ( i ) B t ( i ) ⟩ , we note that [ ∇ f ( X t )( X t ) ⊤ ] B t ( i ) B t ( i ) = [ ∇ f ( X t )]( B t ( i ) , :)[( X t ) ⊤ ](: , B t ( i ) ) = [ ∇ f ( X t )]( B t ( i ) , :)[( X t )( B t ( i ) , :)] ⊤ , which just use the information of block B t ( i ) . The proof ends.

573

C.4 Proof of Lemma 2.5

Proof. Part ( a ). For the purpose of analysis, we define the following: ∀ i ∈ [ n 2 ] , K i = U B ( i ) ( V i -574 I 2 ) U ⊤ B ( i ) X . 575

where step ① uses the definition of K i and the assumption that B ∈ Υ ; step ② uses the definition of 576 Squared Frobenius Norm; step ③ uses the definition of K i . 577

578

579

Part ( b ). Using the update rule for X + = X +[ ∑ n/ 2 i =1 U B ( i ) ( V i -I 2 ) U ⊤ B ( i ) ] X ∈ R n × n , we have the following inequalities:

where step ① uses the conclusion of Part ( a ); step ② uses the same proof process of Part ( b ) of lemma 580 2.1. 581

Part ( c ). We derive the following results:

where step ① uses the conclusion of Part ( a ); step ② uses the same proof process of Part ( c ) of lemma 582 2.1. 583

Part ( d ). We derive the following results: 584

where step ① uses ∀ A , B , 1 2 ∥ A -B ∥ 2 F = 1 2 ∥ A ∥ 2 F + 1 2 ∥ B ∥ 2 F -⟨ A , B ⟩ ≥ 0 , with A = ∥ X + -X ∥ 2 F and B = ∥ [ ∇ f ( X ) -˜ G ] ∥ 2 F ; step ② uses the conclusion of Part ( b ).

D Proofs for Section 3

D.1 Proof of Lemma 3.1

Proof. We consider the Lagrangian function of problem (1):

Setting the gradient of L ( X , Λ) w.r.t. X to zero yields:

Part (a) . Multiplying both sides by X T and using the fact that X T JX = J , we have J Λ = X T ∇ f ( X ) . Multiplying both sides by J T and using J ⊤ J = I , we have Λ = JX ⊤ ∇ f ( X ) . Since Λ is symmetric, we have Λ = ∇ f ( X ) T XJ . Putting this equality into Equality (21) yields the following first-order optimality condition for Problem (1):

Part (b) . We let G = ∇ f ( X ) . We derive the following results:

where step ① uses the results of left-multiplying both sides by JX T ; step ② uses J · X T JX = JJ = I ; step ③ uses the results of left-multiplying both sides by X and subsequently right-multiplying them by X T ; ④ uses G = JXG T XJ ; step ⑤ uses the the results of right-multiplying both sides by JX ; step ⑥ uses JJ = I and X T JX = J ; step ⑦ uses the results of left-multiply both sides by J and right-multiplied by J .

Given Equality (23), we conclude that the critical point condition is equivalent to the requirement that the matrix X ∇ f ( ˇ X ) T J is symmetric, which is expressed as XG T J = [ XG T J ] T .

D.2 Proof of Theorem 3.3

Proof. We use ¨ X and ˇ X to denote any BS-point and critical point, respectively.

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

For all B ∈ Ω ≜ {B 1 , B 2 , . . . , B C 2 n } , we have: 605

The Euclidean gradient of G ( V ; ¨ X , B ) can be computed as: 607

Given Lemma 3.1, we set the Riemannian gradient of G ( V ; ¨ X , B ) w.r.t. V to zero, leading to the 608 following first-order optimality condition: 609

Letting V = I 2 , and using the definition of ¨ G , we have: 610

where step ① uses U ⊤ B J = J BB U ⊤ B and JU B = U B J BB ; step ② uses the the following results for any 611 W ∈ R n × n : 612

step ③ uses the fact that both sides are left-multiplied by J . We conclude that the matrix J ∇ f ( ¨ X ) ¨ X T 613 is symmetric. Using Claim ( b ) of Lemma 3.1, we conclude that ¨ X is a also a critical point. 614

615

616

617

Notably, the condition in Equation (25) is a necessary but not sufficient condition. This is because BS-point is the global minimum of Problem: arg min V ∈J B G ( V ; ¨ X , B ) , according to Definition 3.2.

618

619

E Proofs for Section 4

E.1 Proof of Lemma 4.5

Proof. By the definition of ˜ G t , we have 620

where step ① uses formula (9); step ② uses that ˜ G t -1 -∇ f ( X t -1 ) is measurable w.r.t. ι t -1 and 621 E ι t [ ∥ 1 b ′ ∑ b ′ i =1 ( ∇ f i ( X t ) -∇ f i ( X t -1 )) -∇ f ( X t ) + ∇ f ( X t -1 ) ∥ 2 F ] = 0 . We further have 622

where step ① uses that for any random variable X , E [( X -E [ X ]) 2 ] ≤ E [ X 2 ] ; step ② uses lemma 623 A.2; step ③ uses assumption 4.3, Inequality (2) and Part ( b ) of lemma 2.5. 624

625

626

E.2 Proof of theorem 4.6

Proof. For simplicity, we use B instead of B t . We will show that the following inequality holds :

Since ¯ V t is the global optimal solution of Problem (5), we have:

Letting V = I 2 , we have: G ( ¯ V t ; X t , B ) ≤ G ( I 2 ; X t , B ) . We further obtain: 627

Using Inequality (2) with N = 1 and Part ( c ) of Lemma 2.1, we have: 628

Adding Inequality (29) and (30) together, we obtain the inequality in (28). Using the result of Part ( b ) 629 in Lemma 2.1 that ∥ X + -X ∥ 2 F ∥ X ∥ 2 F ≤ ∥ V -I 2 ∥ 2 F , we have the following sufficient decrease condition: 630

We now prove the global convergence. Taking the expectation for Inequality (31), we obtain a lower bound on the expected progress made by each iteration for Algorithm 1:

Summing up the inequality above over t = 0 , 1 , . . . , T , we have:

As a result, there exists an index ¯ t with 0 ≤ ¯ t ≤ T such that 631

Furthermore, for any t , we have: 632

Combining Inequality (32) and equality (33), we have the following result: 633

We will give the arithmetic operations of GS-JOBCD . By the chosen parameters and Inequality (34), we have

We define ∆ 0 = f ( X 0 ) -f ( ¯ X ) and set T +1 = 2∆ 0 ϵθ . Denoting m t to be the number of arithmetic operations at t -th iteration, we have for t ≥ 1 :

Then we have for t ≥ 1 , the total number of arithmetic operations M T in T iterations to obtain ϵ -BS-point is

We have ( T +1) N = N 2∆ 0 ϵθ = O ( ∆ 0 N ϵ ) 634

E.3 Proof of Theorem 4.7 635

Proof. For simplicity, we use B instead of B t . Defining ¯ V : t as the global optimal solution of arg min V : T ( V : ; X t , B ) , we have:

Letting V i = I 2 , ∀ i , we have: T ( ¯ V : t ; X t , B ) ≤ T ( I 2 ; X t , B ) . We further obtain: 636

Using the results of telescoping Inequality (2) over i from 1 to N with Part ( c ) of Lemma 2.5, we 637 have: 638

Adding inequality (35), and (36) together, we obtain the inequality in (37). 639

where step ① uses Part ( d ) of Lemma 2.5. 640

Taking expectation on both sides of inequality (37) with respect to all randomness of the algorithm, 641 and adding the inequality in Lemma 4.5 × 1 2 p to (37), we have: 642

Summing up the inequality above over t = 0 , 1 , . . . , T , we have: 643

As a result, there exists an index ¯ t with 0 ≤ ¯ t ≤ T such that 644

Defining ϖ = θ -X 2 2 -L 2 f X 2 (1 -p ) 2 pb ′ , furthermore, for any t and ∀ i , we have: 645

Combining inequality (40) and (41) , we have the following result: 646

By the chosen parameters and Inequality (42), we have

We define ∆ 0 = f ( X 0 ) -f ( ¯ X ) and set T +1 = ∆ 0 ϵϖ . Denoting m i t to be the number of arithmetic operations to update the i -th block at t -th iteration, we have for t ≥ 1

Letting m t be the number of arithmetic operations in the t -the iteration, we have for t ≥ 1

Hence, the total number of arithmetic operations M T in T iterations to obtain ϵ -BS-point is

Since b = N,b ′ = √ b and p = b ′ b + b ′ , ϖ = θ -X 2 2 -L 2 f X 2 (1 -p ) 2 pb ′ = 1 2 ( θ -X 2 -L 2 f X 2 ) , we have

E.4 Proof of Theorem 4.10

Proof. For simplicity, we use B instead of B t . We notice that the Riemannian gradient of T ( V : ; X t , B ) at the point V i = I 2 , ∀ i . Defining G = ˜ G t [ X t ] ⊤ and using JU B = U B J BB , U ⊤ B J = J BB U ⊤ B ,we have:

Then, we prove the following important lemmas.

Lemma E.1. We have the following result for VR-J-JOBCD : E ι t +1 [ ∥ ˜ G t -˜ G t +1 ∥ F ] ≤ p E ι t [ √ u t ] + L f E ι t +1 [ ∥ X t -X t +1 ∥ F ]

Proof. By the definition of ˜ G t , with the choice of b = N , b ′ = √ b and p = b ′ b + b ′ , we have

647

648

649

650

651

652

653

654

655

where step ① uses formula (9); step ② uses norm inequality and 1 b ∑ b i =1 ∇ f i ( X t +1 ) = ∇ f ( X t +1 ) 656 with b = N and norm inequality; step ③ uses triangle inequality that ∥ A -B ∥ F ≤ ∥ A -C ∥ F + 657 ∥ C -B ∥ F , for any A , B and C ; step ④ the definition of u t ; step ⑤ uses Inequality (2) and the results 658 of telescoping it over i from 1 to N . 659

Lemma E.2. (Riemannian gradient Lower Bound for the Iterates Gap) We de660 fine ϕ ≜ (3 X + VX ) G +(1 + V 2 + n 2 ( X 2 + V 2 X 2 )) L f +(1 + V 2 ) θ . It holds that: 661 E ι t +1 [dist( 0 , ∇ J T ( I 2 ; X t +1 , B t +1 ))] ≤ ϕ · E ι t [ ∑ n/ 2 i =1 ∥ ¯ V t i -I 2 ∥ F ] + np √ u t 2 ( X + V 2 X ) . 662

Proof. For notation simplicity, we define: 663

First, using the optimality of ¯ V t i , i ∈ { 1 , · · · , n 2 } for the subproblem, we have: 664

Using the relation that ˜ G i = Υ i 1 +Υ i 2 , we obtain the following results from the above equality: 665

where step ① uses Υ i 2 = Ω i 1 +Ω i 2 . Then we derive the following results: 666

where step ① uses Equality (43) ; step ② uses the fact that both the working set B t and B t +1 are selected 667 randomly and uniformly; step ③ uses the definition of Ω i 0 in (44); step ④ uses -Ω i 1 +Ω i 1 = 0 668 and -Ω ⊤ i 1 +Ω ⊤ i 1 = 0 ; step ⑤ uses the norm inequality; step ⑥ uses the norm inequality; step ⑦ uses 669 the norm inequality; step ⑧ uses Equality (49); step ⑨ uses the norm inequality. We now establish 670 individual bounds for each term for Inequality (50). 671

For the first term 2 E ι t [ ∥ ∑ n/ 2 i =1 Ω i 0 -Ω i 1 ∥ F ] in (50): 672

where step ① uses [ X t -X t ] B i B i = U B ( i ) ( ¯ V t i -I 2 ) U ⊤ B ( i ) X t ; step ② uses the inequality ∥ XY ∥ F ≤ 673 ∥ X ∥ F ∥ Y ∥ F for all X and Y repeatedly and the fact that ∀ t, ∥ ˜ G t ∥ F ≤ G and ∀ t, ∥ X t ∥ F ≤ X; step ③ 674 uses the norm inequality. 675

For the second term E ι t [ ∥ ∑ n/ 2 i =1 J B ( i ) ¯ V t i Υ ⊤ i 1 ¯ V t i J B ( i ) -Υ i 1 ∥ F ] in (50): 676

where step ① uses the triangle inequality; step ② uses the inequality ∥ XY ∥ F ≤ ∥ X ∥ F ∥ Y ∥ F for all 677 X and Y and ∀ t, ∥ V t ∥ F ≤ V; step ③ uses the definition of Υ i 1 ; step ④ uses the choice of Q ⪯ L f I 678 and the norm inequality. 679

For the third term E ι t [ ∥ ∑ n/ 2 i =1 ¯ V t i Ω ⊤ i 1 ¯ V t i -Ω ⊤ i 1 ∥ F ] in (50), we have: 680

where step ① uses the fact that -¯ V t i Ω ⊤ i 1 I 2 + ¯ V t i Ω ⊤ i 1 = 0 ; step ② uses the norm inequality and 681 ∀ t, ∥ V t ∥ F ≤ V; step ③ uses the fact that ∥ Ω i 1 ∥ F = ∥ U ⊤ B ( i ) ˜ G t [ X t ] ⊤ U B ( i ) ∥ F ≤ X ∥ ˜ G t ∥ F , ∀ i which 682 can be derived using the norm inequality ; step ④ uses the fact that ∀ X , ∥ ˜ G t ∥ F ≤ G. 683

For the fourth term E ι t [ ∥ ∑ n/ 2 i =1 J B ( i ) ¯ V t i Ω ⊤ i 2 ¯ V t i J B ( i ) -Ω i 2 ∥ F ] in (50), we have: 684

where step ① uses the triangle inequality; step ② uses the norm inequality and ∀ t, ∥ V t ∥ F ≤ V; step ③ 685 uses the definition of ∀ i, Ω i 2 = U ⊤ B ( i ) [ ˜ G t -˜ G t ][ X t ] ⊤ U B ( i ) in (46); step ④ uses the norm inequality 686 and ∀ t, ∥ X t ∥ F ≤ X; step ⑤ uses Lemma E.1; step ⑥ uses Part ( b ) in Lemma 2.5 and ∀ t, ∥ X t ∥ F ≤ X. 687 In view of( 51), (52), (53), (54), and (50), we have: 688

689

690

691

692

where c 1 = 2 XG , c 2 = (1 + V 2 )( L f + θ ) , c 3 = ( X + VX ) G, and c 4 = n 2 ( X 2 + V 2 X 2 ) L f .

Lemma E.3. We have the following results: dist( 0 , ∇ J f ( X t )) ≤ γ · ∥∇ J T ( I 2 ; X t , B ) ∥ F + 2 X 2 √ E ι t [ u t ] with γ ≜ X √ C 2 n .

Proof. We have the following inequalities:

where step ① uses the definition of ∇ J f ( X t ) ; step ② uses JJ = I and X ⊤ JX = J ⇒ X ⊤ JXJ = 693 JJ = I ; step ③ uses the norm inequality and ; step ④ uses ∀ t, ∥ X t ∥ F ≤ X. 694

t t ⊤ t t ⊤

where step ① uses ∀ A , B , ∥ A ∥ F -∥ B ∥ F ≤ ∥ A -B ∥ F ; step ② uses the norm inequality; step ③ 696 uses ∀ t, ∥ X t ∥ F ≤ X. Thus, 697

where step ① uses Lemma A.1 with W = ˜ G t ( X t ) ⊤ -JX t ( ˜ G t ) ⊤ J and k = 2 ; step ② uses the 698 definition of ∇ J T ( I 2 ; X t , B ) . 699

We now present the following useful lemma. 700

Lemma E.4. We define T X J ≜ { Y ∈ R n × n | A X ( Y ) = 0 } and A X ( Y ) ≜ X ⊤ JY + Y ⊤ JX . For any G ∈ R n × n and X ⊤ JX = J , the unique minimizer of the following optimization problem:

satisify h ( ¯ Y ) ≤ h ( G -JXG ⊤ XJ ) .

701

Proof. We note that ¯ Y = arg min Y ∈ T X J 1 2 ∥ Y -G ∥ 2 F = arg min Y 1 2 ∥ Y -G ∥ 2 F , 702 s.t. X ⊤ JY + Y ⊤ JX = 0 . Introducing a multiplier Λ ∈ R n × n for the linear con703 straints X ⊤ JY + Y ⊤ JX = 0 , we have following Lagrangian function: ˜ L ( Y ; Λ ) = 704

1 2 ∥ Y -G ∥ 2 F + ⟨ X ⊤ JY + Y ⊤ JX , Λ ⟩ . We naturally derive the following first-order optimality 705 condition: Y -G + JXΛ = 0 , X ⊤ JY + Y ⊤ JX = 0 . Incorporating the term Y = G -JXΛ 706 into X ⊤ JY + Y ⊤ JX = 0 , we obtain: 707

Any Λ satisfying formula (55) is a feasible point, so we can easily find : 708

where step ① uses the fact that any matrix X satisfying the J-orthogonality constraint has a determinant of 1 or -1, thus inv( X ) exists; step ② multiply both sides of the equation by XJ ;step ③ uses X T JX = J and JJ = I ; step ④ multiply both sides of the equation by J and uses JJ = I ; step ⑤ uses the fact that Λ is a symmetric matrix.

Therefore, a feasible solution Y can be computed as Y = G -JXΛ = G -JXG ⊤ XJ . Since ¯ Y is the optimal solution, there must be h ( ¯ Y ) ≤ h ( G -JXG ⊤ XJ ) .

We now present the proof of this lemma.

Lemma E.5. For any X ∈ R n × n , it holds that dist( 0 , ∇ f ◦ ( X )) ≤ dist( 0 , ∇ J f ( X )) .

Proof. For the purpose of analysis, we define the nearest J orthogonal matrix to an arbitrary matrix Y ∈ R n × n is given by P J ( X ) . Similarly, we have P T X J ( ∇ f ( X )) for projecting gradient ∇ f ( X ) into space T X J .

We recall that the following first-order optimality conditions are equivalent for all X ∈ R n × n :

Therefore, we derive the following results:

We let G ∈ ∇ f ( X ) and obtain the following results from the above equality: 722

where step ① uses Lemma E.4; step ② uses ∇ J f ( X ) = G -JXG ⊤ XJ with G ∈ ∇ f ( X ) . 723

First of all, since f ◦ ( X ) ≜ f ( X ) + I J ( X ) is a KL function, we have from Proposition 4.8 that: 724

where step ① uses Lemma E.5. Here, φ ( · ) is some certain concave desingularization function. Since 725 φ ( · ) is concave, we have: 726

Applying the inequality above with ∆ = f ( X t ) -f ( ¯ X ) and ∆ + = f ( X t +1 ) -f ( ¯ X ) , we have: 727

709

710

711

712

713

714

715

716

717

718

719

720

721

With the sufficient descent condition as shown in Theorem 4.7, we derive the following inequalities: 728

(67)

where step ① uses ∀ t, ∥ X t ∥ F ≤ X. 729

where step ① uses the sufficient descent condition as shown in Theorem 4.7; step 730 ② uses Inequality (64) and (62) with X ′ = X t and X = ¯ X ; step ③ uses lemma 731 E.3 ; step ④ uses Lemma E.2 ; step ⑤ uses ∀ x i ∈ R , x 1 + ··· + x n n ⩽ √ x 2 1 + ··· + x 2 n n 732 ; step ⑥ applies the inequality that ∀ θ ′ > 0 , a, b, ab ≤ θ ′ a 2 2 + b 2 2 θ ′ with 733 a = √ ∑ n/ 2 i =1 ∥ ¯ V t -1 i -I 2 ∥ 2 F , b = E t γϕ √ n 2 ; a = √ E ι t [ u t ] , b = E t (2 X 2 + γ np 2 X + γ np 2 V 2 X ) ; 734 step ⑦ denote A 2 ≜ (2 X 2 + γ np 2 X + γ np 2 V 2 X ) 2 2 ¯ θ + nγ 2 ϕ 2 4 θ ′ . To simplify the formula, we define 735 ℵ t = ∑ n/ 2 i =1 ∥ ¯ V t i -I 2 ∥ 2 F . 736

Multiplying both sides by 2 and taking the square root of both sides, we have: 737

To recursively eliminate term √ ( ¯ θ +1) E ι t [ u t ] , we take the root of both sides of the Inequality in 738 Lemma 4.5: 739

Adding Inequality √ ¯ θ +1 1 - √ 1 -p × (70) to (69) 740

With the choice √ θ ′ = θ -X 2 2 -√ L 2 f X 2 (1 -p ) b ′ ¯ θ +1 1 - √ 1 -p , we have: 741

Rearranging terms, we have: 742

Summing the inequality above over t = 1 , 2 . . . , T , we have: 743

where step ① uses the fact that E ι T [ u T ] ≥ 0 and E ι 0 [ u 0 ] ≤ N -b b ( N -1) σ 2 ; step ② uses ∀ t, ∥ V ∥ F ≤ V, 744 then, ∥ V i -I 2 ∥ 2 F ≤ ( ∥ V i ∥ F + ∥ I 2 ∥ F ) 2 ≤ ( X + √ 2) 2 and ∑ n/ 2 i =1 ∥ ¯ V 0 i -I 2 ∥ 2 F ≤ n 2 ( V + √ 2) 2 . 745 Define C = √ 1 -p √ ( ¯ θ +1) 1 - √ 1 -p √ N -b b ( N -1) σ 2 + T √ ¯ θ +1 1 - √ 1 -p √ p ( N -b ) b ( N -1) σ 2 and rearrange terms, we have: 746 √

Considering A ∑ T t =1 E t , we have: 747

where step ① uses the definition of E i in (64); step ② uses a basic recursive reduction; step ③ uses the 748 fact the desingularization function φ ( · ) is positive. Combining Inequality (74) and (75), we obtain : 749

Using the inequality that ∥ X + -X ∥ 2 F X 2 ≤ ∥ X + -X ∥ 2 F ∥ X ∥ 2 F ≤ ∑ n/ 2 i =1 ∥ ¯ V i -I 2 ∥ 2 F as shown in Part ( b ) in Lemma 750 2.5, we have: 751

Since b = N,b ′ = √ b and p = b ′ b + b ′ , C = √ 1 -p √ ( ¯ θ +1) 1 - √ 1 -p √ N -b b ( N -1) σ 2 + T √ ¯ θ +1 1 - √ 1 -p √ p ( N -b ) b ( N -1) σ 2 = 0 , we can get the expression for C:

752

753

754

755

756

757

758

759

760

761

762

763

764

E.5 Proof of Theorem 4.9

Proof. For simplicity, we use B instead of B t . Initially, we prove the following important lemmas.

Lemma E.6. (Riemannian gradient Lower Bound for the Iterates Gap) We define ϕ ≜ (3 X + VX ) G + (1 + X 2 + V 2 + V 2 X 2 ) L f + (1 + V 2 ) θ . It holds that: E ξ t +1 [dist( 0 , ∇ J G ( I 2 ; X t +1 , B t +1 ))] ≤ ϕ · E ξ t [ ∥ ¯ V t -I 2 ∥ F ] .

Proof. The proof process is exactly the same as in lemma E.2 and will not be repeated here.

The following lemma is useful to outline the relation of ∥∇ J f ( X t ) ∥ F and ∥∇ J G ( I 2 ; X t , B ) ∥ F .

Proof. We have the following inequalities: 765

766

767

768

769

770

771

where step ① uses the definition of ∇ J f ( X t ) ; step ② uses JJ = I and X ⊤ JX = J ⇒ X ⊤ JXJ = JJ = I ; step ③ uses the norm inequality and ; step ④ uses the definition of W ≜ G t ( X t ) ⊤ -JX t ( G t ) ⊤ J and ∀ t, ∥ X t ∥ F ≤ X ; step ⑤ uses Lemma (A.1) with k = 2 ; step ⑥ uses the definition of ∇ J G ( I 2 ; X t , B ) . Taking the square root of both sides, we finish the proof of this lemma.

Finally, we obtain our main convergence results. First of all, since f ◦ ( X ) ≜ f ( X ) + I J ( X ) is a KL function, we have from Proposition 4.8 that:

where step ① uses Lemma E.5. Here, φ ( · ) is some certain concave desingularization function. Since φ ( · ) is concave, we have:

Applying the inequality above with ∆ = f ( X t ) -f ( ¯ X ) and ∆ + = f ( X t +1 ) -f ( ¯ X ) , we have: 772

We derive the following inequalities: 773

where step ① uses the sufficient descent condition as shown in Theorem 4.6; step ② uses Inequality 774 (77); step ③ uses Inequality (76) with X ′ = X t and X = ¯ X ; step ④ uses Lemma E.7; step ⑤ uses 775 Lemma E.6; step ⑥ applies the inequality that ∀ θ ′ > 0 , a, b, ab ≤ θ ′ a 2 2 + b 2 2 θ ′ with a = ∥ ¯ V t -1 -I 2 ∥ F 776 and b = E t γϕ . 777

Multiplying both sides by 2 and taking the square root of both sides, we have: 778

where step ① uses the inequality that √ a + b ≤ √ a + √ b for all a ≥ 0 and b ≥ 0 . Summing the 779 inequality above over i = 1 , 2 . . . , t , we have: 780

where step ① uses the definition of E i in (77); step ② uses a basic recursive reduction; step ③ uses 781 the fact the desingularization function φ ( · ) is positive. With the choice θ ′ = θ 4 , we have: 782

where step ① uses ∀ t, ∥ V ∥ F ≤ V,then, ∥ V -I 2 ∥ F ≤ ∥ V ∥ F + ∥ I ∥ F ≤ V + √ 2 . Finally, we obtain 783 from Inequality (79): 784

where step ① uses the inequality that ∥ X i +1 -X i ∥ F X ≤ ∥ ¯ V i -I 2 ∥ F as shown in Part ( b ) in Lemma 2.1. Finally, we can get the expression for C :

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

F Additional Experiment Details and Results

F.1 Additional Details for Hyperbolic Structural Probe Problem

To begin with, we give the definition of the Ultrahyperbolic manifold U p,q α , which will be used in Ultra-hyperbolic geodesic distance d α ( x , y ) and Diffeomorphism φ ( · ) .

Ultrahyperbolic manifold. Vectors in an ultrahyperbolic manifold is defined as U p,q α = { x = ( x 1 , x 2 , · · · , x p + q ) ⊤ ∈ R p,q : ∥ x ∥ 2 q = -α 2 } [48], where α is a non-negative real number denoting the radius of curvature. ∥ x ∥ 2 q = ⟨ x , x ⟩ q , ∀ x , y ∈ R p,q , ⟨ x , y ⟩ q = ∑ p i =1 x i y i -∑ p + q j = p +1 x j y j is a norm of the induced scalar product. The hyperbolic and spherical manifolds can be defined as : H α = U p, 1 α , S α = U 0 ,q α .

Ultra-hyperbolic geodesic distance. The ultra-hyperbolic geodesic distance [27][28] d γ ( · , · ) is formulated: ∀ x ∈ U p,q α , y ∈ U p,q α and α > 0 , d α ( x , y ) = { α cosh -1 ( | ⟨ x , y ⟩ q α 2 | ) if | ⟨ x , y ⟩ q α 2 | ≥ 1 α cos -1 ( | ⟨ x , y ⟩ q α 2 | ) otherwise. .

Diffeomorphism. [Theorem 1 Diffeomorphism of [49]]: Any vector x ∈ R p × R q ∗ can be mapped into U p,q α by a double projection φ = ϕ -1 ◦ ϕ , with ψ ( x ) = ( s α t ∥ t ∥ ) , ψ -1 ( z ) = ( v √ α 2 + ∥ v ∥ 2 α u ) , where x = ( s ) ∈ U p,q α with s ∈ R p and t ∈ R q ∗ · z = ( v ) ∈ R p × S q α with v ∈ R p and u ∈ S q α

F.2 Additional application: Ultra-hyperbolic Knowledge Graph Embedding

The J orthogonal matrix can be used as an isometric linear operator in the Ultrahyperbolic manifold, [48] et al. extended the knowledge graph model from hyperbolic space to Ultra-hyperbolic space (named as UltraE ) by this property. The UltraE model is formulated as follows:

where E ∈ R n e × n with E h = E ( h, :) ∈ U p,q α , b ∈ R n r with b h = b ( r ) ∈ R , R ∈ R n r × n × n with R r = R ( r, : , :) ∈ R n × n and J = [ I p 0 0 -I q ] ; ∆ ∈ N N × 3 is the set of positive triplets, ∆ ′ ( h,r,t ) ∈ N N × k × 3 denotes the set of negative triples constructed by corrupting ( h, r, t ) ; δ is a global margin hyper-parameter, σ ( · ) is the sigmoid function, n e represents the number of entities and n r represents the number of relations; d α ( · ) stands for the Ultra-hyperbolic geodesic distance (refer to F.1).

Experiment Details. We selected a batch of FB15K and WN18RR respectively as the data set for the Ultra-hyperbolic Knowledge Graph Embedding problem, (training set size, test set size, number of entities, number of relations) are (719,308,135,22) and (545,233,208,5) respectively. n = 36 , p = 18 , δ = 5 , α = 1 and k = 50 . In order to highlight the difference between J orthogonal optimization, in the UltraE model, all entities and biases of the optimization algorithm are optimized using ADMM by Pytorch , lr = 5 e -4 . We use the Adagrad optimizer in Pytorch to optimize the J-orthogonality constraint variable in the CS model.

F.3 Experiment result

Hyperbolic Eigenvalue Problem. Table 2 and Figure 3, 4, 5 are supplementary experiments for HEVP. Several conclusions can be drawn. (i) GS-JOBCD often greatly improves upon UMCM , ADMM and CSDM . This is because our methods find stronger stationary points than them. (ii) J-JOBCD is a parallel version of GS-JOBCD and thus exhibits significantly faster convergence. (iii) The proposed methods generally give the best performance.

Hyperbolic Structural Probe Problem. Table 3 and Figure 6, 7 are supplementary experiments for HSPP. Several conclusions can be drawn. (i) J-JOBCD often greatly improves upon UMCM , ADMM and CSDM (ii) VR-J-JOBCD is a reduced variance version of J-JOBCD and thus exhibits

825

826

827

828

829

830

831

832

significantly faster convergence for problems with large samples. (iii) The proposed methods generally give the best performance.

▶ Ultra-hyperbolic Knowledge Graph Embedding Problem. Figure 8, 9, 10 and 11 are supplementary experiments for UltraE . Several conclusions can be drawn. (i) In terms of Epoch performance, J-JOBCD and VR-J-JOBCD often greatly improves upon CSDM , thus they show better MRR and hits results. (ii) In models with limited sample sizes, the computational efficiency of VR-J-JOBCD is inferior to that of J-JOBCD . This discrepancy arises because each iteration in VR-J-JOBCD necessitates two instances of backpropagation, thus consuming substantial computational resources.

(iii) The proposed methods generally give the best performance. 833

F.3.1 Hyperbolic Eigenvalue Problem 834

dataname

(m-n-p)

UMCM

ADMM

CSDM

GS-JOBCD

J-JOBCD

UMCM+GS-JOBCD

ADMM+GS-JOBCD

CSDM+GS-JOBCD

time limit=30s

cifar

(1000-100-50)

-1.05e+04(3.0e-09)

-1.05e+04(3.0e-09)

-5.28e+04(4.8e-09)

-7.76e+04(1.6e-08)

-1.19e+05(8.1e-08)

-7.96e+04(1.1e-08)(+)

-5.86e+04(8.4e-09)(+)

-8.50e+04(1.2e-08)(+)

CnnCaltech

(2000-1000-500)

-5.89e+02(2.9e-08)

-5.89e+02(3.1e-10)

-7.86e+02(3.8e-10)

-7.68e+02(5.2e-10)

-3.90e+03(2.3e-08)

-6.71e+02(2.9e-08)(+)

-6.73e+02(3.6e-10)(+)

-8.49e+02(4.3e-10)(+)

gisette

(3000-1000-500)

-3.22e+06(3.1e-10)

-3.22e+06(3.1e-10)

-5.16e+06(3.9e-10)

-7.13e+06(5.9e-10)

-1.15e+07(1.5e-08)

-4.63e+06(3.7e-10)(+)

-4.74e+06(3.7e-10)(+)

-6.31e+06(4.6e-10)(+)

mnist

(1000-780-390)

-8.65e+04(4.1e-10)

-8.65e+04(4.1e-10)

-1.63e+05(4.9e-10)

-2.23e+05(6.6e-10)

-6.99e+05(2.1e-08)

-1.59e+05(4.7e-10)(+)

-1.48e+05(4.7e-10)(+)

-2.04e+05(5.7e-10)(+)

randn10

(10-10-5)

1.29e+02(9.7e-02)

1.29e+02(9.7e-02)

3.03e+02(2.3e-01)

-3.96e+01(9.7e-02)

-2.98e+02(9.7e-02)

-7.05e+01(9.7e-02)(+)

-1.75e+01(9.7e-02)(+)

2.29e+02(2.3e-01)(+)

randn100

(100-100-50)

-1.03e+04(3.0e-09)

-1.03e+04(2.5e-07)

-1.98e+04(5.1e-09)

-1.49e+04(2.7e-08)

-3.44e+04(1.3e-07)

-1.31e+04(2.5e-07)(+)

-2.20e+04(1.8e-08)(+)

(1000-1000-500)

-1.16e+06(3.1e-10)

-1.16e+06(3.1e-10)

-1.47e+06(3.9e-10)

-1.20e+06(4.4e-10)

-4.83e+06(7.3e-08)

-1.18e+06(3.5e-10)(+)

-1.18e+06(3.5e-10)(+)

-1.49e+06(4.7e-10)(+)

sector

(500-1000-500)

-3.61e+03(3.1e-10)

-3.61e+03(3.1e-10)

-5.35e+03(3.9e-10)

-6.68e+03(5.8e-10)

-1.07e+04(1.2e-08)

-4.73e+03(3.7e-10)(+)

-4.85e+03(3.6e-10)(+)

-6.47e+03(4.6e-10)(+)

TDT2

(1000-1000-500)

-4.25e+06(3.1e-10)

-4.25e+06(3.1e-10)

-6.37e+06(4.0e-10)

-8.20e+06(6.0e-10)

-1.32e+07(1.2e-08)

-5.67e+06(3.7e-10)(+)

-5.93e+06(3.7e-10)(+)

-7.85e+06(4.8e-10)(+)

w1a

(2470-290-145)

-3.02e+04(1.1e-04)

-3.02e+04(1.1e-04)

-5.42e+04(4.4e-04)

-5.74e+04(1.1e-04)

-6.73e+05(1.1e-04)

-4.76e+04(1.1e-04)(+)

-4.57e+04(1.1e-04)(+)

-6.32e+04(4.4e-04)(+)

cifar

(1000-100-70)

-7.32e+03(1.9e-09)

-7.32e+03(1.9e-09)

-3.29e+04(3.2e-09)

-6.01e+04(1.5e-08)

-1.12e+05(7.4e-08)

-4.84e+04(1.0e-08)(+)

-4.30e+04(7.6e-09)(+)

-7.52e+04(1.3e-08)(+)

CnnCaltech

(2000-1000-700)

-4.33e+02(2.1e-08)

-4.33e+02(2.2e-10)

-5.43e+02(2.5e-10)

-5.69e+02(3.6e-10)

-2.88e+03(1.6e-08)

-4.86e+02(2.1e-08)(+)

-4.85e+02(2.6e-10)(+)

-5.98e+02(3.0e-10)(+)

gisette

(1000,700)

-2.45e+06(2.2e-10)

-2.45e+06(2.2e-10)

-3.59e+06(2.5e-10)

-5.02e+06(4.2e-10)

-9.17e+06(1.0e-08)

-3.15e+06(2.5e-10)(+)

-3.25e+06(2.5e-10)(+)

-4.25e+06(2.7e-10)(+)

mnist

(1000-780-500)

-7.05e+04(3.1e-10)

-7.05e+04(3.1e-10)

-1.21e+05(3.6e-10)

-1.81e+05(5.3e-10)

-6.28e+05(1.9e-08)

-1.14e+05(3.5e-10)(+)

-1.21e+05(3.6e-10)(+)

-1.59e+05(4.3e-10)(+)

randn10

(10-10-7)

1.61e+02(5.6e-02)

1.61e+02(5.6e-02)

3.46e+02(1.8e-01)

-4.14e+11(2.1e+02)

-1.64e+00(5.6e-02)

3.69e+01(5.6e-02)(+)

-8.63e+02(5.6e-02)(+)

2.96e+02(1.8e-01)(+)

randn100

(100,70)

-8.00e+03(1.9e-09)

-8.00e+03(1.8e-07)

-1.41e+04(2.9e-09)

-1.10e+04(1.9e-08)

-2.37e+04(8.5e-08)

-9.68e+03(9.3e-09)(+)

-9.75e+03(1.8e-07)(+)

-1.64e+04(1.3e-08)(+)

(1000-1000-700)

-8.88e+05(2.2e-10)

-8.88e+05(2.2e-10)

-1.07e+06(2.7e-10)

-9.15e+05(3.4e-10)

-3.24e+06(4.3e-08)

-9.04e+05(2.6e-10)(+)

-9.05e+05(2.5e-10)(+)

sector

(500-1000-700)

-2.66e+03(2.2e-10)

-2.66e+03(2.2e-10)

-3.63e+03(2.5e-10)

-4.57e+03(3.8e-10)

-8.93e+03(9.1e-09)

-3.16e+03(2.4e-10)(+)

-3.26e+03(2.4e-10)(+)

-4.12e+03(2.8e-10)(+)

TDT2

(1000-1000-700)

-3.15e+06(2.2e-10)

-3.15e+06(2.2e-10)

-4.33e+06(2.5e-10)

-5.32e+06(3.4e-10)

-1.23e+07(9.1e-09)

-3.80e+06(2.5e-10)(+)

-3.75e+06(2.4e-10)(+)

-4.77e+06(2.8e-10)(+)

w1a

(2470-290-200)

-2.77e+04(8.0e-10)

-2.77e+04(8.0e-10)

-3.93e+04(1.1e-09)

-5.19e+04(4.1e-09)

-3.97e+05(5.6e-08)

-4.05e+04(1.8e-09)(+)

-3.73e+04(1.7e-09)(+)

-5.45e+04(2.9e-09)(+)

cifar

(1000-100-90)

-6.42e+03(1.2e-09)

-6.42e+03(1.2e-09)

-1.59e+04(1.2e-09)

-4.51e+04(1.4e-08)

-5.05e+04(5.6e-08)

-2.64e+04(6.3e-09)(+)

-3.35e+04(7.0e-09)(+)

-3.89e+04(6.7e-09)(+)

CnnCaltech

(2000-1000-900)

-3.10e+02(1.1e-08)

-3.10e+02(1.2e-10)

-3.41e+02(1.3e-10)

-3.64e+02(1.7e-10)

-1.65e+03(8.0e-09)

-3.29e+02(1.1e-08)(+)

-3.32e+02(1.3e-10)(+)

-3.61e+02(1.4e-10)(+)

gisette

(3000-1000-900)

-1.74e+06(1.2e-10)

-1.74e+06(1.2e-10)

-2.05e+06(1.2e-10)

-2.57e+06(1.8e-10)

-6.46e+06(8.0e-09)

-2.00e+06(1.3e-10)(+)

-1.99e+06(1.2e-10)(+)

-2.33e+06(1.4e-10)(+)

mnist

(1000-780-650)

-5.19e+04(1.9e-10)

-5.19e+04(1.9e-10)

-6.12e+04(2.2e-10)

-1.02e+05(2.9e-10)

-4.03e+05(1.2e-08)

-6.75e+04(2.1e-10)(+)

-6.58e+04(2.1e-10)(+)

randn10

(10-10-9)

5.33e+02(1.7e-01)

5.33e+02(1.7e-01)

4.28e+02(1.3e-01)

-1.03e+12(5.5e+02)

3.64e+02(1.7e-01)

2.21e+02(1.7e-01)(+)

-3.46e+02(1.7e-01)(+)

1.54e+02(1.3e-01)(+)

randn100

(100-100-90)

-6.14e+03(1.1e-07)

-6.14e+03(1.2e-09)

-8.14e+03(1.4e-09)

-8.31e+03(1.5e-08)

-1.31e+04(5.8e-08)

-7.74e+03(1.2e-07)(+)

-7.77e+03(1.3e-08)(+)

-9.69e+03(1.1e-08)(+)

(1000-1000-900)

-6.33e+05(1.2e-10)

-6.33e+05(1.2e-10)

-6.84e+05(1.3e-10)

-6.46e+05(1.9e-10)

-1.87e+06(1.8e-08)

-6.39e+05(1.3e-10)(+)

-6.39e+05(1.3e-10)(+)

-6.90e+05(1.5e-10)(+)

sector

(500-1000-900)

-1.92e+03(1.2e-10)

-1.92e+03(1.2e-10)

-2.18e+03(1.2e-10)

-2.50e+03(1.6e-10)

-5.84e+03(6.5e-09)

-2.12e+03(1.2e-10)(+)

-2.13e+03(1.2e-10)(+)

-2.34e+03(1.4e-10)(+)

TDT2

(1000-1000-900)

-2.26e+06(1.2e-10)

-2.26e+06(1.2e-10)

-2.58e+06(1.2e-10)

-2.90e+06(1.6e-10)

-7.40e+06(6.6e-09)

-2.53e+06(1.2e-10)(+)

-2.51e+06(1.2e-10)(+)

w1a

(2470-290-250)

-2.03e+04(5.4e-10)

-2.03e+04(5.4e-10)

-2.41e+04(5.9e-10)

-3.74e+04(2.9e-09)

-2.59e+05(3.3e-08)

-2.82e+04(1.2e-09)(+)

-3.17e+04(1.4e-09)(+)

-3.78e+04(1.5e-09)(+)

time limit=60s

cifar

(1000-100-50)

-1.05e+04(3.0e-09)

-1.05e+04(3.0e-09)

-5.28e+04(4.7e-09)

-9.03e+04(2.2e-08)

-1.14e+05(1.1e-07)

-8.07e+04(1.5e-08)(+)

-1.01e+05(1.6e-08)(+)

CnnCaltech

(2000-1000-500)

-5.89e+02(2.9e-08)

-5.89e+02(3.1e-10)

-9.79e+02(4.6e-10)

-9.58e+02(9.9e-10)

-6.68e+03(4.7e-08)

-7.13e+02(2.9e-08)(+)

-7.23e+02(4.3e-10)(+)

-1.05e+03(5.9e-10)(+)

gisette

(3000-1000-500)

-3.22e+06(3.1e-10)

-3.22e+06(3.1e-10)

-6.54e+06(4.3e-10)

-8.84e+06(9.7e-10)

-1.29e+07(2.1e-08)

-5.55e+06(4.1e-10)(+)

-5.67e+06(4.2e-10)(+)

-8.02e+06(6.4e-10)(+)

mnist

(1000-780-390)

-8.65e+04(4.1e-10)

-8.65e+04(4.1e-10)

-2.28e+05(5.3e-10)

-2.75e+05(9.5e-10)

-9.32e+05(3.2e-08)

-1.80e+05(5.4e-10)(+)

-1.88e+05(5.4e-10)(+)

-2.83e+05(6.8e-10)(+)

randn10

(10-10-5)

1.29e+02(9.7e-02)

1.29e+02(9.7e-02)

2.43e+02(2.2e-01)

-3.96e+01(9.7e-02)

-7.07e+01(9.7e-02)

-2.27e+05(9.7e-02)(+)

-3.46e+02(9.7e-02)(+)

randn100

(100-100-50)

-1.03e+04(3.0e-09)

-1.03e+04(2.5e-07)

-1.98e+04(5.4e-09)

-1.81e+04(4.3e-08)

-3.88e+04(1.9e-07)

-1.41e+04(1.9e-08)(+)

-1.44e+04(2.5e-07)(+)

-2.41e+04(2.9e-08)(+)

(1000-1000-500)

-1.16e+06(3.1e-10)

-1.16e+06(3.1e-10)

-1.79e+06(4.7e-10)

-1.21e+06(6.0e-10)

-7.79e+06(1.5e-07)

-1.19e+06(3.9e-10)(+)

-1.19e+06(4.0e-10)(+)

-1.81e+06(5.7e-10)(+)

sector

(500-1000-500)

-3.61e+03(3.1e-10)

-3.61e+03(3.1e-10)

-6.63e+03(4.5e-10)

-8.22e+03(9.1e-10)

-1.11e+04(1.7e-08)

-5.53e+03(4.2e-10)(+)

-5.63e+03(4.3e-10)(+)

-7.83e+03(5.8e-10)(+)

TDT2

(1000-1000-500)

-4.25e+06(3.1e-10)

-4.25e+06(3.1e-10)

-8.31e+06(4.4e-10)

-9.87e+06(8.9e-10)

-1.43e+07(1.7e-08)

-6.51e+06(4.1e-10)(+)

-6.80e+06(4.2e-10)(+)

-9.33e+06(5.3e-10)(+)

w1a

(2470-290-145)

-3.02e+04(1.1e-04)

-3.02e+04(1.1e-04)

-5.63e+04(1.0e-04)

-7.13e+04(1.1e-04)

-2.07e+06(1.1e-04)

-5.44e+04(1.1e-04)(+)

-5.42e+04(1.1e-04)(+)

-6.91e+04(1.0e-04)(+)

cifar

(1000-100-70)

-7.32e+03(1.9e-09)

-7.32e+03(1.9e-09)

-3.29e+04(3.3e-09)

-6.94e+04(1.9e-08)

-9.31e+04(9.6e-08)

-5.69e+04(1.4e-08)(+)

-5.87e+04(1.4e-08)(+)

-8.58e+04(1.8e-08)(+)

CnnCaltech

(2000-1000-700)

-4.33e+02(2.1e-08)

-4.33e+02(2.2e-10)

-6.99e+02(3.2e-10)

-6.74e+02(5.7e-10)

-4.74e+03(3.1e-08)

-5.23e+02(2.1e-08)(+)

-5.21e+02(2.7e-10)(+)

-7.62e+02(4.1e-10)(+)

gisette

(1000,700)

-2.45e+06(2.2e-10)

-2.45e+06(2.2e-10)

-4.99e+06(2.8e-10)

-6.15e+06(6.0e-10)

-1.15e+07(1.6e-08)

-3.71e+06(2.6e-10)(+)

-3.86e+06(2.7e-10)(+)

-5.81e+06(3.7e-10)(+)

mnist

(1000-780-500)

-7.05e+04(3.1e-10)

-7.05e+04(3.1e-10)

-1.62e+05(4.3e-10)

-2.19e+05(6.8e-10)

-8.50e+05(2.6e-08)

-1.34e+05(4.0e-10)(+)

-1.41e+05(3.9e-10)(+)

-2.00e+05(5.1e-10)(+)

randn10

(10-10-7)

1.61e+02(5.6e-02)

1.61e+02(5.6e-02)

3.81e+02(1.6e-01)

-3.45e+16(9.0e+06)

4.37e-01(5.6e-02)

-2.19e+04(5.6e-02)(+)

-2.03e+04(5.6e-02)(+)

3.39e+02(1.6e-01)(+)

randn100

(100,70)

-8.00e+03(1.9e-09)

-8.00e+03(1.8e-07)

-1.41e+04(2.9e-09)

-1.37e+04(3.2e-08)

-2.69e+04(1.4e-07)

-1.04e+04(1.3e-08)(+)

-1.08e+04(1.8e-07)(+)

(1000-1000-700)

-8.88e+05(2.2e-10)

-8.88e+05(2.2e-10)

-1.31e+06(3.0e-10)

-9.25e+05(4.5e-10)

-5.49e+06(9.7e-08)

-9.09e+05(2.8e-10)(+)

-9.09e+05(2.7e-10)(+)

-1.33e+06(3.8e-10)(+)

sector

(500-1000-700)

-2.66e+03(2.2e-10)

-2.66e+03(2.2e-10)

-5.05e+03(3.0e-10)

-5.74e+03(5.6e-10)

-1.13e+04(1.4e-08)

-3.74e+03(2.7e-10)(+)

-3.74e+03(2.7e-10)(+)

-5.65e+03(4.0e-10)(+)

TDT2

(1000-1000-700)

-3.15e+06(2.2e-10)

-3.15e+06(2.2e-10)

-6.13e+06(3.1e-10)

-6.94e+06(5.7e-10)

-1.45e+07(1.4e-08)

-4.55e+06(2.7e-10)(+)

-4.54e+06(2.8e-10)(+)

-6.99e+06(4.1e-10)(+)

w1a

(2470-290-200)

-2.77e+04(8.0e-10)

-2.77e+04(8.0e-10)

-4.00e+04(1.1e-09)

-7.13e+04(1.2e-08)

-3.42e+06(5.3e-07)

-4.71e+04(3.6e-09)(+)

-5.28e+04(4.3e-09)(+)

-6.55e+04(5.1e-09)(+)

cifar

(1000-100-90)

-6.42e+03(1.2e-09)

-6.42e+03(1.2e-09)

-1.60e+04(1.5e-09)

-4.91e+04(2.0e-08)

-4.97e+04(9.3e-08)

-4.12e+04(1.2e-08)(+)

-4.83e+04(1.6e-08)(+)

-4.87e+04(1.7e-08)(+)

CnnCaltech

(2000-1000-900)

-3.10e+02(1.1e-08)

-3.10e+02(1.2e-10)

-3.15e+02(1.2e-10)

-4.25e+02(3.1e-10)

-2.37e+03(1.4e-08)

-3.44e+02(1.1e-08)(+)

-3.41e+02(1.4e-10)(+)

-3.45e+02(1.4e-10)(+)

gisette

(3000-1000-900)

-1.74e+06(1.2e-10)

-1.74e+06(1.2e-10)

-2.59e+06(1.4e-10)

-3.09e+06(2.6e-10)

-8.40e+06(1.2e-08)

-2.21e+06(1.3e-10)(+)

-2.19e+06(1.3e-10)(+)

-2.79e+06(1.7e-10)(+)

mnist

(1000-780-650)

-5.19e+04(1.9e-10)

-5.19e+04(1.9e-10)

-6.81e+04(2.4e-10)

-1.35e+05(4.4e-10)

-5.92e+05(2.0e-08)

-7.34e+04(2.2e-10)(+)

-7.07e+04(2.3e-10)(+)

-9.21e+04(2.8e-10)(+)

randn10

(10-10-9)

5.33e+02(1.7e-01)

5.33e+02(1.7e-01)

5.07e+02(2.3e-01)

-1.28e+19(4.2e+09)

3.41e+02(1.7e-01)

-2.71e+04(1.7e-01)(+)

4.49e+02(1.7e-01)(+)

2.41e+02(2.3e-01)(+)

randn100

(100-100-90)

-6.14e+03(1.1e-07)

-6.14e+03(1.2e-09)

-8.14e+03(1.7e-09)

-1.08e+04(2.6e-08)

-1.56e+04(9.7e-08)

-8.18e+03(1.2e-07)(+)

-8.15e+03(1.2e-08)(+)

-1.12e+04(2.1e-08)(+)

(1000-1000-900)

-6.33e+05(1.2e-10)

-6.33e+05(1.2e-10)

-7.65e+05(1.5e-10)

-6.49e+05(2.4e-10)

-2.77e+06(3.5e-08)

-6.42e+05(1.4e-10)(+)

-6.41e+05(1.4e-10)(+)

-7.72e+05(1.8e-10)(+)

sector

(500-1000-900)

-1.92e+03(1.2e-10)

-1.92e+03(1.2e-10)

-2.66e+03(1.4e-10)

-2.80e+03(2.1e-10)

-7.48e+03(1.1e-08)

-2.23e+03(1.3e-10)(+)

-2.32e+03(1.4e-10)(+)

-2.82e+03(1.7e-10)(+)

TDT2

(1000-1000-900)

-2.26e+06(1.2e-10)

-2.26e+06(1.2e-10)

-3.18e+06(1.4e-10)

-3.33e+06(2.0e-10)

-8.81e+06(9.7e-09)

-2.72e+06(1.4e-10)(+)

-2.73e+06(1.3e-10)(+)

-3.37e+06(1.7e-10)(+)

w1a

(2470-290-250)

-2.03e+04(5.4e-10)

-2.03e+04(5.4e-10)

-2.42e+04(7.0e-10)

-4.96e+04(7.1e-09)

-1.56e+06(2.5e-07)

-3.55e+04(3.2e-09)(+)

-3.67e+04(2.9e-09)(+)

-4.36e+04(2.8e-09)(+)

time limit=90s

cifar

(1000-100-50)

-1.05e+04(3.0e-09)

-1.05e+04(3.0e-09)

-5.28e+04(5.4e-09)

-1.03e+05(2.6e-08)

-1.11e+05(1.4e-07)

-8.38e+04(1.7e-08)(+)

-1.24e+05(2.6e-08)(+)

CnnCaltech

(2000-1000-500)

-5.89e+02(2.9e-08)

-5.89e+02(3.1e-10)

-1.11e+03(5.2e-10)

-1.07e+03(1.3e-09)

-9.16e+03(6.9e-08)

-7.44e+02(2.9e-08)(+)

-7.51e+02(4.6e-10)(+)

-1.15e+03(6.9e-10)(+)

gisette

(3000-1000-500)

-3.22e+06(3.1e-10)

-3.22e+06(3.1e-10)

-8.53e+06(4.9e-10)

-9.49e+06(1.2e-09)

-1.36e+07(2.6e-08)

-6.21e+06(4.6e-10)(+)

-6.23e+06(4.9e-10)(+)

-9.65e+06(7.9e-10)(+)

mnist

(1000-780-390)

-8.65e+04(4.1e-10)

-8.65e+04(4.1e-10)

-2.56e+05(5.6e-10)

-3.14e+05(1.2e-09)

-1.20e+06(4.1e-08)

-2.05e+05(5.8e-10)(+)

-2.11e+05(6.1e-10)(+)

-3.06e+05(7.6e-10)(+)

randn10

(10-10-5)

1.29e+02(9.7e-02)

1.29e+02(9.7e-02)

2.45e+02(2.3e-01)

-3.96e+01(9.7e-02)

-3.97e+02(9.7e-02)

1.17e+01(9.7e-02)(+)

-2.66e+09(1.1e+00)(+)

1.55e+01(2.3e-01)(+)

randn100

(100-100-50)

-1.03e+04(3.0e-09)

-1.03e+04(2.5e-07)

-1.98e+04(4.4e-09)

-2.28e+04(5.6e-08)

-4.37e+04(2.6e-07)

-1.50e+04(2.5e-08)(+)

-1.54e+04(2.6e-07)(+)

-2.41e+04(4.2e-08)(+)

(1000-1000-500)

-1.16e+06(3.1e-10)

-1.16e+06(3.1e-10)

-1.93e+06(5.0e-10)

-1.22e+06(6.9e-10)

-1.04e+07(2.3e-07)

-1.19e+06(4.1e-10)(+)

-1.95e+06(6.7e-10)(+)

sector

(500-1000-500)

-3.61e+03(3.1e-10)

-3.61e+03(3.1e-10)

-7.90e+03(4.9e-10)

-9.24e+03(1.3e-09)

-1.06e+04(2.0e-08)

-5.56e+03(4.3e-10)(+)

-5.69e+03(4.3e-10)(+)

-8.51e+03(6.4e-10)(+)

TDT2

(1000-1000-500)

-4.25e+06(3.1e-10)

-4.25e+06(3.1e-10)

-9.39e+06(4.8e-10)

-1.05e+07(1.1e-09)

-1.42e+07(2.1e-08)

-6.65e+06(4.4e-10)(+)

-6.89e+06(4.2e-10)(+)

-1.04e+07(6.5e-10)(+)

w1a

(2470-290-145)

-3.02e+04(1.1e-04)

-3.02e+04(1.1e-04)

-5.72e+04(2.7e-05)

-9.21e+04(1.1e-04)

-9.32e+06(1.1e-04)

-5.74e+04(1.1e-04)(+)

-6.40e+04(1.1e-04)(+)

-7.94e+04(2.7e-05)(+)

cifar

(1000-100-70)

-7.32e+03(1.9e-09)

-7.32e+03(1.9e-09)

-3.29e+04(3.1e-09)

-7.19e+04(2.1e-08)

-1.36e+05(1.2e-07)

-6.05e+04(1.8e-08)(+)

-6.42e+04(1.8e-08)(+)

-9.76e+04(2.5e-08)(+)

CnnCaltech

(2000-1000-700)

-4.33e+02(2.1e-08)

-4.33e+02(2.2e-10)

-7.42e+02(3.1e-10)

-7.76e+02(8.4e-10)

-6.63e+03(4.7e-08)

-5.23e+02(2.1e-08)(+)

-5.38e+02(3.0e-10)(+)

-8.08e+02(4.2e-10)(+)

gisette

(1000,700)

-2.45e+06(2.2e-10)

-2.45e+06(2.2e-10)

-5.56e+06(3.1e-10)

-6.76e+06(7.5e-10)

-1.13e+07(1.9e-08)

-3.90e+06(3.0e-10)(+)

-4.02e+06(3.0e-10)(+)

-6.62e+06(4.4e-10)(+)

mnist

(1000-780-500)

-7.05e+04(3.1e-10)

-7.05e+04(3.1e-10)

-1.71e+05(4.5e-10)

-2.51e+05(8.4e-10)

-1.09e+06(3.5e-08)

-1.58e+05(4.3e-10)(+)

-1.46e+05(4.4e-10)(+)

-2.23e+05(5.9e-10)(+)

randn10

(10-10-7)

1.61e+02(5.6e-02)

1.61e+02(5.6e-02)

3.33e+02(1.7e-01)

6.45e+01(5.6e-02)

5.58e+01(5.6e-02)

1.15e+02(5.6e-02)(+)

8.86e+01(5.6e-02)(+)

1.67e+02(1.7e-01)(+)

randn100

(100,70)

-8.00e+03(1.9e-09)

-8.00e+03(1.8e-07)

-1.41e+04(3.0e-09)

-1.43e+04(3.0e-08)

-2.60e+04(1.7e-07)

-1.05e+04(1.4e-08)(+)

-1.08e+04(1.8e-07)(+)

-1.87e+04(3.2e-08)(+)

(1000-1000-700)

-8.88e+05(2.2e-10)

-8.88e+05(2.2e-10)

-1.36e+06(3.1e-10)

-9.25e+05(4.6e-10)

-7.05e+06(1.4e-07)

-9.10e+05(2.8e-10)(+)

-9.10e+05(2.8e-10)(+)

-1.37e+06(4.2e-10)(+)

sector

(500-1000-700)

-2.66e+03(2.2e-10)

-2.66e+03(2.2e-10)

-5.64e+03(3.2e-10)

-5.40e+03(5.2e-10)

-1.53e+04(1.8e-08)

-3.67e+03(2.6e-10)(+)

-3.80e+03(2.8e-10)(+)

TDT2

(1000-1000-700)

-3.15e+06(2.2e-10)

-3.15e+06(2.2e-10)

-6.78e+06(3.2e-10)

-6.43e+06(5.0e-10)

-1.33e+07(1.6e-08)

-4.49e+06(2.9e-10)(+)

-4.51e+06(2.6e-10)(+)

-7.19e+06(4.2e-10)(+)

w1a

(2470-290-200)

-2.77e+04(8.0e-10)

-2.77e+04(8.0e-10)

-4.00e+04(1.1e-09)

-7.60e+04(1.2e-08)

-8.13e+06(1.3e-06)

-5.33e+04(4.8e-09)(+)

-5.40e+04(4.5e-09)(+)

-6.47e+04(5.6e-09)(+)

cifar

(1000-100-90)

-6.42e+03(1.2e-09)

-6.42e+03(1.2e-09)

-1.60e+04(1.3e-09)

-3.78e+04(1.3e-08)

-5.24e+04(1.2e-07)

-2.77e+04(6.1e-09)(+)

-2.53e+04(4.8e-09)(+)

-3.60e+04(7.8e-09)(+)

CnnCaltech

(2000-1000-900)

-3.10e+02(1.1e-08)

-3.10e+02(1.2e-10)

-4.05e+02(1.6e-10)

-3.99e+02(2.6e-10)

-3.16e+03(2.0e-08)

-3.45e+02(1.1e-08)(+)

-3.42e+02(1.4e-10)(+)

-4.36e+02(2.0e-10)(+)

gisette

(3000-1000-900)

-1.74e+06(1.2e-10)

-1.74e+06(1.2e-10)

-2.84e+06(1.4e-10)

-2.76e+06(1.9e-10)

-9.30e+06(1.6e-08)

-2.16e+06(1.3e-10)(+)

-2.15e+06(1.3e-10)(+)

-3.06e+06(1.6e-10)(+)

mnist

(1000-780-650)

-5.19e+04(1.9e-10)

-5.19e+04(1.9e-10)

-6.85e+04(2.4e-10)

-1.28e+05(4.0e-10)

-7.18e+05(2.5e-08)

-8.43e+04(2.3e-10)(+)

-8.21e+04(2.3e-10)(+)

-1.06e+05(2.7e-10)(+)

randn10

(10-10-9)

5.33e+02(1.7e-01)

5.33e+02(1.7e-01)

4.54e+02(2.1e-01)

-2.28e+05(1.7e-01)

4.62e+02(1.7e-01)

1.80e+01(1.7e-01)(+)

-6.43e+00(1.7e-01)(+)

-1.06e+02(2.1e-01)(+)

randn100

(100-100-90)

-6.14e+03(1.1e-07)

-6.14e+03(1.2e-09)

-8.14e+03(1.5e-09)

-1.12e+04(2.5e-08)

-1.36e+04(1.2e-07)

-8.09e+03(1.2e-07)(+)

-8.48e+03(1.7e-08)(+)

-1.11e+04(2.1e-08)(+)

(1000-1000-900)

-6.33e+05(1.2e-10)

-6.33e+05(1.2e-10)

-7.87e+05(1.5e-10)

-6.50e+05(2.5e-10)

-3.36e+06(4.5e-08)

-6.42e+05(1.5e-10)(+)

-6.44e+05(1.5e-10)(+)

-7.94e+05(2.0e-10)(+)

sector

(500-1000-900)

-1.92e+03(1.2e-10)

-1.92e+03(1.2e-10)

-2.89e+03(1.5e-10)

-2.80e+03(1.9e-10)

-8.81e+03(1.3e-08)

-2.26e+03(1.3e-10)(+)

-2.30e+03(1.3e-10)(+)

-2.99e+03(1.7e-10)(+)

TDT2

(1000-1000-900)

-2.26e+06(1.2e-10)

-2.26e+06(1.2e-10)

-3.46e+06(1.5e-10)

-3.38e+06(2.0e-10)

-1.29e+07(1.4e-08)

-2.73e+06(1.3e-10)(+)

-2.65e+06(1.3e-10)(+)

-3.57e+06(1.7e-10)(+)

w1a

(2470-290-250)

-2.03e+04(5.4e-10)

-2.03e+04(5.4e-10)

-2.42e+04(6.2e-10)

-5.60e+04(8.5e-09)

-9.37e+06(1.1e-06)

-3.95e+04(3.5e-09)(+)

-3.90e+04(3.5e-09)(+)

-5.02e+04(3.4e-09)(+)

0

20

40

60

80

Time (seconds)

-7

-6

-5

-4

-3

-2

-1

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(a) Cifar(1000-100-50)

0

25

50

75

100

125

150

175

Time (seconds)

-2.2

-2.0

-1.8

-1.6

-1.4

-1.2

-1.0

-0.8

-0.6

Ob ective

×10 3

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(b) CnnCaltech(2000-1000-500)

0

25

50

75

100

125

150

175

Time (seconds)

-8

-7

-6

-5

-4

-3

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(c) gisette(3000-1000-500)

0

20

40

60

80

100

120

Time (seconds)

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

Objective

×10 5

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(d) mnist(1000-780-390)

0

20

40

60

80

Time (seconds)

0

2

4

6

8

Objective

×10 2

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(e) randn(10-10-5)

0

20

40

60

80

Time (sec nds)

-1.8

-1.6

-1.4

-1.2

-1.0

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(f) randn(100-100-50)

0

25

50

75

100

125

150

175

Time (seconds)

-2.0

-1.8

-1.6

-1.4

-1.2

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(g) randn(1000-1000-500)

0

25

50

75

100

125

150

175

Time (seconds)

-7.5

-7.0

-6.5

-6.0

-5.5

-5.0

-4.5

-4.0

-3.5

Objective

×10 3

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(h) sector(500-1000-500)

0

25

50

75

100

125

150

175

Time (seconds)

-9

-8

-7

-6

-5

-4

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(i) TDT2(1000-1000-500)

0

20

40

60

80

100

Time (seconds)

-6.5

-6.0

-5.5

-5.0

-4.5

-4.0

-3.5

-3.0

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(j) w1a(2470-290-145)

0

20

40

60

80

Time (seco ds)

-5

-4

-3

-2

-1

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(a) Cifar(1000-100-70)

0

25

50

75

100

125

150

175

Time (seconds)

-1.6

-1.4

-1.2

-1.0

-0.8

-0.6

-0.4

Ob ective

×10 3

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(b) CnnCaltech(2000-1000-700)

0

25

50

75

100

125

150

175

Time (seco ds)

-7

-6

-5

-4

-3

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(c) gisette(3000-1000-700)

0

20

40

60

80

100

120

Time (seconds)

-3.0

-2.5

-2.0

-1.5

-1.0

Objective

×10 5

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(d) mnist(1000-780-500)

0

20

40

60

80

Time (seconds)

0

1

2

3

4

Objective

×10 2

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(e) randn(10-10-7)

0

20

40

60

80

Time (seconds)

-1.4

-1.3

-1.2

-1.1

-1.0

-0.9

-0.8

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(f) randn(100-100-70)

0

25

50

75

100

125

150

175

Time (seconds)

-1.6

-1.5

-1.4

-1.3

-1.2

-1.1

-1.0

-0.9

Ob ective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(g) randn(1000-1000-700)

0

25

50

75

100

125

150

175

Time (seco ds)

-7

-6

-5

-4

-3

Objective

×10 3

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(h) sector(500-1000-700)

0

25

50

75

100

125

150

175

Time (seconds)

-8

-7

-6

-5

-4

-3

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(i) TDT2(1000-1000-700)

0

20

40

60

80

100

Time (seconds)

-6.0

-5.5

-5.0

-4.5

-4.0

-3.5

-3.0

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(j) w1a(2470-290-200)

0

20

40

60

80

Time (seconds)

-4.0

-3.5

-3.0

-2.5

-2.0

-1.5

-1.0

-0.5

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(a) Cifar(1000-100-90)

0

25

50

75

100

125

150

175

Time (seconds)

-9

-8

-7

-6

-5

-4

-3

Ob ective

×10 2

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(b) CnnCaltech(2000-1000-900)

0

25

50

75

100

125

150

175

Time (seconds)

-5.5

-5.0

-4.5

-4.0

-3.5

-3.0

-2.5

-2.0

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(c) gisette(3000-1000-900)

0

20

40

60

80

100

120

Time (seco ds)

-2.2

-2.0

-1.8

-1.6

-1.4

-1.2

-1.0

-0.8

-0.6

Objective

×10 5

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(d) mnist(1000-780-650)

0

20

40

60

80

Time (seconds)

-8

-7

-6

-5

-4

-3

-2

-1

0

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(e) randn(10-10-9)

0

20

40

60

80

Time (seconds)

-9.5

-9.0

-8.5

-8.0

-7.5

-7.0

-6.5

-6.0

Object ve

×10 3

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(f) randn(100-100-90)

0

25

50

75

100

125

150

175

Time (seconds)

-9.5

-9.0

-8.5

-8.0

-7.5

-7.0

-6.5

Objective

×10 5

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(g) randn(1000-1000-900)

0

25

50

75

100

125

150

175

Time (seco ds)

-5.0

-4.5

-4.0

-3.5

-3.0

-2.5

-2.0

Objective

×10 3

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(h) sector(500-1000-900)

0

25

50

75

100

125

150

175

Time (seco ds)

-6

-5

-4

-3

Objective

×10 6

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(i) TDT2(1000-1000-900)

0

20

40

60

80

100

Time (seco ds)

-3.6

-3.4

-3.2

-3.0

-2.8

-2.6

-2.4

-2.2

-2.0

Objective

×10 4

GS-JOBCD

J-JOBCD

CSDM

ADMM

UMCM

(j) w1a(2470-290-250)

F.3.2 Hyperbolic Structural Probe Problem 835

datasetname

(m-n-p)

ADMM

UMCM

CSDM

J-JOBCD

VR-J-JOBCD

CSDM+J-JOBCD

time limit=30s

20News

(9423-50-25)

6.47e+04(2.4e-03)

6.47e+04(2.4e-03)

6.45e+04(2.0e-03)

6.40e+04(2.4e-03)

6.26e+04(2.4e-03)

6.31e+04(2.0e-03)(+)

Cifar

(10000-50-25)

1.49e+07(2.4e-03)

1.49e+07(2.4e-03)

1.49e+07(2.0e-03)

1.46e+07(2.4e-03)

1.45e+07(2.5e-03)

1.46e+07(2.0e-03)(+)

cnnCaltech

(3000-96-48)

3.08e+04(3.5e-09)

3.08e+04(3.5e-09)

3.03e+04(2.3e-05)

1.64e+04(4.0e-08)

1.64e+04(4.7e-08)

1.74e+04(2.3e-05)(+)

E2006

(5000-100-50)

5.54e+04(3.5e-09)

5.54e+04(3.5e-09)

5.54e+04(1.9e-05)

5.53e+04(9.0e-08)

5.53e+04(1.2e-07)

5.53e+04(1.9e-05)(+)

gisette

(6000-50-25)

1.61e+05(2.4e-03)

1.61e+05(2.4e-03)

1.52e+05(1.6e-03)

9.06e+04(2.4e-03)

7.48e+04(2.4e-03)

8.65e+04(1.6e-03)(+)

Mnist

(6000-92-46)

6.94e+06(3.6e-09)

6.94e+06(3.6e-09)

6.93e+06(2.1e-05)

6.50e+06(4.0e-05)

6.48e+06(8.7e-05)

6.93e+06(2.1e-05)

(7967-50-25)

6.12e+04(2.4e-03)

6.12e+04(2.4e-03)

6.12e+04(2.0e-03)

6.12e+04(2.4e-03)

6.12e+04(2.4e-03)

6.12e+04(2.0e-03)

randn5000

(5000-100-50)

1.84e+06(3.5e-09)

1.84e+06(3.5e-09)

1.84e+06(2.0e-05)

1.77e+06(1.4e-07)

1.56e+06(9.9e-05)

1.84e+06(2.0e-05)

randn10000

(10000-50-25)

2.66e+06(2.4e-03)

2.66e+06(2.4e-03)

2.66e+06(2.5e-03)

2.64e+06(2.4e-03)

2.34e+06(2.5e-03)

2.66e+06(2.5e-03)

w1a

(2477-100-50)

2.71e+04(3.5e-09)

2.71e+04(3.5e-09)

2.47e+04(1.6e-05)

2.05e+04(3.6e-08)

1.92e+04(5.1e-08)

2.47e+04(1.6e-05)

20News

(9423-50-35)

7.89e+04(3.7e-09)

7.89e+04(3.7e-09)

7.85e+04(2.4e-05)

7.85e+04(8.6e-09)

7.71e+04(2.7e-08)

7.64e+04(2.4e-05)(+)

Cifar

(10000-50-35)

1.48e+07(5.5e-03)

1.48e+07(5.5e-03)

1.48e+07(6.2e-03)

1.43e+07(5.5e-03)

1.43e+07(5.7e-03)

1.43e+07(6.2e-03)(+)

cnnCaltech

(3000-96-70)

3.75e+04(2.0e-09)

3.75e+04(2.0e-09)

3.71e+04(1.3e-05)

2.40e+04(2.5e-08)

1.87e+04(3.4e-08)

2.15e+04(1.3e-05)(+)

E2006

(5000-100-75)

6.79e+04(1.0e-03)

6.79e+04(6.3e-04)

6.79e+04(5.8e-04)

6.65e+04(6.3e-04)

6.34e+04(6.3e-04)

6.69e+04(5.8e-04)(+)

gisette

(6000-50-35)

1.89e+05(5.5e-03)

1.89e+05(5.5e-03)

1.83e+05(4.7e-03)

1.24e+05(5.5e-03)

1.03e+05(5.5e-03)

1.17e+05(4.7e-03)(+)

Mnist

(6000-92-70)

6.75e+06(2.0e-09)

6.75e+06(2.0e-09)

6.74e+06(1.4e-05)

6.30e+06(7.7e-05)

6.20e+06(3.7e-04)

6.24e+06(7.0e-05)(+)

(7967-50-35)

7.26e+04(5.5e-03)

7.26e+04(5.5e-03)

7.26e+04(4.9e-03)

7.26e+04(5.5e-03)

7.25e+04(5.5e-03)

7.26e+04(4.9e-03)

randn5000

(5000-100-75)

1.75e+06(6.3e-04)

1.75e+06(6.3e-04)

1.75e+06(6.1e-04)

1.44e+06(6.6e-04)

1.70e+06(6.3e-04)

1.75e+06(6.1e-04)

randn10000

(10000-50-36)

2.56e+06(3.7e-09)

2.56e+06(3.7e-09)

2.56e+06(2.0e-05)

2.56e+06(5.9e-09)

2.54e+06(2.2e-08)

2.56e+06(2.0e-05)(+)

w1a

(2477-100-75)

3.36e+04(6.3e-04)

3.36e+04(6.3e-04)

3.16e+04(5.5e-04)

2.53e+04(6.3e-04)

2.48e+04(6.3e-04)

2.90e+04(5.5e-04)(+)

20News

(9423-50-45)

8.79e+04(4.8e-03)

8.79e+04(4.8e-03)

8.73e+04(4.6e-03)

8.62e+04(4.8e-03)

8.60e+04(4.8e-03)

8.68e+04(4.6e-03)(+)

Cifar

(10000-50-45)

1.47e+07(4.8e-03)

1.47e+07(4.8e-03)

1.47e+07(4.6e-03)

1.46e+07(4.8e-03)

1.41e+07(7.2e-03)

1.46e+07(4.6e-03)(+)

cnnCaltech

(3000-96-85)

4.13e+04(9.2e-04)

4.14e+04(4.4e-04)

4.11e+04(6.5e-04)

2.59e+04(4.4e-04)

2.25e+04(4.4e-04)

2.65e+04(6.5e-04)(+)

E2006

(5000-100-90)

7.44e+04(1.2e-09)

7.44e+04(1.2e-09)

7.44e+04(7.5e-06)

6.95e+04(6.5e-08)

6.92e+04(3.8e-07)

6.93e+04(7.5e-06)(+)

gisette

(6000-50-45)

2.25e+05(4.8e-03)

2.25e+05(4.8e-03)

2.10e+05(5.2e-03)

1.42e+05(4.8e-03)

1.50e+05(4.8e-03)

1.54e+05(5.2e-03)(+)

Mnist

(6000-92-85)

6.73e+06(1.0e-03)

6.73e+06(1.0e-03)

6.73e+06(1.1e-03)

6.67e+06(1.0e-03)

6.22e+06(1.1e-03)

6.61e+06(1.1e-03)(+)

(7967-50-45)

8.24e+04(4.8e-03)

8.24e+04(4.8e-03)

8.24e+04(4.5e-03)

8.24e+04(4.8e-03)

8.24e+04(4.8e-03)

8.24e+04(4.5e-03)

randn5000

(5000-100-85)

1.72e+06(1.3e-03)

1.72e+06(1.3e-03)

1.72e+06(1.4e-03)

1.70e+06(1.3e-03)

1.56e+06(1.3e-03)

1.72e+06(1.4e-03)

randn10000

(10000-50-45)

2.54e+06(3.3e-09)

2.54e+06(3.3e-09)

2.54e+06(1.9e-05)

2.53e+06(4.9e-09)

2.46e+06(3.3e-08)

2.54e+06(1.9e-05)

w1a

(2477-100-90)

4.10e+04(1.2e-09)

4.10e+04(1.2e-09)

3.76e+04(6.4e-06)

3.77e+04(4.2e-09)

3.27e+04(1.1e-08)

3.48e+04(6.4e-06)(+)

time limit=60s

20News

(9423-50-25)

6.47e+04(2.4e-03)

6.47e+04(2.4e-03)

6.42e+04(1.9e-03)

6.32e+04(2.4e-03)

6.35e+04(2.4e-03)

6.29e+04(1.9e-03)(+)

Cifar

(10000-50-25)

1.49e+07(2.4e-03)

1.49e+07(2.4e-03)

1.49e+07(1.7e-03)

1.45e+07(2.5e-03)

1.45e+07(2.5e-03)

1.45e+07(1.7e-03)(+)

cnnCaltech

(3000-96-48)

3.08e+04(3.5e-09)

3.08e+04(3.5e-09)

3.00e+04(2.4e-05)

1.92e+04(3.8e-08)

1.52e+04(4.8e-08)

1.67e+04(2.4e-05)(+)

E2006

(5000-100-50)

5.54e+04(3.5e-09)

5.54e+04(3.5e-09)

5.54e+04(2.3e-05)

5.51e+04(1.6e-07)

5.23e+04(3.9e-07)

5.51e+04(2.3e-05)(+)

gisette

(6000-50-25)

1.61e+05(2.4e-03)

1.61e+05(2.4e-03)

1.46e+05(2.1e-03)

9.76e+04(2.4e-03)

7.20e+04(2.4e-03)

8.40e+04(2.1e-03)(+)

Mnist

(6000-92-46)

6.94e+06(3.6e-09)

6.94e+06(3.6e-09)

6.93e+06(2.1e-05)

6.47e+06(1.4e-04)

6.45e+06(3.0e-04)

6.93e+06(2.1e-05)

(7967-50-25)

6.12e+04(2.4e-03)

6.12e+04(2.4e-03)

6.12e+04(2.0e-03)

6.12e+04(2.4e-03)

6.12e+04(2.4e-03)

6.12e+04(2.0e-03)

randn5000

(5000-100-50)

1.84e+06(3.5e-09)

1.84e+06(3.5e-09)

1.84e+06(1.9e-05)

1.59e+06(8.3e-05)

1.51e+06(4.9e-04)

1.84e+06(1.9e-05)

randn10000

(10000-50-25)

2.66e+06(2.4e-03)

2.66e+06(2.4e-03)

2.66e+06(2.5e-03)

2.58e+06(2.4e-03)

2.37e+06(2.4e-03)

2.66e+06(2.5e-03)

w1a

(2477-100-50)

2.71e+04(3.5e-09)

2.71e+04(3.5e-09)

2.41e+04(1.5e-05)

2.33e+04(1.3e-08)

1.89e+04(6.0e-08)

2.41e+04(1.5e-05)

20News

(9423-50-35)

7.89e+04(3.7e-09)

7.89e+04(3.7e-09)

7.84e+04(2.1e-05)

7.63e+04(3.3e-08)

7.60e+04(5.3e-08)

7.84e+04(2.1e-05)

Cifar

(10000-50-35)

1.48e+07(5.5e-03)

1.48e+07(5.5e-03)

1.48e+07(6.5e-03)

1.43e+07(5.6e-03)

1.42e+07(5.7e-03)

1.43e+07(6.5e-03)(+)

cnnCaltech

(3000-96-70)

3.75e+04(2.0e-09)

3.75e+04(2.0e-09)

3.69e+04(1.1e-05)

1.93e+04(3.2e-08)

1.89e+04(3.6e-08)

2.19e+04(1.1e-05)(+)

E2006

(5000-100-75)

6.79e+04(1.0e-03)

6.79e+04(6.3e-04)

6.79e+04(5.6e-04)

6.48e+04(6.3e-04)

6.36e+04(6.3e-04)

6.39e+04(5.6e-04)(+)

gisette

(6000-50-35)

1.89e+05(5.5e-03)

1.89e+05(5.5e-03)

1.80e+05(4.4e-03)

1.14e+05(5.5e-03)

9.97e+04(5.5e-03)

1.24e+05(4.4e-03)(+)

Mnist

(6000-92-70)

6.75e+06(2.0e-09)

6.75e+06(2.0e-09)

6.74e+06(1.3e-05)

6.21e+06(5.4e-04)

6.17e+06(2.0e-04)

6.21e+06(3.4e-04)(+)

(7967-50-35)

7.26e+04(5.5e-03)

7.26e+04(5.5e-03)

7.26e+04(4.8e-03)

7.25e+04(5.5e-03)

7.25e+04(5.5e-03)

7.25e+04(4.8e-03)(+)

randn5000

(5000-100-75)

1.75e+06(6.3e-04)

1.75e+06(6.3e-04)

1.75e+06(6.1e-04)

1.44e+06(7.2e-04)

1.35e+06(9.9e-04)

1.75e+06(6.1e-04)

randn10000

(10000-50-36)

2.56e+06(3.7e-09)

2.56e+06(3.7e-09)

2.56e+06(2.0e-05)

2.56e+06(5.4e-09)

2.41e+06(2.6e-07)

2.56e+06(2.0e-05)

w1a

(2477-100-75)

3.36e+04(6.3e-04)

3.36e+04(6.3e-04)

3.10e+04(5.4e-04)

3.32e+04(6.3e-04)

2.57e+04(6.3e-04)

3.10e+04(5.4e-04)

20News

(9423-50-45)

8.79e+04(4.8e-03)

8.79e+04(4.8e-03)

8.71e+04(4.5e-03)

8.73e+04(4.8e-03)

8.57e+04(4.8e-03)

8.66e+04(4.5e-03)(+)

Cifar

(10000-50-45)

1.47e+07(4.8e-03)

1.47e+07(4.8e-03)

1.47e+07(4.5e-03)

1.46e+07(4.8e-03)

1.46e+07(4.8e-03)

1.46e+07(4.5e-03)(+)

cnnCaltech

(3000-96-85)

4.13e+04(9.2e-04)

4.14e+04(4.4e-04)

4.09e+04(7.2e-04)

2.69e+04(4.4e-04)

2.44e+04(4.4e-04)

2.40e+04(7.2e-04)(+)

E2006

(5000-100-90)

7.44e+04(1.2e-09)

7.44e+04(1.2e-09)

7.44e+04(6.1e-06)

6.92e+04(8.2e-08)

6.90e+04(1.3e-07)

6.94e+04(6.1e-06)(+)

gisette

(6000-50-45)

2.25e+05(4.8e-03)

2.25e+05(4.8e-03)

2.10e+05(5.2e-03)

1.63e+05(4.8e-03)

1.37e+05(4.8e-03)

2.10e+05(5.2e-03)

Mnist

(6000-92-85)

6.73e+06(1.0e-03)

6.73e+06(1.0e-03)

6.73e+06(1.1e-03)

6.65e+06(1.0e-03)

6.12e+06(1.3e-03)

6.73e+06(1.1e-03)

(7967-50-45)

8.24e+04(4.8e-03)

8.24e+04(4.8e-03)

8.24e+04(4.4e-03)

8.24e+04(4.8e-03)

8.24e+04(4.8e-03)

8.24e+04(4.4e-03)

randn5000

(5000-100-85)

1.72e+06(1.3e-03)

1.72e+06(1.3e-03)

1.72e+06(1.4e-03)

1.65e+06(1.3e-03)

1.56e+06(1.3e-03)

1.72e+06(1.4e-03)

randn10000

(10000-50-45)

2.54e+06(3.3e-09)

2.54e+06(3.3e-09)

2.54e+06(1.9e-05)

2.53e+06(9.0e-09)

2.34e+06(2.5e-07)

2.54e+06(1.9e-05)

w1a

(2477-100-90)

4.10e+04(1.2e-09)

4.10e+04(1.2e-09)

3.64e+04(6.0e-06)

3.45e+04(1.0e-08)

3.16e+04(1.0e-08)

3.34e+04(6.0e-06)(+)

time limit=90s

20News

(9423-50-25)

6.47e+04(2.4e-03)

6.47e+04(2.4e-03)

6.41e+04(1.8e-03)

6.31e+04(2.4e-03)

6.25e+04(2.4e-03)

6.41e+04(1.8e-03)

Cifar

(10000-50-25)

1.49e+07(2.4e-03)

1.49e+07(2.4e-03)

1.49e+07(2.0e-03)

1.45e+07(2.5e-03)

1.45e+07(2.6e-03)

1.45e+07(2.0e-03)(+)

cnnCaltech

(3000-96-48)

3.08e+04(3.5e-09)

3.08e+04(3.5e-09)

2.98e+04(2.2e-05)

1.89e+04(4.0e-08)

1.52e+04(4.5e-08)

1.74e+04(2.2e-05)(+)

E2006

(5000-100-50)

5.54e+04(3.5e-09)

5.54e+04(3.5e-09)

5.54e+04(2.1e-05)

5.34e+04(2.7e-07)

5.22e+04(3.0e-07)

5.30e+04(2.1e-05)(+)

gisette

(6000-50-25)

1.61e+05(2.4e-03)

1.61e+05(2.4e-03)

1.42e+05(2.3e-03)

9.50e+04(2.4e-03)

8.14e+04(2.4e-03)

7.44e+04(2.3e-03)(+)

Mnist

(6000-92-46)

6.94e+06(3.6e-09)

6.94e+06(3.6e-09)

6.93e+06(2.0e-05)

6.45e+06(3.6e-04)

6.42e+06(1.4e-03)

6.93e+06(2.0e-05)

(7967-50-25)

6.12e+04(2.4e-03)

6.12e+04(2.4e-03)

6.12e+04(1.9e-03)

6.12e+04(2.4e-03)

6.12e+04(2.4e-03)

6.12e+04(1.9e-03)

randn5000

(5000-100-50)

1.84e+06(3.5e-09)

1.84e+06(3.5e-09)

1.84e+06(1.9e-05)

1.63e+06(1.4e-05)

1.52e+06(6.9e-04)

1.84e+06(1.9e-05)

randn10000

(10000-50-25)

2.66e+06(2.4e-03)

2.66e+06(2.4e-03)

2.66e+06(2.5e-03)

2.65e+06(2.4e-03)

2.25e+06(2.7e-03)

2.66e+06(2.5e-03)

w1a

(2477-100-50)

2.71e+04(3.5e-09)

2.71e+04(3.5e-09)

2.34e+04(1.5e-05)

2.24e+04(1.9e-08)

2.00e+04(4.3e-08)

1.90e+04(1.5e-05)(+)

20News

(9423-50-35)

7.89e+04(3.7e-09)

7.89e+04(3.7e-09)

7.84e+04(2.3e-05)

7.66e+04(4.5e-08)

7.57e+04(1.6e-07)

7.70e+04(2.3e-05)(+)

Cifar

(10000-50-35)

1.48e+07(5.5e-03)

1.48e+07(5.5e-03)

1.48e+07(7.0e-03)

1.43e+07(5.6e-03)

1.42e+07(6.2e-03)

1.43e+07(7.1e-03)(+)

cnnCaltech

(3000-96-70)

3.75e+04(2.0e-09)

3.75e+04(2.0e-09)

3.67e+04(1.4e-05)

2.33e+04(2.3e-08)

2.29e+04(2.5e-08)

2.16e+04(1.4e-05)(+)

E2006

(5000-100-75)

6.79e+04(1.0e-03)

6.79e+04(6.3e-04)

6.79e+04(5.5e-04)

6.37e+04(6.3e-04)

6.36e+04(6.3e-04)

6.38e+04(5.5e-04)(+)

gisette

(6000-50-35)

1.89e+05(5.5e-03)

1.89e+05(5.5e-03)

1.78e+05(4.2e-03)

1.16e+05(5.5e-03)

9.77e+04(5.5e-03)

1.10e+05(4.2e-03)(+)

Mnist

(6000-92-70)

6.75e+06(2.0e-09)

6.75e+06(2.0e-09)

6.74e+06(1.2e-05)

6.18e+06(7.5e-04)

6.15e+06(2.4e-03)

6.22e+06(6.7e-04)(+)

(7967-50-35)

7.26e+04(5.5e-03)

7.26e+04(5.5e-03)

7.26e+04(4.6e-03)

7.25e+04(5.5e-03)

7.25e+04(5.5e-03)

7.25e+04(4.6e-03)(+)

randn5000

(5000-100-75)

1.75e+06(6.3e-04)

1.75e+06(6.3e-04)

1.75e+06(6.1e-04)

1.39e+06(1.2e-03)

1.38e+06(1.0e-03)

1.75e+06(6.1e-04)

randn10000

(10000-50-36)

2.56e+06(3.7e-09)

2.56e+06(3.7e-09)

2.56e+06(2.0e-05)

2.56e+06(4.4e-09)

2.51e+06(3.5e-08)

2.56e+06(2.0e-05)

w1a

(2477-100-75)

3.36e+04(6.3e-04)

3.36e+04(6.3e-04)

3.04e+04(5.2e-04)

2.64e+04(6.3e-04)

2.58e+04(6.3e-04)

2.56e+04(5.2e-04)(+)

20News

(9423-50-45)

8.79e+04(4.8e-03)

8.79e+04(4.8e-03)

8.70e+04(4.4e-03)

8.66e+04(4.8e-03)

8.58e+04(4.8e-03)

8.61e+04(4.4e-03)(+)

Cifar

(10000-50-45)

1.47e+07(4.8e-03)

1.47e+07(4.8e-03)

1.47e+07(4.4e-03)

1.46e+07(4.8e-03)

1.41e+07(8.6e-03)

1.41e+07(5.7e-03)(+)

cnnCaltech

(3000-96-85)

4.13e+04(9.2e-04)

4.14e+04(4.4e-04)

4.07e+04(8.2e-04)

2.47e+04(4.4e-04)

2.38e+04(4.4e-04)

2.91e+04(8.2e-04)(+)

E2006

(5000-100-90)

7.44e+04(1.2e-09)

7.44e+04(1.2e-09)

7.44e+04(8.7e-06)

6.89e+04(9.4e-08)

6.99e+04(7.9e-08)

7.09e+04(8.7e-06)(+)

gisette

(6000-50-45)

2.25e+05(4.8e-03)

2.25e+05(4.8e-03)

2.09e+05(5.2e-03)

1.83e+05(4.8e-03)

1.34e+05(4.8e-03)

2.09e+05(5.2e-03)

Mnist

(6000-92-85)

6.73e+06(1.0e-03)

6.73e+06(1.0e-03)

6.73e+06(1.1e-03)

6.72e+06(1.0e-03)

6.14e+06(1.1e-03)

6.73e+06(1.1e-03)

(7967-50-45)

8.24e+04(4.8e-03)

8.24e+04(4.8e-03)

8.24e+04(4.2e-03)

8.24e+04(4.8e-03)

8.24e+04(4.8e-03)

8.24e+04(4.2e-03)

randn5000

(5000-100-85)

1.72e+06(1.3e-03)

1.72e+06(1.3e-03)

1.72e+06(1.4e-03)

1.46e+06(1.3e-03)

1.55e+06(1.3e-03)

1.72e+06(1.4e-03)

randn10000

(10000-50-45)

2.54e+06(3.3e-09)

2.54e+06(3.3e-09)

2.54e+06(1.9e-05)

2.53e+06(5.8e-09)

2.52e+06(9.7e-09)

2.54e+06(1.9e-05)

w1a

(2477-100-90)

4.10e+04(1.2e-09)

4.10e+04(1.2e-09)

3.56e+04(6.5e-06)

3.86e+04(2.3e-09)

2.95e+04(1.7e-08)

3.07e+04(6.5e-06)(+)

Objective

0

25

50

75

100

125

150

175

Epoch

8.64

8.66

8.68

8.70

8.72

8.74

8.76

8.78

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(a) 20News(9423-50-45)

0

25

50

75

100

125

150

175

Epoch

1.41

1.42

1.43

1.44

1.45

1.46

1.47

Objective

×10 7

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(b) Cifar(10000-50-45)

0

100

200

300

400

500

600

Epoch

2.50

2.75

3.00

3.25

3.50

3.75

4.00

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(c) CnnCaltech(3000-96-85)

0

50

100

150

200

250

300

350

400

Epoch

6.9

7.0

7.1

7.2

7.3

7.4

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(d) E2006(5000-100-90)

0

50

100

150

200

250

300

350

Epoch

1.2

1.4

1.6

1.8

2.0

2.2

Objective

×10 5

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(e) Gisette(6000-50-45)

0

50

100

150

200

250

300

350

Epoch

6.45

6.50

6.55

6.60

6.65

6.70

Objective

×10 6

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(f) Mnist(6000-92-85)

0

50

100

150

200

250

Epoch

8.240

8.241

8.242

8.243

8.244

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(g) News20(7967-50-45)

0

50

100

150

200

250

300

350

400

Epoch

1.40

1.45

1.50

1.55

1.60

1.65

1.70

Objective

×10 6

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(h) randn(5000-100-85)

0

25

50

75

100

125

150

175

Epoch

2.375

2.400

2.425

2.450

2.475

2.500

2.525

Objective

×10 6

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(i) randn(10000-50-45)

0

100

200

300

400

500

600

Epoch

3.2

3.4

3.6

3.8

4.0

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(j) W1a(2477-100-90)

Objective

0

20

40

60

80

Time (seconds)

8.64

8.66

8.68

8.70

8.72

8.74

8.76

8.78

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(a) 20News(9423-50-45)

0

20

40

60

80

Time (seconds)

1.41

1.42

1.43

1.44

1.45

1.46

1.47

Objective

×10 7

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(b) Cifar(10000-50-45)

0

20

40

60

80

Time (seconds)

2.50

2.75

3.00

3.25

3.50

3.75

4.00

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(c) CnnCaltech(3000-96-85)

0

20

40

60

80

Time (seconds)

6.9

7.0

7.1

7.2

7.3

7.4

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(d) E2006(5000-100-90)

0

20

40

60

80

Time (seconds)

1.2

1.4

1.6

1.8

2.0

2.2

Objective

×10 5

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(e) Gisette(6000-50-45)

0

20

40

60

80

Time (seconds)

6.45

6.50

6.55

6.60

6.65

6.70

Objective

×10 6

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(f) Mnist(6000-92-85)

0

20

40

60

80

Time (seconds)

8.240

8.241

8.242

8.243

8.244

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(g) News20(7967-50-45)

0

20

40

60

80

Time (seconds)

1.40

1.45

1.50

1.55

1.60

1.65

1.70

Objective

×10 6

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(h) randn(5000-100-85)

0

20

40

60

80

Time (seconds)

2.375

2.400

2.425

2.450

2.475

2.500

2.525

Objective

×10 6

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(i) randn(10000-50-45)

0

20

40

60

80

Time (seconds)

3.2

3.4

3.6

3.8

4.0

Objective

×10 4

J-JOBCD

VR-J-JOBCD

CSDM

ADMM

UMCM

(j) W1a(2477-100-90)

F.3.3 Ultra-hyperbolic Knowledge Graph Embedding 836

0

20

40

60

80

100

Epoch

0

200

400

600

800

1000

1200

1400

Cumulative loss

J-JOBCD

VR-J-JOBCD

CSDM

(a) Train accumulated loss

0

20

40

60

80

100

Epoch

0.05

0.10

0.15

0.20

0.25

0.30

MRR

J-JOBCD

VR-J-JOBCD

CSDM

(b) Test MRR

H@10

H@3

H@1

(c) Test Hits

0

1000 2000 3000 4000 5000 6000 7000

Time (seconds)

0

200

400

600

800

1000

1200

1400

Cumulative loss

J-JOBCD

VR-J-JOBCD

CSDM

(a) Train accumulated loss

0

1000 2000 3000 4000 5000 6000 7000

Time (seconds)

0.05

0.10

0.15

0.20

0.25

0.30

MRR

J-JOBCD

VR-J-JOBCD

CSDM

(b) Test MRR

+#GLYPH<c=20,font=/FHFXGW+TimesNewRomanPS-BoldMT>GLYPH<c=19,font=/FHFXGW+TimesNewRomanPS-BoldMT>

+#

3

+#GLYPH<c=20,font=/FHFXGW+TimesNewRomanPS-BoldMT>

(c) Test Hits

0

20

40

60

80

100

Epoch

0

200

400

600

800

1000

1200

1400

Cumulative loss

J-JOBCD

VR-J-JOBCD

CSDM

(a) Train accumulated loss

0

20

40

60

80

100

Epoch

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

MRR

J-JOBCD

VR-J-JOBCD

CSDM

(b) Test MRR

+#GLYPH<c=20,font=/FHFXGW+TimesNewRomanPS-BoldMT>GLYPH<c=19,font=/FHFXGW+TimesNewRomanPS-BoldMT>

+#

3

+#GLYPH<c=20,font=/FHFXGW+TimesNewRomanPS-BoldMT>

(c) Test Hits

0

1000

2000

3000

4000

5000

Time (seconds)

0

200

400

600

800

1000

1200

1400

Cumulative loss

J-JOBCD

VR-J-JOBCD

CSDM

(a) Train accumulated loss

0

1000

2000

3000

4000

5000

Time (seconds)

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

MRR

J-JOBCD

VR-J-JOBCD

CSDM

(b) Test MRR

+#GLYPH<c=20,font=/FHFXGW+TimesNewRomanPS-BoldMT>GLYPH<c=19,font=/FHFXGW+TimesNewRomanPS-BoldMT>

+#

3

+#

1

(c) Test Hits

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we highlighted our contributions, including algorithm development, theoretical analysis, and empirical study.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the assumptions made for the optimization problem outlined in the introduction and Section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have added a hyperlink before each theoretical result, which points to the complete proof located in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided sufficient details for reproducing the results of the paper, such as parameter settings, runtime environments, and dataset descriptions.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included all the code and data in the supplemental materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided sufficient details for solving the optimization problem, encompassing hyperparameter settings and dataset generation.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: For simplicity, we only demonstrate the convergence behavior of the objective function by varying the time or iterations. Our methods exhibit clear advantages over the compared methods. Such results have demonstrated significance in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines: We have outlined the types of compute workers, detailing CPU and memory specifications.

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research aligns with the ethical guidelines outlined by NeurIPS.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper addresses theoretical questions on algorithm complexity, which, to the best of our knowledge, pose no negative social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The dataset used in the experiments is published on an open site without license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The experiments do not involve new datasets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or human object is involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or human object is involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

1148

1149

1150

1151

1152

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces the JOBCD (J-Orthogonal Block Coordinate Descent) algorithm, a novel method designed to tackle optimization problems under J-orthogonality constraints. JOBCD includes two variants: GS-JOBCD (Gauss-Seidel strategy) and VR-J-JOBCD (Jacobi strategy with variance reduction). Theoretical analyses establish the algorithms' complexity and convergence, while extensive experiments show JOBCD's superior performance compared to state-of-the-art methods in various applications.

STRENGTHS:
The strengths of this paper are listed as follows:

Originality: This paper introduces JOBCD as a novel approach to handling J-orthogonality constraints. It offers GS-JOBCD and VR-J-JOBCD, showcasing flexibility and innovation in optimization strategies.

Quality: This paper provides comprehensive complexity and convergence analyses. Extensive experiments demonstrate superior performance on real-world and synthetic data.

Clarity: The structure is logical.

Significance: This work is relevant to various statistical learning and data science fields.

WEAKNESSES:
Some proofs for Section 4 are hard to follow.

QUESTIONS:
1. Do the optimization problems in (S3) of Algorithm 1 and Algorithm 2 require an especially exact solution? What happens if they are solved approximately?
2. Line 34: The notation $\mathcal{Q}(X^+; X)$ is undefined.
3. Line 109: Should the notation $\mathrm{B}$ be replaced by $\mathrm{B^t}$?
4. LIne 124, formula (4): Should $I_2$ be replaced by $I_4$?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper proposes two Block Coordinate gradient descent methods(BCD) for solving J-orthogonal constrained problem. One is Gauss-Seidel type, the other one is Jocobi type as well as addressing finite sum problem using variance reduction strategies. Convergence guarantees are proved with KL conditions. Numerical experiments show the advantages of the  proposed algorithm.

STRENGTHS:
The proposed algorithm decomposes the matrix variable into row block structure, yielding a block coordinate descent algorithm with a small size subproblem. The numerical performance is very impressive.

WEAKNESSES:
1. This paper is based on the paper " [51] Ganzhao Yuan. A block coordinate descent method for nonsmooth composite optimization
under orthogonality constraints. ArXiv, abs/2304.03641, 2023."
The main difference is the constraint in this paper becomes J-orthogonality constraint. However, the framework follows almost the same as [51]. The authors should highlight the novelty of the algorithm or difficulty in the extension.

2. The authors of the reference [31] may be wrong. Besides, the UMCM algorithm in [31] solves orthogonal constrained problem. Is there any difference in implmenting in solving J-orthogonality problem? The objective value of UMCM is far from the JOBCD method. I'm curious about the reasons.

3. In numerical experiment, how do you select subset from the dataset, see line 309.

QUESTIONS:
1. Why cannot we design variance reduced algorithm for Gauss-Seidel type BCD?
2. line 323: it claims that the lower objective achieved by GS-JOBCD can be explained by the stronger stationary point result. However, it is based on KL condition. I think it is not the main reason?
3. line 324: infinite sum?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

comments: SUMMARY:
The paper proposes two block coordinate descent methods for minimization of a finite-sum subject to the J-Orthogonality constraints — one based on Gauss-Seidel strategy, the other based on variance reduction and Jacobi strategy. The convergence is proved, with a global convergence rate of O(N/\epsilon) and O(\sqrt{N}/\epsilon) respectively, and a local convergence rate that depends on the desingularization in the KL-condition assumption.

STRENGTHS:
The algorithms proposed are novel and might be useful in practice: the update rules involve solving a small size problem thereby is very simple, and the convergence is proved theoretically under reasonable assumptions.

WEAKNESSES:
The paper is relatively dense, and I find it a bit hard to keep track of all the terms introduced. For instance, the parameter theta is used in the algorithms but I’m not sure where it is introduced; in Assumption 4.8 KL function is mentioned but it’s not defined…

QUESTIONS:
The blocks in the proposed JOBCD algorithms consist of just 2 indices. I’m wondering if there is benefit in updating more than 2 indices in each iteration? 

I might miss something while reading the paper… what is the parameter theta in the algorithm 1?

LIMITATIONS:
Yes, the paper discusses the assumptions of the theorems.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a block coordinate descent method for solving optimization problems with J-orthogonality constraints. Several variants of the method are introduced within this framework, and convergence results are established. Extensive numerical results are also presented to demonstrate the efficiency of the proposed methods. However, I have some concerns regarding the novelty of this paper as well as the numerical results.\n\nSTRENGTHS:\nIt appears that optimization with J-orthogonality constraints has not been thoroughly studied in the literature. This paper proposes an efficient method for addressing this problem.\n\nWEAKNESSES:\n1. My major concern is that the novelty of this paper might be insufficient since the row-based approach is very similar to that in [51], even though the two papers tackle different problems.\n\n2.  For the numerical results shown in Table 1, the proposed method fails to return a feasible solution for some instances, such as randn(10-10-5) and w1a (2470-290-145), as well as some other instances in the appendix. This is strange since the paper describes a BCD-type method, which should always return a feasible solution.\n\n3. The information of the reference [31] might be incorrect. \n\n4. For the GS-JOBCD method, there are two options for choosing $Q$, whereas J-JOBCD only has one option. The authors should provide an explanation for this difference.\n\n5. The presentation could be further improved. Here are a few examples: the formulation of $P_i$  after equation (12) could be simplified by removing the notation $\\mathrm{mat}$; it is unclear if the requirement on $\\underline{Q}$ in equation (4) is sufficient to guarantee convergence (probably not, since $\\underline{Q} = 0$  also satisfies this condition).\n\nQUESTIONS:\n1. How to perform and choose the parameters in ADMM for solving this problem? It seems that the corresponding method lacks a convergence guarantee.\n\n2. The authors should provide more details on the implementation of J-JOBCD and GS-JOBCD. Did you use parallel techniques in J-JOBCD? What would happen if these techniques were not used? Should you compare the iteration numbers of the methods? How do you choose the parameter in equation (12)? Moreover, if J-JOBCD consistently performs much better than GS-JOBCD, it seems unnecessary to introduce GS-JOBCD.\n\nLIMITATIONS:\nAt the beginning of the paper, the authors claim that equation (2) can imply \n$\\\\|\\nabla f_i(X) - \\nabla_i f(X^+)\\\\| \\leq L_f \\\\|X - X^+\\\\|$, which is incorrect. Note that the converse is correct. The other assumptions in Assumptions 4.1 and 4.2 essentially assume the compactness of the iterates.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}